RC stats in 1.27.2

I kind of wanted to wait for official announcement, but it looks like there might be none. Maybe because 1.27.3 is just around the corner and it also requires full replay? Since 1.27.2 was already deployed on some servers (api.hive.blog among them), I guess there is no point to wait anymore.

While I was analyzing how RC works during time before HF26, I was using block_data_export plugin to get access to the RC data. I had to optimize it a bit, but even then I had to deal with enormous amount of data, since it printed usage and cost for every single transaction separately. It became apparent that something better is needed. I've noticed that collecting statistics in daily "buckets" seems to be the golden mean - smaller grade just introduces noise and the raw weight of the data makes it hard to deal with, coarser grade can be useful but loses some fine details. Dailies are light enough. Based on that observation I decided to add new tool to the code - daily reports on RC. The tool is to provide the same kind of data that I ended up using during my work. Now not only are those reports available to node operators, the data on last complete day is also given out through new API calls.

New call: rc_api.get_rc_operation_stats

Let's start with something that can be useful even for ordinary users - detailed data on specific type of operation.

curl \
  --request POST \
  --url https://api.hive.blog/ \
  --header 'Content-Type: application/json' \
  --data '{"jsonrpc": "2.0", "method": "rc_api.get_rc_operation_stats", "params": {"operation":"custom_json_operation"}, "id": 1}'

The request will give statistics collected for selected operation (technically for transactions that contained single operation of that type) executed during last complete day (most recent transactions are part of pending report and are not available). The call has only one argument - operation - which can be provided either as a name of operation or alternatively as more compact tag number (refer to this list for proper names and tag values). Note that only actual operations have stats, not virtual ones. For the sake of completeness you can also ask for multiop which covers data for transactions that contained more than one operation. Since that averages out transactions as different as curation bot voting for many posts, wallet combining comment with options or a whale claiming 100 account tokens, the resulting information is not very useful.

The result will look similar to the following:

{
  "count": 2030228, // number of such operations executed during last day
  "avg_cost_rc": 378730420, // average RC cost of single operation
  "resource_cost": { // average RC cost split between various resources
    "history_rc": 354717108,
    "tokens_rc": 0,
    "market_rc": 0,
    "state_rc": 31667,
    "exec_rc": 23981643
  },
  "resource_cost_share": { // share of resource cost in average final cost (expressed in basis points)
    "history_bp": 9365,
    "tokens_bp": 0,
    "market_bp": 0,
    "state_bp": 0,
    "exec_bp": 633
  },
  "resource_usage": { // average consumption of resources per operation
    "history_bytes": 335, // - size of transaction in bytes
    "tokens": "0.0000", // - number of tokens (always 0 or 1 (with exception of multiop) - tokens are internally expressed with 4 digit precision
    "market_bytes": 0, // - size of transaction in bytes when it belongs to market category or 0 otherwise
    "state_hbytes": 358, // - hour-bytes of state
    "exec_ns": 102335 // - nanoseconds of execution time
  }
}

As you can see in this particular case - custom_json_operation - the dominant source of RC cost is in history resource (almost 94%). Therefore by comparing size of your transaction with average size from above results (335 bytes) you can estimate the cost that you are likely to be paying. In general case it is not that easy though, so you should just broadcast transaction and see what happens. Either it fails or it passes, in both cases you will get information of actual cost (the cost of recently executed transaction is latest addition to cli_wallet and transaction_status_api).

If you are interested in various factors that influence RC costs, see this post.

A side note for whales claiming account tokens: due to related pool being nearly dry, the price of tokens swings a lot, so above API call is not very useful as a source of expected cost, because it likely changed already (remember, the info is an average and from last completed day). It is better to get the feel of it by mentally connecting cost with pool fill level - the current up-to-date value is accessible with rc_api.get_resource_pool.

New call: rc_api.get_rc_stats

The information on single operation is detailed, because it is taken from FULL report and additionally recalculated to provide data in human readable form. But when asking for all operations at once, such details are not normally needed. The get_rc_stats gives last daily REGULAR report, where operations have just average costs.

curl \
  --request POST \
  --url https://api.hive.blog/ \
  --header 'Content-Type: application/json' \
  --data '{"jsonrpc": "2.0", "method": "rc_api.get_rc_stats", "params": {}, "id": 1}'

The report will look similar to the following:

{
  "rc_stats": {
    "block": 70300800, // starting block of the day covered by the report
    "regen": "2061225430819", // global RC regen at starting block
    "budget": [43403, 797, 72338, 43546196, 40000000], // block-budget for each resource
    "pool": ["11291839369", 11534593, 1960955579, "24306169848870", "51095679598"], // content of each resource pool at starting block
    "share": [6178, 10000, 247, 1172, 2401], // resource popularity share at starting block
    "vote": 178722332, // average cost of vote at starting block
    "comment": 1420508072, // average cost of comment at starting block
    "transfer": 287317787, // average cost of transfer at starting block

    "ops": { // basic stats for each type of operation that was executed during reported day - number of operations executed and average cost
      "vote_operation": {
         "count": 291732,
         "avg_cost": 181587340
      },
      "comment_operation": {
        "count": 22787,
        "avg_cost": 1418928099
      },
      "transfer_operation": {
        "count": 11739,
        "avg_cost": 263934250
      },
      ...
      "custom_json_operation": {
        "count": 2030228,
        "avg_cost": 378730420,
      },
      ...
    },

    "payers": [ // stats for users that paid for transactions during reported day, split between ranks 0..7 (levels of "RC wealth")
      {
        "rank": 0,
        "count": 579680,
        "lt5": 8674,
        "lt10": 21124,
        "lt20": 109323,
        "cant_afford": {
          "vote": 6792,
          "comment": 181156,
          "transfer": 8680
        }
      },
      ...
      {
       "rank": 7,
       "count": 1431
      }
    ]
  }
}

The report consists of three sections: the general state of RC system, the operations and the payers. General data is collected at the start of report day and reflects the state at that moment. Average costs of three basic operations (vote, comment, transfer) are included. These are values from previous day that are used in affordability filter presented in last section of the report. Second section consists of data on operations that were executed during report day - number of executions and average cost. If some operation is missing, it means it was not executed that day. Note that costs for vote, comment and transfer can be slightly different than those from first section, because they were collected on different days and reflect different actual transactions. The last section gives overview of users that paid RC for transactions. They are split between 8 ranks, from 0 to 7. To qualify as rank 0, payer has to have less than 10 billion max RC mana (from own stake or delegations). Above 10B but less than 100B is rank 1 and so on - account gets ten times bigger (wealthier) with each rank. Account that has 10'000T max RC or more is rank 7 (like ocdb). Data on each class of payer consists of number of transactions paid by users of that rank and optionally how many times that user had less than 5%, 10% or 20% of their max RC mana after transaction, as well as how many times they would not be able to immediately afford another vote, comment and/or transfer (with costs from first section used as filter). This way we can monitor how frequently users have to wait before they can transact again.

Daily report management options

Node operators can choose what kind of RC report they want and where to place it, similar to how block stats are managed. Command line or config.ini options for that are:

• rc-stats-report-type - choice between NONE, MINIMAL (just first section), REGULAR (like above) or FULL (sums of usage per resource and costs per resource instead of average cost on transactions and similar extra info for payers); the default is REGULAR and it only takes couple megabytes in the log for all historical RC data
• rc-stats-report-output - choice between DLOG, ILOG (default) and NOTIFY (recommended for FULL report with redirection to external database)

Summary

Information taken from daily reports can be used to provide more accurate data on RC costs, f.e. most recent costs of basic operations can be used to estimate how much user can afford instead of some hardcoded value. I'm pretty sure someone will pick up the data to post various trends like with HIVE price and inflation. Finally analyzing RC no longer requires hundreds of gigabytes of log, scripts that run for hours or large databases. It is going to be needed as in near future RC will be pushed towards becoming part of consensus code, so we need a tool that can help in catching potential bugs and discrepancies between nodes (for that reason I'd strongly recommend node operators to not turn off the reports - they might be needed).

Just a final note: since RC is still not part of consensus, costs and therefore daily reports can be subject to retroactive changes (1.27.3 will bring such correction).