Setup a Hive witness using the new rolling block log feature
48 comments
Running a Hive witness is typically done on rented hardware due to the demanding space and reliability requirements. Up until now, this required storing the entire block log on disk which currently requires 507GB with an additional 100GB for artifacts and ~24GB for the state. This requires almost 650GB of very fast storage, ideally NVME. This puts it out of reach of most vendors that offer VPS and dedicated server offerings.
Starting with 1.27.7, you are able to use what I call a "rolling block log" or specifically the new block-log-split config option. This allows you to only keep X blocks on the file system reducing the space requirements of a node by hundreds of GB. While this is little use for a full node that needs access to all chain data, for a witness this is a huge improvement.
I have been testing this feature and have tested it using multiple configurations, to which there are two main options to consider.
- Shared Memory on disk
- Shared Memory on tmpfs
The first option requires very little ram (~4GB) but is slower to to get up and running. The second option requires the entire state file to be in ram but gives you the fastest possible performance. Currently, this requires ~24GB of ram just for the state file and a little more for the OS and processes.
You can also use a hybrid option where you get a node to the head block using tmpfs and then take the node down and move the state to the file system, although this only really helps if you want to reboot the node easily, as you can't temporary access additional ram.
There is very little you need to do to make this work, for the most part all you need to do is change two lines in your config.ini and sync your node. Using a snapshot, I have been able to get a witness node online from nothing in around 7 minutes. This requires having a compatible snapshot available.
block-log-split option
This is the parameter that enables the new rolling block log feature, which officially is known as block log split. This feature splits the block_log into pieces and determines how many pieces you want to save. Each "piece" is 1M blocks, and using a block-log-split = 0 will save no pieces of the block_log to disk. This means the only space requirements is the binaries, configuration file, and the state file as well as a few logs. Using block-log-split = 1 will allow you to have limited access to recent blocks using get_blocks() if you want to interact with the node.
You will need to either use a compatible snapshot, or resync your node to use this new feature. Before you do, it is recommended you use at least version 1.27.8 as a few bugs have been fixed.
The easiest way to get 1.27.8 is grab the binary from @gtg.
wget https://gtg.openhive.network/get/bin/hived-1.27.8
You can also build the binary yourself using this guide.
There currently are not any compatible snapshots, so I created my own. You can find it here.
This snapshot is only compatible with the following settings:
plugin = witness condenser_api network_broadcast_api account_by_key database_api state_snapshot
You will need to run the same plugins or you will have problems restarting your node. This is an open issue and should be resolved in the future.
block-log-split = 0
This snapshot only works with block-log-split = 0. If you need to access your node to get blocks and monitor blockchain events from it, you will need at least block_log_split = 1 and will require you do a full sync.
For the most part, all you need to do for most people is make sure you are running the same snapshots as I do, and adding the block-log-split parameter. You can then just load the snapshot above using the following to launch your node.
`./bin/hived-1.27.8 -d /home/hived/hived --load-snapshot=witness'
You will need to create a snapshot directory under your hived folder and unpack the witness.tgz (4.2GB) file there for it to load the snapshot. You can remove the witness.tgz once you have unpacked it.
Before doing this, you will need to decide if you want to run the shared memory on disk or tmpfs (ram). Here is what the requirements look like in the two configurations.
Disk
- ~4GB-8GB ram
- ~30GB disk space
Tmpfs
- ~28GB ram
- ~1GB disk space
Keep in mind, you need some breathing room for both configurations to download snapshot, and any additional copies of anything you are working with.
Example config.ini
log-appender = {"appender":"stderr","stream":"std_error"}
log-appender = {"appender":"p2p","file":"logs/p2p/p2p.log"}
log-logger = {"name":"default","level":"info","appender":"stderr"}
log-logger = {"name":"p2p","level":"warn","appender":"p2p"}
backtrace = yes
plugin = witness condenser_api network_broadcast_api account_by_key database_api state_snapshot
#shared-file-dir = "/run/hive"
shared-file-size = 24G
shared-file-full-threshold = 9500
shared-file-scale-rate = 1000
enable-stale-production = false
required-participation = 33
p2p-seed-node = hive-seed.arcange.eu:2001 # @arcange (BE, Wavre) (ASN: Proximus NV)
p2p-seed-node = anyx.io:2001 # @anyx (CA) (ASN: OVH SAS)
p2p-seed-node = seed.thecryptodrive.com:2001 # @thecryptodrive (DE, Esslingen) (ASN: Hetzner Online GmbH)
p2p-seed-node = seed.buildteam.io:2001 # @buildteam (DE) (ASN: Hetzner Online GmbH)
p2p-seed-node = seed.hivekings.com:2001 # @drakos (DE, Nuremberg) (ASN: Contabo GmbH)
p2p-seed-node = gtg.steem.house:2001 # @gtg (PL) (ASN: OVH SAS)
p2p-seed-node = seed.openhive.network:2001 # @gtg (PL) (ASN: OVH SAS)
p2p-seed-node = hive-seed.roelandp.nl:2001 # @roelandp (FI, Helsinki) (ASN: Hetzner Online GmbH)
p2p-seed-node = node.mahdiyari.info:2001 # @mahdiyari (FI, Helsinki) (ASN: Hetzner Online GmbH)
p2p-seed-node = hived.hive-engine.com:2001 # @aggroed (FI) (ASN: Hetzner Online GmbH)
p2p-seed-node = hived.splinterlands.com:2001 # @aggroed (FI) (ASN: Hetzner Online GmbH)
p2p-seed-node = rpc.ausbit.dev:2001 # @ausbitbank (FI, Tuusula) (ASN: Hetzner Online GmbH)
p2p-seed-node = seed.chitty.me:2001 # @chitty (FI) (ASN: Hetzner Online GmbH)
p2p-seed-node = hiveseed-fin.privex.io:2001 # @privex (FI, Helsinki) (ASN: Hetzner Online GmbH)
p2p-seed-node = seed.liondani.com:2016 # @liondani (GB, Maidstone) (ASN: HostDime.com, Inc.)
p2p-seed-node = hiveseed-se.privex.io:2001 # @privex (SE, Stockholm) (ASN: Privex Inc.)
webserver-thread-pool-size = 32
p2p-endpoint = 0.0.0.0:35000
block-log-split = 0
You will need to make sure /run is large enough (25GB-30GB is good) to store the shared memory file if you choose to use tmpfs.
Using my hardware, I was able copy the files over, unpack the snapshot, and get a node to head block in around 7 minutes using tmpfs and under 20 minutes using disk (dual NVME raid 0).
You also can opt to resync your node and not use a snapshot which will typically take 24-48 hours. At this point, you can create your own snapshot using --dump-snapshot= parameter when starting your node.
1.27.9 should be considered stable soon, and by the time you decide to give this a try, you may want to use 1.27.9 instead.
Comments