[Stacks 2.1] Epoch 2.0 boot data is not emitted to event-observers

[zone117x]

The boot contracts (e.g. bns, pox, seed account balances, etc) are ignored / not emitted through the event-observer interface when bootstrapping a neon/krypton Stacks node to Epoch 2.1.

This appears to be only a bug in the event-emitter code, because while boot contracts & balances are not emitted, they are still visible through RPC / v2/<endpoint> calls.

This behavior breaks event-observers like the API which end up with negative STX balances for config.toml seed accounts, missing contract ABIs for pox-1 and bns, etc..

Using something like the following to bootstrap a Stacks node into krypton mode (this is paired with a bitcoind instance booted into regtest mode and an regtest auto-mining script):

[node]
name = "krypton-node"
rpc_bind = "0.0.0.0:20443"
p2p_bind = "0.0.0.0:20444"
working_dir = "$DATA_DIR"

seed = "$MINER_SEED"
local_peer_seed = "$MINER_SEED"

miner = true
use_test_genesis_chainstate = true
pox_sync_sample_secs = 0
wait_time_for_microblocks = 50
microblock_frequency = 5000
# mine_microblocks = true
# max_microblocks = 10

[miner]
first_attempt_time_ms = 5000
subsequent_attempt_time_ms = 5000

[connection_options]
disable_block_download = true
disable_inbound_handshakes = true
disable_inbound_walks = true

[burnchain]
chain = "bitcoin"
mode = "krypton"
poll_time_secs = 1
pox_2_activation = 104 # note that this is based off starting with a regtest bitcoind node that mines 101 blocks for a BTC balance, where something like 104 is is stacks_block=1, 105 is stacks_block=2, etc.. this can also be set to a value lower than min bitcoin-regtest confirmation blocks, but will result in various burnblock_height values being extra buggy

### bitcoind-regtest connection info
peer_host = "$BITCOIN_PEER_HOST"
peer_port = "$BITCOIN_PEER_PORT"
rpc_port = "$BITCOIN_RPC_PORT"
rpc_ssl = false
username = "$BITCOIN_RPC_USER"
password = "$BITCOIN_RPC_PASS"
timeout = 30

[[burnchain.epochs]]
epoch_name = "1.0"
start_height = 0

[[burnchain.epochs]]
epoch_name = "2.0"
start_height = 1

[[burnchain.epochs]]
epoch_name = "2.05"
start_height = 2

[[burnchain.epochs]]
epoch_name = "2.1"
start_height = 103 # note that this is based off starting with a regtest bitcoind node that mines 101 blocks for a BTC balance, where something like 104 is is stacks_block=1, 105 is stacks_block=2, etc.. this can also be set to a value lower than min bitcoin-regtest confirmation blocks, but will result in various burnblock_height values being extra buggy

[[ustx_balance]]
address = "STB44HYPYAT2BB2QE513NSP81HTMYWBJP02HPGK6"
amount = 10000000000000000

[[ustx_balance]]
address = "ST11NJTTKGVT6D1HY4NJRVQWMQM7TVAR091EJ8P2Y"
amount = 10000000000000000

[gregorycoppola]

Can we specify which observer endpoints, and possibly even which fields, are affected?

[zone117x]

Can we specify which observer endpoints, and possibly even which fields, are affected?

Yes it's missing data from the genesis (first) /new_block event payload. In particular, the boot contracts (e.g. bns, pox, etc) no longer have their associated contract-deploy txs in the genesis block. Similarly, seed accounts no longer have stx-transfer events in the genesis block -- these are the ones that can be added to config.toml e.g.

[[ustx_balance]]
address = "STB44HYPYAT2BB2QE513NSP81HTMYWBJP02HPGK6"
amount = 10000000000000000

[gregorycoppola]

I think I am seeing the right events possibly. I'll upload the toml file and the log from my listener.
epochs-output.zip

[zone117x]

Thanks for the testing @gregorycoppola!

From comparing your config to mine, it looks like this is what probably makes the difference:
(your config)

[[burnchain.epochs]]
epoch_name = "2.0"
start_height = 105

[[burnchain.epochs]]
epoch_name = "2.05"
start_height = 106

[[burnchain.epochs]]
epoch_name = "2.1"
start_height = 107

The <2.1 epochs are each configured to mine at least 1 stacks block (if perfectly in sync with the burnchain block height). My config has those set to the minimum possible values taken from the original PR where this config was introduced, which allows booting into an epoch 2.1 Stacks node ASAP. I'll look into adjust the regtest env setup and API to use this config and verify the results.

We should consider fixing/implementing a "fast epoch2.1" config to avoid this issue, because while it's useful to test the epoch transitions right now, as soon as 2.1 launches, this extra ceremony will just become a developer burden.

[zone117x]

The root of this issue appears to be related to the restarting the miner after it has synced any bitcoind chainstate (but before it has mined the first block).

The regtest docker image I'm using performs the following in the image build steps:

1. Run bitcoind regtest and mine 100 blocks
2. Run a krypton stacks node, wait for it to sync the 100 bitcoin blocks
3. Gracefully shutdown the stacks node and bitcoind, persisting the chainstate data in the docker image

Later, when the docker image is ran, both the stacks node and bitcoind are started from the existing chainstate and start mining stacks blocks immediately. This setup saves significant time during API development iteration cycles, as well as during CI. It has been working well except for a bug related to the restart step causing the epoch 2.0 event data to not be emitted once the miner is resumed from existing chainstate.

My guess is that the burnchain.epochs config isn't fully applied throughout the codebase (perhaps around the event-emitter logic?) when a stacks-node + regtest environment is started from existing chainstate.

[gregorycoppola]

Is it meant to be supported to stop and re-start the node like this?

Which exact command are you using to actually start the regtest program?

Is there any output to show how the result of running this flow is wrong?

[zone117x]

Is it meant to be supported to stop and re-start the node like this?

Not sure, seems like a bug to me.

Which exact command are you using to actually start the regtest program?

The commands and config are fairly self-contained here: https://github.com/zone117x/stacks-regtest-env/blob/5ef3683275f49e43209d4b0cb2280df1b0615ed3/Dockerfile.e2e#L97-L113

Is there any output to show how the result of running this flow is wrong?

Here's the output of the event stream when attaching an event-observer that prints the results:
https://gist.github.com/zone117x/4c3b800b0886bcb6895dbb128c7c2300

[gregorycoppola]

I'm looking into this actively now.

I'm trying to figure out the intended output, by making a docker script that does not do the snapshot step.

[gregorycoppola]

i see the difference... probably relates to this code...

[gregorycoppola]

Issue_3367_A_bug_with_genesis_boot_events_that_seemingly_requires_a_difficult_refactor.pdf

@jcnelson I investigated this bug and wrote the above report about 1) what seems to be wrong, and 2) potential fixes.

It seems like all potential fixes are difficult/risky.

I would really like to get your opinion at least before trying any of them.

And, arguably you would be the best person to refactor if we do a trickier option.

[lgalabru]

If the goal of this effort is to create a "pre-heated" chainstate for faster devnet/regtest boots, we could add a cli flag that would gracefully exit the node right after the bitcoin sync/leader key registration, but before genesis block.
With that, your stacks-node can boot on top of a bitcoin chain state initialized, and the genesis block can be tweaked.
Thoughts?

[gregorycoppola]

With that, your stacks-node can boot on top of a bitcoin chain state initialized, and the genesis block can be tweaked.

@lgalabru What do you mean by "tweaked" here?

[lgalabru]

Ability to customize genesis balances.

[gregorycoppola]

Well, also the boot contracts don't get output.

One of the three suggestions in the doc is to move the generation of these events to run at the time of "stacks block 1", which I think is kind of what you are saying... but I wasn't confident to make this kind of change by myself.

[jcnelson]

I'm going to make it so that the system just pushes the boot receipts for a designated genesis block (whose height is 0). The boot receipts will be pushed immediately after the chainstate DB is instantiated, but before any bitcoin blocks get downloaded. Note that this is different from how it works now, where the boot receipts are pushed with the first non-genesis block (whose height is 1).

[jcnelson]

Completed!

[blockstack-devops]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.