Having the wrong start block generates non-intuitive error messages

[matthewdarwin]

firehose-antelope is hard-coded to start at block 2.
In this chain, the first start block is actually 3.

After upgrading to latest firehose-core, the substreams tier1 node doesn't start. Log is below. Settting common-first-streamable-block: 3 allows the substreams tier1 node to start

In earlier versions of the software I guess having the correct start block was not checked.
Perhaps the error message could be more clear?

Maybe print a message it is checking the start block can be found? Not sure what is best. It took a bit to figure out the problem from the debug messages available.

Jan 26 03:27:29 telostest-sfsf69 sf[196934]: 2024-01-26T03:27:29.194Z INFO (dtracing) registering development exporters from environment variables
Jan 26 03:27:29 telostest-sfsf69 sf[196934]: 2024-01-26T03:27:29.194Z DEBG (fireantelope) ulimit max open files before adjustment {"current_value": 1000000}
Jan 26 03:27:29 telostest-sfsf69 sf[196934]: 2024-01-26T03:27:29.194Z DEBG (fireantelope) no need to update ulimit as it's already higher than our good enough value {"good_enough_value": 1000000}
Jan 26 03:27:29 telostest-sfsf69 sf[196934]: 2024-01-26T03:27:29.194Z INFO (fireantelope) setting GOMEMLIMIT relative to available memory {"percent": 95}
Jan 26 03:27:29 telostest-sfsf69 sf[196934]: 2024-01-26T03:27:29.195Z DEBG (fireantelope) fireantelope binary started {"data_dir": "/var/lib/dfuse"}
Jan 26 03:27:29 telostest-sfsf69 sf[196934]: 2024-01-26T03:27:29.195Z INFO (fireantelope) starting Firehose on Antelope with config file '/etc/dfuse/config.yml'
Jan 26 03:27:29 telostest-sfsf69 sf[196934]: 2024-01-26T03:27:29.196Z DEBG (fireantelope) launcher created
Jan 26 03:27:29 telostest-sfsf69 sf[196934]: 2024-01-26T03:27:29.196Z INFO (fireantelope) launching applications: substreams-tier1
Jan 26 03:27:29 telostest-sfsf69 sf[196934]: 2024-01-26T03:27:29.196Z DEBG (fireantelope) creating application {"app": "substreams-tier1"}
Jan 26 03:27:29 telostest-sfsf69 sf[196934]: 2024-01-26T03:27:29.196Z INFO (substreams-tier1) setting up grpc authenticator {"config_url": "grpc://pinax-authenticator-grpc.service.march.consul.eosn.io:9000?continuous=true", "en
dpoint": "pinax-authenticator-grpc.service.march.consul.eosn.io:9000", "continuous_auth": true}
Jan 26 03:27:29 telostest-sfsf69 sf[196934]: 2024-01-26T03:27:29.196Z DEBG (fireantelope) launching app {"app": "substreams-tier1"}
Jan 26 03:27:29 telostest-sfsf69 sf[196934]: 2024-01-26T03:27:29.196Z INFO (substreams-tier1) running substreams-tier1 {"config": {"MergedBlocksStoreURL":"s3://ceph-gw.service.march.consul.eosn.io:7480/telostest-sf-v3-merged-bl
ocks?region=none&insecure=true","OneBlocksStoreURL":"s3://ceph-gw.service.march.consul.eosn.io:7480/telostest-sf-v3-one-blocks?region=none&insecure=true","ForkedBlocksStoreURL":"s3://ceph-gw.service.march.consul.eosn.io:7480/te
lostest-sf-v3-forked-blocks?region=none&insecure=true","BlockStreamAddr":"dns:///telostest-sf-relayer-v3.service.march.consul.eosn.io:9000","GRPCListenAddr":":9999","GRPCShutdownGracePeriod":1000000000,"ServiceDiscoveryURL":nul
l,"StateStoreURL":"/var/lib/dfuse/localdata","StateStoreDefaultTag":"","StateBundleSize":1000,"MaxSubrequests":4,"SubrequestsEndpoint":"localhost:5000","SubrequestsInsecure":false,"SubrequestsPlaintext":true,"WASMExtensions":nu
ll,"PipelineOptions":null,"Tracing":false}}
Jan 26 03:27:29 telostest-sfsf69 sf[196934]: 2024-01-26T03:27:29.197Z DEBG (dstore) sanitized base path {"original_base_path": "/var/lib/dfuse/localdata", "sanitized_base_path": "/var/lib/dfuse/localdata"}
Jan 26 03:27:29 telostest-sfsf69 sf[196934]: 2024-01-26T03:27:29.197Z INFO (substreams-clients) setting up basic grpc client factory (no XDS bootstrap)
Jan 26 03:27:29 telostest-sfsf69 sf[196934]: 2024-01-26T03:27:29.197Z DEBG (substreams-service) setting up firehose source
Jan 26 03:27:29 telostest-sfsf69 sf[196934]: 2024-01-26T03:27:29.197Z INFO (substreams-service) creating new joining source {"cursor": "<nil>", "start_block_num": 2}
Jan 26 03:27:29 telostest-sfsf69 sf[196934]: 2024-01-26T03:27:29.197Z DEBG (bstream) error getting source_from_block_num {"error": "no lib"}
Jan 26 03:27:29 telostest-sfsf69 sf[196934]: 2024-01-26T03:27:29.197Z DEBG (bstream) source connecting {"endpoint_url": "dns:///telostest-sf-relayer-v3.service.march.consul.eosn.io:9000"}
Jan 26 03:27:29 telostest-sfsf69 sf[196934]: 2024-01-26T03:27:29.203Z DEBG (bstream) new incoming file {"filename": "0000000000"}
Jan 26 03:27:29 telostest-sfsf69 sf[196934]: 2024-01-26T03:27:29.203Z DEBG (substreams-service) launching processing of file {"base_filename": "0000000000"}
Jan 26 03:27:29 telostest-sfsf69 sf[196934]: 2024-01-26T03:27:29.203Z DEBG (substreams-service) open files {"count": 1, "filename": "0000000000"}
Jan 26 03:27:29 telostest-sfsf69 sf[196934]: 2024-01-26T03:27:29.203Z DEBG (substreams-service) feeding from incoming file {"filename": "0000000000"}
Jan 26 03:27:29 telostest-sfsf69 sf[196934]: 2024-01-26T03:27:29.213Z DEBG (bstream) error getting source_from_block_num {"error": "no lib"}
Jan 26 03:27:29 telostest-sfsf69 sf[196934]: 2024-01-26T03:27:29.213Z DEBG (substreams-service) source shutting down {"error": "stop block reached"}
Jan 26 03:27:29 telostest-sfsf69 sf[196934]: 2024-01-26T03:27:29.213Z ERRO (fireantelope)
Jan 26 03:27:29 telostest-sfsf69 sf[196934]: ################################################################
Jan 26 03:27:29 telostest-sfsf69 sf[196934]: Fatal error in app substreams-tier1:
Jan 26 03:27:29 telostest-sfsf69 sf[196934]: getting block type from stream factory: stop block reached
Jan 26 03:27:29 telostest-sfsf69 sf[196934]: ################################################################

[matthewdarwin]

Note also that the firehose grpc service starts fine with the wrong start block. Probably it should do the same check as substreams?

[sduchesneau]

That "check" at the launch of substreams was a bad idea. We now have the flag --substreams-tier1-block-type to specify the type of block to expose natively in substreams (this flag was actually "just" renamed yesterday)
Checking merged-blocks just to find a "block type" is now the fallback mechanism if none were set.
fireeth now specifies the blocktype in the config, if you run a custom binary for antelope (fireantelope), you could set the fixed antelope block type in the "chain config", like this:
https://github.com/streamingfast/firehose-ethereum/blob/71064c8f67534d5418e86672422d311c1199e70c/cmd/fireeth/main.go#L47

With this (either hardcoded in fireantelope or passed as a parameter to firecore), the program will no longer try to load blocks at boot time and fail miserably, which used to cause more harm than good.

It doesn't solve your need to automatically detect a config error (which will manifest later when trying to get blocks) but it will probably bubble up as errors that make more sense (like "no linkable block" or something like that).

A runtime check of the existing block is IMHO a bad idea, it causes quite a few edge cases when bootstrapping chains or starting up with a chain that doesn't have the first 100 merged blocks file yet

[matthewdarwin]

Just to note, Johnathan from my team ran into this problem again recently.

[matthewdarwin]

See also: #62