v1.1.1

• Added FORCE_FINALITY_AFTER_BLOCKS environment variable to override block finality information at the reader/poller level. This allows an operator to pretend that finality is still progressing, N blocks behind HEAD, in the case where a beacon chain fails to do so and is intended as a workaround for deprecated chains like Goerli.

v1.1.0

• Updated substreams and dgrpc to latest versions to reduce logging.

• Tools printing Firehose Block model to JSON now have --proto-paths take higher precedence over well-known types and even the chain itself, the order is --proto-paths > chain > well-known (so well-known is lookup last).

• The tools print one-block now works correctly on blocks generated by omni-chain firecore binary.

• The various health endpoint now sets Content-Type: application/json header prior sending back their response to the client.

• The firehose, substreams-tier1 and substream-tier2 health endpoint now respects the common-system-shutdown-signal-delay configuration value meaning that the health endpoint will return false now if SIGINT has been received but we are still in the shutdown unready period defined by the config value. If you use some sort of load balancer, you should make sure they are configured to use the health endpoint and you should common-system-shutdown-signal-delay to something like 15s.

• The firecore.ConsoleReader gained the ability to print stats as it ingest blocks.

• The firecore.ConsoleReader has been made stricter by ensuring Firehose chain exchange protocol is respected.

• Changed reader logger back to reader-node to fit with the app's name which is reader-node.

• Fix -c "" not working properly when no arguments are present when invoking start command.

• Fix tools compare-blocks that would fail on new format.

• Fix substreams to correctly delete .partial files when serving a request that is not on a boundary.

• Add Antelope types to the blockchain's known types.

v1.0.0

This is a major release.

Operators

Important

When upgrading your stack to firehose-core v1.0.0, be sure to upgrade all components simultaneously because the block encapsulation format has changed.
Blocks that are merged using the new merger will not be readable by previous versions.
Blocks that are sent on the wire (ex: through the relayer) with the previous version will not be readable by the new components (ex: firehose)
This will require either a "Stop the world" upgrade, or a very careful execution of the following steps:

1. stop merger
2. start a new reader (making sure it is not picked up by the relayer
3. stop relayer+firehose+substreams (DOWNTIME STARTS HERE)
4. launch new version of relayer+firehose+substreams that connects to the new reader (DOWNTIME ENDS HERE)
5. start new merger
6. replace other components like index-builder

Added

• New binary firecore which can run all firehose components (reader, reader-stdin, merger, relayer, firehose, substreams-tier1|2) in a chain-agnostic way. This is not mandatory (it can still be used as a library) but strongly suggested when possible.

• Current Limitations on Ethereum:

  • The firecore firehose app does not support transforms (filters, header-only --for graph-node compatibility--) so you will want to continue running this app from fireeth
  • The firecore substreams apps do not support eth_calls so you will want to continue running them from fireeth
  • The firecore reader does not support the block format output by the current geth firehose instrumentation, so you will want to continue running it from fireeth

• New BlockPoller library to facilitate the implementation of rpc-poller-based chains, taking care of managing reorgs

• Considering that firehose-core is chain-agnostic, it's not aware of the different of the different block types. To be able to use tools around block decoding/printing,
  there are two ways to provide the type definition:

  1. the 'protoregistry' package contains well-known block type definitions (ethereum, near, solana, bitcoin...), you won't need to provide anything in those cases.
  2. for other types, you can provide additional protobuf files using --proto-path flag

Changed

• Merged blocks storage format has been changed. Current blocks will continue to be decoded, but new merged blocks will not be readable by previous software versions.
• The code from the following repositories have been merged into this repo. They will soon be archived.
  • github.com/streamingfast/node-manager
  • github.com/streamingfast/merger
  • github.com/streamingfast/relayer
  • github.com/streamingfast/firehose
  • github.com/streamingfast/index-builder

v0.2.4

• Fixed SF_TRACING feature (regression broke the ability to specify a tracing endpoint)
• Firehose connections rate-limiting will now force a delay of between 1 and 4 seconds (random value) before refusing a connection when under heavy load
• Fixed substreams GRPC/Connect error codes not propagating correctly

v0.2.3

Fixed

• fixed typo in check-merged-blocks preventing its proper display of missing ranges

v0.2.2

Added

• Firehose logs now include auth information (userID, keyID, realIP) along with blocks + egress bytes sent.

Fixed

• Filesource validation of block order in merged-blocks now works correctly when using indexes in firehose Blocks queries

Removed

• Flag substreams-rpc-endpoints removed, this was present by mistake and unused actually.
• Flag substreams-rpc-cache-store-url removed, this was present by mistake and unused actually.
• Flag substreams-rpc-cache-chunk-size removed, this was present by mistake and unused actually.

v0.2.1

Integrators

Note

For integrations updating firehose-core, check the upgrade procedure to go to v0.1.x -> v0.2.0

Operators

Important

We have had reports of older versions of this software creating corrupted merged-blocks-files (with duplicate or extra out-of-bound blocks)
This release adds additional validation of merged-blocks to prevent serving duplicate blocks from the firehose or substreams service.
This may cause service outage if you have produced those blocks or downloaded them from another party who was affected by this bug.

• Find the affected files by running the following command (can be run multiple times in parallel, over smaller ranges)

  tools check merged-blocks-batch <merged-blocks-store> <start> <stop>
  

• If you see any affected range, produce fixed merged-blocks files with the following command, on each range:

  tools fix-bloated-merged-blocks <merged-blocks-store> <output-store> <start>:<stop>
  

• Copy the merged-blocks files created in output-store over to the your merged-blocks-store, replacing the corrupted files.

Removed

• Removed the --dedupe-blocks flag on tools download-from-firehose as it can create confusion and more issues.

Fixed

• Bumped bstream: the filesource will now refuse to read blocks from a merged-files if they are not ordered or if there are any duplicate.
• The command tools download-from-firehose will now fail if it is being served blocks "out of order", to prevent any corrupted merged-blocks from being created.
• The command tools print merged-blocks did not print the whole merged-blocks file, the arguments were confusing: now it will parse <start_block> as a uint64.
• The command tools unmerge-blocks did not cover the whole given range, now fixed

Added

• Added the command tools fix-bloated-merged-blocks to try to fix merged-blocks that contain duplicates and blocks outside of their range.
• Command tools print one-block and merged-blocks now supports a new --output-format jsonl format.
  Bytes data can now printed as hex or base58 string instead of base64 string.

Changed

• Changed tools check merged-blocks-batch argument syntax: the output-to-store is now optional.

v0.1.12

• Added tools check merged-blocks-batch to simplify checking blocks continuity in batched mode, writing results to a store
• Bumped substreams to v1.1.20 with a fix for some minor bug fixes related to start block processing

v0.1.11

• Bumped substreams to v1.1.18 with a regression fix for when a substreams has a start block in the reversible segment

v0.1.10

Added

The --common-auth-plugin got back the ability to use secret://<expected_secret>?[user_id=<user_id>]&[api_key_id=<api_key_id>] in which case request are authenticated based on the Authorization: Bearer <actual_secret> and continue only if <actual_secret> == <expected_secret>.

Changed

• Bumped substreams to v1.1.17 with provider new metrics substreams_active_requests and substreams_counter