v1.3.0

Highlights

• Support new networks configuration block in substreams.yaml to override modules' params and initial_block. Network can be specified at run-time, avoiding the need for separate spkg files for each chain.
• [BREAKING CHANGE] Remove the support for the deriveFrom overrides. The imports, along with the new networks feature, should provide a better mechanism to cover the use cases that deriveFrom tried to address.

Note

These changes are all handled in the substreams CLI, applying the necessary changes to the package before sending the requests. The Substreams server endpoints do not need to be upgraded to support it.

Added

• Added networks field at the top level of the manifest definition, with initialBlock and params overrides for each module. See the substreams.yaml.example file in the repository or https://substreams.streamingfast.io/reference-and-specs/manifests for more details and example usage.
• The networks params and `initialBlock`` overrides for the chosen network are applied to the module directly before being sent to the server. All network configurations are kept when packing an .spkg file.
• Added the --network flag for choosing the network on run, gui and alpha service deploy commands. Default behavior is to use the one defined as network in the manifest.
• Added the --endpoint flag to substreams alpha service serve to specify substreams endpoint to connect to
• Added endpoints for Antelope chains
• Command 'substreams info' now shows the params

Removed

• Removed the handling of the DeriveFrom keyword in manifest, this override feature is going away.
• Removed the `--skip-package-validation`` option only on run/gui/inspect/info

Changed

• Added the --params flag to alpha service deploy to apply per-module parameters to the substreams before pushing it.
• Renamed the --parameters flag to --deployment-params in alpha service deploy, to clarify the intent of those parameters (given to the endpoint, not applied to the substreams modules)
• Small improvement on substreams gui command: no longer reads the .spkg multiple times with different behavior during its process.

v1.2.0

Client

• Fixed bug in substreams init with numbers in ABI types

Backend

• Return the correct GRPC code instead of wrapping it under an "Unknown" error. "Clean shutdown" now returns CodeUnavailable. This is compatible with previous substreams clients like substreams-sql which should retry automatically.
• Upgraded components to manage the new block encapsulation format in merged-blocks and on the wire required for firehose-core v1.0.0

v1.1.22

alpha service deployments

• Fix fuzzy matching when endpoint require auth headers
• Fix panic in "serve" when trying to delete a non-existing deployment
• Add validation check of substreams package before sending deploy request to server

v1.1.21

Changed

• Codegen: substreams-database-change to v1.3, properly generates primary key to support chain reorgs in postgres sink.

• Sink server commands all moved from substreams alpha sink-* to substreams alpha service *

• Sink server: support for deploying sinks with DBT configuration, so that users can deploy their own DBT models (supported on postgres and clickhouse sinks). Example manifest file segment:

  [...]
  
  sink:
    module: db_out
    type: sf.substreams.sink.sql.v1.Service
    config:
      schema: "./schema.sql"
      wire_protocol_access: true
      postgraphile_frontend:
        enabled: true
      pgweb_frontend:
        enabled: true
      dbt:
        files: "./dbt"
        run_interval_seconds: 60

  where "./dbt" is a folder containing the dbt project.

• Sink server: added REST interface support for clickhouse sinks. Example manifest file segment:

  [...]
    
  sink:
    module: db_out
    type: sf.substreams.sink.sql.v1.Service
    config:
      schema: "./schema.clickhouse.sql"
      wire_protocol_access: true
      engine: clickhouse
      postgraphile_frontend:
        enabled: false
      pgweb_frontend:
        enabled: false
      rest_frontend:
        enabled: true

Fixed

• Fix substreams info cli doc field which wasn't printing any doc output

v1.1.20

• Optimized start of output stream in developer mode when start block is in reversible segment and output module does not have any stores in its dependencies.
• Fixed bug where the first streamable block of a chain was not processed correctly when the start block was set to the default zero value.

v1.1.19

Changed

• Codegen: Now generates separate substreams.{target}.yaml files for sql, clickhouse and graphql sink targets.

Added

• Codegen: Added support for clickhouse in schema.sql

Fixed

• Fixed metrics for time spent in eth_calls within modules stats (server and GUI)
• Fixed undo json message in 'run' command
• Fixed stream ending immediately in dev mode when start/end blocks are both 0.
• Sink-serve: fix missing output details on docker-compose apply errors
• Codegen: Fixed pluralized entity created for db_out and graph_out

v1.1.18

Fixed

• Fixed a regression where start block was not resolved correctly when it was in the reversible segment of the chain, causing the substreams to reprocess a segment in tier 2 instead of linearly in tier 1.

v1.1.17

Fixed

• Missing decrement on metrics substreams_active_requests

v1.1.16

Added

• substreams_active_requests and substreams_counter metrics to substreams-tier1

Changed

• evt_block_time in ms to timestamp in lib.rs, proto definition and schema.sql

v1.1.15

Highlights

• This release brings the substreams init command out of alpha! You can quickly generate a Substreams from an Ethereum ABI:
  

• New Alpha feature: deploy your Substreams Sink as a deployable unit to a local docker environment!
  

• See those two new features in action in this tutorial

Added

• Sink configs can now use protobuf annotations (aka Field Options) to determine how the field will be interpreted in
  substreams.yaml:

  • load_from_file will put the content of the file directly in the field (string and bytes contents are supported).
  • zip_from_folder will create a zip archive and put its content in the field (field type must be bytes).

  Example protobuf definition:

  import "sf/substreams/v1/options.proto";
  
  message HostedPostgresDatabase {
    bytes schema = 1 [ (sf.substreams.v1.options).load_from_file = true ];
    bytes extra_config_files = 2 [ (sf.substreams.v1.options).zip_from_folder = true ];
  }
  

  Example manifest file:

  [...]
  network: mainnet
  
  sink:
    module: main:db_out
    type: sf.substreams.sink.sql.v1.Service
    config:
      schema: "./schema.sql"
      wire_protocol_access: true
      postgraphile_frontend:
        enabled: true
      pgweb_frontend:
        enabled: true

• substreams info command now properly displays the content of sink configs, optionally writing the fields that were bundled from files to disk with --output-sinkconfig-files-path=</some/path>

Changed

• substreams alpha init renamed to substreams init. It now includes db_out module and schema.sql to support the substreams-sql-sink directly.

• The override feature has been overhauled. Users may now override an existing substreams by pointing to an override file in run or gui command.
  This override manifest will have a deriveFrom field which points to the original substreams which is to be overriden.
  This is useful to port a substreams to one network to another.
  Example of an override manifest:

  deriveFrom: path/to/mainnet-substreams.spkg #this can also be a remote url
  
  package:
    name: "polygon-substreams"
    version: "100.0.0"
  
  network: polygon
  
  initialBlocks:
    module1: 17500000
  params:
    module1: "address=2a75ca72679cf1299936d6104d825c9654489058"
  

• The substreams run and substreams gui commands now determine the endpoint from the 'network' field in the manifest if no value is passed in the --substreams-endpoint flag.

• The endpoint for each network can be set by using an environment variable SUBSTREAMS_ENDPOINTS_CONFIG_<network_name>, ex: SUBSTREAMS_ENDPOINTS_CONFIG_MAINNET=my-endpoint:443

• The substreams alpha init has been moved to substreams init

Fixed

• fixed the substreams gui command to correctly compute the stop-block when given a relative value (ex: '-t +10')