Skip to content

v3.0.0
Compare
Choose a tag to compare
@sduchesneau sduchesneau released this 27 Sep 20:02
v3.0.0
cd6d9ac

Highlights

This release brings a major refactoring enabling support for multiple database drivers and not just Postgres anymore. Our first newly supported driver is Clickhouse which defines itself as The fastest and most resource efficient open-source database for real-time apps and analytics. In the future, further database driver could be supported like MySQL, MSSQL and any other that can talk the SQL protocol.

Now that we support multiple driver, keeping the substreams-sink-postgres didn't make sense anymore. As such, we have renamed the project from substreams-sink-postgresql to substreams-sink-sql since it now supports Clickhouse out of the box. The binary and Go modules have been renamed in consequence.

Another major change brought by this release is the usage of Substreams "Deployable Unit". What we call a "Deployable Unit" is a Substreams manifest that fully defines a deployment packaged as a single artifact. This change how the sink is operated; the SQL schema, output module and "Network" identifer are now passed in the "SinkConfig" section of the Substreams manifest instead of being accepted at command line.

Read the Operators section below to learn how to migrate to this new version.

Operators

Passing theschema and the module_name to the run and setup commands is no longer accepted via arguments, they need to be written to the substreams.yaml file.

Before:

substreams-sink-sql setup <dsn> "path/to/schema.sql"
substreams-sink-sql run <dsn> mainnet.eth.streamingfast.io:443 https://github.com/streamingfast/substreams-eth-block-meta/releases/download/v0.5.1/substreams-eth-block-meta-v0.5.1.spkg db_out [<range>]

Now:

  • Create a deployable unit file, let's call it substreams.prod.yaml with content:
specVersion: v0.1.0
package:
  name: "<name>"
  version: v0.0.1

imports:
  sql: https://github.com/streamingfast/substreams-sink-sql/releases/download/protodefs-v1.0.1/substreams-sink-sql-protodefs-v1.0.1.spkg
  main: https://github.com/streamingfast/substreams-eth-block-meta/releases/download/v0.5.1/substreams-eth-block-meta-v0.5.1.spkg

network: mainnet

sink:
  module: main:db_out
  type: sf.substreams.sink.sql.v1.Service
  config:
    schema: "./path/to/schema.sql"

In this <name> is the same name as what <manifest> defines was, https://github.com/streamingfast/substreams-eth-block-meta/releases/download/v0.5.1/substreams-eth-block-meta-v0.5.1.spkg is the current manifest you deploy.

The ./path/to/schema.sql would point to your schema file (path resolved relative to parent directory of substreams.prod.yaml).

The 'network: mainnet' will be used to resolve to an endpoint. You can configure each network to have its own endpoint via environment variables SUBSTREAMS_ENDPOINTS_CONFIG_<NETWORK> or override this mechanism completely by using the --endpoint flag. Most used networks have default endpoints.

  • Setup your database:
substreams-sink-sql setup <dsn> substreams.prod.yaml
  • Run the sink:
substreams-sink-sql run <dsn> substreams.prod.yaml [<range>]

Similar changes have been applied to other commands as well.

Assets 7
Loading