Skip to content

Latest commit

 

History

History
478 lines (327 loc) · 21.4 KB

CHANGELOG.md

File metadata and controls

478 lines (327 loc) · 21.4 KB
Raw

Change log

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

0.6.0

List of changes

  • Enabled the 'skip_empty_output' instruction in all maps and stores by default, unless the macro is called with the keep_empty_output parameter, like this: #[substreams::handlers::map(keep_empty_output)]
  • Bumped the prost dependencies from 0.11 to 0.13.3

Upgrading your substreams project

Because of code generated by "buf neoeinstein-prost" plugin and since Rust allows multiple versions of a dependency to be loaded, but we export public functions to the wasm interface, it is very important to align all the dependencies that may depend on this substreams library.

Here are the changes that you must perform in your substreams project to upgrade to v0.6.0:

  1. in buf.gen.yaml
  • bump buf.build/community/neoeinstein-prost:v0.4.0
  • bump buf.build/community/neoeinstein-prost-crate:v0.4.1
  1. in Cargo.toml, update the dependencies/build_deps only for the modules that you already have:
  • bump substreams = "0.6"
  • bump prost = "0.13"
  • bump prost-types = "0.13"
  • bump prost-build = "0.13"
  • bump substreams-ethereum = "0.10"
  • bump substreams-antelope = "0.6"
  • bump substreams-database-change = "2"
  • bump substreams-entity-change = "2"
  1. Run substreams build again

Debugging errors after the upgrade

Unimplemented prost::message::Message

10  | #[substreams::handlers::map]
    | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ the trait `prost::message::Message` is not implemented for `MyData`

This means that you have a mismatch between the generated protobuf bindings (usually generated by the buf neoinstein-prost and neoinstein-prost-crate plugins) and the prost/prost-types/prost-build version in one of your Cargo.toml. Make sure that your buf.gen.yaml file exists and points to the versions specified above, as well as the versions for prost libraries in every Cargo.toml.

Two different versions of crate being used

error[E0308]: mismatched types
    --> src/lib.rs:17:36
     |
17   |     my_data.block_timestamp = Some(blk.timestamp().to_owned());
     |                               ---- ^^^^^^^^^^^^^^^^^^^^^^^^^^ expected `Timestamp`, found `prost_types::protobuf::Timestamp`
     |                               |
     |                               arguments to this enum variant are incorrect
     |
     = note: `prost_types::protobuf::Timestamp` and `Timestamp` have similar names, but are actually distinct types
note: `prost_types::protobuf::Timestamp` is defined in crate `prost_types`
    --> /Users/you/.cargo/registry/src/index.crates.io-6f17d22bba15001f/prost-types-0.11.9/src/protobuf.rs:2296:1
     |
2296 | pub struct Timestamp {
     | ^^^^^^^^^^^^^^^^^^^^
note: `Timestamp` is defined in crate `prost_types`
    --> /Users/you/.cargo/registry/src/index.crates.io-6f17d22bba15001f/prost-types-0.13.3/src/protobuf.rs:2253:1
     |
2253 | pub struct Timestamp {
     | ^^^^^^^^^^^^^^^^^^^^
     = note: perhaps two different versions of crate `prost_types` are being used?

This means that one of your dependencies uses the wrong version of prost-types (0.11.9 in this example). You probably forgot to bump a dependency in one of your Cargo.toml, for example substreams-ethereum or substreams-antelope.

Symbol multiply defined

warning: Linking globals named 'alloc': symbol multiply defined!

error: failed to load bitcode of module "substreams-cc542fa47b990c4e.substreams.f89693ec9899ca95-cgu.05.rcgu.o": 

warning: `my_project` (lib) generated 1 warning
error: could not compile `my_project` (lib) due to 1 previous error; 1 warning emitted

This means that the this module (substreams) is linked many times with different versions. This is invalid because of the global exports, make sure that only the version 0.6.0 of substreams-rs is used by any of your dependencies.

0.5.22

  • Added ExprMatcher, and constructor(s) expr_matcher and ExprMatcher::new. This can be used to parse an expression once and re-used it to run multiple matches_keys against different keys:

      let matcher = substreams::expr_matcher(&query);

  transactions.flat_map(|trx| {
      trx.instructions()
          .filter(|view| matcher.matches_keys(&vec![format!("program:{}", view.program_id().to_string())]))
  });

0.5.21

  • Add skip_empty_output intrinsic (requires substreams server version v1.9.0 or later)

0.5.20

  • Accepts - character in expression's parsing expect at the beginning of the expression. (Ex: -test is failing, test-8 is not)

0.5.19

  • The return of store type StoreSetSum which allows both summing and setting a value in a store. This is useful for storing aggregated values in a store. Note: to read deltas from this store, you will need to use the Deltas type. The delta values will come prefixed with "set:" or "sum:" depending on the operation, followed by the string representation of the value. It is up to the user to decode the values as necessary.

0.5.18

  • Accept all characters within index keys except -,', ", &&, ||, ), ( and spaces of all kind.

0.5.17

  • Removed StoreSetSum. (Reverted changes from 0.5.16 and 0.5.15)

0.5.16

  • Add StoreSetSum to the WRITABLE_STORE array

0.5.15

  • Add new store type StoreSetSum which allows both summing and setting a value in a store. This is useful for storing aggregated values in a store.

0.5.14

  • Add index keys protobuf in substreams crate
  • Add matches_keys_in_parsed_expr function returning a bool. It returns true, if the set of keys provided, matches the expression. (Ex: expression: (key1 || key2), if the set of keys contains key1 or contains key2, matches_keys_in_parsed_expr(keys, expression) returns true, else returns false)

0.5.13

Added support for specifying let mut <param>: <Type> when using the #[substreams::handlers::map] macros, this enables in-place trimming of received data.

0.5.12

Added back some lost binary operations on BigInt like BigInt - BigInt for example.

0.5.11

Highlights

In this release we add further arithmetics and logics operators possibilities on BigInt type, namely:

  • BitAnd
  • BitAndAssign
  • BitOr
  • BitOrAssign
  • BitXor
  • Rem
  • Pow
  • Shl
  • ShlAssign
  • ShrAssign

The BigInt now also has a new method div_rem that performs division by another BigInt and returns the quotient and the remainder.

We also improved error messages on a variety of error cases for example BigInt::to_u32 and BigInt::to_u64 will now print a much more useful message that should make it clear that the value is too big.

Changed

  • Removed all internal usage of .unwrap() and replaced them with .expect(<message>) or .unwrap_or_else(|| panic!(<message>)) improving the error message on different edge cases that can happen from time to time.

0.5.10

Highlights

This release brings:

  • You can now pass hexadecimal strings with the 0x prefix to the Hex::decode function. Hexadecimal strings without the prefix are still supported.

Example

...
use substreams::Hex;

let hex = Hex::decode("0x6e60bCdF52078A250932CF9FeC174c5F67348845")
...

Changed

  • The Hex::decode function now accepts hexadecimal strings with the 0x prefix. Hexadecimal strings without the prefix are still supported.
  • Fixed warning that std::mem::forget was called a reference while it should receive an owned object.

0.5.9

Highlights

This release brings:

  • #[substreams::handlers::map] now handles Result<Option<T>, Error> and Option<T>
  • Improved Error Handling

#[substreams::handlers::map] now handles Result<Option<T>, Error> and Option<T>

It's now possible to avoid sending back output from your mapper entirely by using Option<T> or Result<Option<T>>. This should be used whenever you are not returning something every block. This can make some use cases easier to "view" and comes with a small improved speed as the Protobuf encoding of an "empty" object will be avoided completely and a WASM intrinsic call will be avoided. Here an example of a Result<Option<T>, Error>:

#[substreams::handlers::map]
fn map_transfers(blk: eth::Block) -> Result<Option<erc721::Transfers>, substreams::errors::Error> {
    if some_condition {
        return Ok(None);
    }

    Ok(Some(erc721::Transfers {
        // ...
    }))
}

And plain Option:

#[substreams::handlers::map]
fn map_transfers(blk: eth::Block) -> Option<erc721::Transfers> {
    if some_condition {
        return None;
    }

    Some(erc721::Transfers {
        // ...
    })
}

Improved Error Handling

The substreams::errors::Error is now a plain alias to anyhow::Error which means it much easier to create generic errors, contextualize existing one and we gain the ability to be converted from any error that implements std:error:Error interface which is the majority of errors out there. This enables proper usage of the ? Rust operator.

#[substreams::handlers::map]
fn map_transfers(params: String, block: Block) -> Result<Transfers, substreams::errors::Error> {
    let address = Hex::decode(params)?;

    // ...
}

Here, a decoding error returned by Hex::decode will be converted to substreams::errors::Error and an early return will happen at that point. This will make error handling and reporting much easier.

The Rust anyhow library can now be used seamlessly to quickly write ad-hoc error as well as adding context to errors. First add anyhow as a dependency:

cargo add anyhow

Then use this code to contextualize another error:

use anyhow::Context;

#[substreams::handlers::map]
fn map_transfers(params: String, block: Block) -> Result<Transfers, substreams::errors::Error> {
    let address = Hex::decode(&params).with_context(|| format!("invalid address '{}'", &params))?;

    // ...
}

This should be a seamless upgrade for the vast majority of users. This change comes at the price that Error::Unexpected("msg".to_string()) is not available anymore. Add anyhow as a dependency to your project:

cargo add anyhow

And then convert substreams::errors:Error::Unexpected usage with:

use anyhow::anyhow;

#[substreams::handlers::map]
fn map_transfers(block: Block) -> Result<Transfers, substreams::errors::Error> {
    if block.number == 0 {
        return Err(anyhow!("invalid block #{}", block.number))
    }

    // ...
}

Changed

  • Added support Result<Option<>> and Option<> in substreams::handlers::map macro.

  • Breaking substreams::errors:Error is now an alias to anyhow:Error. This has been done for improving dealing with errors within Substreams Rust handler. If you were using substreams::errors:Error::Unexpected, now use Err(anyhow!("invalid block #{}", block.number)) (add anyhow = "1" as a dependency of your project).

0.5.8

Highlights

This is a re-packaging of https://github.com/streamingfast/substreams-rs/release/tag/v0.5.7 with a small removal that was actually wrong. Please see 0.5.7 release notes for highlights of previous release.

Fixed

  • Removed impl<I: Iterator> from Deltas, this was implemented using pop which returns deltas in reverse order.

0.5.7

Highlights

  • New helpers to work with store and delta keys.
  • Improved a bit performance of delta implementation.
  • BigInt and BigDecimal quality of life improvements.

In this release we add various helpers to more easily decode store keys and extract meaningful information from them as well as dealing with store deltas.

In a lot of use cases, you will encode data into your keys for example user:<address> or position:<pool>:<id>. The new helpers make it easier than before to work with those. The Substreams default key key format is now to use the : segment separator to separate logical part of a key.

Import at the top of your module the use substreams::store::DeltaExt; trait and gain access to key_segment_at_eq, key_first_segment_eq, key_last_segment_eq, key_first_segment_in and key_last_segment_in on iterator of type Delta.

The new key module can then be used to extract useful part of the key:

use substreams::key;
use substreams::store::{Delta, DeltaExt, Deltas, DeltaBigDecimal};
fn db_out(store: Deltas<DeltaBigDecimal>) {
    for delta in store.key_first_segment_eq("user") {
        let address = key::segment_at(delta.get_key(), 1);
        // Do something for this delta where the key was in format `user:<address>`
    }
}

Or when filtering for multiple segments:

use substreams::key;
use substreams::store::{Delta, DeltaExt, Deltas, DeltaBigDecimal};
fn db_out(store: Deltas<DeltaBigDecimal>) {
    for delta in store.key_first_segment_in(["user", "contract"]) {
        // Do something for this delta where the key was in format `(user|contract):...`
    }
}

The DeltaExt trait also brings in operation_eq and operation_not_eq to filter Deltas based on the actual operation.

use substreams::key;
use substreams::pb::substreams::store_delta::Operation;
use substreams::store::{Delta, DeltaExt, Deltas, DeltaBigDecimal};
fn db_out(store: Deltas<DeltaBigDecimal>) {
    for delta in store
        .iter()
        .operation_eq(Operation::Create)
        .key_first_segment_in(["user", "contract"])
        .key_last_segment_eq("token0")
   {
        // Do something for Create delta(s) where the key was in format `(user|contract):...:token0`
    }
}

Added

  • Core: Added key module which contains extractor segment_at, first_segment, last_segment, try_segment_at, try_first_segment and try_last_segment to extract parts of a key.
  • Stores: Added store::DeltaExt trait which contains predicates key_segment_at_eq, key_first_segment_eq, key_last_segment_eq, key_first_segment_in, key_last_segment_in, operation_eq and operation_not_eq for filtering of delta's keys.
  • Stores: Added get_key, get_operation to the Delta trait, implemented for all Delta implementations.
  • Macros: Add support for Option<T> and T as supported map output types, in addition to Result<T, ...>.
  • Scalars: BigInt and BigDecimal types now implement the std Default trait (defaults to 0) to be able to use unwrap_or_default().
  • Scalar: Added absolute method on BigInt and BigDecimal types.
  • Scalar: Added to_i32() on BigInt

Improved

  • Stores: Reduced amount of clone performed in the store module which should improve speed a bit.

Changed

  • Stores: Breaking Delta trait method new() has been removed, removed by a trait bound From<StoreDelta> on Deltas, shouldn't affect anyone.
  • Stores: Breaking Deltas now require the trait bound From<StoreDelta> implemented for all Delta implementations, shouldn't affect anyone.

0.5.6

  • Macros: Add StoreSetIfNotExists* to the list of supported stores.

0.5.5

  • Macros: Use std::mem::ManuallyDrop to manage memory instead of std::mem::forget for String input parameters.
  • Macros: Add StoreSetIfNotExistsString to the list of supported stores.

0.5.4

  • Bugfix: Fixed a bug where memory was not properly freed when using String input parameters in macros.
  • Added has_at, has_first and has_last methods to StoreGet.

0.5.3

Changed

  • Support common-sense cross-type arithmetic operations for BigInt and BigDecimal. (e.g. BigInt + f64 -> BigDecimal)

0.5.2

  • Fixed tests
  • Macros: Add support for String input parameters

0.5.1

  • Add from<usize> for BigDecimal
  • Added new method for BigInt and BigDecimal.
  • Removed forced precision of 100 when returning a BigDecimal from a store.
  • Fixed a bug where empty byte arrays were not properly handled

0.5.0

Added

  • Added BigInt::from_unsigned_bytes_be to create the BigInt from unsigned big endian bytes.

Changed

  • Breaking Changed signature of BigInt::from_store_bytes(bytes: Vec<u8>) to BigInt::from_store_bytes(bytes: &[u8]).
  • Breaking Changed signature of BigDecimal::from_store_bytes(bytes: Vec<u8>) to BigDecimal::from_store_bytes(bytes: &[u8]).
  • Improved implementation of BigDecimal::divide_by_decimals to rely on BigDecimal instead of a padded string.
  • Reduced allocation performed when using Store::set_if_not_exists_many, Store::set_many and Store::add_many functions.
  • Removed a bunch of unnecessary clones and removed some useless conversion which should increase overall speed of various Store and Scalar operations.

0.4.0

  • Renaming StoreSetIfNotExistsI64, StoreI64, DeltaI32, DeltaI64 to StoreSetIfNotExistsInt64, StoreInt64, DeltaInt32 and DeltaInt64.
  • Adding StoreSetString, StoreGetString and StoreGetArray typed stores.

0.3.2

  • Adding DeltaI32, DeltaBool and DeltaBytes.

0.3.1

  • Made Windows target(s) able to run tests when depending on substreams crate.

0.3.0

  • Abstraction of StoreDelete to implement delete_prefix and StoreNew.

  • Removing config flag wasm32.

0.2.1

  • Added conditional compilation to make sure code that is linked to wasm modules can only be compiled when a wasm target is specified. Non-wasm targets will skip compiling the linked code allowing the crate to be compiled with any target.

0.2.0

Breaking changes

  • Renamed all {Types}StoreGet (e.g.: BigDecimalStoreGet, BigIntStoreGet, etc.) and {Types}StoreSet (e.g.: BigDecimalStoreSet, BigIntStoreSet, etc.) to StoreGet{Types} and StoreSet{Types}
  • Renamed all {Types}Delta (e.g.: DeltaBigDecimal, DeltaBigInt) to Delta{Types}
  • Added StoreGetI64 and StoreSetI64

0.1.0

Breaking changes

  • StoreGet, StoreSet and StoreSetIfNotExists have all been changed from a struct to a trait
    • Multiple implementations for StoreGet, StoreSet and StoreSetIfNotExists have been added. Notably:
      • BigDecimalStoreGet, BigDecimalStoreSet, BigIntStoreGet, BigIntStoreSet. These stores are typed, meaning the user does not need to think about the encoding and the decoding as it's done for you. The user only needs to create a BigDecimal and store it. Storing and reading it will work out of the box for the users. No need to decode it.
      • ProtoStoreGet<ProtobufType>, ProtoStoreSet<ProtobufType> and ProtoStoreSetIfNotExists<ProtobufType>. All these implementations of proto have to be typed. example: 
        #[derive(Clone, PartialEq, ::prost::Message)]
pub struct ProtobufType {
    [...] // your attributes defined in your proto
}

#[substreams::handlers::map]
pub fn map_my_substreams(store: ProtoStoreGet<ProtobufType>) -> Result<[...]> {
    [...]
}
    • The previous StoreGet, StoreSet and StoreSetIfNotExists can still be used, but they need to be used as RawStoreGet, RawStoreSet and RawStoreSetIfNotExists which all use bytes as input/outputs instead of a typed value. Using any of the Raw stores has the same behaviour as before this release meaning if you have a RawStore of BigInt, BigDecimal or ProtobufType you would need to decode/encode them.
    • When fetching data from a typed Store, the user will not need to decode the returned value. Meaning a method call to get_last(), get_at() will all already return the decoded ProtobufType specified by the user.
  • Custom BigInt and BigDecimal have been added to be able to add synthetic sugar and make the code more readable in a substreams.
    • Instead of doing manipulations like BigDecimal::from_str(your_str_representation_of_big_decimal.as_str()).unwrap() the user can do let bd: BigDecimal = your_str_representation_of_big_decimal.indo(). Much clearer and less convoluted

0.0.21

  • Ported rust modules from github.com/streamingfast/substreams to this repository