Skip to content

v0.0.7-beta
Compare
Choose a tag to compare
@maoueh maoueh released this 30 May 16:54
· 3199 commits to develop since this release
v0.0.7-beta
59215b0

Summary

  • Changed startBlock to initialBlock in substreams.yaml manifests.

  • code: is now defined in the binaries section of the manifest, instead of in each module. A module can select which binary with the binary: field on the Module definition.

  • Added substreams inspect ./substreams.yaml or inspect some.spkg to see what's inside. Requires protoc to be installed (which you should have anyway).

  • Added command substreams protogen that writes a temporary buf.gen.yaml and generates Rust structs based on the contents of the provided manifest or package.

  • Added substreams::handlers macros to reduce boilerplate when create substream modules.

    substreams::handlers::map is used for the handlers corresponding to modules of type map. Modules of type map should return a Result where the error is of type Error

    /// Map module example
#[substreams::handlers::map]
pub fn map_module_func(blk: eth::Block) -> Result<erc721::Transfers, Error> {
     ...
}

    substreams::handlers::store is used for the handlers corresponding to modules of type store. Modules of type store should have no return value.

    /// Map module example
#[substreams::handlers::store]
pub fn store_module(transfers: erc721::Transfers, s: store::StoreAddInt64, pairs: store::StoreGet, tokens: store::StoreGet) {
      ...
}

Changelog

  • eceec63 Added --exclude-paths flag on substream protocgen
  • 4945217 Added Substreams crate documentation
  • 82ae6b7 Added protogen
  • c02a5b4 Added some missing well known proto descriptors in the system.pb, removed statik replaced by
  • 621ba88 Added support for the sf.susbtreams.v1.Clock in inputs.
  • 1a779ca Added top-level doc.. really thin though :)
  • 42517ea Avoid panic in state builder when missing deltas is encountered
  • 81a642c Better print
  • c9742a0 Bunch of renames and refactors. BREAK BACKWARDS COMPATIBILITY. Needs to be redeployed everywhere. Packages need to be rebuilt and repackaged. New substreams needs to be used to interact with server.
  • db26ea7 Change all those store methods helpers to be u64 instead of i64, as that's the type we use in all ordinals all over the place (Eth, and Solana).
  • 153b2cf Cleaned up some unecessary deps
  • ce857aa Cleaned up substream lib and exported macros
  • c7debad Correctly handle error on call to grpcClientFactory
  • d380853 Doc fixes.
  • 29bdd3f Ensure that tests passes in rust code
  • f9a32d9 First draft at documentaiton.
  • 9dca46b Fix manifest info formatting
  • 280b9e1 Fix pack output message
  • 5545177 Fix proto import dependancy
  • 134a5f7 Fix typo
  • ec40b6c Fixed up docs. Added set_many for a few.
  • 59215b0 Fixed wrong version in Cargo on substreams crate
  • c2633a4 Fixup protogen
  • c260fca Fixup range.
  • 0286a7f Fixup some retries.
  • 0dc3c0a GitBook: [#10] No subject
  • 68efa88 GitBook: [#12] No subject
  • f4ab67a GitBook: [#13] Modules and packages
  • cd2ae90 GitBook: [#20] No subject
  • 43158ee GitBook: [#23] getting-started
  • ee01b07 GitBook: [#24] Better docs, merge strategies, improvements to modules.
  • bf2ee07 GitBook: [#25] getting-started
  • 709ee88 GitBook: [#26] No subject
  • 07382c8 GitBook: [#27] No subject
  • 26cddf4 GitBook: [#28] getting-started
  • 931c9de GitBook: [#29] No subject
  • dae3ffc GitBook: [#2] title modification
  • 97e2c49 GitBook: [#4] Substreams doc revamp
  • ef4cf64 GitBook: [#5] No subject
  • 408ed30 Improved hex encoder and added decoding from hex library
  • 9aec010 Merge branch 'develop' into wasm-refactor
  • 59f299e Merge branch 'develop' of github.com:streamingfast/substreams into develop
  • 4ddec88 Merge branch 'develop' of github.com:streamingfast/substreams into develop
  • c78ffc2 Merge branch 'develop' of github.com:streamingfast/substreams into feature/progress_wip
  • eac4b1b Merge branch 'develop' of github.com:streamingfast/substreams into feature/progress_wip
  • d9d39d9 Merge branch 'develop' of github.com:streamingfast/substreams into feature/progress_wip
  • e9da6a8 Merge branch 'develop' of github.com:streamingfast/substreams into feature/progress_wip
  • b5730c9 Merge branch 'feature/parallel_partial_sub_req' of github.com:streamingfast/substreams into feature/parallel_partial_sub_req
  • 5766c32 Merge branch 'feature/publish' into develop
  • deabc1f Merge branch 'manifest-reader' into develop
  • e7412d3 Merge branch 'rust-macros' into develop
  • 77fe753 Merge branch 'wasm-refactor' into develop
  • c174526 Merge pull request #13 from streamingfast/feature/parallel_partial_sub_req
  • 3682925 Merge pull request #14 from streamingfast/feature/broken_cache
  • af7288b One response func instead of many.
  • f023ca0 Preparing for release of CLI + crate
  • 654b58e Refactor substream macros
  • c885e3e Refactored instrensiv names
  • c3fcf05 Remove --partial-mode and --no-return-handler .. we can put --partial-mode back if necessary to ease debugging of backend services. We could use SUBSTREAMS_PARTIAL_MODE=true, as to not bloat the end user's experience.
  • 89c8a0f Remove ETH reference
  • cd4a4f6 Remove a Grrr
  • 67dafd8 Remove the Ethereum stuff
  • f3f52dc Removed blank lines in errors
  • dd2dd59 Renamed loadDetla to loadDeltas, replaced some panic with errors, updated some zlog statements
  • 450855f Review the Progress signaling.
  • c0e477d Some renames
  • 4ed5ae8 Take out Partial from the proto.
  • 45c5647 Take out non working install instructions.
  • 836ef82 Try again
  • 1b2672c Try to pad things. Fix the proto import priority.
  • ff67ce5 Trying to fix cargo publish command
  • 70d042e Tweaked docs.
  • cd1bd78 Updated CHANGELOG
  • 33abcc2 Updated CHANGELOG, with substream macros
  • 06f8f35 Updated README.md
  • 9309b53 Updated Summary
  • eb8171a Updated explicit traits
  • b74178a Updated latest rendered substreams version
  • ca3c4b0 Updated packages metadata
  • 3883630 WIP
  • beb843e WIP: cleaned up args
  • 8ffb631 Working
  • 3b3e008 Wrap up protogen. Fix tests.
  • 76b46f4 added an early exit when no catchup request are needed
  • 893e016 added couple of MarshalLogObject func
  • 5b8cf2d added lock to squasher
  • 1cecae2 added logs to debug context stuff
  • 381202c added missing return
  • 8edee92 added missing return worker to pool
  • 1da7057 added retry around dstore access
  • 6c92140 added retry on sub request
  • 3c4fd6a added some log
  • 2261fa9 added unlock to squasher ;-)
  • 022b2b7 adding config for scheduler's subrequests and block range size
  • 228b828 adding information on the progress bar
  • fe447ba adding log for request created and adding nil check on if the partial file doesn't exist
  • 7b47ce6 adding missing pointer for squasher in scheduler struct
  • 47e11a7 adding pipeline and scheduler wip
  • e3ff44c adding progress print for catch up of parallel requests
  • 54b8c54 adding return for scheduler workers
  • 70e1c39 better logging
  • 6345d61 better logging and grpc stream handling
  • a1e1517 better logging and remove sched config
  • 9c1e030 broken wip
  • 9629eab builder, etc: replace builder start block and end block with block range object
  • ed57b8a change block range split algo
  • 77001af clean up save stores snapshots
  • e654b3c cleaned up fix macros
  • 3eb410e client: fix package name overwrite
  • 339fed9 delete partial file after squash
  • 2767c43 do not reuse grpc client create a new for each sub request
  • 30b4096 fix "substreams-partial-mode" request header better logging fix early exit of workers
  • 1387ff2 fix NO_RETURN_HANDLER
  • 996c163 fix all merge tests
  • d5768da fix delta output
  • 1076c00 fix error message for max subrequest range size
  • cdbbb22 fix init builder block range
  • 1b5bf42 fix init builder block range
  • 09883e1 fix initialize store so only module outputs one can be initialized has partial fix builder initialization with deltas
  • 62790e9 fix load deltas when cache file is empty
  • 24869bb fix load state info
  • 420c037 fix logs
  • df92975 fix output caching
  • b76b0de fix partial file naming
  • 327d094 fix range split
  • 2d4c509 fix reader
  • c01e1bd fix request pop order from list
  • 78daa28 fix retry code scope
  • cd64eeb fix scheduler error handling
  • 4682ce6 fix some retry code
  • 83ecdc6 fix squasher not finding the right file
  • 3393f9b fix squashing
  • 41ee189 fix summary layout
  • 8270603 fix test
  • 54f2fa5 fix test
  • 89f8a2a fix test and remove Squash func from builder
  • e8898d5 fix worker threading issue
  • 63fa967 fixing mermaid
  • b28b287 flip builder merger func
  • a8778d5 handle grpc client getter error
  • a11432c merge develop
  • d8adcb3 merge values into struct
  • 587ce1b more logs
  • 783b45e move scheduler and squasher to orchestrator package
  • 082aa25 move some things around
  • 5f49840 moving stuff around and creating own concept for progress, instead of having 2 different files doing similarly the same thing
  • d6865d7 new global worker pool
  • 8252c5b orchestrator refactoring
  • fd9c3de orchestrator: add basic first test for squasher
  • 8fdc673 orchestrator: add priority queue to pool to re-order requests in the wait buffer
  • 057389c orchestrator: better context handling
  • 1915fa7 orchestrator: change lock to rw mutex
  • 6ebca76 orchestrator: first draft of ordered strategy
  • 6793cdc orchestrator: fix pool tests
  • bd2c674 orchestrator: fix race conditions in pool
  • 8c55852 orchestrator: fixed close of closed channel on waiter item, switch pipeline to ordered strategy
  • f237c49 orchestrator: more testing
  • 7abd562 orchestrator: more tests for block waiter
  • 28abc3a orchestrator: pass map of already processed blocks to Waiter for early exit condition
  • 791efcd orchestrator: priority queue now uses csp instead of polling, with some additional renamings and cleanup
  • 193ba09 orchestrator: renames
  • e65e8fa orchestrator: request stream handles context errors
  • d2b4d1a orchestrator: smarter locking for priority queue
  • 5888a3a orchestrator: tests and refactorings
  • a0b473d padding computation
  • adc78f4 pipeline: InitializeStores uses different store methods depending on partial mode
  • efafe7e pipeline: add option to limit size of subrequest range
  • 7c62709 pipeline: call squasher.Close
  • 5fa453f pool: add comment
  • 50e0c6a pool: use rw-lock in notify method
  • 420db82 prevent context override
  • f1aff5d proto: add partial mode to request, pipeline does not syncronize stores in partial mode
  • 5e127ea range: code cleanup
  • 63d16b6 refactor scheduler: use strategy
  • ec0cd7e refactor to adapt to new worker pool changes
  • 093cc59 remove build()unused parameters
  • 80292b6 remove build()unused parameters 2
  • 5116a00 remove bytes option when generating proto
  • 1a412a7 remove dum log and comments
  • e0386a2 remove todo
  • 5c86a8f removing bar thingy
  • 1d6b58b removing useless var
  • 5b3a354 scheduler squasher wip fun
  • ec0f358 service: remove old unused worker pool
  • 301482b squash is no more deleting partial files
  • 41f1c67 squashable: fix bug checking first item of empty ranges slice
  • 33e1c8f squasher do not failed anymore when not able to delete partial file
  • 1de7727 squasher first edition
  • f6de7e6 squasher first release candidate
  • 9163124 squasher: code structure for clarity
  • 84f1b47 squasher: refactor to add some clarity
  • 7774ab7 state: fix info save
  • a0e90a3 state: split up massive builder.go file into more manageable pieces
  • c1cdb4e state: update MarshalLogObject
  • 1a480f0 upgrading progress and not including first range for the squasher
  • 6fa7396 use ordered list of state builders instead of map
  • a6983ef wip
  • ef1dbe9 wip
  • 615f6dc wip
  • 9e5e9e5 wip
  • 3b2ec56 wip
  • 160678b wip
  • 45b0e37 wip
  • 2380fb5 wip
  • e76394e wip for squasher changes
  • 33155db worker pool borrow refactoring
Assets 7
Loading