Rearchitect exporter to use BullMQ as a background job queueing system #5
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…yet loaded block height.
…block heights anymore.
…ailed jobs 3 times.
…t waiting in the future since it will be set to 0 if not found. Also clean up trace exporter queue promise.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This moves the brunt of the exporting work to a background job system called BullMQ which uses Redis.
This makes the exporter script itself very lightweight, now only responsible for reading the trace events in the FIFO, detecting which state events are relevant, and queueing them for export in batches by block. This ensures that the exporting does not get in the way of the node running, such that it is safe to run an indexer on top of a validator without missing blocks.
This is also more fault tolerant AND debuggable, since queued jobs can be replayed when they fail with their data persisted. If the indexer errors, the events are no longer lost to time, so events should not be missed.