Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR aims to introduce a batch data job, that is ran every few hours and populates a temporary table in the analytics db with all the necessary data that are meant to be synced on Dune. This decouples the computation of this processed table from the actual uploading of the data on Dune, which will take place via a separate dune-syn-v2 job.
There are two temporary tables in the db, named
raw_batch_data_latest_even_month_mainnet
andraw_batch_data_latest_odd_month_mainnet
. If the current month is an odd month, then the relevant entries are stored in the ....odd_month... table and the ...even_month... table then contains the entries of the previous month.The split between months happens based on the timestamp of the block_deadline of an auction.
One way to test this branch is actually run it and observe the changes in the temp tables in the db
python -m src.main --sync-table batch_data