Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add raw batch data syncing #1

Open
wants to merge 6 commits into
base: main
Choose a base branch
from
Open

Add raw batch data syncing #1

wants to merge 6 commits into from

Conversation

harisang
Copy link
Collaborator

@harisang harisang commented Nov 28, 2024

This PR aims to introduce a batch data job, that is ran every few hours and populates a temporary table in the analytics db with all the necessary data that are meant to be synced on Dune. This decouples the computation of this processed table from the actual uploading of the data on Dune, which will take place via a separate dune-syn-v2 job.

There are two temporary tables in the db, named raw_batch_data_latest_even_month_mainnet and raw_batch_data_latest_odd_month_mainnet. If the current month is an odd month, then the relevant entries are stored in the ....odd_month... table and the ...even_month... table then contains the entries of the previous month.

The split between months happens based on the timestamp of the block_deadline of an auction.

One way to test this branch is actually run it and observe the changes in the temp tables in the db

python -m src.main --sync-table batch_data

Copy link

@fleupold fleupold left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I still think this script would have a good place in https://github.com/cowprotocol/solver-rewards (inside a data_processing_jobs subfolder) to keep all the solver rewards logic in one place (I'm not sure we will ever see another job added to this repository), but fine I don't want to block you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants