-
Notifications
You must be signed in to change notification settings - Fork 14
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add export all history command #220
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the speedy turnaround with the changes. Looks like a good enough temporary fix, so I kept the comments to a minimum.
How does the runtime for all history_*
exports compare with running a single export command?
} | ||
|
||
outFileOperations.Close() | ||
cmdLogger.Info("Number of bytes written: ", totalNumBytes) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If we get frequent transient errors, more logging that identifies which file/export process failed might be helpful. Since this is a temp fix, i'll leave it up to you whether that's worthwhile to add.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah it's probably not that useful. Any issue would more easily be resolved by running the original pipeline instead of this hack
Co-authored-by: sydneynotthecity <[email protected]>
I didn't run this in prod but in test it adds about 10s - 20s more time (so completes in 40s compared to 20s-30s). I think the runtime ratio will be much better in prod because the part that will benefit the most is the captive core catchup portion |
Adding temporary command
export_all_history
to mitigate the high egress costs hitting the history archives.This combines the export for operations, trades, effects, transactions, and diagnostic events to use only a single captive core instead of each spinning up their own for data export.
TODO: This should be removed as soon as LedgerExporter is complete and a TxMeta datalake is available