Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[epic ] Automatically backfill and reap history #196

Closed
2opremio opened this issue Jun 4, 2024 · 1 comment
Closed

[epic ] Automatically backfill and reap history #196

2opremio opened this issue Jun 4, 2024 · 1 comment

Comments

@2opremio
Copy link
Contributor

2opremio commented Jun 4, 2024

Right now there is:

  1. No automatic backfill (e.g. if the history is empty on start, it will be filled organically as ledgers close).
  2. Very basic reaping. It happens after ingesting every ledger, synchronuously and I am not sure it takes into account any ledgers behind the current history window.

RPC should automatically reconcile the existing history state with the desired state (desired window), accounting for things like changes in the commandline parameters.

RPC should do so asynchronously, in the background without disturbing the normal operation of serving method calls.

Also, RPC should strive to leave the state consistent at all times (e.g. no history gaps should be left as a result of crashes or restarts).

Subtasks (by dependency order):

  1. Unify events and transactions retention window #199 (dealing with different window sizes is too complicated)
  2. Add a fast and reliable way to check history #200
  3. Add a way to reset the database #204
  4. Implement synchronous history backfilling #203
  5. Separate reaping from ingestion #201
  6. Orchestrate ingestion, reaping and backfilling #202
  7. Add ingestion, reaping and backfilling tests #205
@mollykarcher mollykarcher moved this from Backlog to To Do in Platform Scrum Jun 4, 2024
@mollykarcher mollykarcher added this to the platform sprint 47 milestone Jun 4, 2024
@mollykarcher mollykarcher changed the title Automtically backfill history Automatically backfill history Jun 4, 2024
@2opremio 2opremio changed the title Automatically backfill history Automatically backfill and reap history Jun 6, 2024
@mollykarcher mollykarcher changed the title Automatically backfill and reap history [epic ] Automatically backfill and reap history Jun 6, 2024
@mollykarcher mollykarcher removed this from the platform sprint 47 milestone Jun 6, 2024
@mollykarcher
Copy link
Contributor

I'm going to close all these issues. We're unsure of the product need. Will recreate if/when we decide to move forward

@github-project-automation github-project-automation bot moved this from To Do to Done in Platform Scrum Jul 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: Done
Development

No branches or pull requests

2 participants