Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create process to maintain a queue of MetabaseCache migrations #21

Open
preaction opened this issue Jun 13, 2018 · 0 comments
Open

Create process to maintain a queue of MetabaseCache migrations #21

preaction opened this issue Jun 13, 2018 · 0 comments

Comments

@preaction
Copy link
Member

We have 80,000,000 reports in the old metabase cache that need to be upgraded to the new test report format. We have CPAN::Testers::Backend::Migrate::MetabaseCache to do this, but there are too many reports to migrate all in one go.

We need a way to use Minion to process these reports, at a low priority (so that new incoming test reports are processed first). This new process should:

  1. Check how many Metabase cache upgrade jobs are in the queue
  2. Look for unprocessed cache entries that do not already have a job in the queue
  3. Add new jobs to the queue until there are a set number of jobs (starting with 1000)

Every time the process is run, it should try to refill the queue with new jobs. This process will be run periodically to keep refilling the queue until every piece of data is processed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant