fix(backend): ignore deleted sequences in processing pipeline updates #3253
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
corneliusroemer
requested review from
chaoran-chen,
fengelniederhammer,
theosanderson and
anna-parker
corneliusroemer
added
backend
|
Superseded by #3259 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
fixes #3250 independently of #3252, but that PR is worth it anyways as we don't want processed data lying around for deleted sequence entries.
preview URL: https://ignore-deleted.loculus.org
Summary
Solve the issue of #3250 narrowly, where we noticed it. This is to make sure that even if #3252 doesn't delete 100% of entries and some slip through it has no effect on processing pipeline updates. This also means existing instances don't need to clean up processed data to avoid #3250.
It'd be nice if we could rewrite the single big SQL update statement into chunks so we can do some informative logging of intermediate results to help with debugging updates that are in process, see #3249 - but that's for another time.
Once we have foreign key constraints we can probably remove the inner join again.
Testing
I tested deleting some sequences then updating prepro version and it worked (PR #3256)