Use Unchanging Identifiers for Worker Processes #308
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Previously, worker jobs used score set URNs to identify resources on which they should operate. Due to the nature of these async jobs and the fact that URNs can change on score set publication, this was a poor choice for processes that might defer execution until a moment in time after the resource identifier had changed. This commit updates worker processes to operate on internal database identifiers to identify the resources on which they should work. This should protect us from instances where the URN changes and the worker process no longer can access the resource on which it was spawned.
Providing a score set to the mapping manager implied it was managing the passed score set, when it was really managing the head of the mapping queue. Eliminates that argument to the manager to make the function signature more idiomatic to the job purpose.
bencap
added
app: worker
sallybg
approved these changes
Oct 7, 2024
There was a problem hiding this comment.
This looks great! I like that this removes the score set argument to the variant mapper manager, it's much cleaner this way.
We talked about this offline but putting it here for visibility: If the score set is published after the mapping job starts, the job will fail because the mapper returns mapped variants associated with variant urns, and changing the score set urn also changes the variant urns. It's fine that this scenario causes the mapping to fail; the mapping job will be retried, so it will download the new variant urns upon retry and the job should then succeed.
bencap
deleted the
bugfix/bencap/301/use-unchanging-identifiers-for-async-jobs
branch
Merged
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Previously, worker jobs used score set URNs to identify resources on which they should operate. Due to the nature of these async jobs and the fact that URNs can change on score set publication, this was a poor choice for processes that might defer execution until a moment in time after the resource identifier had changed.
This commit updates worker processes to operate on internal database identifiers to identify the resources on which they should work. This should protect us from instances where the URN changes and the worker process no longer can access the resource on which it was spawned.