Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Race condition in metadata extraction #253

Open
candleindark opened this issue Oct 11, 2023 · 0 comments
Open

Race condition in metadata extraction #253

candleindark opened this issue Oct 11, 2023 · 0 comments

Comments

@candleindark
Copy link
Collaborator

While race conditions relating to accessing local dataset cache and the RepoUrl table have been addressed properly in #238, an extremely unlikely race condition is still possible in the extraction of mete-data through the execution of the extract_ds_meta Celery task. Should this race condition occurs, two or more URLMetadata records with the same extractor_name and dataset_version can be written to the database, creating duplication of data in the database.

Provide a solution to prevent this race condition or eliminate the affect of affect of it. Try to avoid using locks in the solution for more locks make dead locks more likely.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant