Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

64 update re id tool.py to not rely on the changesheets submit endpoint #65

Merged
merged 14 commits into from
Feb 23, 2024
Merged

64 update re id tool.py to not rely on the changesheets submit endpoint #65

merged 14 commits into from
Feb 23, 2024

Conversation

mbthornton-lbl
Copy link
Contributor

@mbthornton-lbl mbthornton-lbl commented Feb 23, 2024 •
edited
Loading

This PR provides updates to the commands in re_id_tool.py

  • update ingest-records to not rely on change-sheets or their API endpoint, but to update the has_output array for OmicsProcessing directly via PyMongo
  • update extract-records to fail data objects and their parent workflow execution when data_object_type cannot be determined
  • update delete-old-records to batch delete where possible to improve performance

Also, this PR includes data and log files for the remaining re-ID process steps for all remaining studies

@mbthornton-lbl
switch back to using pymongo
8f8ff83
@mbthornton-lbl
use PyMongo as a workaround
d456954
@mbthornton-lbl
Merge remote-tracking branch 'origin/64-update-re_id_toolpy-to-not-re…
27bdb96 
…ly-on-the-changesheetssubmit-endpoint' into 58-refactor-re_id_tools-ingest-records-to-not-use-deprecated-v1workflowsactivities-endpoint
@mbthornton-lbl
Output from ingest-records and delete-odl-records for Spruce study
f4d69a3
@mbthornton-lbl
add nmdc:sty-11-076c9980_delete_old_records2.log
faf0970
@mbthornton-lbl
update delete-old-records to do batch delete by set_name
3f9f608
@mbthornton-lbl
finish delete-old-records for nmdc:sty-11-076c9980
baece8e
@mbthornton-lbl
Merge branch '58-refactor-re_id_tools-ingest-records-to-not-use-depre…
53a83ec 
…cated-v1workflowsactivities-endpoint' into 64-update-re_id_toolpy-to-not-rely-on-the-changesheetssubmit-endpoint
@mbthornton-lbl
update to handle data objects with missing file_size_bytes or md5_che…
6d9544f 
…cksum
@mbthornton-lbl
update to fail data objects that cannot be typed and remove reliance …
8b63110 
…on changesheets
@mbthornton-lbl
delete obsolete changesheet data
2bdc0c4
@mbthornton-lbl
log files for ingest / delete EMP study
23ebaa2
@mbthornton-lbl
data and logfiles for nmdc:sty-11-dcqce727
35862a4
@mbthornton-lbl mbthornton-lbl linked an issue Feb 23, 2024 that may be closed by this pull request
Update re_id_tool.py to not rely on the changesheets:submit endpoint #64
Closed
Michal-Babins
Michal-Babins approved these changes Feb 23, 2024
View reviewed changes
Copy link

@Michal-Babins Michal-Babins left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added work around for submit:json not working, removed exlplicit password in mongo_uri, update omics_processing_record with new has_output via PyMongo, updated logic to handle failing data objects.

@Michal-Babins Michal-Babins merged commit 3a9a05f into main Feb 23, 2024
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Michal-Babins Michal-Babins Michal-Babins approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Update re_id_tool.py to not rely on the changesheets:submit endpoint
2 participants
@mbthornton-lbl @Michal-Babins