296 scheduler fails unable to find assembly contigs #300

mbthornton-lbl · 2024-11-23T23:42:48Z

This PR provides fixes to the scheduler responsible for incorrect behaviors around finding new jobs and creating a new entry in the jobs collection
#296

Main cause of failure was a lack of matching workflow_execution db records to their workflow_config by type in workflow_process.py
this was not previously necessary, since each workflow_execution type was in its own collextion.

Other updates include

More detailed unit tests and fixtures replicating the issue in Scheduler fails Unable to find Assembly Contigs #296
Refactoring for clarity
update dependencies to nmdc-schema 11,1,0

Test Results running scheduler in local environment with local copy of NMDC DB:

INFO:root:Initializing Scheduler
INFO:root:Found 1 new jobs for nmdc:wfmgan-11-6x59p192.2
INFO:root:JOB RECORD: nmdc:56493f34-a9f7-11ef-b8be-d61879508757

Job Record created in local jobs table:

{
    "_id" : ObjectId("67426cfc21a22e114d84b5c3"),
    "workflow" : {
        "id" : "MAGs: v1.3.12"
    },
    "id" : "nmdc:56493f34-a9f7-11ef-b8be-d61879508757",
    "created_at" : ISODate("2024-11-23T16:02:04.000+0000"),
    "config" : {
        "git_repo" : "https://github.com/microbiomedata/metaMAGs",
        "release" : "v1.3.12",
        "wdl" : "mbin_nmdc.wdl",
        "activity_id" : "nmdc:wfrbt-11-7r0srx57.9",
        "activity_set" : "workflow_execution_set",
        "was_informed_by" : "nmdc:omprc-11-cegmwy02",
        "trigger_activity" : "nmdc:wfmgan-11-6x59p192.2",
        "iteration" : NumberInt(9),
        "input_prefix" : "nmdc_mags",
        "inputs" : {
            "proj" : "{workflow_execution_id}",
            "contig_file" : "https://data.microbiomedata.org/data/nmdc:omprc-11-cegmwy02/nmdc:wfmgan-11-6x59p192.2/nmdc_wfmgan-11-6x59p192.2_contigs.fna",
            "sam_file" : "https://data.microbiomedata.org/data/nmdc:omprc-11-cegmwy02/nmdc:wfmgas-11-jchk0x71.1/nmdc_wfmgas-11-jchk0x71.1_pairedMapped_sorted.sam.gz",
            "gff_file" : "https://data.microbiomedata.org/data/nmdc:omprc-11-cegmwy02/nmdc:wfmgan-11-6x59p192.2/nmdc_wfmgan-11-6x59p192.2_functional_annotation.gff",
            "proteins_file" : "https://data.microbiomedata.org/data/nmdc:omprc-11-cegmwy02/nmdc:wfmgan-11-6x59p192.2/nmdc_wfmgan-11-6x59p192.2_proteins.faa",
            "cog_file" : "https://data.microbiomedata.org/data/nmdc:omprc-11-cegmwy02/nmdc:wfmgan-11-6x59p192.2/nmdc_wfmgan-11-6x59p192.2_cog.gff",
            "ec_file" : "https://data.microbiomedata.org/data/nmdc:omprc-11-cegmwy02/nmdc:wfmgan-11-6x59p192.2/nmdc_wfmgan-11-6x59p192.2_ec.tsv",
            "ko_file" : "https://data.microbiomedata.org/data/nmdc:omprc-11-cegmwy02/nmdc:wfmgan-11-6x59p192.2/nmdc_wfmgan-11-6x59p192.2_ko.tsv",
            "pfam_file" : "https://data.microbiomedata.org/data/nmdc:omprc-11-cegmwy02/nmdc:wfmgan-11-6x59p192.2/nmdc_wfmgan-11-6x59p192.2_pfam.gff",
            "tigrfam_file" : "https://data.microbiomedata.org/data/nmdc:omprc-11-cegmwy02/nmdc:wfmgan-11-6x59p192.2/nmdc_wfmgan-11-6x59p192.2_tigrfam.gff",
            "crispr_file" : "https://data.microbiomedata.org/data/nmdc:omprc-11-cegmwy02/nmdc:wfmgan-11-6x59p192.2/nmdc_wfmgan-11-6x59p192.2_crt.crisprs",
            "product_names_file" : "https://data.microbiomedata.org/data/nmdc:omprc-11-cegmwy02/nmdc:wfmgan-11-6x59p192.2/nmdc_wfmgan-11-6x59p192.2_product_names.tsv",
            "gene_phylogeny_file" : "https://data.microbiomedata.org/data/nmdc:omprc-11-cegmwy02/nmdc:wfmgan-11-6x59p192.2/nmdc_wfmgan-11-6x59p192.2_gene_phylogeny.tsv",
            "lineage_file" : "https://data.microbiomedata.org/data/nmdc:omprc-11-cegmwy02/nmdc:wfmgan-11-6x59p192.2/nmdc_wfmgan-11-6x59p192.2_scaffold_lineage.tsv",
            "map_file" : "https://data.microbiomedata.org/data/nmdc:omprc-11-cegmwy02/nmdc:wfmgan-11-6x59p192.2/nmdc_wfmgan-11-6x59p192.2_contig_names_mapping.tsv"
        },

    },
    "claims" : [

    ]
}

nmdc_automation/workflow_automation/sched.py

aclum

There doesn't appear to be a way to configure which yaml gets passed to sched.py

aclum

There is not a way to specify what yaml sched.py uses.

aclum · 2024-11-25T18:14:01Z

Based on your local results there are issues:

the minted typecode is not correct
"activity_id" : "nmdc:wfrbt-11-7r0srx57.9", <- this should use a mag type code 'wfmag'
"inputs" : {
"proj" : "{workflow_execution_id}", <- this variable isn't be resolved to an identifier, it should match the activity_id

removed comments about output ID variable resolution, based on prod agent.state files this is expected.

…contigs

mbthornton-lbl · 2024-11-25T19:08:33Z

Local Testing results

nmdc-omprc-13-01jx8727_mags_job.json

mbthornton-lbl added 15 commits November 21, 2024 10:46

refactor Scheduler args and get_mongo_db() to work with local db

11fe8b9

refactoring for clarity and added test.

33977ca

normalize record for DataObject construction

4a842fa

refactor for readability

3e5473e

get basic test case working with realistic data

b720b97

update unit test

d5abe2a

fix FileTypeEnum issue

3677acb

clean up comments

19f4e31

update fixtures and get unit tests working

845130b

test that parent / child is set correctly for the node

f9a0e4e

add unit test for find_new_jobs

65060d2

add assertions for scheduler.create_job

62a9570

update log statement

9fe2335

ensure record tupe and workflow type match

b5ced79

update tests and logging

c457d75

mbthornton-lbl linked an issue Nov 23, 2024 that may be closed by this pull request

Scheduler fails Unable to find Assembly Contigs #296

Closed

mbthornton-lbl added 2 commits November 23, 2024 15:45

remove unused api method

d809078

restore args in sched.main

eda7251

mbthornton-lbl requested review from shreddd and aclum November 23, 2024 23:55

aclum reviewed Nov 25, 2024

View reviewed changes

nmdc_automation/workflow_automation/sched.py Outdated Show resolved Hide resolved

aclum approved these changes Nov 25, 2024

View reviewed changes

aclum self-requested a review November 25, 2024 17:47

aclum requested changes Nov 25, 2024

View reviewed changes

fix main() argument handling

fc622db

mbthornton-lbl linked an issue Nov 25, 2024 that may be closed by this pull request

Scheduler assigning incorrect ID type for new jobs #301

Closed

mbthornton-lbl added 3 commits November 25, 2024 10:41

Match was_informed_by AND workflow.type to find earlier db entries

85a5bac

Merge branch 'main' into 296-scheduler-fails-unable-to-find-assembly-…

5e6e73e

…contigs

fix config.inputs.proj not getting workflow_execution id

bc5a534

mbthornton-lbl requested a review from aclum November 25, 2024 19:11

aclum approved these changes Nov 25, 2024

View reviewed changes

mbthornton-lbl merged commit 2aec24f into main Nov 25, 2024
1 check passed

mbthornton-lbl deleted the 296-scheduler-fails-unable-to-find-assembly-contigs branch November 25, 2024 19:14

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

296 scheduler fails unable to find assembly contigs #300

296 scheduler fails unable to find assembly contigs #300

mbthornton-lbl commented Nov 23, 2024 •

edited

Loading

aclum left a comment

aclum left a comment

aclum commented Nov 25, 2024 •

edited

Loading

mbthornton-lbl commented Nov 25, 2024 •

edited

Loading

296 scheduler fails unable to find assembly contigs #300

296 scheduler fails unable to find assembly contigs #300

Conversation

mbthornton-lbl commented Nov 23, 2024 • edited Loading

aclum left a comment

Choose a reason for hiding this comment

aclum left a comment

Choose a reason for hiding this comment

aclum commented Nov 25, 2024 • edited Loading

mbthornton-lbl commented Nov 25, 2024 • edited Loading

mbthornton-lbl commented Nov 23, 2024 •

edited

Loading

aclum commented Nov 25, 2024 •

edited

Loading

mbthornton-lbl commented Nov 25, 2024 •

edited

Loading