nf-casereports restarting at sage/fragcounter upon rerun - processes already finished successfully

[kevinmhadi]

Description of the bug

Rerun is restarting at fragcounter which is finished from previous run?

Command used and terminal output

cd "/gpfs/data/imielinskilab/projects/Clinical_NYU/Nextflow/2025-01-10___500to200X_Subsample/results" && module load singularity && module load aws-cli && module load nextflow/23.04.4 && nextflow run /gpfs/data/imielinskilab/Git/nf-casereports \
    -c "/gpfs/data/imielinskilab/Git/nf-casereports/nextflow.config" \
    -c "/gpfs/data/imielinskilab/projects/Clinical_NYU/Nextflow/2025-01-10___500to200X_Subsample/results/nf-samplesheet-2025-01-10-500to200X_subsample.nyu.config" \
    -params-file "/gpfs/data/imielinskilab/projects/Clinical_NYU/Nextflow/2025-01-10___500to200X_Subsample/results/nf-samplesheet-2025-01-10-500to200X_subsample.params.json" \
    --tumor_only true \
    -profile singularity \
    -with-report "/gpfs/data/imielinskilab/projects/Clinical_NYU/Nextflow/2024-12-17/results/report_$(date +'%Y%m%d_%H%M%S').html" \
    -with-trace \
    -dump-channels \
    -resume

Relevant files

/gpfs/data/imielinskilab/projects/Clinical_NYU/Nextflow/2024-12-17/results/.nextflow.log

Jan-10 16:21:26.370 [Task monitor] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for
  task: name=MSKILABORG_NFCASEREPORTS:NFCASEREPORTS:TUMOR_FRAGCOUNTER:FRAGCOUNTER (TM22-144-0081-B22-2339); work-dir=/gpfs/data/imielinskilab/projects/Clinical_NYU/Nextflow/2024-12-17/results/work/8b/b4a2abae7292cb8ecd6927e169e775
  error [nextflow.exception.ProcessFailedException]: Process `MSKILABORG_NFCASEREPORTS:NFCASEREPORTS:TUMOR_FRAGCOUNTER:FRAGCOUNTER (TM22-144-0081-B22-2339)` terminated with an error exit status (137)
Jan-10 16:21:26.371 [Task monitor] INFO  nextflow.processor.TaskProcessor - [8b/b4a2ab] NOTE: Process `MSKILABORG_NFCASEREPORTS:NFCASEREPORTS:TUMOR_FRAGCOUNTER:FRAGCOUNTER (TM22-144-0081-B22-2339)` terminated with an error exit status (137) -- Execution is retried (1)
Jan-10 16:21:26.457 [Task submitter] DEBUG nextflow.executor.GridTaskHandler - [SLURM] submitted process MSKILABORG_NFCASEREPORTS:NFCASEREPORTS:TUMOR_FRAGCOUNTER:FRAGCOUNTER (TM22-144-0081-B22-2339) > jobId: 58050734; workDir: /gpfs/data/imielinskilab/projects/Clinical_NYU/Nextflow/2024-12-17/results/work/78/dc3c9559de71731a1632751d40b2c0

These new fragcounter jobs are spawning, when the outputs exist in:
/gpfs/data/imielinskilab/projects/Clinical_NYU/Nextflow/2024-12-17/results/coverage/fragcounter_tumor/TM22-144-0081-B22-2339

Also: multiple fragcounter work dirs spawned during the rerun step, not clear why multiple workdirs are getting created:
/gpfs/data/imielinskilab/projects/Clinical_NYU/Nextflow/2024-12-17/results/work/78/dc3c9559de71731a1632751d40b2c0
/gpfs/data/imielinskilab/projects/Clinical_NYU/Nextflow/2024-12-17/results/work/8b/b4a2abae7292cb8ecd6927e169e775

System information

No response

[shihabdider]

Reruns on a resume are dependent on whether the work/ directory contains the output, not the results/. New work directories will be generated for each process for every new run (but the flies in them will be symlinked where possible to avoid redundancy).