Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Prevent Silent Errors #666

Merged
merged 6 commits into from
Nov 8, 2024
+15 −2
Merged

Prevent Silent Errors #666

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
2a3da07
tada
sage-wright Oct 16, 2024
76f2af6
Merge branch 'main' into smw-pipefail-dev
sage-wright Oct 31, 2024
c425b04
two more
sage-wright Oct 31, 2024
5833b9d
Merge branch 'main' into smw-pipefail-dev
sage-wright Nov 8, 2024
7894c64
update
sage-wright Nov 8, 2024
46a27f9
Merge branch 'main' into smw-pipefail-dev
sage-wright Nov 8, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions tasks/utilities/data_export/task_export_two_tsvs.wdl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ task export_two_tsvs {
volatile: true
}
command <<<
set -euo pipefail
python3 /scripts/export_large_tsv/export_large_tsv.py --project ~{terra_project1} --workspace ~{terra_workspace1} --entity_type ~{datatable1} --tsv_filename "~{datatable1}_table1.tsv"

# check if second project is provided; if not, use first
Expand Down
2 changes: 2 additions & 0 deletions tasks/utilities/data_handling/task_summarize_data.wdl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,8 @@ task summarize_data {
volatile: true
}
command <<<
set -euo pipefail

# when running on terra, comment out all input_table mentions
python3 /scripts/export_large_tsv/export_large_tsv.py --project "~{terra_project}" --workspace "~{terra_workspace}" --entity_type ~{terra_table} --tsv_filename ~{terra_table}-data.tsv

Expand Down
2 changes: 2 additions & 0 deletions tasks/utilities/data_handling/task_theiacov_fasta_batch.wdl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,8 @@ task sm_theiacov_fasta_wrangling { # the sm stands for supermassive
Int memory = 4
}
command <<<
set -euo pipefail

# check if nextclade json file exists
if [ -f ~{nextclade_json} ]; then
# this line splits into individual json files
Expand Down
4 changes: 4 additions & 0 deletions tasks/utilities/data_import/task_create_terra_table.wdl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -146,6 +146,10 @@ task create_terra_table {
done <filelist-fullpath.txt

echo "DEBUG: terra table created, now beginning upload"

# set error handling to exit if the subsequent import_large_tsv.py task fails
set -euo pipefail

python3 /scripts/import_large_tsv/import_large_tsv.py --project "~{terra_project}" --workspace "~{terra_workspace}" --tsv terra_table_to_upload.tsv
>>>
output {
Expand Down
2 changes: 2 additions & 0 deletions tasks/utilities/file_handling/task_transfer_files.wdl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,8 @@ task transfer_files {
volatile: true
}
command <<<
set -euo pipefail

file_path_array="~{sep=' ' files_to_transfer}"

gsutil -m cp -n ${file_path_array[@]} ~{target_bucket}
Expand Down
2 changes: 2 additions & 0 deletions tasks/utilities/submission/task_submission.wdl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,8 @@ task prune_table {
volatile: true
}
command <<<
set -euo pipefail

# when running on terra, comment out all input_table mentions
python3 /scripts/export_large_tsv/export_large_tsv.py --project "~{project_name}" --workspace "~{workspace_name}" --entity_type ~{table_name} --tsv_filename ~{table_name}-data.tsv

Expand Down
2 changes: 1 addition & 1 deletion tests/workflows/theiaprok/test_wf_theiaprok_illumina_pe.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -627,7 +627,7 @@
- path: miniwdl_run/wdl/tasks/taxon_id/contamination/task_midas.wdl
md5sum: 64caaaff5910ac0036e2659434500962
- path: miniwdl_run/wdl/tasks/utilities/data_export/task_broad_terra_tools.wdl
md5sum: 8c97c5bd65e2787239f12ef425d479ae
md5sum: 850ad97598aca5c28eb36e6a5c13c2fc
- path: miniwdl_run/wdl/workflows/theiaprok/wf_theiaprok_illumina_pe.wdl
md5sum: d8db687487a45536d4837a540ed2a135
- path: miniwdl_run/wdl/workflows/utilities/wf_merlin_magic.wdl
Expand Down
2 changes: 1 addition & 1 deletion tests/workflows/theiaprok/test_wf_theiaprok_illumina_se.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -590,7 +590,7 @@
- path: miniwdl_run/wdl/tasks/taxon_id/contamination/task_midas.wdl
md5sum: 64caaaff5910ac0036e2659434500962
- path: miniwdl_run/wdl/tasks/utilities/data_export/task_broad_terra_tools.wdl
md5sum: 8c97c5bd65e2787239f12ef425d479ae
md5sum: 850ad97598aca5c28eb36e6a5c13c2fc
- path: miniwdl_run/wdl/workflows/theiaprok/wf_theiaprok_illumina_se.wdl
md5sum: 4111a758490174325ae8ea52a95319e9
- path: miniwdl_run/wdl/workflows/utilities/wf_merlin_magic.wdl
Expand Down