kuberun doesn't put data into the right place

[olgabot]

Hello,
Hope you are doing well. I am testing nextflow on a kubernetes cluster, and am getting errors. It seems that the test data that is bundled with the repo is not getting copied to the pod. Do you know what may be happening?
Thank you!
Warmest,
Olga

The nextflow.config file:

k8s {
   namespace = 'default'
   serviceAccount = 'nextflow'
   storageClaimName = 'nextflow-pvc'
}

process.scratch = true

nextflow kuberun command and output

(nf-core)
 ✘  Wed  1 Jun - 22:30  ~ 
 olgabot@ip-172-31-9-54  nextflow -c ./nextflow.config kuberun nextflow-io/rnaseq-nf -v nextflow-pvc:/mnt/nextflow -profile docker
zsh: correct 'docker' to '.docker' [nyae]? n
Pod started: peaceful-mccarthy
N E X T F L O W  ~  version 22.04.0
Launching `https://github.com/nextflow-io/rnaseq-nf` [peaceful-mccarthy] DSL2 - revision: 37c5039435 [master]
 R N A S E Q - N F   P I P E L I N E
 ===================================
 transcriptome: null/data/ggal/ggal_1_48850000_49020000.Ggal71.500bpflank.fa
 reads        : null/data/ggal/ggal_gut_{1,2}.fq
 outdir       : results

No files match pattern `ggal_gut_{1,2}.fq` at path: null/data/ggal/


Oops .. something went wrong

[olgabot]

If I override the paths by using the pulled repo in ~/.nextflow/assets/nextflow-io/rnaseq-nf, then I get similar issues:

(nf-core)
 ✘  Wed  1 Jun - 22:27  ~ 
 olgabot@ip-172-31-9-54  nextflow -c ./nextflow.config kuberun olgabot/rnaseq-nf -v nextflow-pvc:/mnt/nextflow --reads ~/.nextflow/assets/nextflow-io/rnaseq-nf/data/ggal/'ggal_*_{1,2}.fq' --transcriptome ~/.nextflow/assets/nextflow-io/rnaseq-nf/data/ggal/ggal_1_48850000_49020000.Ggal71.500bpflank.fa --multiqc ~/.nextflow/assets/nextflow-io/rnaseq-nf/multiqc -profile docker
zsh: correct 'docker' to '.docker' [nyae]? n
Pod started: tender-jones
N E X T F L O W  ~  version 22.04.0
Launching `https://github.com/olgabot/rnaseq-nf` [tender-jones] DSL2 - revision: 1af088b52f [master]
 R N A S E Q - N F   P I P E L I N E
 ===================================
 transcriptome: /home/olgabot/.nextflow/assets/nextflow-io/rnaseq-nf/data/ggal/ggal_1_48850000_49020000.Ggal71.500bpflank.fa
 reads        : /home/olgabot/.nextflow/assets/nextflow-io/rnaseq-nf/data/ggal/ggal_*_{1,2}.fq
 outdir       : results

No files match pattern `ggal_*_{1,2}.fq` at path: /home/olgabot/.nextflow/assets/nextflow-io/rnaseq-nf/data/ggal/


Oops .. something went wrong

Interestingly, the quotes must not include the home directory: ~/.nextflow/assets/nextflow-io/rnaseq-nf/data/ggal/'ggal_*_{1,2}.fq'

If I use '~/.nextflow/assets/nextflow-io/rnaseq-nf/data/ggal/ggal_*_{1,2}.fq', then I see /root for the home directory for the reads:

(nf-core)
✘  Wed 1 Jun - 21:32  ~ 
olgabot@ip-172-31-9-54  nextflow -c ./nextflow.config kuberun olgabot/rnaseq-nf -v nextflow-pvc:/mnt/nextflow --reads '~/.nextflow/assets/nextflow-io/rnaseq-nf/data/ggal/ggal_gut_{1,2}.fq' --transcriptome ~/.nextflow/assets/nextflow-io/rnaseq-nf/data/ggal/ggal_1_48850000_49020000.Ggal71.500bpflank.fa --multiqc ~/.nextflow/assets/nextflow-io/rnaseq-nf/multiqc
Pod started: distraught-payne
N E X T F L O W ~ version 22.04.0
Launching `[https://github.com/olgabot/rnaseq-nf`](https://github.com/olgabot/rnaseq-nf%60) [distraught-payne] DSL2 - revision: 1af088b52f [master]
R N A S E Q - N F P I P E L I N E
===================================
transcriptome: /home/olgabot/.nextflow/assets/nextflow-io/rnaseq-nf/data/ggal/ggal_1_48850000_49020000.Ggal71.500bpflank.fa
reads : /root/.nextflow/assets/nextflow-io/rnaseq-nf/data/ggal/ggal_gut_{1,2}.fq
outdir : results



No files match pattern `ggal_gut_{1,2}.fq` at path: /root/.nextflow/assets/nextflow-io/rnaseq-nf/data/ggal/




Oops .. something went wrong
WARN: [RNASEQ:INDEX (ggal_1_48850000_49020000)] Unable to resume cached task -- See log file for details

Is there a setting I'm missing for the pod to get its own copy of the pipeline repo?

[bentsherman]

Got here from the issue you linked. The problem is that implicit variables like baseDir don't work with kuberun when you use it in nextflow.config. The only way to make it work is (1) create a submitter pod manually and launch nextflow from there with k8s executor or (2) replace baseDir with the actual path, which I believe is /workspace/assets/nextflow-io/rnaseq-nf (it's either assets or projects).

You can use kuberun login to figure out where the pipeline code is downloaded.