run wont start. output is in same directory as data

[omarkr8]

I'm trying to start a run. filling the details in the webpage returns me to the top with an error about the output being the same directory as data (i'll edit in the exact wording when i try it again, currently reinstalling)

My data is on a USB stick with the following structure:

harddisk/ samples/ barcode01/ A.fastq

output location is meant to be a folder the same place as samples, but not inside.
is this an issue with naming, or file structure?
does the pipeline require exact file structures as minknow/guppy output?

[Psy-Fer]

Hello,

What is the data path set in the config on the home page?
And what is selected when you are on the parameters page?

[omarkr8]

okay so..
base filepath where data is located: /media/labx/harddisk/interartictest
base filepath where sample-barcode csv: same as above

input data directory: testfastq
(which is in the interartictest)

the warning says: Output directory will be in the same folder as data, please check data structure info in documentation.

error still shows even if i change output elsewhere. am on a WSL ubuntu

[Psy-Fer]

If the folder that is then selected in the parameters page, also contains the fastq files, it will throw that error. Quick fix should be to throw the data into another folder so it's deeper in the file structure

[omarkr8]

hmm well that let the job through. so that's good.

immediately faced with a IsADirectoryError [Errno21] :/media/labx/harddisk/interartictest

was something wrong with my pathing?

[Psy-Fer]

Sorry could you elaborate on that?

If you could screenshot the errors or copy/paste the wording. It helps me find it in the code.

[omarkr8]

IsADirectoryError

IsADirectoryError: [Errno 21] Is a directory: '/media/lab6/B28C-CBE9/intertest/'
Traceback (most recent call last)

File "/home/lab6/interartic_bin/lib/python3.7/site-packages/flask/app.py", line 2464, in __call__

return self.wsgi_app(environ, start_response)

File "/home/lab6/interartic_bin/lib/python3.7/site-packages/flask/app.py", line 2450, in wsgi_app

response = self.handle_exception(e)

File "/home/lab6/interartic_bin/lib/python3.7/site-packages/flask/app.py", line 1867, in handle_exception

reraise(exc_type, exc_value, tb)

File "/home/lab6/interartic_bin/lib/python3.7/site-packages/flask/_compat.py", line 39, in reraise

raise value

File "/home/lab6/interartic_bin/lib/python3.7/site-packages/flask/app.py", line 2447, in wsgi_app

response = self.full_dispatch_request()

File "/home/lab6/interartic_bin/lib/python3.7/site-packages/flask/app.py", line 1952, in full_dispatch_request

rv = self.handle_user_exception(e)

File "/home/lab6/interartic_bin/lib/python3.7/site-packages/flask/app.py", line 1821, in handle_user_exception

reraise(exc_type, exc_value, tb)

File "/home/lab6/interartic_bin/lib/python3.7/site-packages/flask/_compat.py", line 39, in reraise

raise value

File "/home/lab6/interartic_bin/lib/python3.7/site-packages/flask/app.py", line 1950, in full_dispatch_request

rv = self.dispatch_request()

File "/home/lab6/interartic_bin/lib/python3.7/site-packages/flask/app.py", line 1936, in dispatch_request

return self.view_functions[rule.endpoint](**req.view_args)

File "/home/lab6/interartic_bin/main.py", line 940, in parameters

new_job = qSys.newJob(job_name, input_folder, read_file, primer_scheme_dir, primer_scheme, primer_type, output_folder, normalise, num_threads, pipeline, min_length, max_length, bwa, skip_nanopolish, dry_run, override_data, num_samples, guppyplex, barcode_type, input_name, csv_filepath, primer_select, input_name)

File "/home/lab6/interartic_bin/src/system.py", line 25, in newJob

return Job(job_name, input_folder, read_file, primer_scheme_dir, primer_scheme, primer_type, output_folder, normalise, num_threads, pipeline, min_length, max_length, bwa, skip_nanopolish, dry_run, override_data, num_samples, guppyplex, barcode_type, run_name, csv_file, primer_select, input_name)

File "/home/lab6/interartic_bin/src/job.py", line 39, in __init__

self._guppyplex_cmd = self.__generateGuppyplexCmd()

File "/home/lab6/interartic_bin/src/job.py", line 218, in __generateGuppyplexCmd

with open(self._csv_file,'rt')as f:

IsADirectoryError: [Errno 21] Is a directory: '/media/lab6/B28C-CBE9/intertest/'

[Psy-Fer]

Ahh that looks like an issue with the sample CSV file selection. The folder that actually contains the sample CSV should be what is in the home configuration setup. The on the parameters page, the drop down when you hit multiple samples, should have the sample.csv file as the selection, not a folder.

Looks like I need to add some checks on that to make sure its a file being select.

[omarkr8]

okay so that worked. I adjusted the filepath for the sample csv. had to also fix some malformations.
but the job went through and completed!

however, up till this point. i had been selecting 'single sample' not 'multiple sample', so i didnt get the prompt to specify the sample.csv

does the pipeline automatically assume its demux if it sees a 'barcode01' folder?

in any case, the initial issues i had are resolved.
many thanks, and good to have a dev that is active at the same time for once :)

[Psy-Fer]

Yea, the single sample option is rarely used, so much so I'm tempted to auto-populate the multiple-sample option by default. That and the sample.csv type setup seems to be the main complaint from users.

Perhaps something more interactive, that auto-detects directory structure and shows it to the user, with some confirmation of each sample to analyse and denote negative controls to help with error handling. I can always keep the sample.csv as a second option.

Thanks for your feedback. :)

[rebeelouise]

Hi @Psy-Fer - I am having this problem too - trying to play with our data that automatically gets transferred to our server from our runs. But we only take the the run_id over... e.g. 20211221_1411_X3_FAR98500_6ab03da1 - which are all stored in our gridion directory. So it wont let me run this data due to how it's structured!!

[Psy-Fer]

Hmm okay I'll do something about it. Is this urgent?

[rebeelouise]

@Psy-Fer no not at all. Just having a look atm!! Thank you!