Add a safe guard to integron #5132
Conversation
|
@pimarin can you maybe have a look at this one? |
|
I will check yes ! |
|
So you check that length of input won't be so big for integron due to number tmp file need ? |
|
Yes, we only allow fasta files with less than 10.000 sequences. Which will already create 30-000 tp 40-000 files in one directory. |
|
Oky it's okay to me, test pass so it's okay, maybe an issue to the integronfinder to change it coulbe be better for galaxy in the futur |
|
Commented on an already existing issue: gem-pasteur/Integron_Finder#90 |
|
Oky it's good |
|
@bernt-matthias can you please review and maybe merge? Thanks. |
| @@ -35,7 +35,9 @@ | |||
| && mv Results_Integron_Finder_* Results_Integron_Finder | |||
| ]]></command> | |||
| <inputs> | |||
| <param type="data" name="sequence" format="fasta" label="Replicon file" help="Replicon can be entire chromosome, contif, PCR fragments..." /> | |||
| <param type="data" name="sequence" format="fasta" label="Replicon file" help="Replicon can be entire chromosome, contif, PCR fragments..."> | |||
There was a problem hiding this comment.
What kind of problems does integron finder have with large files?
Do you have an upstream issue that we could link here?
There was a problem hiding this comment.
integron finder make 1 tmp directory per analyzed contigs from the fasta file, applied to metagenome or huge genome it make thousand of tmp directories
There was a problem hiding this comment.
for a smallest genome test (e faecalis) i have 100 directories x 10 file into them, for a contamined genome I obtained 1708 directories
There was a problem hiding this comment.
Just wondering if we really need the hard limit (bu I would be fine with it).
Maybe we can add doc (to the parameters help or the tool help) how to deal with larger data.
For your reference: https://integronfinder.readthedocs.io/en/latest/user_guide/tutorial.html#for-big-data-people
There was a problem hiding this comment.
Do you have an upstream issue that we could link here?
See the PR description ;)
Its the pure amount of small files that are created, we had 10.000.000 in one directory. We can discuss if we should limit to 10.000 or 100.000 but we need to limit it imho.
|
Attention: deployment skipped! https://github.com/galaxyproject/tools-iuc/actions/runs/4201763505 |
|
Should we have bumped the version? I stopped deployment... |
|
We did with VERSION_SUFFIX correct? |
|
My bad. Will restart |
This is needed because of gem-pasteur/Integron_Finder#90 which results in a directory with multi-million of files.