Add ability for indexed sequence file to be uploaded and used in an a…

…ssembly (#432)

* Start implementing file upload|download|get

* Add delete command and various small edits

* Add FileCommand handling file operations

* Handle default profile via undefined

* Download file as stream

* Handling of gzip input and bgzip'd & indexed fasta

* CLI can upload and add assembly from gzip files (#405 and #407)

* CLI can add assembly in non-editable mode with `--no-db` flag, i.e. without loading
  sequence to mongodb (#406). Only bgzip'd & indexed fasta file are
  supported, not plain fa/fai files.

See tests in test.py: `testFeatureChecksIndexed`, `testFileUploadGzip`,
`testAddAssemblyWithoutLoadingInMongo`

* Fix linter

* Formatting

* Add description for file CLI topic

* Deduplicate sequence fetching code

* Reference File schema from Assembly schema

* Rename commands

* Remove -i option since file is positional

* Rename option --no-db and some other fixes

* Add additional type and make input arg required

* Tmp commit

* fix app start issue

* fix lint

* Tmp commit addressing #432

Fix #424

* Tmp commit

* This should address comments in #432

But most likely there will be changes to clunky code

* Temp commit to fix buffer

* Fix filehandle type

* Use files service to get filehandles

* Fix fileId vs fileIds and other issues for #432

All tests except those for chekcs passing

* Allow fasta input to be gzip

* Fixes after rebase

* Move LocalFileGzip to collab server package

* Match selector to other updated tests

---------

Co-authored-by: dariober <[email protected]>
Co-authored-by: Shashank Budhanuru Ramaraju <[email protected]>

.github/workflows/reload_demo_data.yml

@@ -84,8 +84,8 @@ jobs:
 
           apollo assembly get | apollo assembly delete -v -a -
 
-          apollo assembly add-fasta -i $demoDataDir/schistosoma_haematobium.TD2_PRJEB44434.WBPS19.genomic.fa -a schistosoma_haematobium -f
-          apollo assembly add-fasta -i $demoDataDir/schistosoma_mansoni.PRJEA36577.WBPS19.genomic.fa -a schistosoma_mansoni -f
+          apollo assembly add-from-fasta $demoDataDir/schistosoma_haematobium.TD2_PRJEB44434.WBPS19.genomic.fa -a schistosoma_haematobium -f
+          apollo assembly add-from-fasta $demoDataDir/schistosoma_mansoni.PRJEA36577.WBPS19.genomic.fa -a schistosoma_mansoni -f
 
           apollo feature import -i $demoDataDir/schistosoma_haematobium.TD2_PRJEB44434.WBPS19.annotations.gff3 -a schistosoma_haematobium -d
           apollo feature import -i $demoDataDir/schistosoma_mansoni.PRJEA36577.WBPS19.annotations.gff3 -a schistosoma_mansoni -d

docs/automated_setup.md

@@ -22,9 +22,9 @@ wget https://ftp.ebi.ac.uk/pub/databases/wormbase/parasite/releases/WBPS19/speci
 
 gunzip *.gz
 
-apollo assembly add-fasta ${P} -i schistosoma_haematobium.TD2_PRJEB44434.WBPS19.genomic.fa -a schistosoma_haematobium -f
-apollo assembly add-fasta ${P} -i schistosoma_mansoni.PRJEA36577.WBPS19.genomic.fa -a schistosoma_mansoni -f
+apollo assembly add-from-fasta ${P} schistosoma_haematobium.TD2_PRJEB44434.WBPS19.genomic.fa -a schistosoma_haematobium -f
+apollo assembly add-from-fasta ${P} schistosoma_mansoni.PRJEA36577.WBPS19.genomic.fa -a schistosoma_mansoni -f
 
-apollo feature import ${P} -i schistosoma_haematobium.TD2_PRJEB44434.WBPS19.annotations.gff3 -a schistosoma_haematobium -d
-apollo feature import ${P} -i schistosoma_mansoni.PRJEA36577.WBPS19.annotations.gff3 -a schistosoma_mansoni -d
+apollo feature import ${P} schistosoma_haematobium.TD2_PRJEB44434.WBPS19.annotations.gff3 -a schistosoma_haematobium -d
+apollo feature import ${P} schistosoma_mansoni.PRJEA36577.WBPS19.annotations.gff3 -a schistosoma_mansoni -d
 ```