feat(ena-submission): Create ena assembly

[anna-parker]

resolves #2397

preview URL: https://create-ena-assembly.loculus.org/

Summary

Uses same principal as create_ena_projects to keep submission state in DB - please review that PR first :-)

Summary

This adds the following rule to the ena-submission snakemake file :

• create_assembly rule . This function will continuously (in a loop) scan for new sequences where a sample needs to be created and trigger their creation. It will also update both the submission_table and the assembly_table.
• In contrast to create_project and create_sample we need to use the webin-cli.jar for assembly submission. This had to be added to the docker image (and its dependencies).
• In contrast to create_project and create_sample we also need to wait for assemblies to be approved and only then we will receive the assembly accessions: https://ena-docs.readthedocs.io/en/latest/submit/assembly/genome.html#assigned-accession-numbers This means there is one additional state in the assembly_table: WAITING.
• It also adds some basic unit tests for sub-functions used by create_assembly.

High level overview of assembly_creation:

In a loop:

1. Get sequences in submission_table in state SUBMITTED_SAMPLE

• if (there exists an entry in the sample_table for the corresponding (accession, version)):
  -- if (entry is in status SUBMITTED): update submission_table to SUBMITTED_ASSEMBLY.
  -- else: update submission_table to SUBMITTING_ASSEMBLY.
• else: create assembly entry in assembly_table for (accession, version).

2. Get sequences in submission_table in state SUBMITTING_SAMPLE

• if (corresponding assembly_table entry is in state SUBMITTED): update entries to state SUBMITTED_ASSEMBLY.

3. Get sequences in assembly_table in state READY, prepare files: we need chromosome_list, fasta files and a manifest file, set status to WAITING

• if (submission succeeds): set status to WAITING and fill in results: ena-internal "erz_accession"
• else: set status to HAS_ERRORS and fill in errors

4. Get sequences in assembly_table in state WAITING, every 5minutes (to not overload ENA) check if ENA has processed the assemblies and assigned them "gca_accession". If so update the table to status SUBMITTED and fill in results
5. Get sequences in assembly_table in state HAS_ERRORS for over 15min and sequences in status SUBMITTING for over 15min, or in state WAITING for over 48hours: #TODO (handle failure ena-submission: Recover from failed project/sample/assembly submission #2311), currently just throw an error

ENA Assembly

As we are submitting https://ena-docs.readthedocs.io/en/latest/submit/assembly/genome.html#chromosome-assembly
We need to submit 3 files for each (multi-segmented) sequence:

1. Manifest files: https://ena-docs.readthedocs.io/en/latest/submit/assembly/genome.html#manifest-files, here there are some mandatory metadata fields which we must set, I map the fields as follows:

ASSEMBLY_TYPE: default=ISOLATE (mandatory)
PROGRAM: loculus_field=sequencing_instrument (mandatory, default=Unknown)
PLATFORM: loculus_field=sequencing_protocol (mandatory, default=Unknown)
COVERAGE: loculus_field=depth_of_coverage (mandatory AND non-negative float, default=1)
MOLECULETYPE: MoleculeType(loculus_field=molecule_type), default=None

3. Chromosome List Files: https://ena-docs.readthedocs.io/en/latest/submit/fileprep/assembly.html#chromosome-list-file
4. Fasta Files

PR Checklist

• feat(ena-submission): Create ena projects #2293 is submitted
• feat(ena-submission): Create ena samples #2312 is submitted
• Currently this PR requires COVERAGE to be set - figure out if we can not set this: feat(ena-submission): How to handle mandatory assembly/ sequencing metadata fields #2330 => default is 1
• Send a message to our slack channel when submission fails in unhandled cases, works locally:

• Test submission locally on ENA dev instance
• Are able to query ENA for state of assembly submission: Automate querying ENA for state of assembly submission #2408
• Test submission on preview on ENA dev instance - as seen in create_ena_samples the ENA dev instance is still down, preventing sample submission: I can however verify slack notifications work:

I can trigger also trigger assembly creation by manually setting the sra_run_accession and the biosample_accession to an already existing sample: "sra_run_accession": "ERS20170050", "biosample_accession": "SAMEA115670243". This leads the sample submission to succeed and the assembly submission to also succeed and go into state waiting:

here I can again set the table to the erz_accession of an already existing sample to verify check_ena works for getting the gca_accessions after processing:

this then puts the submission_table in status SUBMITTED_ALL

[anna-parker]

Closing as this was part of #2417