feat: add opt-in rich fasta headers with metadata

[corneliusroemer]

resolves #2284

preview URL: https://fasta-header.loculus.org

Summary

Allow unaligned sequences to be downloaded with a rich FASTA header (configured at organism level for simplicity), e.g. displayName instead of accessionVersion.

This requires creating a new astro API endpoint, for now /[organism]/api/sequences because LAPIS does not want to implement a dynamic FASTA header feature. This endpoint fetches metadata and sequences separately and joins them before responding with the result.

To prevent excessive memory consumption, there's a hardcoded limit of 5k sequences that can be accessed via this endpoint (enforced via an initial request to /aggregated).

Compression is not supported for this endpoint but this is currently not worth the effort (rich header is for ad hoc analysis/viewing not for download of full data).

Screenshot

PR Checklist

• All necessary documentation has been adapted.
• The implemented feature is covered by an appropriate test.

[theosanderson]

Thanks for looking into this. I just tried it in the preview at https://fasta-header.loculus.org/ebola-sudan/search? , and it didn't seem to yield a difference?

I don't know if you saw, but I hacked something similar together at https://github.com/theosanderson/metadata-sequence-combiner to allow #3439 .

I'm a bit uncertain about whether we should add this to Loculus when it only support <5000 sequences. I do think it would be possible to stream from LAPIS from metadata and sequences and combine them on the fly in a stream, scaling to an unlimited number of sequences. I think my vote might be to keep this out of Loculus until we are able to do that, and rely on hacks on Pathoplexus to allow Nextclade and other integrations - which I think are the main use case atm?

[corneliusroemer]

I hadn't drilled through the new value into the website config (locally I'd just edited the config json directly), should be fixed now.

I'm a bit uncertain about whether we should add this to Loculus when it only support <5000 sequences. I do think it would be possible to stream from LAPIS from metadata and sequences and combine them on the fly in a stream, scaling to an unlimited number of sequences. I think my vote might be to keep this out of Loculus until we are able to do that, and rely on hacks on Pathoplexus to allow Nextclade and other integrations - which I think are the main use case atm?

I think I disagree on the first part (not doing it unless streaming) because imperfect but working is better than not having it all and if there's enough demand one can add streaming later. I see it like that: Right now one can download only 0 sequences with rich header, now one can do 5k, clear improvement, 90+% of total benefit achieved.

I do really like the tools UI/UX you've come up with, very smooth. I'm just a bit worried about deviating from our monorepo pattern for something that can be considered a core feature. But nothing stops us from going down the tools route and then insourcing it later, so happy either wray.

[rneher]

I do think this is useful. Ideally we would allow custom composition of the header. This could be a little button that opens dialog where one can pick separator and metadata field selection/order.

[theosanderson]

Removing preview for now due to being out of previews

[anna-parker]

I would maybe make the option Accession or Display Name (as we do label the display name as display name on the sequence view pages and which metadata fields are added is ambiguous)

[theosanderson]

I updated my version of this endpoint with a new version that streams the data and so should scale indefinitely (as long as we let the Astro endpoint have as long as it needs). Here is a URL that demonstrates how it is used.

So we could adapt my approach to mash with what Cornelius has done here if we wanted to implement this?

[anna-parker]

I updated my version of this endpoint with a new version that streams the data and so should scale indefinitely (as long as we let the Astro endpoint have as long as it needs). Here is a URL that demonstrates how it is used.

So we could adapt my approach to mash with what Cornelius has done here if we wanted to implement this?

Omg this looks amazing Theo!! I would vote for adding this into Loculus if that is possible as I do think this is quite useful (wasn't 100% sure if your suggestion was to update this PR to point to your endpoint or to implement your endpoint here)

[emmahodcroft]

Technically I can't comment much but having a version that scales seems like a nice idea as we won't have to come whack this mole again later. I assume that for anything streaming the compression is not really possible so this is a clear trade off, but one that's ok.
It seems like this is likely largely aligned now, at least in a minimal version, and we can add customization to the metadata header later?