Is getSequences endpoint really used in full by anything? It seems no longer fit for purpose: should it be replaced?

[corneliusroemer]

get-sequences endpoint currently does all of the following:

• A) for a given group and organism: return status of all sequence entries (received/processing/processed/released)
• B) for a given group and organism: return processing results for sequences in status "processed" (no issues/warnings/errors)
• C) for a given group and organism and optionally filtered by status and/or processing result and optionally paginated:

                SequenceEntriesView.accessionColumn,
                SequenceEntriesView.versionColumn,
                SequenceEntriesView.submissionIdColumn,
                SequenceEntriesView.statusColumn,
                SequenceEntriesView.isRevocationColumn,
                SequenceEntriesView.groupIdColumn,
                SequenceEntriesView.submitterColumn,
                SequenceEntriesView.organismColumn,
                SequenceEntriesView.submittedAtTimestampColumn,
                SequenceEntriesView.errorsColumn,
                SequenceEntriesView.warningsColumn,
                SequenceEntriesView.processingResultColumn,
                DataUseTermsTable.dataUseTermsTypeColumn,
                DataUseTermsTable.restrictedUntilColumn,
  

Parts of it are used in the following places by the website (afaict):

• Submission portal: uses A) to show how many sequences are awaiting review [the fact that getSequences returns so much more was behind that pages slowness, fixed in a kind of hacky way by fix(backend,website): calculate status counts efficiently & request only aggregates for submission portal #3279, by simply not asking for any of the voluminous data
• Review page: uses A) and B) to show counts of how many sequences are in which states and uses only the accession and version columns of C) to be able to then query individual entries

I couldn't find anything that uses all of the other columns: in particular the data use terms seem to be joined for no reason. There's no user of it.

It looks like we could serve the current website much leanly by:

• Having an endpoint that simply serves aggregate stats for a given group and organism and optionally returns accessions but nothing else

It's better to make a new dedicated endpoint if we need more stuff rather than having huge beasts that do lots of things but are slow and complicated as a result.

[corneliusroemer]

I looked into it and we do use almost all the fields, including data use terms. Current situation is not a problem anymore now that we've got #3279 - if we find it getting slow again we can reconsider.

[corneliusroemer]

If we separated the counts and the data to show on the cards, we could cache the data on the cards. That way we wouldn't have to repeatedly transfer large amounts of data every second, only do it initially.

Would be easy to use caching similar to what we've got already:

• no need to refresh if there haven't been any database changes (table update tracker)
• if there has been change, one can just calculate a hash of the result to save bandwidth
• sequence entries could return all metadata (but not sequence data - as we don't show that)
• that would be enough to create all the cards in one go without 50 requests

50 requests aren't that bad, but they are also not great.

Overall this is now a small issue, not a big one as we've solved perf.