Overhaul review page: pagination and better filtering

[theosanderson]

As a reviewer I'd like to have a fast and efficient review page. Currently the review page for submitters works in a non-ideal way:

• it makes a big request that gets every sequence satisfying the requirements, which may be thousands leading to OOM Add pagination (or lazy load) to review page to a) load faster and b) reduce website memory requirements #2884 Website crashes with OOM when reviewing west nile as ingest user due to 16k errored sequences #2881.

• it also gets additional data in a lazy-load approach for everything on the page, making a lot of requests per page

• also it allows filtering for errors but does not allow filtering for warnings. User should be able to find any sequences that led to warnings easily Review page: allow filtering on warnings #1372

It would be great to overhaul this

• allowing filtering of errors and warnings (independently, or together) [on the server side]
• probably with the backend providing all the data needed to render a single paginated page of the review page in a single request (rework get-sequences endpoint to return metadata #1000).

Other related issues:

• When discarding, there's no signal anything is happening #2561
• Review page cards: use display names for fields not technical names #3116
• Review page: total number of sequences fluctuates and can be too high #3095

Acceptance criteria

• All relevant data for the review table can be requested from the backend in a single request (requests shouldn't scale up with number of rows)
• The backend doesn't crash, i.e. pagination should also be done at the DB request level
• There is a way to view only sequences with warnings
• All public and dev documentation adapted to new statuses, for example ripgrep "AWAITING_APPROVAL" should no longer show results (except for false positives)

[theosanderson]

Currently the filtering on errors relies on the fact that HAS_ERRORS has its own special status state. Warnings on the other hand don't have a status associated with them. A refactor could involve:

1. not changing these statuses
2. adding a HAS_WARNINGS_BUT_NOT_ERRORS status
3. merging AWAITING_APPROVAL and HAS_ERRORS together into a common PROCESSED status, then using different property(s) to monitor whether the sequence has errors or warnings.

My personal vote would be number 3.

[theosanderson]

Addressing this may close #1000 (whether directly or indirectly)

[theosanderson]

We may want to consider #2208

[fhennig]

I don't fully understand what needs to be done here, and which code areas are affected. I think it'd be best to have a call about this to refine together. After looking into this, I have these thoughts:

• pagination: The review page seems to already have pagination, is this still relevant or not?
• what exactly is lazy-loaded? I didn't see it with test sequences What would be a good way to fix this?
• Where are the in-processing sequences and their statuses fetched from? Where would a new warning status need to be added? (I could go look into the code but it's probably much faster if someone can show me a bit here)

EDIT: We had a chat and I updated the ticket, I think I know now what needs to be done!

[fhennig]

related: #3095

[corneliusroemer]

Note: I added the following point to the acceptance criteria

• All public and dev documentation adapted to new statuses, for example ripgrep "AWAITING_APPROVAL" should no longer show results (except for false positives)

(Also I fixed WAITING_FOR_APPROVAL -> AWAITING_APPROVAL in the tasks section above)

[theosanderson]

The individual-requests thing is a lower priority than the filtering-on-warnings thing. I was connecting them because I thought it might make sense to tackle them together but now we've decided not to do that in one PR I'm open to postponing the multiple-requests thing. It's basically a question of whether it's much easier to do now while it's fresh which might argue in favour of it, or not super different in which case it might make sense to move onto other tasks.

[fhennig]

I want to have a look at that, I think it won't take long, and like you said now I'm already kind of in the headspace for it

[fhennig]

Filtering for warnings is now supported: #3125

the labels on the review cards metadata now use display names: #3116

These smaller issues were fixed:

• a visual bug with the group selector (Review page filters show above group dropdown #3198)
• the tooltips were showing ‘underneath’ the filter (Tooltip clipped on review page #1447)
• the discard release buttons now have a confirmation popup (review page: Any action to discard/release sequences should require confirmation #2566)

We tried to rework the endpoints but it wasn't as easy as we thought so we stopped for now (#3189).

This is it for now, we decided to close this ticket with these changes.

[corneliusroemer]

I have been wondering what happened to the slow loading review page problem.

It's confusing that various issues pointing out problems with load times were closed in favor of this issue here but then this issue was closed without that being fixed.

It's of course ok to change plans if it turns out something is harder than expected but we should then reopen issues that weren't fixed.

I'll reopen a few issues that weren't fixed.