Test update submissions are exported correctly

[mshadbolt]

As a data wrangler I would like to test that the exporter currently exports updates correctly to the GCP staging area so that downstream components are able to update the data records in the TDR and analysis outputs contain the most up to date metadata.

Tests I will try:

• Make edits to the project level metadata

  • delete a contributor
  • edit the content of a field
  • add a field that was formerly empty

• Make edits to biomaterial or protocol entities

  • delete a field
  • edit a field
  • add a field that was formerly empty
  • add a new experimental subgraph i.e. donor_organism->sequence_file

• Do a full update to a project

  • do the updates outlined in ticket AUDR Multiple updates for the Spatio-temporal immune zonation of the human kidney dataset in prod #81

For each update the DCP2 MVP schema specifies:

• entity_id/uuid of all entities remains the same as the previous version
• version of each entity is updated to the current time
• the path to any data file remains unchanged (because it reuses the previously uploaded file

[mshadbolt]

First test I will use the HCA_only project I created when testing the project states.

project uuid: a4804fe4-6135-447a-82da-0a7e1fc55524

I edited the project metadata directly in the UI

Changes made:

• deleted GEO accession
• changed ArrayExpress accession to E-MTAB-1234
• added the supplementary link http://celltag.org
• deleted the contributor John Doe

[mshadbolt]

First test didn't work because currently the system will not propagate updates done in the UI to either the exporter or to archiving.

The system also ignores project metadata in a spreadsheet so it doesn't seem like there is a clear path to update project level metadata in the current system.

I will move on to try updating other metadata entities.

[mshadbolt]

For fun I decided to do a spreadsheet submission to project where I altered the project information, can confirm that the changes to project metadata didn't get propagated into the UI and the submission seemed to get stuck in pending, perhaps because there was nothing to validate?

reported some weirdness in this ticket ebi-ait/hca-ebi-dev-team#197

When I upload using the all submissions route I can submit, but information doesn't get updated in the UI.

And results in 3 new projects in my projects view. Even though I specified the submission should go to HCA the submission stays in submitted state and doesn't progress to exporting->exported

[mshadbolt]

Now I'll try to update some other entities to see whether that would work

I submitted a new project here: https://ui.ingest.dev.archive.data.humancellatlas.org/projects/detail?uuid=ed7cc937-5065-40cb-8e1c-400691715619

I downloaded the spreadsheet and made the following changes:

• added new sub-graph
  • donor 2 specimen 2 etc. -- > pbmc4k_1000_S2_L001 * 3
• deleted weight fields from donor 1
• added a selected cell type to donor 1
• changed specimen organ part to hippocampus alveus

Uploaded via 'submit to project'

Doesn't seem to work and I get the error:
:An error occurred importing the file.

"A link from a biomaterial with id specimen_ID_2 to a protocol with id, "collection_protocol_1", is not found in the spreadsheet"

I'm going to assume this is some issue with trying to add new entities and try only doing edits rather than adding whole new entities.

I edited the spreadsheet to remove the new entities and stuck with the edits as above.

I submitted and it seems to get stuck in 'submitted' state and doesn't proceed to exporting even though I ticket box to send to HCA (I think)

https://ui.ingest.dev.archive.data.humancellatlas.org/submissions/detail?uuid=4e081b6e-9cef-40d9-8f7d-4d7952a23ce2&project=ed7cc937-5065-40cb-8e1c-400691715619

[mshadbolt]

@aaclan-ebi noted that due to some of the changes made to the UI, updating or adding to a project is no longer possible through the UI

[mshadbolt]

Alegria mentioned that the addition submissions should still work so I decided to test this by making a submission of an extra specimen, cell suspension, sequence files that link to the existing donor and protocols.

https://ui.ingest.dev.archive.data.humancellatlas.org/submissions/detail?id=5efdff8601cb5e4efec8bd90&project=ed7cc937-5065-40cb-8e1c-400691715619

Submission went Valid and proceeded to 'exporting' submission state, it seems to be stuck in this state but will check back tomorrow.

Alegria created a ticket to address updates, this ticket is blocked until that work is completed ebi-ait/hca-ebi-dev-team#201

[mshadbolt]

rolando confirmed the 'add' submission was stuck in exporting and fixed a bug in the exporter and now the project is 'Complete'

[lauraclarke]

@mshadbolt Is this still blocked?

[mshadbolt]

yes it won't be unblocked until we can do update submissions

[lauraclarke]

Okay, no updates are possible at all right now then?

[mshadbolt]

see comment above #82 (comment)

due to some of the changes made to the UI, updating or adding to a project is no longer possible through the UI

[lauraclarke]

Thanks, @clairerye this sounds like we need a ticket for the fix here and to prioritize this work against other issues

@mshadbolt it is reasonable to assume that there can be no further progress made on updates till this issue is fixed. If that is the case it probably makes sense to close this issue and @clairerye can restart testing when the fix is prioritized

[mshadbolt]

yes there is already a dev ticket which I mentioned above. Will leave up to Claire whether she wants to keep this open and blocked so that the work that is blocked is visible or whether to close and remake a ticket later

[clairerye]

I think its useful having it showing that it is blocked the way Marion has marked it and links to the dev ticket that it is dependent on.