Mongo DB Storage

[southeo]

Data Storage

We've changed from a postgres rdb to a mongo doc store! 🎉

Each pid record is a single document, identified by its handle. Each field in the pid record needs some metadata, which is now represented in a json:

• index
• type
• data
  • for regular fields: format (always "string"), and value
  • for HS_ADMIN: format="admin", index=200, value=0.NA/{prefix}, permissions = 011111110011
• ttl
• timestamp

An example document is found below:

{
  "_id": "20.5000.1025/ABC",
  "normalisedSpecimenObjectId": "06211:BIRD-COLLECTION-1:RMS-123",
  "values": [
    {
      "index": 1,
      "type": "digitalObjectType",
      "data": {
        "format": "string",
        "value": "https://hdl.handle.net/21.T11148/532ce6796e2828dd2be6"
      },
      "ttl": 86400,
      "timestamp": "2024-06-28 04:32:04Z"
    },
    {
      "index": 203,
      "type": "normalisedPrimarySpecimenObjectId",
      "data": {
        "format": "string",
        "value": "06211:BIRD-COLLECTION-1:RMS-123"
      },
      "ttl": 86400,
      "timestamp": "2024-06-28 04:32:04Z"
    }
  ]
}

Indexing Local Identifiers

3 kinds of objects have local identifiers we can use to check if a handle has already been created for that kind of objects:

• Annotations: annotationHash
• specimens: normalisedSpecimenId
• Media: mediaUrl (called primaryMediaId in the handle API)

When we create a specimen, we first search on normalisedSpecimenId to see if a handle has already been created for that object. By putting normalisedSpecimenId on the top-level of the mongo document, we can index the field and quickly find if the handle exists.

We don't do this for annotations or media, but I've included the functionality, if we want to make the check.

Changes to Updates

We no longer support partial updates. Now, the request must contain all fields required to build a new

• We keep a few fields from the previous version: pidIssueDate, fdoType, license, PID, PidStatus. These fields will have their original timestamp
• We increment pidIssueNumber
• We replace all other fields, even if they are unchanged, updating the timestamp

Changes to Tombstone

Tombstoned records are not truncated anymore. Now, they have all the fields they had before they were tombstoned, but with additional tombstone metadata.

JsonSchemaValidator

• When we allowed partial updates, we had separate validation protocols for updates and creates because we didn't require the user to provide all the fields for an update. Now, all fields are required for both kinds of requests, so the validator got simplified
• Pretty sure we can remove this class honestly. We can the validator hassle by just accepting a specific object instead of a raw json. trying to keep this PR within its original scope though

Other

• Cleaned up testsUtils
• Refactored some names

[samleeflang]

How is the performance, do you feel it is equal to the db?
Really like how easy the mongodb repo looks

[southeo]

Haven't tested large scale, but small tests are performing well. While we can't use the psql batch insert function anymore, i don't think handle creation will be a bottleneck. We now only insert one document instead of up to 30 rows per specimen. Updates are a lot better too, since we had something like update (ATTRIBUTE, where HANDLE = xyz and INDEX = 111) for EACH field we wanted to update. Now we just replace the document on the _id. And adding the local ids to the top-level is really good.
🥳

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
94.4% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

[samleeflang]

Perfect