Restructure of openDS datamodels and new versions #97

samleeflang · 2024-06-12T12:37:49Z

Ok massive changes.

All folder names to singular (before it was inconsistent)
Version has moved a level down, this way we can have a version for each type and not for all types together
Datamodel changes for:
- Source system
- Mapping
- Machine Annotation Service
- Annotations
Captured and changed datamodel of:
- CreateUpdateTombstone Event
Update Dockerfile to include lastest endpoint

southeo

🌟

southeo · 2024-06-12T13:16:55Z

data-model/Dockerfile

+COPY fdo-type/virtual-collection/0.1.0/schema schema-root/schemas/fdo-type/vitual-collection/0.1.0
+
+# Versioned FDO Type schemas


Latest, not versioned

southeo · 2024-06-12T13:20:09Z

data-model/developer-schema/annotation/0.2.0/schema/annotation-event.json

@@ -0,0 +1,55 @@
+{
+  "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "$id": "https://schemas.dissco.tech/schemas/developer-schema/annotation/0.2.0/annotation-for-mas.json",


should be annotation-event.json

data-model/developer-schema/annotation/0.1.0/schema/annotation-event-for-web.json

southeo · 2024-06-12T13:26:49Z

data-model/developer-schema/annotation/0.2.0/schema/annotation-event.json

+      "type": "array",
+      "items": {
+        "type": "object",
+        "properties": {


please add additionalproperties = false

southeo · 2024-06-12T13:29:59Z

data-model/developer-schema/annotation/0.2.0/schema/annotation-event.json

+        "description": "The type of the object, in this case ods:BatchMetaData",
+        "const": "ods:AnnotationEvent"
+      }
+    },


does this need an @id property?

I don't think so, what would it be, the jobID?

southeo · 2024-06-12T13:32:33Z

data-model/developer-schema/annotation/0.2.0/schema/annotation-for-mas.json

+            "https://doi.org/21.T11148/894b1e6cad57e921764e"
+          ]
+        },
+        "oa:hasSelector": {


1 - if this isn't an array, do we need the hasXXX structuring?
2 - I don't think we can change an external namespace

this applies to oa:hasTarget, ac:hasROI, oa:hasBody

If we need to add the hasXXX structuring, I don't think we can change the name of the term. We'd probably have to nest it like we do with specimens

ods:hasSelector: { "oa:selector":{ ... } }

Yes I thought about this but we cannot change an externally borrowed term. What we could do is create our own term ods:Selector and then map it but that is ugly and confusing as well. I think we are just stuck with the oa term which doesn't fit our naming convention.
So 1. no we wouldn't need a hasXXX structure but we got 2. this is how this is called in oa

southeo · 2024-06-12T13:41:06Z

data-model/fdo-type/annotation/0.3.0/schema/annotation.json

+    "@type": {
+      "type": "string",
+      "description": "The type of the object, in this case oa:Annotation",
+      "const": "oa:Annotation"


can we call it oa:Annotation if we've made changes? Maybe ods:Annotation?

yes. good point. technically, this const is not in the oa name space.

Changed this to ods:Annotation

southeo · 2024-06-12T13:43:28Z

data-model/fdo-type/annotation/0.3.0/schema/annotation.json

+      "additionalProperties": false
+    },
+    "oa:hasBody": {
+      "type": "object",


might not be able to use hasXXX here either

Taking over the hasBody over from oa, I think that is the better choice here

sharifX · 2024-06-13T19:41:29Z

data-model/developer-schema/annotation/0.1.0/schema/annotation-event.json

@@ -1,12 +1,12 @@
 {
  "$schema": "https://json-schema.org/draft/2020-12/schema",
-  "$id": "https://schemas.dissco.tech/schemas/annotations/0.1.0/annotation-event.json",
-  "description": "Object sent to annotation processing service from core backend. Internal use only.",
+  "$id": "https://schemas.dissco.tech/schemas/developer-schema/annotation/0.1.0/annotation-event.json",


what is "deverloper-schema" here?

That are schema's used for systems who want to interact to our system. In this case for example the MAS. The MAS doesn't needs to produces annotations which are slightly different than what the annotations endpoint produces. For example, the MAS does not need to indicate the generator part as this gets added by the annotation-processing-service. However, if the MAS supports batch, it does need to provide the batching information. So we put these schemas under deverloper-schema to indicate that only developers building on DiSSCo platform need these schemas.

sharifX · 2024-06-13T19:45:41Z

data-model/fdo-type/annotation/0.3.0/schema/annotation.json

+    "@type": {
+      "type": "string",
+      "description": "The type of the object, in this case oa:Annotation",
+      "const": "oa:Annotation"


yes. good point. technically, this const is not in the oa name space.

sharifX · 2024-06-13T19:47:03Z

data-model/developer-schema/annotation/0.2.0/schema/annotation-event.json

+            "description": "The type of the object, in this case ods:BatchMetaData",
+            "const": "ods:BatchMetaData"
+          },
+          "ods:placeInBatch": {


placeInBatchqueue be better?

Don't think this is related to the position in the queue, @southeo can probably better answer this.

There are two components of an AnnotationEvent sent from the MAS to the processing service: BatchMetadata and the Annotations. Both of these can be lists, however, so the processing service needs a way to figure out which Annotation is associated with which BatchMetadata. The field placeInBatch is the value that links these two fields.

(The batch metadata are the terms and values used to determine the search queries for to create annotations)

TomDijkema · 2024-06-14T07:10:38Z

data-model/developer-schema/annotation/0.2.0/schema/annotation-for-mas.json

+                  "type": "string",
+                  "description": "The full jsonPath of the field being annotated. Following: https://goessner.net/articles/JsonPath/index.html#e2",
+                  "examples": [
+                    "$.occurrences[0].location.dwc:country"


Update this example? Since occurrences is not present any more in the new data model

Thanks, fixed this everywhere

TomDijkema · 2024-06-14T07:11:05Z

data-model/developer-schema/annotation/0.2.0/schema/annotation-for-mas.json

+                  "type": "string",
+                  "description": "The full jsonPath of the class being annotated. Following: https://goessner.net/articles/JsonPath/index.html#e2",
+                  "examples": [
+                    "$.occurrences[0].location.georeference"


Also update this example since occurrences is not in the data model anymore

TomDijkema · 2024-06-14T07:26:20Z

data-model/fdo-type/machine-annotation-service/0.3.0/examples/mas-example.json

+      "Other Geodiversity",
+      "Unclassified"
+    ],
+    "$.occurrences[*].location.dwc:locality": [


Same here, occurrences does not exist anymore right

TomDijkema · 2024-06-14T07:28:06Z

data-model/fdo-type/machine-annotation-service/0.3.0/schema/mas.json

+            "Other Geodiversity",
+            "Unclassified"
+          ],
+          "$.occurrences[*].location.dwc:locality": [


Same here, occurrences does not exist anymore right

TomDijkema

🍌

samleeflang added 2 commits June 12, 2024 14:34

Restructure of openDS datamodels and new versions

718592d

Fix FDO profile $ids

00af4c9

samleeflang requested review from southeo, TomDijkema, sharifX and wouteraddink June 12, 2024 12:46

southeo reviewed Jun 12, 2024

View reviewed changes

sharifX reviewed Jun 13, 2024

View reviewed changes

TomDijkema reviewed Jun 14, 2024

View reviewed changes

samleeflang requested review from southeo, TomDijkema and sharifX June 17, 2024 06:18

Code review

44f4883

TomDijkema approved these changes Jun 17, 2024

View reviewed changes

samleeflang added 2 commits June 17, 2024 14:42

Add dataset ID, was missing for some reason

ed46d1e

Switch oa:Annotation to ods:Annotation

7ef5f01

samleeflang merged commit 78a489a into master Jun 17, 2024
1 check passed

samleeflang deleted the feature/all-objects-to-0.3.0 branch June 17, 2024 12:49

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Restructure of openDS datamodels and new versions #97

Restructure of openDS datamodels and new versions #97

samleeflang commented Jun 12, 2024

southeo left a comment

southeo Jun 12, 2024

samleeflang Jun 17, 2024

southeo Jun 12, 2024

samleeflang Jun 17, 2024

southeo Jun 12, 2024

samleeflang Jun 17, 2024

southeo Jun 12, 2024

samleeflang Jun 17, 2024

southeo Jun 12, 2024

southeo Jun 12, 2024

samleeflang Jun 17, 2024

southeo Jun 12, 2024

sharifX Jun 13, 2024

samleeflang Jun 17, 2024

southeo Jun 12, 2024

samleeflang Jun 17, 2024

sharifX Jun 13, 2024

samleeflang Jun 17, 2024 •

edited

Loading

sharifX Jun 13, 2024

sharifX Jun 13, 2024

samleeflang Jun 17, 2024

southeo Jun 18, 2024

TomDijkema Jun 14, 2024

samleeflang Jun 17, 2024

TomDijkema Jun 14, 2024

samleeflang Jun 17, 2024

TomDijkema Jun 14, 2024

samleeflang Jun 17, 2024

TomDijkema Jun 14, 2024

samleeflang Jun 17, 2024

TomDijkema left a comment

		COPY fdo-type/virtual-collection/0.1.0/schema schema-root/schemas/fdo-type/vitual-collection/0.1.0

		# Versioned FDO Type schemas

Restructure of openDS datamodels and new versions #97

Restructure of openDS datamodels and new versions #97

Conversation

samleeflang commented Jun 12, 2024

southeo left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

samleeflang Jun 17, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TomDijkema left a comment

Choose a reason for hiding this comment

samleeflang Jun 17, 2024 •

edited

Loading