From 719fe56f018d4f81efd1e9701eb1020336176cf3 Mon Sep 17 00:00:00 2001 From: Sebastien Besson Date: Wed, 25 Nov 2020 09:06:29 +0100 Subject: [PATCH 01/18] Add expected hierarchy for HCS data --- latest/index.bs | 60 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 60 insertions(+) diff --git a/latest/index.bs b/latest/index.bs index 149c5b45..cbb1cf94 100644 --- a/latest/index.bs +++ b/latest/index.bs @@ -99,6 +99,11 @@ is represented here as it would appear locally but could equally be stored on a web server to be accessed via HTTP or in object storage like S3 or GCS. +### Images + +The following layout describes the expected Zarr hierarchy for images with +multiple levels of resolutions and optionally associated labels. + ``` . # Root folder, potentially in S3, │ # with a flat list of images by image ID. @@ -144,6 +149,61 @@ like S3 or GCS. └── n ``` +### High-content screening + +The following layout should describe the hierarchy for a high-content screening +dataset. There are exactly four levels of hierarchies above the images: + +- the top-level group defines the plate, it MUST implement the plate + specification defined below +- the second group defines all acquisitions performed on a single plate. If + only one acquisition was performed, a single group must be used. +- the third group defines all the well rows available for an acquisition +- the fourth group defines all the well columns available for a given well row +- the fifth group defined all the individual fields of views for a given well. + The fields of views are images, SHOULD implement the "multiscales" + specification, MAY implement the "omero" specification and MAY contain + labels. + +``` +. # Root folder, potentially in S3, +│ +├── 123.zarr # One image (id=123) converted to Zarr. +│ +└── 5966.zarr # One plate (id=5966) converted to Zarr + ├── .zgroup + ├── .zattrs # Implements "plate" specification + │ + ├── 2020-10-10 # First acquisition + │ │ + │ ├── .zgroup + │ ├── .zattrs + │ │ + │ ├── A # First row of acquisition 2020-10-10 + │ │ ├── .zgroup + │ │ ├── .zattrs + │ │ │ + │ │ ├── 1 # First column of row A + │ │ │ ├── .zgroup + │ │ │ ├── .zattrs + │ │ │ │ + │ │ │ ├── Field_1 # First field of view of well A1 + │ │ │ │ │ + │ │ │ │ ├── .zgroup + │ │ │ │ ├── .zattrs # Implements "multiscales", "omero" + │ │ │ │ ├── 0 + │ │ │ │ │ ... # Resolution levels + │ │ │ │ ├── n + │ │ │ │ └── labels + │ │ │ ├── ... # Fields of view + │ │ │ └── Field_m + │ │ ├── ... # Columns + │ │ └── 12 + │ ├── ... # Rows + │ └── H + ├── ... # Acquisitions + └── l + Metadata {#metadata} ==================== From 56914b8da0bf765e33bd9405cbdccb0d5ab9f1ed Mon Sep 17 00:00:00 2001 From: Sebastien Besson Date: Wed, 25 Nov 2020 09:10:03 +0100 Subject: [PATCH 02/18] Add metadata specification for HCS data --- latest/index.bs | 79 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 79 insertions(+) diff --git a/latest/index.bs b/latest/index.bs index cbb1cf94..015a6455 100644 --- a/latest/index.bs +++ b/latest/index.bs @@ -333,6 +333,80 @@ above). ] ``` +"plate" metadata {#plate-md} +---------------------------- + +For high-content screening datasets, the plate layout can be found under the +custom attributes of the plate group under the `plate` key. + +plateAcquisitions + A list of JSON objects defining the acquisitions for a given plate. + Each acquisition object MUST contain a `path` key identifying the path to the + acquisition group. + +rows + An integer defining the number of rows i.e. the first dimension of + a two-dimensional array of wells. + +row_names + A list of strings defining the names of the rows + +rows + An integer defining the number of columns i.e. the first dimension of + a two-dimensional array of wells. + +column_names + A list of strings defining the names of the columns + +images + A list of JSON objects defining the images (or well samples) containing in + the plate. + Each image object MUST contain a `path` key identifying the path to the + individual image. + + +For example the following JSON object encodes a plate with one acquisition and +6 wells containing up to two fields of view each. + +```json +"plate": + { + "rows": 2, + "columns": 3, + "row_names": ["A", "B"], + "column_names": ["1", "2", "3"], + "plateAcquisitions": [ + {"path": "2020-10-10"} + ], + "images": [ + { + "path": "2020-10-10/A/1/Field_1" + }, + { + "path": "2020-10-10/A/1/Field_2" + }, + { + "path": "2020-10-10/A/2/Field_1" + }, + { + "path": "2020-10-10/A/2/Field_2" + }, + { + "path": "2020-10-10/A/3/Field_1" + }, + { + "path": "2020-10-10/B/1/Field_1" + }, + { + "path": "2020-10-10/B/2/Field_1" + }, + { + "path": "2020-10-10/B/3/Field_1" + } + ] + } +``` + Citing {#citing} ================ @@ -353,6 +427,11 @@ Version History {#history} Description + + 0.1.3-dev5 + TBD + Add the HCS specification + 0.1.3-dev4 2020-09-14 From d047bacde01f27abf3926ea5ed1a9852d4415f0b Mon Sep 17 00:00:00 2001 From: Sebastien Besson Date: Wed, 25 Nov 2020 09:11:08 +0100 Subject: [PATCH 03/18] Use Markdown definition lists --- latest/index.bs | 45 ++++++++++++++++++++------------------------- 1 file changed, 20 insertions(+), 25 deletions(-) diff --git a/latest/index.bs b/latest/index.bs index 015a6455..90cb4e62 100644 --- a/latest/index.bs +++ b/latest/index.bs @@ -339,31 +339,26 @@ above). For high-content screening datasets, the plate layout can be found under the custom attributes of the plate group under the `plate` key. -plateAcquisitions - A list of JSON objects defining the acquisitions for a given plate. - Each acquisition object MUST contain a `path` key identifying the path to the - acquisition group. - -rows - An integer defining the number of rows i.e. the first dimension of - a two-dimensional array of wells. - -row_names - A list of strings defining the names of the rows - -rows - An integer defining the number of columns i.e. the first dimension of - a two-dimensional array of wells. - -column_names - A list of strings defining the names of the columns - -images - A list of JSON objects defining the images (or well samples) containing in - the plate. - Each image object MUST contain a `path` key identifying the path to the - individual image. - +
+
plateAcquisitions
+
A list of JSON objects defining the acquisitions for a given plate. + Each acquisition object MUST contain a `path` key identifying the path + to the acquisition group.
+
rows
+
An integer defining the number of rows i.e. the first dimension of + a two-dimensional array of wells.
+
row_names
+
A list of strings defining the names of the rows
+
rows
+
An integer defining the number of columns i.e. the first dimension of + a two-dimensional array of wells.
+
column_names
+
A list of strings defining the names of the columns
+
images
+
A list of JSON objects defining the images (or well samples) containing in the plate. + Each image object MUST contain a `path` key identifying the path to the + individual image.
+
For example the following JSON object encodes a plate with one acquisition and 6 wells containing up to two fields of view each. From 4f8c6e4ef1e8f6e90dc545d498b80664c032b924 Mon Sep 17 00:00:00 2001 From: Sebastien Besson Date: Wed, 25 Nov 2020 09:12:49 +0100 Subject: [PATCH 04/18] Add name field to plate specification --- latest/index.bs | 3 +++ 1 file changed, 3 insertions(+) diff --git a/latest/index.bs b/latest/index.bs index 90cb4e62..d329c718 100644 --- a/latest/index.bs +++ b/latest/index.bs @@ -340,6 +340,8 @@ For high-content screening datasets, the plate layout can be found under the custom attributes of the plate group under the `plate` key.
+
name
+
A string defining the name of the plate.
plateAcquisitions
A list of JSON objects defining the acquisitions for a given plate. Each acquisition object MUST contain a `path` key identifying the path @@ -366,6 +368,7 @@ For example the following JSON object encodes a plate with one acquisition and ```json "plate": { + "name": "test", "rows": 2, "columns": 3, "row_names": ["A", "B"], From 36f6ec3986c5ebc9473695284f603a185f918dce Mon Sep 17 00:00:00 2001 From: Sebastien Besson Date: Wed, 25 Nov 2020 09:18:32 +0100 Subject: [PATCH 05/18] Capture new layout of HCS specification --- latest/index.bs | 119 +++++++++++++++++++++++++++++------------------- 1 file changed, 73 insertions(+), 46 deletions(-) diff --git a/latest/index.bs b/latest/index.bs index d329c718..69eeaf91 100644 --- a/latest/index.bs +++ b/latest/index.bs @@ -151,18 +151,22 @@ multiple levels of resolutions and optionally associated labels. ### High-content screening -The following layout should describe the hierarchy for a high-content screening +The following layout describes the hierarchy for a high-content screening dataset. There are exactly four levels of hierarchies above the images: -- the top-level group defines the plate, it MUST implement the plate +- the top-level group defines a single plate and MUST implement the plate specification defined below -- the second group defines all acquisitions performed on a single plate. If +- the second group defines all acquisitions performed on the plate. If only one acquisition was performed, a single group must be used. -- the third group defines all the well rows available for an acquisition -- the fourth group defines all the well columns available for a given well row +- the third group defines all the rows available for an acquisition. The group + name must match the name of one of the row objects defined in the plate + metadata. +- the fourth group defines all the wells available for a given row. The group + name must match the name of one of the columd objects defined in the plate + metadata. The well group MUST implement the well specification defined below - the fifth group defined all the individual fields of views for a given well. The fields of views are images, SHOULD implement the "multiscales" - specification, MAY implement the "omero" specification and MAY contain + specification, MAY implement the "omero" specification and MAY contain labels. ``` @@ -185,9 +189,9 @@ dataset. There are exactly four levels of hierarchies above the images: │ │ │ │ │ ├── 1 # First column of row A │ │ │ ├── .zgroup - │ │ │ ├── .zattrs + │ │ │ ├── .zattrs # Implements "well" specification │ │ │ │ - │ │ │ ├── Field_1 # First field of view of well A1 + │ │ │ ├── 0 # First field of view of well A1 │ │ │ │ │ │ │ │ │ ├── .zgroup │ │ │ │ ├── .zattrs # Implements "multiscales", "omero" @@ -196,7 +200,7 @@ dataset. There are exactly four levels of hierarchies above the images: │ │ │ │ ├── n │ │ │ │ └── labels │ │ │ ├── ... # Fields of view - │ │ │ └── Field_m + │ │ │ └── m │ │ ├── ... # Columns │ │ └── 12 │ ├── ... # Rows @@ -342,65 +346,88 @@ custom attributes of the plate group under the `plate` key.
name
A string defining the name of the plate.
-
plateAcquisitions
+
acquisitions
A list of JSON objects defining the acquisitions for a given plate. - Each acquisition object MUST contain a `path` key identifying the path - to the acquisition group.
+ Each acquisition object MUST contain a `path` key identifying the path + to the acquisition subgroup.
rows
-
An integer defining the number of rows i.e. the first dimension of - a two-dimensional array of wells.
-
row_names
-
A list of strings defining the names of the rows
-
rows
-
An integer defining the number of columns i.e. the first dimension of - a two-dimensional array of wells.
-
column_names
-
A list of strings defining the names of the columns
-
images
-
A list of JSON objects defining the images (or well samples) containing in the plate. - Each image object MUST contain a `path` key identifying the path to the - individual image.
+
A list of JSON defining the rows of the plate. Each row object defines + the properties of the row at the index of the object in the list. If + not empty, it MUST contain a `name` key specifying the row name.
+
columns
+
A list of JSON defining the columns of the plate. Each column object + defines the properties of the column at the index of the object in the + list. If not empty, it MUST contain a `name` key specifying the column + name.
+
field_count
+
An integer defining the maximum number of fields per view across all + wells.
+
wells
+
A list of JSON objects defining the wells of the plate. Each well object + MUST contain a `path` key identifying the path to the well subgroup.
-For example the following JSON object encodes a plate with one acquisition and -6 wells containing up to two fields of view each. +For example the following JSON object defines a plate with one acquisition and +6 wells (2 rows and 3 columns), containing up to three fields of view each. ```json -"plate": - { +"plate": { + "acquisitions": [{"path": "0"}], + "columns": [{"name": "1" }, {"name": "2" }, {"name": "3" }], + "field_count": 3, "name": "test", - "rows": 2, - "columns": 3, - "row_names": ["A", "B"], - "column_names": ["1", "2", "3"], - "plateAcquisitions": [ - {"path": "2020-10-10"} - ], - "images": [ + "rows": [{"name": "A" }, {"name": "B" }], + "wells": [ { - "path": "2020-10-10/A/1/Field_1" + "path": "2020-10-10/A/1" }, { - "path": "2020-10-10/A/1/Field_2" + "path": "2020-10-10/A/2" }, { - "path": "2020-10-10/A/2/Field_1" + "path": "2020-10-10/A/3" }, { - "path": "2020-10-10/A/2/Field_2" + "path": "2020-10-10/B/1" }, { - "path": "2020-10-10/A/3/Field_1" + "path": "2020-10-10/B/2" }, { - "path": "2020-10-10/B/1/Field_1" + "path": "2020-10-10/B/3" + } + ] + } +``` + +### "well" metadata + +For high-content screening datasets, the metadata about all fields of views +under a given well can be found under the "well" key in the attributes of the +well group. + +
+
images
+
A list of JSON objects defining the fields of views for a given well. + Each object MUST contain a `path` key identifying the path to the + field of view.
+
+ +For example the following JSON object defines a well with three fields of +views: + +```json +"well": { + "images": [ + { + "path": "0" }, { - "path": "2020-10-10/B/2/Field_1" + "path": "1" }, { - "path": "2020-10-10/B/3/Field_1" - } + "path": "2" + }, ] } ``` From fbf7d4fbdff291332e2196929abe6e8b3b7f755d Mon Sep 17 00:00:00 2001 From: jmoore Date: Wed, 25 Nov 2020 09:24:15 +0100 Subject: [PATCH 06/18] Correct formatting for bikeshed --- latest/index.bs | 28 ++++++++++++++++------------ 1 file changed, 16 insertions(+), 12 deletions(-) diff --git a/latest/index.bs b/latest/index.bs index 69eeaf91..9488042f 100644 --- a/latest/index.bs +++ b/latest/index.bs @@ -99,7 +99,8 @@ is represented here as it would appear locally but could equally be stored on a web server to be accessed via HTTP or in object storage like S3 or GCS. -### Images +Images {#image-layout} +---------------------- The following layout describes the expected Zarr hierarchy for images with multiple levels of resolutions and optionally associated labels. @@ -149,25 +150,26 @@ multiple levels of resolutions and optionally associated labels. └── n ``` -### High-content screening +High-content screening {#hcs-layout} +------------------------------------ The following layout describes the hierarchy for a high-content screening dataset. There are exactly four levels of hierarchies above the images: - the top-level group defines a single plate and MUST implement the plate - specification defined below + specification defined below - the second group defines all acquisitions performed on the plate. If - only one acquisition was performed, a single group must be used. + only one acquisition was performed, a single group must be used. - the third group defines all the rows available for an acquisition. The group - name must match the name of one of the row objects defined in the plate - metadata. + name must match the name of one of the row objects defined in the plate + metadata. - the fourth group defines all the wells available for a given row. The group - name must match the name of one of the columd objects defined in the plate - metadata. The well group MUST implement the well specification defined below + name must match the name of one of the columd objects defined in the plate + metadata. The well group MUST implement the well specification defined below - the fifth group defined all the individual fields of views for a given well. - The fields of views are images, SHOULD implement the "multiscales" - specification, MAY implement the "omero" specification and MAY contain - labels. + The fields of views are images, SHOULD implement the "multiscales" + specification, MAY implement the "omero" specification and MAY contain + labels. ``` . # Root folder, potentially in S3, @@ -207,6 +209,7 @@ dataset. There are exactly four levels of hierarchies above the images: │ └── H ├── ... # Acquisitions └── l +``` Metadata {#metadata} ==================== @@ -400,7 +403,8 @@ For example the following JSON object defines a plate with one acquisition and } ``` -### "well" metadata +"well" metadata {#well-md} +-------------------------- For high-content screening datasets, the metadata about all fields of views under a given well can be found under the "well" key in the attributes of the From 564506e87d43106f19b3be75bad883857ea024e0 Mon Sep 17 00:00:00 2001 From: Sebastien Besson Date: Wed, 25 Nov 2020 11:15:42 +0000 Subject: [PATCH 07/18] Update HCS layout to drop acquisition group Also revert the order of the group description for clarity --- latest/index.bs | 76 +++++++++++++++++++------------------------------ 1 file changed, 30 insertions(+), 46 deletions(-) diff --git a/latest/index.bs b/latest/index.bs index 9488042f..239300eb 100644 --- a/latest/index.bs +++ b/latest/index.bs @@ -153,62 +153,46 @@ multiple levels of resolutions and optionally associated labels. High-content screening {#hcs-layout} ------------------------------------ -The following layout describes the hierarchy for a high-content screening -dataset. There are exactly four levels of hierarchies above the images: - -- the top-level group defines a single plate and MUST implement the plate - specification defined below -- the second group defines all acquisitions performed on the plate. If - only one acquisition was performed, a single group must be used. -- the third group defines all the rows available for an acquisition. The group - name must match the name of one of the row objects defined in the plate - metadata. -- the fourth group defines all the wells available for a given row. The group - name must match the name of one of the columd objects defined in the plate - metadata. The well group MUST implement the well specification defined below -- the fifth group defined all the individual fields of views for a given well. - The fields of views are images, SHOULD implement the "multiscales" - specification, MAY implement the "omero" specification and MAY contain - labels. +The following specification defines the hierarchy for a high-content screening +dataset. Three groups must be defined above the images: + +- the group above the images defines the well and MUST implement the + [well specification](#well-md). All images contained in a well are fields + of view of the same well +- the group above the well defines a row of wells +- the group above the well row defines an entire plate i.e. a two-dimensional + collection of wells organized in rows and columns. It MUST implement the + [plate specification](#plate-md) + ``` -. # Root folder, potentially in S3, -│ -├── 123.zarr # One image (id=123) converted to Zarr. +. # Root folder, potentially in S3, │ -└── 5966.zarr # One plate (id=5966) converted to Zarr +└── 5966.zarr # One plate (id=5966) converted to Zarr ├── .zgroup - ├── .zattrs # Implements "plate" specification - │ - ├── 2020-10-10 # First acquisition - │ │ + ├── .zattrs # Implements "plate" specification + ├── A # First row of the plate │ ├── .zgroup │ ├── .zattrs │ │ - │ ├── A # First row of acquisition 2020-10-10 + │ ├── 1 # First column of row A │ │ ├── .zgroup - │ │ ├── .zattrs + │ │ ├── .zattrs # Implements "well" specification │ │ │ - │ │ ├── 1 # First column of row A - │ │ │ ├── .zgroup - │ │ │ ├── .zattrs # Implements "well" specification + │ │ ├── 0 # First field of view of well A1 │ │ │ │ - │ │ │ ├── 0 # First field of view of well A1 - │ │ │ │ │ - │ │ │ │ ├── .zgroup - │ │ │ │ ├── .zattrs # Implements "multiscales", "omero" - │ │ │ │ ├── 0 - │ │ │ │ │ ... # Resolution levels - │ │ │ │ ├── n - │ │ │ │ └── labels - │ │ │ ├── ... # Fields of view - │ │ │ └── m - │ │ ├── ... # Columns - │ │ └── 12 - │ ├── ... # Rows - │ └── H - ├── ... # Acquisitions - └── l + │ │ │ ├── .zgroup + │ │ │ ├── .zattrs # Implements "multiscales", "omero" + │ │ │ ├── 0 + │ │ │ │ ... # Resolution levels + │ │ │ ├── n + │ │ │ └── labels # Labels (optional) + │ │ ├── ... # Fields of view + │ │ └── m + │ ├── ... # Columns + │ └── 12 + ├── ... # Rows + └── H ``` Metadata {#metadata} From 9aa266f53bdb9b1aa0e266294e72d7bc8402f61d Mon Sep 17 00:00:00 2001 From: Sebastien Besson Date: Wed, 25 Nov 2020 11:44:03 +0000 Subject: [PATCH 08/18] Update plate and well metadata to comply with latest HCS proposal --- latest/index.bs | 69 +++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 58 insertions(+), 11 deletions(-) diff --git a/latest/index.bs b/latest/index.bs index 239300eb..9c7694fb 100644 --- a/latest/index.bs +++ b/latest/index.bs @@ -335,8 +335,14 @@ custom attributes of the plate group under the `plate` key.
A string defining the name of the plate.
acquisitions
A list of JSON objects defining the acquisitions for a given plate. - Each acquisition object MUST contain a `path` key identifying the path - to the acquisition subgroup.
+ Each acquisition object MUST contain an `id` key providing an unique + identifier within the context of the plate to which fields of view can + refer to. It SHOULD contain a `name` key identifying the name of the + acquisition. It SHOULD contain a `maximumfieldcount` key indicating the + maximum number of fields of view for the acquisition. It MAY contain a + `description` key providing a description for the acquisition. It MAY + contain a `startime` and/or `endtime` key specifying the start and/or + end timestamp of the acquisition using an epoch string.
rows
A list of JSON defining the rows of the plate. Each row object defines the properties of the row at the index of the object in the list. If @@ -354,16 +360,46 @@ custom attributes of the plate group under the `plate` key. MUST contain a `path` key identifying the path to the well subgroup.
-For example the following JSON object defines a plate with one acquisition and -6 wells (2 rows and 3 columns), containing up to three fields of view each. +For example the following JSON object defines a plate with two acquisition and +6 wells (2 rows and 3 columns), containing up 2 fields of view per acquistion. ```json "plate": { - "acquisitions": [{"path": "0"}], - "columns": [{"name": "1" }, {"name": "2" }, {"name": "3" }], - "field_count": 3, + "acquisitions": [ + { + "id": 1, + "maximumfieldcount": 2, + "name": "Meas_01(2012-07-31_10-41-12)", + "starttime": 1343731272000 + }, + { + "id": 2, + "maximumfieldcount": 2, + "name": "Meas_02(201207-31_11-56-41)", + "starttime": 1343735801000 + } + ], + "columns": [ + { + "name": "1" + }, + { + "name": "2" + }, + { + "name": "3" + } + ], + "field_count": 4, "name": "test", - "rows": [{"name": "A" }, {"name": "B" }], + "rows": [ + { + "name": "A" + }, + { + "name": "B" + } + ], "wells": [ { "path": "2020-10-10/A/1" @@ -398,24 +434,35 @@ well group.
images
A list of JSON objects defining the fields of views for a given well. Each object MUST contain a `path` key identifying the path to the - field of view.
+ field of view. If multiple acquisitions were performed in the plate, it + SHOULD contain an `acquisition` key identifying the id of the + acquisition which must match one of acquisition JSON objects defined in + the plate metadata. -For example the following JSON object defines a well with three fields of -views: +For example the following JSON object defines a well with four fields of +views. The first two fields of view were part of the first acquisition while +the last two fields of view were part of the second acquisition. ```json "well": { "images": [ { + "acquisition": 1, "path": "0" }, { + "acquisition": 1, "path": "1" }, { + "acquisition": 2, "path": "2" }, + { + "acquisition": 2, + "path": "3" + } ] } ``` From 734e563a8c30506d5ca5a8bd3af9dccede4bcf89 Mon Sep 17 00:00:00 2001 From: Sebastien Besson Date: Wed, 25 Nov 2020 11:44:59 +0000 Subject: [PATCH 09/18] Test editor addition --- latest/index.bs | 1 + 1 file changed, 1 insertion(+) diff --git a/latest/index.bs b/latest/index.bs index 9c7694fb..902ad5b0 100644 --- a/latest/index.bs +++ b/latest/index.bs @@ -14,6 +14,7 @@ Local Boilerplate: copyright yes Boilerplate: style-darkmode off Markup Shorthands: markdown yes Editor: Josh Moore, Open Microscopy Environment (OME) https://www.openmicroscopy.org +Editor: Sébastien Besson, Open Microscopy Environment (OME) https://www.openmicroscopy.org Abstract: This document contains next-generation file format (NGFF) Abstract: specifications for storing bioimaging data in the cloud. Abstract: All specifications are submitted to the https://image.sc community for review. From 2aef9cea4d9915b19cc860337cd26dee6b7abb52 Mon Sep 17 00:00:00 2001 From: Sebastien Besson Date: Thu, 26 Nov 2020 10:56:29 +0000 Subject: [PATCH 10/18] Use today's date --- latest/index.bs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/latest/index.bs b/latest/index.bs index 48d893ec..a715bf6d 100644 --- a/latest/index.bs +++ b/latest/index.bs @@ -526,7 +526,7 @@ Version History {#history} 0.1.3-dev5 - TBD + 2020-11-26 Add the HCS specification From 5d3f53ccbaed6464b1582d127b86576fd5131abb Mon Sep 17 00:00:00 2001 From: Sebastien Besson Date: Thu, 26 Nov 2020 12:36:28 +0000 Subject: [PATCH 11/18] Remove unused .zattrs for now --- latest/index.bs | 1 - 1 file changed, 1 deletion(-) diff --git a/latest/index.bs b/latest/index.bs index a715bf6d..9987a1c5 100644 --- a/latest/index.bs +++ b/latest/index.bs @@ -174,7 +174,6 @@ dataset. Three groups must be defined above the images: ├── .zattrs # Implements "plate" specification ├── A # First row of the plate │ ├── .zgroup - │ ├── .zattrs │ │ │ ├── 1 # First column of row A │ │ ├── .zgroup From b34966bab4254ff3e4e438ace2fd0eed203ddd19 Mon Sep 17 00:00:00 2001 From: Sebastien Besson Date: Thu, 26 Nov 2020 12:36:49 +0000 Subject: [PATCH 12/18] Fix plate definitions --- latest/index.bs | 34 ++++++++++++++++++---------------- 1 file changed, 18 insertions(+), 16 deletions(-) diff --git a/latest/index.bs b/latest/index.bs index 9987a1c5..7646fa14 100644 --- a/latest/index.bs +++ b/latest/index.bs @@ -334,24 +334,26 @@ custom attributes of the plate group under the `plate` key.
name
A string defining the name of the plate.
acquisitions
-
A list of JSON objects defining the acquisitions for a given plate. - Each acquisition object MUST contain an `id` key providing an unique - identifier within the context of the plate to which fields of view can - refer to. It SHOULD contain a `name` key identifying the name of the - acquisition. It SHOULD contain a `maximumfieldcount` key indicating the - maximum number of fields of view for the acquisition. It MAY contain a - `description` key providing a description for the acquisition. It MAY - contain a `startime` and/or `endtime` key specifying the start and/or - end timestamp of the acquisition using an epoch string.
+
An optional list of JSON objects defining the acquisitions for a given + plate. Each acquisition object MUST contain an `id` key providing an + unique identifier within the context of the plate to which fields of + view can refer to. It SHOULD contain a `name` key identifying the name + of the acquisition. It SHOULD contain a `maximumfieldcount` key + indicating the maximum number of fields of view for the acquisition. It + MAY contain a `description` key providing a description for the + acquisition. It MAY contain a `startime` and/or `endtime` key specifying + the start and/or end timestamp of the acquisition using an epoch + string.
rows
-
A list of JSON defining the rows of the plate. Each row object defines - the properties of the row at the index of the object in the list. If - not empty, it MUST contain a `name` key specifying the row name.
-
columns
-
A list of JSON defining the columns of the plate. Each column object - defines the properties of the column at the index of the object in the - list. If not empty, it MUST contain a `name` key specifying the column +
A list of JSON objects defining the rows of the plate. Each row object + defines the properties of the row at the index of the object in the + list. If not empty, it MUST contain a `name` key specifying the row name.
+
columns
+
A list of JSON objects defining the columns of the plate. Each column + object defines the properties of the column at the index of the object + in the list. If not empty, it MUST contain a `name` key specifying the + column name.
field_count
An integer defining the maximum number of fields per view across all wells.
From 2d4ade0180720111f456172434b02ce4766080ef Mon Sep 17 00:00:00 2001 From: Sebastien Besson Date: Thu, 26 Nov 2020 12:38:01 +0000 Subject: [PATCH 13/18] Reorganize definitions alphabetically --- latest/index.bs | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/latest/index.bs b/latest/index.bs index 7646fa14..c66c4063 100644 --- a/latest/index.bs +++ b/latest/index.bs @@ -331,8 +331,6 @@ For high-content screening datasets, the plate layout can be found under the custom attributes of the plate group under the `plate` key.
-
name
-
A string defining the name of the plate.
acquisitions
An optional list of JSON objects defining the acquisitions for a given plate. Each acquisition object MUST contain an `id` key providing an @@ -344,11 +342,6 @@ custom attributes of the plate group under the `plate` key. acquisition. It MAY contain a `startime` and/or `endtime` key specifying the start and/or end timestamp of the acquisition using an epoch string.
-
rows
-
A list of JSON objects defining the rows of the plate. Each row object - defines the properties of the row at the index of the object in the - list. If not empty, it MUST contain a `name` key specifying the row - name.
columns
A list of JSON objects defining the columns of the plate. Each column object defines the properties of the column at the index of the object @@ -357,6 +350,13 @@ custom attributes of the plate group under the `plate` key.
field_count
An integer defining the maximum number of fields per view across all wells.
+
name
+
A string defining the name of the plate.
+
rows
+
A list of JSON objects defining the rows of the plate. Each row object + defines the properties of the row at the index of the object in the + list. If not empty, it MUST contain a `name` key specifying the row + name.
wells
A list of JSON objects defining the wells of the plate. Each well object MUST contain a `path` key identifying the path to the well subgroup.
From 0255833f39e390d75e03aae662eaff83104306f0 Mon Sep 17 00:00:00 2001 From: Sebastien Besson Date: Thu, 26 Nov 2020 12:38:59 +0000 Subject: [PATCH 14/18] Add string keys --- latest/index.bs | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/latest/index.bs b/latest/index.bs index c66c4063..c4ecdcb8 100644 --- a/latest/index.bs +++ b/latest/index.bs @@ -357,6 +357,8 @@ custom attributes of the plate group under the `plate` key. defines the properties of the row at the index of the object in the list. If not empty, it MUST contain a `name` key specifying the row name. +
version
+
A string defining the version of the specification.
wells
A list of JSON objects defining the wells of the plate. Each well object MUST contain a `path` key identifying the path to the well subgroup.
@@ -440,6 +442,8 @@ well group. SHOULD contain an `acquisition` key identifying the id of the acquisition which must match one of acquisition JSON objects defined in the plate metadata. +
version
+
A string defining the version of the specification.
For example the following JSON object defines a well with four fields of From 7f7e446c6ad3492b45ca3499e859e0b91a3013e9 Mon Sep 17 00:00:00 2001 From: Sebastien Besson Date: Thu, 26 Nov 2020 12:41:38 +0000 Subject: [PATCH 15/18] Add version keys to the JSON examples --- latest/index.bs | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/latest/index.bs b/latest/index.bs index c4ecdcb8..a563b309 100644 --- a/latest/index.bs +++ b/latest/index.bs @@ -404,6 +404,7 @@ For example the following JSON object defines a plate with two acquisition and "name": "B" } ], + "version": "0.1", "wells": [ { "path": "2020-10-10/A/1" @@ -469,7 +470,8 @@ the last two fields of view were part of the second acquisition. "acquisition": 2, "path": "3" } - ] + ], + "version": "0.1" } ``` From fff9ba00b48fb74d352d82ae80b8890f0cb7e53e Mon Sep 17 00:00:00 2001 From: Sebastien Besson Date: Thu, 26 Nov 2020 14:35:40 +0000 Subject: [PATCH 16/18] Collapse version history to have labels and HCS --- latest/index.bs | 18 ++++-------------- 1 file changed, 4 insertions(+), 14 deletions(-) diff --git a/latest/index.bs b/latest/index.bs index a563b309..acac80b6 100644 --- a/latest/index.bs +++ b/latest/index.bs @@ -532,24 +532,14 @@ Version History {#history} - 0.1.3-dev5 + 0.1.4 2020-11-26 - Add the HCS specification + Add HCS specification - 0.1.3-dev4 + 0.1.3 2020-09-14 - Add the image-label object - - - 0.1.3-dev3 - 2020-09-01 - Convert labels to multiscales - - - 0.1.3-dev2 - 2020-08-18 - Rename masks as labels + Add labels specification 0.1.3-dev1 From deef548732397725f50f23cb366a6a09f8519c0b Mon Sep 17 00:00:00 2001 From: Sebastien Besson Date: Thu, 26 Nov 2020 14:37:53 +0000 Subject: [PATCH 17/18] Merge latest spec including HCS into 0.1 --- 0.1/index.bs | 255 ++++++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 244 insertions(+), 11 deletions(-) diff --git a/0.1/index.bs b/0.1/index.bs index 8a64973c..e48181f1 100644 --- a/0.1/index.bs +++ b/0.1/index.bs @@ -13,6 +13,7 @@ Local Boilerplate: copyright yes Boilerplate: style-darkmode off Markup Shorthands: markdown yes Editor: Josh Moore, Open Microscopy Environment (OME) https://www.openmicroscopy.org +Editor: Sébastien Besson, Open Microscopy Environment (OME) https://www.openmicroscopy.org Abstract: This document contains next-generation file format (NGFF) Abstract: specifications for storing bioimaging data in the cloud. Abstract: All specifications are submitted to the https://image.sc community for review. @@ -98,6 +99,12 @@ is represented here as it would appear locally but could equally be stored on a web server to be accessed via HTTP or in object storage like S3 or GCS. +Images {#image-layout} +---------------------- + +The following layout describes the expected Zarr hierarchy for images with +multiple levels of resolutions and optionally associated labels. + ``` . # Root folder, potentially in S3, │ # with a flat list of images by image ID. @@ -143,6 +150,50 @@ like S3 or GCS. └── n ``` +High-content screening {#hcs-layout} +------------------------------------ + +The following specification defines the hierarchy for a high-content screening +dataset. Three groups must be defined above the images: + +- the group above the images defines the well and MUST implement the + [well specification](#well-md). All images contained in a well are fields + of view of the same well +- the group above the well defines a row of wells +- the group above the well row defines an entire plate i.e. a two-dimensional + collection of wells organized in rows and columns. It MUST implement the + [plate specification](#plate-md) + + +``` +. # Root folder, potentially in S3, +│ +└── 5966.zarr # One plate (id=5966) converted to Zarr + ├── .zgroup + ├── .zattrs # Implements "plate" specification + ├── A # First row of the plate + │ ├── .zgroup + │ │ + │ ├── 1 # First column of row A + │ │ ├── .zgroup + │ │ ├── .zattrs # Implements "well" specification + │ │ │ + │ │ ├── 0 # First field of view of well A1 + │ │ │ │ + │ │ │ ├── .zgroup + │ │ │ ├── .zattrs # Implements "multiscales", "omero" + │ │ │ ├── 0 + │ │ │ │ ... # Resolution levels + │ │ │ ├── n + │ │ │ └── labels # Labels (optional) + │ │ ├── ... # Fields of view + │ │ └── m + │ ├── ... # Columns + │ └── 12 + ├── ... # Rows + └── H +``` + Metadata {#metadata} ==================== @@ -272,6 +323,193 @@ above). ] ``` +"plate" metadata {#plate-md} +---------------------------- + +For high-content screening datasets, the plate layout can be found under the +custom attributes of the plate group under the `plate` key. + +
+
acquisitions
+
An optional list of JSON objects defining the acquisitions for a given + plate. Each acquisition object MUST contain an `id` key providing an + unique identifier within the context of the plate to which fields of + view can refer to. It SHOULD contain a `name` key identifying the name + of the acquisition. It SHOULD contain a `maximumfieldcount` key + indicating the maximum number of fields of view for the acquisition. It + MAY contain a `description` key providing a description for the + acquisition. It MAY contain a `startime` and/or `endtime` key specifying + the start and/or end timestamp of the acquisition using an epoch + string.
+
columns
+
A list of JSON objects defining the columns of the plate. Each column + object defines the properties of the column at the index of the object + in the list. If not empty, it MUST contain a `name` key specifying the + column name.
+
field_count
+
An integer defining the maximum number of fields per view across all + wells.
+
name
+
A string defining the name of the plate.
+
rows
+
A list of JSON objects defining the rows of the plate. Each row object + defines the properties of the row at the index of the object in the + list. If not empty, it MUST contain a `name` key specifying the row + name.
+
version
+
A string defining the version of the specification.
+
wells
+
A list of JSON objects defining the wells of the plate. Each well object + MUST contain a `path` key identifying the path to the well subgroup.
+
+ +For example the following JSON object defines a plate with two acquisition and +6 wells (2 rows and 3 columns), containing up 2 fields of view per acquistion. + +```json +"plate": { + "acquisitions": [ + { + "id": 1, + "maximumfieldcount": 2, + "name": "Meas_01(2012-07-31_10-41-12)", + "starttime": 1343731272000 + }, + { + "id": 2, + "maximumfieldcount": 2, + "name": "Meas_02(201207-31_11-56-41)", + "starttime": 1343735801000 + } + ], + "columns": [ + { + "name": "1" + }, + { + "name": "2" + }, + { + "name": "3" + } + ], + "field_count": 4, + "name": "test", + "rows": [ + { + "name": "A" + }, + { + "name": "B" + } + ], + "version": "0.1", + "wells": [ + { + "path": "2020-10-10/A/1" + }, + { + "path": "2020-10-10/A/2" + }, + { + "path": "2020-10-10/A/3" + }, + { + "path": "2020-10-10/B/1" + }, + { + "path": "2020-10-10/B/2" + }, + { + "path": "2020-10-10/B/3" + } + ] + } +``` + +"well" metadata {#well-md} +-------------------------- + +For high-content screening datasets, the metadata about all fields of views +under a given well can be found under the "well" key in the attributes of the +well group. + +
+
images
+
A list of JSON objects defining the fields of views for a given well. + Each object MUST contain a `path` key identifying the path to the + field of view. If multiple acquisitions were performed in the plate, it + SHOULD contain an `acquisition` key identifying the id of the + acquisition which must match one of acquisition JSON objects defined in + the plate metadata.
+
version
+
A string defining the version of the specification.
+
+ +For example the following JSON object defines a well with four fields of +views. The first two fields of view were part of the first acquisition while +the last two fields of view were part of the second acquisition. + +```json +"well": { + "images": [ + { + "acquisition": 1, + "path": "0" + }, + { + "acquisition": 1, + "path": "1" + }, + { + "acquisition": 2, + "path": "2" + }, + { + "acquisition": 2, + "path": "3" + } + ], + "version": "0.1" + } +``` + +Implementations {#implementations} +================================== + +Projects which support reading and/or writing OME-NGFF data include: + +
+ +
[omero-ms-zarr](https://github.com/ome/omero-ms-zarr)
+
A microservice for OMERO.server that converts images stored in OMERO to OME Zarr files on the fly, served via a web API.
+ +
[idr-zarr-tools](https://github.com/IDR/idr-zarr-tools)
+
A full workflow demonstrating the conversion of IDR images to OME Zarr images on S3.
+ +
[OMERO CLI Zarr plugin](https://github.com/ome/omero-cli-zarr)
+
An OMERO CLI plugin that converts images stored in OMERO.server into a local Zarr file.
+ +
[ome-zarr-py](https://github.com/ome/ome-zarr-py)
+
A napari plugin for reading ome-zarr files.
+ +
[bioformats2raw](https://github.com/glencoesoftware/bioformats2raw)
+
A performant, Bio-Formats image file format converter.
+ +
[vizarr](https://github.com/hms-dbmi/vizarr/)
+
A minimal, purely client-side program for viewing Zarr-based images with Viv & ImJoy.
+ +
+ +Diagram of related projects + +All implementations prevent an equivalent representation of a dataset which can be downloaded or uploaded freely. An interactive +version of this diagram is available from the [OME2020 Workshop](https://downloads.openmicroscopy.org/presentations/2020/Dundee/Workshops/NGFF/zarr_diagram/). +Mouseover the blackboxes representing the implementations above to get a quick tip on how to use them. + +Note: If you would like to see your project listed, please open an issue or PR on the [ome/ngff](https://github.com/ome/ngff) repository. + Citing {#citing} ================ @@ -293,19 +531,14 @@ Version History {#history} - 0.1.3-dev4 - 2020-09-14 - Add the image-label object - - - 0.1.3-dev3 - 2020-09-01 - Convert labels to multiscales + 0.1.4 + 2020-11-26 + Add HCS specification - 0.1.3-dev2 - 2020-08-18 - Rename masks as labels + 0.1.3 + 2020-09-14 + Add labels specification 0.1.3-dev1 From 0669bab4788968464494354c9ae757451a38b763 Mon Sep 17 00:00:00 2001 From: Sebastien Besson Date: Thu, 26 Nov 2020 15:37:20 +0000 Subject: [PATCH 18/18] Drop 0.1.3-dev1 in favor of 0.1.3 --- 0.1/index.bs | 5 ----- latest/index.bs | 5 ----- 2 files changed, 10 deletions(-) diff --git a/0.1/index.bs b/0.1/index.bs index e48181f1..b60204f1 100644 --- a/0.1/index.bs +++ b/0.1/index.bs @@ -540,11 +540,6 @@ Version History {#history} 2020-09-14 Add labels specification - - 0.1.3-dev1 - 2020-07-07 - Add mask metadata - 0.1.2 2020-05-07 diff --git a/latest/index.bs b/latest/index.bs index 05ec2771..31752fcb 100644 --- a/latest/index.bs +++ b/latest/index.bs @@ -542,11 +542,6 @@ Version History {#history} 2020-09-14 Add labels specification - - 0.1.3-dev1 - 2020-07-07 - Add mask metadata - 0.1.2 2020-05-07