From 3b9cbc561067cb693fcb57e3c0f49ec8b867ae7f Mon Sep 17 00:00:00 2001 From: Johan Bergsma <29785380+JPBergsma@users.noreply.github.com> Date: Mon, 26 Jun 2023 10:22:18 +0200 Subject: [PATCH 01/34] Intermediate progress writing property ranges parameter. --- optimade.rst | 30 +++++++++++++++++++++++++++++- 1 file changed, 29 insertions(+), 1 deletion(-) diff --git a/optimade.rst b/optimade.rst index b91b89d43..7870af3a2 100644 --- a/optimade.rst +++ b/optimade.rst @@ -459,6 +459,9 @@ The default partial data format is named "jsonlines" and is described in the App An implementation SHOULD always include this format as one of the partial data formats provided for a property that has been omitted from the response to the initial query. Implementations MAY provide links to their own non-standard formats, but non-standard format names MUST be prefixed by a database-provider-specific prefix. +To enable requesting only a part of the data for such properties via the :param:`property\_ranges` query parameter, there is additional meta data stored under the field :field:`meta..range`. +The metadata field of the ranged property, :field:`meta..range`, MUST include these fields: + Below follows an example of the :field:`data` and :field:`meta` parts of a response using the JSON response format that communicates that the property value has been omitted from the response, with three different links for different partial data formats provided. .. code:: jsonc @@ -493,6 +496,7 @@ Below follows an example of the :field:`data` and :field:`meta` parts of a respo // ... } + Metadata properties ------------------- @@ -1077,6 +1081,30 @@ Standard OPTIONAL URL query parameters not in the JSON:API specification: If provided, these fields MUST be returned along with the REQUIRED fields. Other OPTIONAL fields MUST NOT be returned when this parameter is present. Example: :query-url:`http://example.com/optimade/v1/structures?response_fields=last_modified,nsites` +- **property\_ranges**: A set of slices for partial data properties. + This parameter is used to select a subrange of partial data properties so only a part of the data needs to be returned. + In general support is OPTIONAL, property definitions may however deviate from this and place stricter requirements on servers. + It consists of the name of a dimension directly followed by the requested slice in this dimension. + A slice consists of a pair of brackets ("(", ASCII 40(0x28)) and (")", ASCII 41(0x29)) enclosing three integers, which are separated by commas (",", ASCII 91(0x5B)) + It is defined in the same manner as the `slice object`_. + The first integer specifies the first index in that dimension for which values should be returned. + The second integer specifies the last index for which values should be returned. + The third integer specifies the step size in that dimension. + Slices can be specified for multiple dimensions by separating them with a comma. + The slices are 0 based, i.e. the first value has index 0, and inclusive i.e. for the range :val:`(10,20,1)` the last value returned belongs to index 20. + Databases MUST use these ranges for properties where the dimension is listed under indexable_dimensions, if this is not the case the database MAY return more data than was specified in the range. + + If a dimension is not specified, it is assumed that the whole range in that dimension is requested. + If a value is not present at a set of the indexes, no value SHOULD be returned. + However, when a value is explicitly set to :val:`null` and :val:`null` has a meaning beyond indicating that no value has been defined :val:`null` MUST be returned. + Incase the requested property_range covers more data than the server wants to return the server may choose to return only a part of the data. + For each combination of indexes for which data is returned all the values for all requested properties however need to be returned. + If the server does not return all the requested data, a link MUST be provided in the :field:`next` field, that applies to an entry as a whole, from which the remainder of the data can be retrieved. + + + + + Additional OPTIONAL URL query parameters not described above are not considered to be part of this standard, and are instead considered to be "custom URL query parameters". These custom URL query parameters MUST be of the format "". @@ -3621,7 +3649,7 @@ The strings below contain Extended Regular Expressions (EREs) to recognize ident #BEGIN ERE identifiers [a-z_][a-z_0-9]* #END ERE identifiers - +== #BEGIN ERE numbers [-+]?([0-9]+(\.[0-9]*)?|\.[0-9]+)([eE][-+]?[0-9]+)? #END ERE numbers From eae167938a1c10bf3722ea754960123192340b0d Mon Sep 17 00:00:00 2001 From: Johan Bergsma <29785380+JPBergsma@users.noreply.github.com> Date: Tue, 27 Jun 2023 15:12:12 +0200 Subject: [PATCH 02/34] Added metadata fields to be able to use property ranges query parameter. --- optimade.rst | 50 +++++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 47 insertions(+), 3 deletions(-) diff --git a/optimade.rst b/optimade.rst index 7870af3a2..30f8fb062 100644 --- a/optimade.rst +++ b/optimade.rst @@ -459,10 +459,34 @@ The default partial data format is named "jsonlines" and is described in the App An implementation SHOULD always include this format as one of the partial data formats provided for a property that has been omitted from the response to the initial query. Implementations MAY provide links to their own non-standard formats, but non-standard format names MUST be prefixed by a database-provider-specific prefix. -To enable requesting only a part of the data for such properties via the :param:`property\_ranges` query parameter, there is additional meta data stored under the field :field:`meta..range`. -The metadata field of the ranged property, :field:`meta..range`, MUST include these fields: +If the server supports the :query-param:`property_ranges` query parameter, as described in section `Entry Listing URL Query Parameters`_ additional metadata needs to be present under :field:`property_metadata` field, for each property that is returned in the partial data format. +This meta data allows clients to request specific subsections of properties. -Below follows an example of the :field:`data` and :field:`meta` parts of a response using the JSON response format that communicates that the property value has been omitted from the response, with three different links for different partial data formats provided. +If the field can be sliced with the :query-param:`property_ranges`, the following fields MUST be defined under :field:`property_metadata` Otherwise implementation is OPTIONAL: + +- :field:`range_ids`: List of Strings. + A list with an identifier for each dimension of the property. + The outermost dimension of a nested array should come first. + If, within one entry, dimensions for two or more properties share the same :field:`range_id` those dimensions should be thought of as the same dimension. + For example, both the :property:`energy` and :property:`cartesian_site_positions` of a molecular dynamics trajectory share a range_id of :val:`frames`. + This means that the energy at index x(in the dimension labelled by this range_id) belongs to the cartesian_site_positions at the same index x. + +- :field:`indexable_dim`: List of Strings. + The list of range_ids of the dimensions for which slicing is supported, i.e. the client can request a subrange via the :query-param:`property_ranges` query parameter. + +- :field:`data_range`: List of Slice Objects. + This field describes how the values are distributed in the different dimensions. + It consists of a slice object for each dimension of the property. + The order of the slice objects must be the same as in the :field:`range_ids` field. + +- :field:`nvalues`: Integer. + The total number of values in the property. + SHOULD be a queryable property with support for all mandatory filter features. + +- :field:`"layout"`: String. + A string either equal to :val:`"dense"` or :val:`"sparse"` to indicate whether the property is returned in a dense or sparse layout. + +Below follows an example of the :field:`data` and :field:`meta` parts of a response using the JSON response format. It communicates that the property value has been omitted from the response and includes three different links for different partial data formats provided and metadata that makes it possible to only request only a part of the data for this property. .. code:: jsonc @@ -475,6 +499,25 @@ Below follows an example of the :field:`data` and :field:`meta` parts of a respo "a": null } "meta": { + "property_metadata":{ + "a":{ + "range":{ + "range_ids":["frames","particles"], + "indexable_dim": ["frames"], + "data_range": [{ + "start": 1, + "step": 1, + "stop": 200, + },{ + "start": 1, + "step": 1, + "stop": 3, + }], + "format":"1.2", + "layout":"dense", + } + } + }, "partial_data_links": { "a": [ { @@ -3680,6 +3723,7 @@ The dictionary has the following OPTIONAL fields: The default is 1, i.e., every value in the range indicated by :field:`start` and :field:`stop` is included in the slice. Hence, a value of 2 denotes a slice of every second value in the array. + For example, for the array :val:`["a", "b", "c", "d", "e", "f", "g", "h", "i", "j"]` the slice object :val:`{"start": 1, "end": 7, "step": 3}` refers to the items :val:`["b", "e", "h"]`. Furthermore, we also define the following special markers: From f00b52daf4693aba3348d3db8db6c98183cc0597 Mon Sep 17 00:00:00 2001 From: Johan Bergsma <29785380+JPBergsma@users.noreply.github.com> Date: Wed, 28 Jun 2023 15:36:55 +0200 Subject: [PATCH 03/34] Small corrections. --- optimade.rst | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-) diff --git a/optimade.rst b/optimade.rst index 30f8fb062..810aa18cf 100644 --- a/optimade.rst +++ b/optimade.rst @@ -462,29 +462,34 @@ Implementations MAY provide links to their own non-standard formats, but non-sta If the server supports the :query-param:`property_ranges` query parameter, as described in section `Entry Listing URL Query Parameters`_ additional metadata needs to be present under :field:`property_metadata` field, for each property that is returned in the partial data format. This meta data allows clients to request specific subsections of properties. -If the field can be sliced with the :query-param:`property_ranges`, the following fields MUST be defined under :field:`property_metadata` Otherwise implementation is OPTIONAL: +If the field can be sliced with the :query-param:`property_ranges`, the following fields MUST be defined under :field:`property_metadata`, otherwise implementation is OPTIONAL. +Except for the `nvalues` field, the queryability of all fields is OPTIONAL. - :field:`range_ids`: List of Strings. A list with an identifier for each dimension of the property. The outermost dimension of a nested array should come first. + If, within one entry, dimensions for two or more properties share the same :field:`range_id` those dimensions should be thought of as the same dimension. - For example, both the :property:`energy` and :property:`cartesian_site_positions` of a molecular dynamics trajectory share a range_id of :val:`frames`. + For example, if both the :property:`energy` and :property:`cartesian_site_positions` of a molecular dynamics trajectory share a range_id of :val:`frames`. This means that the energy at index x(in the dimension labelled by this range_id) belongs to the cartesian_site_positions at the same index x. - :field:`indexable_dim`: List of Strings. - The list of range_ids of the dimensions for which slicing is supported, i.e. the client can request a subrange via the :query-param:`property_ranges` query parameter. + The list of range_ids of the dimensions for which slicing is supported, i.e. the client can request a subrange in this dimension via the :query-param:`property_ranges` query parameter. + +- :field:`"layout"`: String. + A string either equal to :val:`"dense"` or :val:`"sparse"` to indicate whether the property is returned in a dense or sparse layout. - :field:`data_range`: List of Slice Objects. This field describes how the values are distributed in the different dimensions. It consists of a slice object for each dimension of the property. The order of the slice objects must be the same as in the :field:`range_ids` field. + If the :field:`layout` field is set to :val:`"sparse"` the value of the :field:`step` sub-field has no meaning. - :field:`nvalues`: Integer. The total number of values in the property. SHOULD be a queryable property with support for all mandatory filter features. -- :field:`"layout"`: String. - A string either equal to :val:`"dense"` or :val:`"sparse"` to indicate whether the property is returned in a dense or sparse layout. + Below follows an example of the :field:`data` and :field:`meta` parts of a response using the JSON response format. It communicates that the property value has been omitted from the response and includes three different links for different partial data formats provided and metadata that makes it possible to only request only a part of the data for this property. @@ -513,7 +518,6 @@ Below follows an example of the :field:`data` and :field:`meta` parts of a respo "step": 1, "stop": 3, }], - "format":"1.2", "layout":"dense", } } From 2a0bef8770fe17863ae134defe6f0d950a8160c2 Mon Sep 17 00:00:00 2001 From: Johan Bergsma <29785380+JPBergsma@users.noreply.github.com> Date: Thu, 29 Jun 2023 14:07:16 +0200 Subject: [PATCH 04/34] moved property ranges to single_entryendpoint query param + further refinement text --- optimade.rst | 51 +++++++++++++++++++++++++++------------------------ 1 file changed, 27 insertions(+), 24 deletions(-) diff --git a/optimade.rst b/optimade.rst index 810aa18cf..4dc9d4698 100644 --- a/optimade.rst +++ b/optimade.rst @@ -1128,30 +1128,6 @@ Standard OPTIONAL URL query parameters not in the JSON:API specification: If provided, these fields MUST be returned along with the REQUIRED fields. Other OPTIONAL fields MUST NOT be returned when this parameter is present. Example: :query-url:`http://example.com/optimade/v1/structures?response_fields=last_modified,nsites` -- **property\_ranges**: A set of slices for partial data properties. - This parameter is used to select a subrange of partial data properties so only a part of the data needs to be returned. - In general support is OPTIONAL, property definitions may however deviate from this and place stricter requirements on servers. - It consists of the name of a dimension directly followed by the requested slice in this dimension. - A slice consists of a pair of brackets ("(", ASCII 40(0x28)) and (")", ASCII 41(0x29)) enclosing three integers, which are separated by commas (",", ASCII 91(0x5B)) - It is defined in the same manner as the `slice object`_. - The first integer specifies the first index in that dimension for which values should be returned. - The second integer specifies the last index for which values should be returned. - The third integer specifies the step size in that dimension. - Slices can be specified for multiple dimensions by separating them with a comma. - The slices are 0 based, i.e. the first value has index 0, and inclusive i.e. for the range :val:`(10,20,1)` the last value returned belongs to index 20. - Databases MUST use these ranges for properties where the dimension is listed under indexable_dimensions, if this is not the case the database MAY return more data than was specified in the range. - - If a dimension is not specified, it is assumed that the whole range in that dimension is requested. - If a value is not present at a set of the indexes, no value SHOULD be returned. - However, when a value is explicitly set to :val:`null` and :val:`null` has a meaning beyond indicating that no value has been defined :val:`null` MUST be returned. - Incase the requested property_range covers more data than the server wants to return the server may choose to return only a part of the data. - For each combination of indexes for which data is returned all the values for all requested properties however need to be returned. - If the server does not return all the requested data, a link MUST be provided in the :field:`next` field, that applies to an entry as a whole, from which the remainder of the data can be retrieved. - - - - - Additional OPTIONAL URL query parameters not described above are not considered to be part of this standard, and are instead considered to be "custom URL query parameters". These custom URL query parameters MUST be of the format "". @@ -1242,6 +1218,33 @@ The rules for which properties are to be present for an entry in the response ar Single Entry URL Query Parameters ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +- **property\_ranges**: A set of slices which specify which range of a property should be returned for partial data properties. + The property to which this range should be applied can be specified via the :query-param:`response_fields` query parameter. + Each slice consists of the name of a dimension directly followed by the requested slice in this dimension. + The dimension name and the start, stop and step values of the slice are separated by colons (":", ASCII 58(0x3A)) + The slice is defined in the same manner as the `slice object`_. + If no value is placed between the colons for a component of the slice the default value is used. + The first integer specifies the start of the slice, i.e. the first index in that dimension for which values should be returned. The default is the index of the first value. + The second integer specifies the end/stop of the slice, i.e. the last index for which values should be returned. The default is the index of the last value of the property. + And the third integer specifies the step size in that dimension. The default is :val:`1` + Multiple dimensional slices can be defined by specifying a range for each dimension. These ranges are separated by a comma (",", ASCII 44(0x2C)). + The slices are 0 based, i.e. the first value has index 0, and inclusive i.e. for the range :val:`:10:20:1` the last value returned belongs to index 20. + + In general support is OPTIONAL, property definitions may however deviate from this and place stricter requirements on servers. + Databases must use the methods described in the section `Transmission of large property values`_ to return the requested sub-range of a property. +Databases MUST use these ranges for properties where the dimension is listed under indexable_dimensions, if this is not the case the database MAY return more data than was specified in the range. + + If a dimension is not specified, it is assumed that the whole range in that dimension is requested. + If one or more values are not present at one of the requested combination of indexes, the server MAY either decide to return null or if possible adjust the returned range so the indexes for which no value is defined are no longer part of the range. + The latter is only allowed when no defined values would be lost. + For dense arrays that may mean that the field :field:`returned_range` differs from the requested range. + When a value is, however, explicitly set to :val:`null` and :val:`null` has a meaning beyond indicating that no value has been defined :val:`null` MUST be returned + + Example: In this example the Cartesian site positions are requested for particles 30 through 70 for 1 out of every 10 frames of the first 1000 frames of a trajectory. + +:query-url:`http://example.com/optimade/v1/structures/id_12345?response_fields=cartesian_site_positions&property_ranges=frames::1000:10,particles:30:70::` + + The client MAY provide a set of additional URL query parameters for this endpoint type. URL query parameters not recognized MUST be ignored. While the following URL query parameters are OPTIONAL for clients, API implementations MUST accept and handle them: From 4a00af4d4f1005576bb334142768e3ee6ba2ac4b Mon Sep 17 00:00:00 2001 From: Johan Bergsma <29785380+JPBergsma@users.noreply.github.com> Date: Thu, 29 Jun 2023 18:21:24 +0200 Subject: [PATCH 05/34] Some small changes after proof reading. --- optimade.rst | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/optimade.rst b/optimade.rst index 4dc9d4698..b93221ac4 100644 --- a/optimade.rst +++ b/optimade.rst @@ -459,8 +459,8 @@ The default partial data format is named "jsonlines" and is described in the App An implementation SHOULD always include this format as one of the partial data formats provided for a property that has been omitted from the response to the initial query. Implementations MAY provide links to their own non-standard formats, but non-standard format names MUST be prefixed by a database-provider-specific prefix. -If the server supports the :query-param:`property_ranges` query parameter, as described in section `Entry Listing URL Query Parameters`_ additional metadata needs to be present under :field:`property_metadata` field, for each property that is returned in the partial data format. -This meta data allows clients to request specific subsections of properties. +If the server supports the :query-param:`property_ranges` query parameter, as described in section `Single Entry URL Query Parameters`_, additional metadata SHOULD be present in the property specific field under the :field:`property_metadata` field, for each property that is returned in the partial data format. +This metadata allows clients to request specific subsections of properties. If the field can be sliced with the :query-param:`property_ranges`, the following fields MUST be defined under :field:`property_metadata`, otherwise implementation is OPTIONAL. Except for the `nvalues` field, the queryability of all fields is OPTIONAL. @@ -468,7 +468,7 @@ Except for the `nvalues` field, the queryability of all fields is OPTIONAL. - :field:`range_ids`: List of Strings. A list with an identifier for each dimension of the property. The outermost dimension of a nested array should come first. - + If, within one entry, dimensions for two or more properties share the same :field:`range_id` those dimensions should be thought of as the same dimension. For example, if both the :property:`energy` and :property:`cartesian_site_positions` of a molecular dynamics trajectory share a range_id of :val:`frames`. This means that the energy at index x(in the dimension labelled by this range_id) belongs to the cartesian_site_positions at the same index x. @@ -519,6 +519,7 @@ Below follows an example of the :field:`data` and :field:`meta` parts of a respo "stop": 3, }], "layout":"dense", + "nvalues": 600 } } }, @@ -1231,8 +1232,8 @@ Single Entry URL Query Parameters The slices are 0 based, i.e. the first value has index 0, and inclusive i.e. for the range :val:`:10:20:1` the last value returned belongs to index 20. In general support is OPTIONAL, property definitions may however deviate from this and place stricter requirements on servers. - Databases must use the methods described in the section `Transmission of large property values`_ to return the requested sub-range of a property. -Databases MUST use these ranges for properties where the dimension is listed under indexable_dimensions, if this is not the case the database MAY return more data than was specified in the range. + Databases MUST use the methods described in the section `Transmission of large property values`_ to return the requested sub-range of a property. + Databases MUST use these ranges for properties where the dimension is listed under indexable_dimensions, if this is not the case the database MAY return more data than was specified in the range. If a dimension is not specified, it is assumed that the whole range in that dimension is requested. If one or more values are not present at one of the requested combination of indexes, the server MAY either decide to return null or if possible adjust the returned range so the indexes for which no value is defined are no longer part of the range. @@ -1242,7 +1243,7 @@ Databases MUST use these ranges for properties where the dimension is listed und Example: In this example the Cartesian site positions are requested for particles 30 through 70 for 1 out of every 10 frames of the first 1000 frames of a trajectory. -:query-url:`http://example.com/optimade/v1/structures/id_12345?response_fields=cartesian_site_positions&property_ranges=frames::1000:10,particles:30:70::` + :query-url:`http://example.com/optimade/v1/structures/id_12345?response_fields=cartesian_site_positions&property_ranges=frames::1000:10,particles:30:70::` The client MAY provide a set of additional URL query parameters for this endpoint type. From 6905d7f208cccd1bc0bc7214ab590772e1a42bd3 Mon Sep 17 00:00:00 2001 From: Johan Bergsma <29785380+JPBergsma@users.noreply.github.com> Date: Thu, 29 Jun 2023 19:55:33 +0200 Subject: [PATCH 06/34] added that fields should be placed in the range dictionary. --- optimade.rst | 63 ++++++++++++++++++++++++++-------------------------- 1 file changed, 32 insertions(+), 31 deletions(-) diff --git a/optimade.rst b/optimade.rst index b93221ac4..6c1d0f18b 100644 --- a/optimade.rst +++ b/optimade.rst @@ -459,37 +459,38 @@ The default partial data format is named "jsonlines" and is described in the App An implementation SHOULD always include this format as one of the partial data formats provided for a property that has been omitted from the response to the initial query. Implementations MAY provide links to their own non-standard formats, but non-standard format names MUST be prefixed by a database-provider-specific prefix. -If the server supports the :query-param:`property_ranges` query parameter, as described in section `Single Entry URL Query Parameters`_, additional metadata SHOULD be present in the property specific field under the :field:`property_metadata` field, for each property that is returned in the partial data format. -This metadata allows clients to request specific subsections of properties. - -If the field can be sliced with the :query-param:`property_ranges`, the following fields MUST be defined under :field:`property_metadata`, otherwise implementation is OPTIONAL. -Except for the `nvalues` field, the queryability of all fields is OPTIONAL. - -- :field:`range_ids`: List of Strings. - A list with an identifier for each dimension of the property. - The outermost dimension of a nested array should come first. - - If, within one entry, dimensions for two or more properties share the same :field:`range_id` those dimensions should be thought of as the same dimension. - For example, if both the :property:`energy` and :property:`cartesian_site_positions` of a molecular dynamics trajectory share a range_id of :val:`frames`. - This means that the energy at index x(in the dimension labelled by this range_id) belongs to the cartesian_site_positions at the same index x. - -- :field:`indexable_dim`: List of Strings. - The list of range_ids of the dimensions for which slicing is supported, i.e. the client can request a subrange in this dimension via the :query-param:`property_ranges` query parameter. - -- :field:`"layout"`: String. - A string either equal to :val:`"dense"` or :val:`"sparse"` to indicate whether the property is returned in a dense or sparse layout. - -- :field:`data_range`: List of Slice Objects. - This field describes how the values are distributed in the different dimensions. - It consists of a slice object for each dimension of the property. - The order of the slice objects must be the same as in the :field:`range_ids` field. - If the :field:`layout` field is set to :val:`"sparse"` the value of the :field:`step` sub-field has no meaning. - -- :field:`nvalues`: Integer. - The total number of values in the property. - SHOULD be a queryable property with support for all mandatory filter features. - - +If the server supports the :query-param:`property_ranges` query parameter, as described in section `Single Entry URL Query Parameters`_, additional metadata SHOULD be present for each field that is returned via the partial data mechanism. +The client needs this metadata to be able to select only a part of the data of a property. +This metadata is stored in a dictionary in the field :field:`range` which is stored as per property metadata as described in section `Metadata properties`_ + +- :field:`range`: Dictionary + A dictionary that contains the data necessary for the client to select only a sub_range of a property. + If the property_ranges query parameter is supported for this property the following keys MUST be present. + Otherwise these keys are optional. + + - :field:`range_ids`: List of Strings. + A list with an identifier for each dimension of the property. + The outermost dimension of a nested array should come first. + + If, within one entry, dimensions for two or more properties share the same :field:`range_id` those dimensions should be thought of as the same dimension. + For example, if both the :property:`energy` and :property:`cartesian_site_positions` of a molecular dynamics trajectory share a range_id of :val:`frames`. + This means that the energy at index x(in the dimension labelled by this range_id) belongs to the cartesian_site_positions at the same index x. + + - :field:`indexable_dim`: List of Strings. + The list of range_ids of the dimensions for which slicing is supported, i.e. the client can request a subrange in this dimension via the :query-param:`property_ranges` query parameter. + + - :field:`"layout"`: String. + A string either equal to :val:`"dense"` or :val:`"sparse"` to indicate whether the property is returned in a dense or sparse layout. + + - :field:`data_range`: List of Slice Objects. + This field describes how the values are distributed in the different dimensions. + It consists of a slice object for each dimension of the property. + The order of the slice objects must be the same as in the :field:`range_ids` field. + If the :field:`layout` field is set to :val:`"sparse"` the value of the :field:`step` sub-field has no meaning. + + - :field:`nvalues`: Integer. + The total number of values in the property. + SHOULD be a queryable property with support for all mandatory filter features. Below follows an example of the :field:`data` and :field:`meta` parts of a response using the JSON response format. It communicates that the property value has been omitted from the response and includes three different links for different partial data formats provided and metadata that makes it possible to only request only a part of the data for this property. From d579c6ece94a74402d4c43896451f2cccac97373 Mon Sep 17 00:00:00 2001 From: Johan Bergsma <29785380+JPBergsma@users.noreply.github.com> Date: Fri, 30 Jun 2023 08:56:20 +0200 Subject: [PATCH 07/34] Some more sentences that had to be moved to a new line. --- optimade.rst | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/optimade.rst b/optimade.rst index 6c1d0f18b..963caeb19 100644 --- a/optimade.rst +++ b/optimade.rst @@ -1226,10 +1226,14 @@ Single Entry URL Query Parameters The dimension name and the start, stop and step values of the slice are separated by colons (":", ASCII 58(0x3A)) The slice is defined in the same manner as the `slice object`_. If no value is placed between the colons for a component of the slice the default value is used. - The first integer specifies the start of the slice, i.e. the first index in that dimension for which values should be returned. The default is the index of the first value. - The second integer specifies the end/stop of the slice, i.e. the last index for which values should be returned. The default is the index of the last value of the property. - And the third integer specifies the step size in that dimension. The default is :val:`1` - Multiple dimensional slices can be defined by specifying a range for each dimension. These ranges are separated by a comma (",", ASCII 44(0x2C)). + The first integer specifies the start of the slice, i.e. the first index in that dimension for which values should be returned. + The default is the index of the first value. + The second integer specifies the end/stop of the slice, i.e. the last index for which values should be returned. + The default is the index of the last value of the property. + And the third integer specifies the step size in that dimension. + The default is :val:`1` + Multiple dimensional slices can be defined by specifying a range for each dimension. + These ranges are separated by a comma (",", ASCII 44(0x2C)). The slices are 0 based, i.e. the first value has index 0, and inclusive i.e. for the range :val:`:10:20:1` the last value returned belongs to index 20. In general support is OPTIONAL, property definitions may however deviate from this and place stricter requirements on servers. From 679296214e098d3a5c1ac6de80a3e6684551a381 Mon Sep 17 00:00:00 2001 From: Johan Bergsma <29785380+JPBergsma@users.noreply.github.com> Date: Fri, 30 Jun 2023 08:53:59 +0200 Subject: [PATCH 08/34] Update optimade.rst Make sure each sentence starts on a new line. Co-authored-by: Antanas Vaitkus --- optimade.rst | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/optimade.rst b/optimade.rst index 963caeb19..d334d5b1d 100644 --- a/optimade.rst +++ b/optimade.rst @@ -492,7 +492,8 @@ This metadata is stored in a dictionary in the field :field:`range` which is sto The total number of values in the property. SHOULD be a queryable property with support for all mandatory filter features. -Below follows an example of the :field:`data` and :field:`meta` parts of a response using the JSON response format. It communicates that the property value has been omitted from the response and includes three different links for different partial data formats provided and metadata that makes it possible to only request only a part of the data for this property. +Below follows an example of the :field:`data` and :field:`meta` parts of a response using the JSON response format. +It communicates that the property value has been omitted from the response and includes three different links for different partial data formats provided and metadata that makes it possible to request only a part of the data for this property. .. code:: jsonc From 93ecd0b80b9dc8c71aa713d35756d96f97f465bc Mon Sep 17 00:00:00 2001 From: Johan Bergsma <29785380+JPBergsma@users.noreply.github.com> Date: Fri, 30 Jun 2023 08:58:23 +0200 Subject: [PATCH 09/34] removed unintentional changes in appendix. --- optimade.rst | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/optimade.rst b/optimade.rst index d334d5b1d..5629ed741 100644 --- a/optimade.rst +++ b/optimade.rst @@ -3706,7 +3706,7 @@ The strings below contain Extended Regular Expressions (EREs) to recognize ident #BEGIN ERE identifiers [a-z_][a-z_0-9]* #END ERE identifiers -== + #BEGIN ERE numbers [-+]?([0-9]+(\.[0-9]*)?|\.[0-9]+)([eE][-+]?[0-9]+)? #END ERE numbers @@ -3737,7 +3737,6 @@ The dictionary has the following OPTIONAL fields: The default is 1, i.e., every value in the range indicated by :field:`start` and :field:`stop` is included in the slice. Hence, a value of 2 denotes a slice of every second value in the array. - For example, for the array :val:`["a", "b", "c", "d", "e", "f", "g", "h", "i", "j"]` the slice object :val:`{"start": 1, "end": 7, "step": 3}` refers to the items :val:`["b", "e", "h"]`. Furthermore, we also define the following special markers: From a84c72e423f797cfd355e7ba61e3ac0d82e70c0f Mon Sep 17 00:00:00 2001 From: Johan Bergsma <29785380+JPBergsma@users.noreply.github.com> Date: Tue, 4 Jul 2023 16:24:23 +0200 Subject: [PATCH 10/34] Seperated property_ranges query parameter from the partial data. --- optimade.rst | 99 +++++++++++++++++++++++++++++++++++----------------- 1 file changed, 67 insertions(+), 32 deletions(-) diff --git a/optimade.rst b/optimade.rst index 5629ed741..7b8c7d1d4 100644 --- a/optimade.rst +++ b/optimade.rst @@ -459,33 +459,65 @@ The default partial data format is named "jsonlines" and is described in the App An implementation SHOULD always include this format as one of the partial data formats provided for a property that has been omitted from the response to the initial query. Implementations MAY provide links to their own non-standard formats, but non-standard format names MUST be prefixed by a database-provider-specific prefix. -If the server supports the :query-param:`property_ranges` query parameter, as described in section `Single Entry URL Query Parameters`_, additional metadata SHOULD be present for each field that is returned via the partial data mechanism. +Below follows an example of the :field:`data` and :field:`meta` parts of a response using the JSON response format. +It communicates that the property value has been omitted from the response and includes three different links for different partial data formats provided. + +.. code:: jsonc + + { + // ... + "data": { + "type": "structures", + "id": "2345678", + "attributes": { + "a": null + } + "meta": { + "partial_data_links": { + "a": [ + { + "format": "jsonlines", + "link": "https://example.org/optimade/v1.2/extensions/partial_data/structures/2345678/a/default_format" + }, + { + "format": "_exmpl_bzip2_jsonlines", + "link": "https://db.example.org/assets/partial_values/structures/2345678/a/bzip2_format" + }, + { + "format": "_exmpl_hdf5", + "link": "https://cloud.example.org/ACCHSORJGIHWOSJZG" + } + ] + } + } + } + // ... + } + + +Metadata for Property_ranges query parameter +-------------------------------------------- + +If the server supports the :query-param:`property_ranges` query parameter, as described in section `Single Entry URL Query Parameters`_, additional metadata SHOULD be present for each field for which the :query-param:`property_ranges` query parameter can be used. The client needs this metadata to be able to select only a part of the data of a property. This metadata is stored in a dictionary in the field :field:`range` which is stored as per property metadata as described in section `Metadata properties`_ - :field:`range`: Dictionary - A dictionary that contains the data necessary for the client to select only a sub_range of a property. - If the property_ranges query parameter is supported for this property the following keys MUST be present. - Otherwise these keys are optional. - - :field:`range_ids`: List of Strings. - A list with an identifier for each dimension of the property. - The outermost dimension of a nested array should come first. - - If, within one entry, dimensions for two or more properties share the same :field:`range_id` those dimensions should be thought of as the same dimension. - For example, if both the :property:`energy` and :property:`cartesian_site_positions` of a molecular dynamics trajectory share a range_id of :val:`frames`. - This means that the energy at index x(in the dimension labelled by this range_id) belongs to the cartesian_site_positions at the same index x. + A dictionary that contains the data necessary for the client to select only a sub_range of a property. + If the property_ranges query parameter is supported for this property the following keys MUST be present. + Otherwise this dictionary and these keys are optional. - :field:`indexable_dim`: List of Strings. The list of range_ids of the dimensions for which slicing is supported, i.e. the client can request a subrange in this dimension via the :query-param:`property_ranges` query parameter. - - :field:`"layout"`: String. + - :field:`layout`: String. A string either equal to :val:`"dense"` or :val:`"sparse"` to indicate whether the property is returned in a dense or sparse layout. - :field:`data_range`: List of Slice Objects. This field describes how the values are distributed in the different dimensions. It consists of a slice object for each dimension of the property. - The order of the slice objects must be the same as in the :field:`range_ids` field. + The order of the slice objects must be the same as in the :field:`range_ids` field in the property definition. If the :field:`layout` field is set to :val:`"sparse"` the value of the :field:`step` sub-field has no meaning. - :field:`nvalues`: Integer. @@ -493,7 +525,7 @@ This metadata is stored in a dictionary in the field :field:`range` which is sto SHOULD be a queryable property with support for all mandatory filter features. Below follows an example of the :field:`data` and :field:`meta` parts of a response using the JSON response format. -It communicates that the property value has been omitted from the response and includes three different links for different partial data formats provided and metadata that makes it possible to request only a part of the data for this property. +It communicates that the property value has been omitted from the response and metadata that makes it possible to request only a part of the data for this property. .. code:: jsonc @@ -526,20 +558,7 @@ It communicates that the property value has been omitted from the response and i } }, "partial_data_links": { - "a": [ - { - "format": "jsonlines", - "link": "https://example.org/optimade/v1.2/extensions/partial_data/structures/2345678/a/default_format" - }, - { - "format": "_exmpl_bzip2_jsonlines", - "link": "https://db.example.org/assets/partial_values/structures/2345678/a/bzip2_format" - }, - { - "format": "_exmpl_hdf5", - "link": "https://cloud.example.org/ACCHSORJGIHWOSJZG" - } - ] + //... } } } @@ -1221,24 +1240,26 @@ The rules for which properties are to be present for an entry in the response ar Single Entry URL Query Parameters ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -- **property\_ranges**: A set of slices which specify which range of a property should be returned for partial data properties. - The property to which this range should be applied can be specified via the :query-param:`response_fields` query parameter. +- **property\_ranges**: A set of slices which specify which sub-range of a property should be returned. + + The property to which this sub-range should be applied can be specified via the :query-param:`response_fields` query parameter. Each slice consists of the name of a dimension directly followed by the requested slice in this dimension. The dimension name and the start, stop and step values of the slice are separated by colons (":", ASCII 58(0x3A)) - The slice is defined in the same manner as the `slice object`_. + The slice is defined in a similar manner as the `slice object`_. If no value is placed between the colons for a component of the slice the default value is used. + The first integer specifies the start of the slice, i.e. the first index in that dimension for which values should be returned. The default is the index of the first value. The second integer specifies the end/stop of the slice, i.e. the last index for which values should be returned. The default is the index of the last value of the property. And the third integer specifies the step size in that dimension. The default is :val:`1` + Multiple dimensional slices can be defined by specifying a range for each dimension. These ranges are separated by a comma (",", ASCII 44(0x2C)). The slices are 0 based, i.e. the first value has index 0, and inclusive i.e. for the range :val:`:10:20:1` the last value returned belongs to index 20. In general support is OPTIONAL, property definitions may however deviate from this and place stricter requirements on servers. - Databases MUST use the methods described in the section `Transmission of large property values`_ to return the requested sub-range of a property. Databases MUST use these ranges for properties where the dimension is listed under indexable_dimensions, if this is not the case the database MAY return more data than was specified in the range. If a dimension is not specified, it is assumed that the whole range in that dimension is requested. @@ -1251,6 +1272,7 @@ Single Entry URL Query Parameters :query-url:`http://example.com/optimade/v1/structures/id_12345?response_fields=cartesian_site_positions&property_ranges=frames::1000:10,particles:30:70::` +| The client MAY provide a set of additional URL query parameters for this endpoint type. URL query parameters not recognized MUST be ignored. @@ -2195,6 +2217,19 @@ A Property Definition MUST be composed according to the combination of the requi - :val:`partial`: the defined property MUST be queryable with support for a subset of the filter language operators as specified by the field :field:`query-support-operators`. - :val:`none`: the defined property does not need to be queryable with any features of the filter language. + - :field:`property_range`: Dictionary. + A dictionary that contains information needed to use the :query-param:`property_range` query parameter + + - :field:`supported`: Boolean. + This field indicates whether the it is possible to use the :query-param:`property_range` query parameter in combination with this field. If true the use of the :query-param:`property_range` is supported + + - :field:`range_ids`: List of Strings. + A list with an identifier for each dimension of the property. + The outermost dimension of a nested array should come first. + If, within one entry, dimensions for two or more properties share the same :field:`range_id` those dimensions should be thought of as the same dimension. + For example, if both the :property:`energy` and :property:`cartesian_site_positions` of a molecular dynamics trajectory share a range_id of :val:`frames`. + This means that the energy at index x(in the dimension labelled by this range_id) belongs to the cartesian_site_positions at the same index x. + - :field:`query-support-operators`: List of Strings. Defines the filter language features supported on this property. Each string in the list MUST be one of :val:`<`, :val:`<=`, :val:`>`, :val:`>=`, :val:`=`, :val:`!=`, :val:`CONTAINS`, :val:`STARTS WITH`, :val:`ENDS WITH`:, :val:`HAS`, :val:`HAS ALL`, :val:`HAS ANY`, :val:`HAS ONLY`, :val:`IS KNOWN`, :val:`IS UNKNOWN` with the following meanings: From 17beeef72512666fe3776583be51a3511b9b218e Mon Sep 17 00:00:00 2001 From: Matthew Evans <7916000+ml-evs@users.noreply.github.com> Date: Thu, 18 Jan 2024 11:52:58 +0000 Subject: [PATCH 11/34] Apply formatting suggestions from code review --- optimade.rst | 66 ++++++++++++++++++++++++++-------------------------- 1 file changed, 33 insertions(+), 33 deletions(-) diff --git a/optimade.rst b/optimade.rst index 49fdc2875..852f02170 100644 --- a/optimade.rst +++ b/optimade.rst @@ -495,28 +495,26 @@ It communicates that the property value has been omitted from the response and i } -Metadata for Property_ranges query parameter +Metadata for :query-param:`property_ranges` query parameter -------------------------------------------- If the server supports the :query-param:`property_ranges` query parameter, as described in section `Single Entry URL Query Parameters`_, additional metadata SHOULD be present for each field for which the :query-param:`property_ranges` query parameter can be used. The client needs this metadata to be able to select only a part of the data of a property. -This metadata is stored in a dictionary in the field :field:`range` which is stored as per property metadata as described in section `Metadata properties`_ +This metadata is given in a dictionary field :field:`range` which stores per property metadata as described in section `Metadata properties`_ -- :field:`range`: Dictionary - - A dictionary that contains the data necessary for the client to select only a sub_range of a property. - If the property_ranges query parameter is supported for this property the following keys MUST be present. - Otherwise this dictionary and these keys are optional. +- :field:`range`: Dictionary. + A dictionary that contains the data necessary for the client to select only a subset or slice of a property. + If the :query-param:`property_ranges` query parameter is supported for this property the following keys MUST be present: - :field:`indexable_dim`: List of Strings. - The list of range_ids of the dimensions for which slicing is supported, i.e. the client can request a subrange in this dimension via the :query-param:`property_ranges` query parameter. + The list of :field:`range_ids` of the dimensions for which slicing is supported, i.e. the client can request a slice in this dimension via the :query-param:`property_ranges` query parameter. - :field:`layout`: String. A string either equal to :val:`"dense"` or :val:`"sparse"` to indicate whether the property is returned in a dense or sparse layout. - - :field:`data_range`: List of Slice Objects. + - :field:`data_range`: List of slice objects. This field describes how the values are distributed in the different dimensions. - It consists of a slice object for each dimension of the property. + It consists of a `slice object`_ for each dimension of the property. The order of the slice objects must be the same as in the :field:`range_ids` field in the property definition. If the :field:`layout` field is set to :val:`"sparse"` the value of the :field:`step` sub-field has no meaning. @@ -535,24 +533,27 @@ It communicates that the property value has been omitted from the response and m "type": "structures", "id": "2345678", "attributes": { - "a": null + "a": null } "meta": { - "property_metadata":{ - "a":{ - "range":{ - "range_ids":["frames","particles"], + "property_metadata": { + "a": { + "range": { + "range_ids": ["frames", "particles"], "indexable_dim": ["frames"], - "data_range": [{ + "data_range": [ + { "start": 1, "step": 1, "stop": 200, - },{ + }, + { "start": 1, "step": 1, "stop": 3, - }], - "layout":"dense", + } + ], + "layout": "dense", "nvalues": 600 } } @@ -1240,39 +1241,38 @@ The rules for which properties are to be present for an entry in the response ar Single Entry URL Query Parameters ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -- **property\_ranges**: A set of slices which specify which sub-range of a property should be returned. +- **property\_ranges**: A set of slice objects which specify which sub-range of a property should be returned. The property to which this sub-range should be applied can be specified via the :query-param:`response_fields` query parameter. Each slice consists of the name of a dimension directly followed by the requested slice in this dimension. The dimension name and the start, stop and step values of the slice are separated by colons (":", ASCII 58(0x3A)) The slice is defined in a similar manner as the `slice object`_. - If no value is placed between the colons for a component of the slice the default value is used. + If no value is placed between the colons for a component of the slice then the default value is used. The first integer specifies the start of the slice, i.e. the first index in that dimension for which values should be returned. The default is the index of the first value. The second integer specifies the end/stop of the slice, i.e. the last index for which values should be returned. The default is the index of the last value of the property. - And the third integer specifies the step size in that dimension. + The third integer specifies the step size in that dimension. The default is :val:`1` - Multiple dimensional slices can be defined by specifying a range for each dimension. + Multidimensional slices can be defined by specifying a range for each dimension. These ranges are separated by a comma (",", ASCII 44(0x2C)). - The slices are 0 based, i.e. the first value has index 0, and inclusive i.e. for the range :val:`:10:20:1` the last value returned belongs to index 20. + The slices are 0-based, i.e. the first value has index 0, and inclusive i.e. for the range :val:`:10:20:1` the last value returned belongs to index 20. - In general support is OPTIONAL, property definitions may however deviate from this and place stricter requirements on servers. - Databases MUST use these ranges for properties where the dimension is listed under indexable_dimensions, if this is not the case the database MAY return more data than was specified in the range. + General support for :field:`property_ranges` is OPTIONAL, however particular property definitions may themselves deviate from this and place stricter requirements on implementations. + Databases MUST use these ranges for properties where the dimension is listed under :field:`indexable_dimensions`, if this is not the case, the database MAY return more data than was specified in the range. If a dimension is not specified, it is assumed that the whole range in that dimension is requested. If one or more values are not present at one of the requested combination of indexes, the server MAY either decide to return null or if possible adjust the returned range so the indexes for which no value is defined are no longer part of the range. The latter is only allowed when no defined values would be lost. For dense arrays that may mean that the field :field:`returned_range` differs from the requested range. - When a value is, however, explicitly set to :val:`null` and :val:`null` has a meaning beyond indicating that no value has been defined :val:`null` MUST be returned + However, when a value is explicitly set to :val:`null` to indicate that the underlying property has no defined values, then :val:`null` MUST be returned. - Example: In this example the Cartesian site positions are requested for particles 30 through 70 for 1 out of every 10 frames of the first 1000 frames of a trajectory. + Example: In this example the Cartesian site positions are requested for particles 30 through 70 for 1 out of every 10 frames of the first 1000 frames of a trajectory with ID :val:`id_12345`. - :query-url:`http://example.com/optimade/v1/structures/id_12345?response_fields=cartesian_site_positions&property_ranges=frames::1000:10,particles:30:70::` + :query-url:`http://example.com/optimade/v1/trajectories/id_12345?response_fields=cartesian_site_positions&property_ranges=frames::1000:10,particles:30:70::` -| The client MAY provide a set of additional URL query parameters for this endpoint type. URL query parameters not recognized MUST be ignored. @@ -2218,17 +2218,17 @@ A Property Definition MUST be composed according to the combination of the requi - :val:`none`: the defined property does not need to be queryable with any features of the filter language. - :field:`property_range`: Dictionary. - A dictionary that contains information needed to use the :query-param:`property_range` query parameter + A dictionary that contains information needed to use the :query-param:`property_range` query parameter. - :field:`supported`: Boolean. - This field indicates whether the it is possible to use the :query-param:`property_range` query parameter in combination with this field. If true the use of the :query-param:`property_range` is supported + This field indicates whether it is possible to use the :query-param:`property_range` query parameter in combination with this field. - :field:`range_ids`: List of Strings. A list with an identifier for each dimension of the property. The outermost dimension of a nested array should come first. If, within one entry, dimensions for two or more properties share the same :field:`range_id` those dimensions should be thought of as the same dimension. For example, if both the :property:`energy` and :property:`cartesian_site_positions` of a molecular dynamics trajectory share a range_id of :val:`frames`. - This means that the energy at index x(in the dimension labelled by this range_id) belongs to the cartesian_site_positions at the same index x. + This means that the energy at index *x* (in the dimension labelled by this :field:`range_id`) belongs to the :field:`cartesian_site_positions` at the same index *x*. - :field:`query-support-operators`: List of Strings. Defines the filter language features supported on this property. From 6d8e24ff58c124039238931c61599f420eaad555 Mon Sep 17 00:00:00 2001 From: Rickard Armiento Date: Fri, 22 Mar 2024 02:07:48 +0100 Subject: [PATCH 12/34] Apply suggestions from review Co-authored-by: Antanas Vaitkus --- optimade.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/optimade.rst b/optimade.rst index bb04a1cf9..2cf31c798 100644 --- a/optimade.rst +++ b/optimade.rst @@ -1265,12 +1265,12 @@ Single Entry URL Query Parameters The slices are 0-based, i.e. the first value has index 0, and inclusive i.e. for the range :val:`:10:20:1` the last value returned belongs to index 20. General support for :field:`property_ranges` is OPTIONAL, however particular property definitions may themselves deviate from this and place stricter requirements on implementations. - Databases MUST use these ranges for properties where the dimension is listed under :field:`indexable_dimensions`, if this is not the case, the database MAY return more data than was specified in the range. + Databases MUST use these ranges for properties where the dimension is listed under :field:`indexable_dim`, if this is not the case, the database MAY return more data than was specified in the range. If a dimension is not specified, it is assumed that the whole range in that dimension is requested. If one or more values are not present at one of the requested combination of indexes, the server MAY either decide to return null or if possible adjust the returned range so the indexes for which no value is defined are no longer part of the range. The latter is only allowed when no defined values would be lost. - For dense arrays that may mean that the field :field:`returned_range` differs from the requested range. + For dense arrays that may mean that the field :field:`returned_ranges` differs from the requested range. However, when a value is explicitly set to :val:`null` to indicate that the underlying property has no defined values, then :val:`null` MUST be returned. Example: In this example the Cartesian site positions are requested for particles 30 through 70 for 1 out of every 10 frames of the first 1000 frames of a trajectory with ID :val:`id_12345`. From 63e382143874c7b6b07110f7302cae78cf8d46b8 Mon Sep 17 00:00:00 2001 From: Rickard Armiento Date: Fri, 22 Mar 2024 12:48:03 +0100 Subject: [PATCH 13/34] Minor corrections from review Co-authored-by: Andrius Merkys --- optimade.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/optimade.rst b/optimade.rst index a88cb8225..7a94c5851 100644 --- a/optimade.rst +++ b/optimade.rst @@ -1295,14 +1295,14 @@ Single Entry URL Query Parameters Multidimensional slices can be defined by specifying a range for each dimension. These ranges are separated by a comma (",", ASCII 44(0x2C)). - The slices are 0-based, i.e. the first value has index 0, and inclusive i.e. for the range :val:`:10:20:1` the last value returned belongs to index 20. + The slices are 0-based, i.e. the first value has index 0, and inclusive i.e. for the dimension :val:`dim` the range :val:`dim:10:20:1` the last value returned belongs to index 20. General support for :field:`property_ranges` is OPTIONAL, however particular property definitions may themselves deviate from this and place stricter requirements on implementations. Databases MUST use these ranges for properties where the dimension is listed under :field:`indexable_dim`, if this is not the case, the database MAY return more data than was specified in the range. If a dimension is not specified, it is assumed that the whole range in that dimension is requested. - If one or more values are not present at one of the requested combination of indexes, the server MAY either decide to return null or if possible adjust the returned range so the indexes for which no value is defined are no longer part of the range. - The latter is only allowed when no defined values would be lost. + If one or more values are not present at one of the requested combination of indexes, the server MAY either decide to return :val:`null` or if possible adjust the returned range so the indexes for which no value is defined are no longer part of the range. + The latter is only allowed when no defined values would be omitted. For dense arrays that may mean that the field :field:`returned_ranges` differs from the requested range. However, when a value is explicitly set to :val:`null` to indicate that the underlying property has no defined values, then :val:`null` MUST be returned. From bf7e5b4acf7cf111e1ebe28c7e9a239ea469ac15 Mon Sep 17 00:00:00 2001 From: Rickard Armiento Date: Thu, 13 Jun 2024 13:12:40 +0200 Subject: [PATCH 14/34] Edit property slice definition [WIP] --- optimade.rst | 194 +++++++++++++++++++++++++++------------------------ 1 file changed, 103 insertions(+), 91 deletions(-) diff --git a/optimade.rst b/optimade.rst index 5ca81834c..4a3bff0d9 100644 --- a/optimade.rst +++ b/optimade.rst @@ -527,78 +527,6 @@ It communicates that the property value has been omitted from the response and i } -Metadata for :query-param:`property_ranges` query parameter --------------------------------------------- - -If the server supports the :query-param:`property_ranges` query parameter, as described in section `Single Entry URL Query Parameters`_, additional metadata SHOULD be present for each field for which the :query-param:`property_ranges` query parameter can be used. -The client needs this metadata to be able to select only a part of the data of a property. -This metadata is given in a dictionary field :field:`range` which stores per property metadata as described in section `Metadata properties`_ - -- :field:`range`: Dictionary. - A dictionary that contains the data necessary for the client to select only a subset or slice of a property. - If the :query-param:`property_ranges` query parameter is supported for this property the following keys MUST be present: - - - :field:`indexable_dim`: List of Strings. - The list of :field:`range_ids` of the dimensions for which slicing is supported, i.e. the client can request a slice in this dimension via the :query-param:`property_ranges` query parameter. - - - :field:`layout`: String. - A string either equal to :val:`"dense"` or :val:`"sparse"` to indicate whether the property is returned in a dense or sparse layout. - - - :field:`data_range`: List of slice objects. - This field describes how the values are distributed in the different dimensions. - It consists of a `slice object`_ for each dimension of the property. - The order of the slice objects must be the same as in the :field:`range_ids` field in the property definition. - If the :field:`layout` field is set to :val:`"sparse"` the value of the :field:`step` sub-field has no meaning. - - - :field:`nvalues`: Integer. - The total number of values in the property. - SHOULD be a queryable property with support for all mandatory filter features. - -Below follows an example of the :field:`data` and :field:`meta` parts of a response using the JSON response format. -It communicates that the property value has been omitted from the response and metadata that makes it possible to request only a part of the data for this property. - -.. code:: jsonc - - { - // ... - "data": { - "type": "structures", - "id": "2345678", - "attributes": { - "a": null - } - "meta": { - "property_metadata": { - "a": { - "range": { - "range_ids": ["frames", "particles"], - "indexable_dim": ["frames"], - "data_range": [ - { - "start": 1, - "step": 1, - "stop": 200, - }, - { - "start": 1, - "step": 1, - "stop": 3, - } - ], - "layout": "dense", - "nvalues": 600 - } - } - }, - "partial_data_links": { - //... - } - } - } - // ... - } - - Metadata properties ------------------- @@ -676,6 +604,85 @@ Example of the corresponding metadata property definition contained in the field } // ... +Slices of array properties +-------------------------- + +# respnse_fields=property_metadata means MUST RETURN ALL METADATA. + +The OPTIMADE standard defines a way for a client to request only a subset of the items of an array, referred to as a slice. +The protocol for this functionality allows the server to allow slicing on a per-entry basis. +It is important to note that this functionality is separate from the protocol described in `Transmission of large property values`. +Slices are used for a client to ask the server to only provide a subset of items, which may result in a small or large set of items. +The protocol for large property values is used by the server implementation to transmit a set of items that it deems too large to provide inside the normal OPTIMADE response. + +If the server supports the :query-param:`property_ranges` query parameter, as described in section `Single Entry URL Query Parameters`_, additional metadata SHOULD be present for each field for which the :query-param:`property_ranges` query parameter can be used. +The client needs this metadata to be able to select only a part of the data of a property. +This metadata is given in a dictionary field :field:`range` which stores per property metadata as described in section `Metadata properties`_ + +- :field:`range`: Dictionary. + A dictionary that contains the data necessary for the client to select only a subset or slice of a property. + If the :query-param:`property_ranges` query parameter is supported for this property the following keys MUST be present: + + - :field:`indexable_dim`: List of Strings. + The list of :field:`range_ids` of the dimensions for which slicing is supported, i.e. the client can request a slice in this dimension via the :query-param:`property_ranges` query parameter. + + - :field:`layout`: String. + A string either equal to :val:`"dense"` or :val:`"sparse"` to indicate whether the property is returned in a dense or sparse layout. + + - :field:`data_range`: List of slice objects. + This field describes how the values are distributed in the different dimensions. + It consists of a `slice object`_ for each dimension of the property. + The order of the slice objects must be the same as in the :field:`range_ids` field in the property definition. + If the :field:`layout` field is set to :val:`"sparse"` the value of the :field:`step` sub-field has no meaning. + + - :field:`nvalues`: Integer. + The total number of values in the property. + SHOULD be a queryable property with support for all mandatory filter features. + +Below follows an example of the :field:`data` and :field:`meta` parts of a response using the JSON response format. +It communicates that the property value has been omitted from the response and metadata that makes it possible to request only a part of the data for this property. + +.. code:: jsonc + + { + // ... + "data": { + "type": "structures", + "id": "2345678", + "attributes": { + "a": null + } + "meta": { + "property_metadata": { + "a": { + "range": { + "range_ids": ["frames", "particles"], + "indexable_dim": ["frames"], + "data_range": [ + { + "start": 1, + "step": 1, + "stop": 200, + }, + { + "start": 1, + "step": 1, + "stop": 3, + } + ], + "layout": "dense", + "nvalues": 600 + } + } + }, + "partial_data_links": { + //... + } + } + } + // ... + } + Responses ========= @@ -1189,6 +1196,10 @@ Standard OPTIONAL URL query parameters not in the JSON:API specification: Other OPTIONAL fields MUST NOT be returned when this parameter is present. Example: :query-url:`http://example.com/optimade/v1/structures?response_fields=last_modified,nsites` + +********* + + Additional OPTIONAL URL query parameters not described above are not considered to be part of this standard, and are instead considered to be "custom URL query parameters". These custom URL query parameters MUST be of the format "". These names adhere to the requirements on implementation-specific query parameters of `JSON:API v1.1 `__ since the database-provider-specific prefixes contain at least two underscores (a LOW LINE character '\_'). @@ -1278,7 +1289,16 @@ The rules for which properties are to be present for an entry in the response ar Single Entry URL Query Parameters ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -- **property\_ranges**: A set of slice objects which specify which sub-range of a property should be returned. +The client MAY provide a set of additional URL query parameters for this endpoint type. +URL query parameters not recognized MUST be ignored. +While the following URL query parameters are OPTIONAL for clients, API implementations MUST accept and handle them: +:query-param:`response_format`, :query-param:`email_address`, :query-param:`response_fields`. +The URL query parameter :query-param:`include` is OPTIONAL for both clients and API implementations. +The meaning of these URL query parameters are as defined above in section `Entry Listing URL Query Parameters`_. + +One additional query parameter :query-param:`property_slices` MUST be handled by the API implementation either as defined below or by returning the error :http-error:`501 Not Implemented`: + +- **property\_slices**: A set of slice objects which specify which sub-range of a property should be returned. The property to which this sub-range should be applied can be specified via the :query-param:`response_fields` query parameter. Each slice consists of the name of a dimension directly followed by the requested slice in this dimension. @@ -1297,26 +1317,15 @@ Single Entry URL Query Parameters These ranges are separated by a comma (",", ASCII 44(0x2C)). The slices are 0-based, i.e. the first value has index 0, and inclusive i.e. for the dimension :val:`dim` the range :val:`dim:10:20:1` the last value returned belongs to index 20. - General support for :field:`property_ranges` is OPTIONAL, however particular property definitions may themselves deviate from this and place stricter requirements on implementations. - Databases MUST use these ranges for properties where the dimension is listed under :field:`indexable_dim`, if this is not the case, the database MAY return more data than was specified in the range. - - If a dimension is not specified, it is assumed that the whole range in that dimension is requested. - If one or more values are not present at one of the requested combination of indexes, the server MAY either decide to return :val:`null` or if possible adjust the returned range so the indexes for which no value is defined are no longer part of the range. - The latter is only allowed when no defined values would be omitted. - For dense arrays that may mean that the field :field:`returned_ranges` differs from the requested range. - However, when a value is explicitly set to :val:`null` to indicate that the underlying property has no defined values, then :val:`null` MUST be returned. + If an array dimension is indicated as slicable, the response MUST include the items in the requested slice. - Example: In this example the Cartesian site positions are requested for particles 30 through 70 for 1 out of every 10 frames of the first 1000 frames of a trajectory with ID :val:`id_12345`. + If an array dimension is not indicated as slicable, the response MUST be one of the following: (i) An error is returned; or (ii) the items in the requested slice is returned. - :query-url:`http://example.com/optimade/v1/trajectories/id_12345?response_fields=cartesian_site_positions&property_ranges=frames::1000:10,particles:30:70::` + If a dimension is not specified, it is assumed that all list items in that dimension is requested. + Example: In the following example, items from the Cartesian site positions array is requested only for the 31st to 71st sites (i.e., with indexes 30 through 70 inclusive) for 1 out of every 10 frames of the first 1000 frames (i.e., taking steps of 10 over indexes 0 through 999 inclusive, which requests the frames with index 0, 10, 20, 30, ..., 990) of a trajectory with ID :val:`id_12345`. -The client MAY provide a set of additional URL query parameters for this endpoint type. -URL query parameters not recognized MUST be ignored. -While the following URL query parameters are OPTIONAL for clients, API implementations MUST accept and handle them: -:query-param:`response_format`, :query-param:`email_address`, :query-param:`response_fields`. -The URL query parameter :query-param:`include` is OPTIONAL for both clients and API implementations. -The meaning of these URL query parameters are as defined above in section `Entry Listing URL Query Parameters`_. + :query-url:`http://example.com/optimade/v1/trajectories/id_12345?response_fields=cartesian_site_positions&property_ranges=dim_frames::999:10,dim_sites:30:70::` Single Entry JSON Response Schema ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -2291,8 +2300,11 @@ A Property Definition MUST be composed according to the combination of the requi - :field:`names`: List of Strings. A list of names of the dimensions of the underlying one or multi-dimensionsional data represented as mutiple levels of lists. The order is that the the first name applies to the outermost list, the next name to the lists embedded in that list, etc. - If, within one entry, dimensions for two or more properties share the same :field:`name` those dimensions represents the same dimension. - For example, if both the :property:`energy` and :property:`cartesian_site_positions` of a molecular dynamics trajectory share a range_id of :val:`frames` this means that the energy at index *i* (in the dimension labelled by this :field:`range_id`) belongs to the :field:`cartesian_site_positions` at the same index *i*. + Dimension names defined by the OPTIMADE standard are prefixed by ``dim_``. + Dimension names defined by database or definition providers MUST be prefixed by the corresponding namespace prefix and SHOULD also be prefixed with ``dim_``, e.g., ``_exmpl_dim_particles``. + If, within one entry, two or more array dimensions in one or more properties share the same :field:`name`, those represent the same dimension. + For example, in a trajectory entry there is a property :property:`cartesian_site_positions` where one of the dimension names is :val:`dim_frames`. + If there is another one-dimensional array property :property:`_exmpl_energy` that specifies the same dimension name in its property definition, then this declares that the energy and site positions at index *i* pertain to the same frame. - :field:`sizes`: List of Integers or :val:`null`. A list of fixed length requirements on the underlying one or multi-dimensionsional data represented as mutiple levels of lists. From d02975663cd4abf9bee4e8d0bfb6854cfed15eef Mon Sep 17 00:00:00 2001 From: Rickard Armiento Date: Thu, 13 Jun 2024 16:16:37 +0200 Subject: [PATCH 15/34] Edit property slice definition [WIP] --- optimade.rst | 101 ++++++++++++++++++++++++++++++++++++--------------- 1 file changed, 72 insertions(+), 29 deletions(-) diff --git a/optimade.rst b/optimade.rst index 4a3bff0d9..49b01bd96 100644 --- a/optimade.rst +++ b/optimade.rst @@ -607,37 +607,80 @@ Example of the corresponding metadata property definition contained in the field Slices of array properties -------------------------- -# respnse_fields=property_metadata means MUST RETURN ALL METADATA. +# response_fields=property_metadata means MUST RETURN ALL METADATA. The OPTIMADE standard defines a way for a client to request only a subset of the items of an array, referred to as a slice. -The protocol for this functionality allows the server to allow slicing on a per-entry basis. -It is important to note that this functionality is separate from the protocol described in `Transmission of large property values`. -Slices are used for a client to ask the server to only provide a subset of items, which may result in a small or large set of items. -The protocol for large property values is used by the server implementation to transmit a set of items that it deems too large to provide inside the normal OPTIMADE response. - -If the server supports the :query-param:`property_ranges` query parameter, as described in section `Single Entry URL Query Parameters`_, additional metadata SHOULD be present for each field for which the :query-param:`property_ranges` query parameter can be used. -The client needs this metadata to be able to select only a part of the data of a property. -This metadata is given in a dictionary field :field:`range` which stores per property metadata as described in section `Metadata properties`_ - -- :field:`range`: Dictionary. - A dictionary that contains the data necessary for the client to select only a subset or slice of a property. - If the :query-param:`property_ranges` query parameter is supported for this property the following keys MUST be present: - - - :field:`indexable_dim`: List of Strings. - The list of :field:`range_ids` of the dimensions for which slicing is supported, i.e. the client can request a slice in this dimension via the :query-param:`property_ranges` query parameter. - - - :field:`layout`: String. - A string either equal to :val:`"dense"` or :val:`"sparse"` to indicate whether the property is returned in a dense or sparse layout. - - - :field:`data_range`: List of slice objects. - This field describes how the values are distributed in the different dimensions. - It consists of a `slice object`_ for each dimension of the property. - The order of the slice objects must be the same as in the :field:`range_ids` field in the property definition. - If the :field:`layout` field is set to :val:`"sparse"` the value of the :field:`step` sub-field has no meaning. - - - :field:`nvalues`: Integer. - The total number of values in the property. - SHOULD be a queryable property with support for all mandatory filter features. +The protocol for this functionality allows the server to allow slicing independently per entry, per array, and per array dimension. +This functionality is separate from the protocol described in `Transmission of large property values`_. +Slices are used for a client to ask the server to only provide a subset of items of an array, which can result in a small or large set of items. +In contrast, the protocol for large property values is used by the server implementation to transmit a set of items that it deems too large to provide inside the normal OPTIMADE response. + +The main mechanism is provided through the query parameter :query-param:`property_slices` defined in section `Single Entry URL Query Parameters`_. +Information relating to the ability of the server to handle this query parameter and the relevant ranges of indexes is provided using metadata property fields (see `Metadata properties`_) of the array property as defined below: + +- :field:`dimensions`: List of Dictionary. + A list of dictionaries which provide information related to the extents of an array property, which is relevant for slicing. + Each item, in order, represents the array dimension as declared in the corresponding property definition. + + Each item MUST be a dictionary with the following REQUIRED fields: + + - :field:`name`: String. + The dimension name of the corresponding array dimension. + + If the request specifies the :query-param:`property_slices` query parameter for any of the array dimensions of this array property the following key MUST be present: + + - :field:`requested_slice`: Dictionary. + A field that describes the requested slice that was provided via the query parameter :query-param:`property_slices`. + The subfields MUST reflect the values provided via the :query-param:`property_slices`. + The implementation MUST preserve the values as given in the query parameter, including the distinction between specific values and default values even when they are equivalent. + It MAY contain the following subfields that are defined according to the specification of a `slice object`_. + + - :field:`start`: Integer or :val:`null`. + + - :field:`stop`: Integer or :val:`null`. + + - :field:`step`: Integer or :val:`null`. + + A :val:`null` value for any of these fields means it has the default value defined for a `slice object`_. + A missing value is equivalent to a :val:`null` value. + (Consequently, the dictionary MAY be empty if all fields take the default values.) + + Examples: + + - For :?:`property_slices=dim_frames:::` some of the equivalent valid representations are: + + - ``{}`` + - ``{"start": null, "stop": null, "step": null}`` + - ``{"start": null}`` + - ``{"stop": null, "step": null}`` + + Whereas, e.g., the following representation is not valid: + + - ``{"start": 0}`` (despite that :val:`0` is the default value of the :field:`start` field.) + But it is instead a valid representation if the query parameter was :?:`property_slices=dim_frames:0::`. + + - :?:`property_slices=dim_frames:3:5:2` the representation MUST be: + + - ``{"start": 3, "stop": 5, "step": 2}`` + + The dictionary MAY contain the following fields: + + - :field:`size`: Integer. + The length of this dimension for the specific entry that this metadata entry pertains to. + + - :field:`sliceable`: Boolean. + If true, the server MUST handle slices for that dimension. + If false (which is the default), the server MAY handle slices for that dimension, or MAY return the error :http-error:`501 Not Implemented` when a client requests a slice involving this dimension. + + - :field:`available_slice`: Dictionary or :val:`null`. + This field describes a `slice object`_ where there MAY be non-null values in the data. + By including this field, the server certifies that there are only :val:`null` values outside this slice. + If not provided, or equal to :val:`null` or an empty dictionary, the client cannot make any assumptions about what part of the array contains :val:`null` values. + + Examples: + + - ``{"start": 3, "stop": 7, "step": 2}`` means the server certifies that values at indexes 0,1,2,4,6 and any index from 8 to the end of the array are :val:`null`. + Below follows an example of the :field:`data` and :field:`meta` parts of a response using the JSON response format. It communicates that the property value has been omitted from the response and metadata that makes it possible to request only a part of the data for this property. From 63f122f635792be59525cb1956499524702f62f3 Mon Sep 17 00:00:00 2001 From: Rickard Armiento Date: Thu, 13 Jun 2024 20:57:17 +0200 Subject: [PATCH 16/34] Edit property slice definition [WIP] --- optimade.rst | 104 ++++++++++++++++++++++++++++++++++----------------- 1 file changed, 70 insertions(+), 34 deletions(-) diff --git a/optimade.rst b/optimade.rst index 49b01bd96..7e70658bd 100644 --- a/optimade.rst +++ b/optimade.rst @@ -610,7 +610,7 @@ Slices of array properties # response_fields=property_metadata means MUST RETURN ALL METADATA. The OPTIMADE standard defines a way for a client to request only a subset of the items of an array, referred to as a slice. -The protocol for this functionality allows the server to allow slicing independently per entry, per array, and per array dimension. +The protocol for this functionality allows the server to allow slicing independently per entry, per array, and per array axis. This functionality is separate from the protocol described in `Transmission of large property values`_. Slices are used for a client to ask the server to only provide a subset of items of an array, which can result in a small or large set of items. In contrast, the protocol for large property values is used by the server implementation to transmit a set of items that it deems too large to provide inside the normal OPTIMADE response. @@ -618,16 +618,16 @@ In contrast, the protocol for large property values is used by the server implem The main mechanism is provided through the query parameter :query-param:`property_slices` defined in section `Single Entry URL Query Parameters`_. Information relating to the ability of the server to handle this query parameter and the relevant ranges of indexes is provided using metadata property fields (see `Metadata properties`_) of the array property as defined below: -- :field:`dimensions`: List of Dictionary. - A list of dictionaries which provide information related to the extents of an array property, which is relevant for slicing. - Each item, in order, represents the array dimension as declared in the corresponding property definition. +- :field:`array_axes`: List of Dictionary. + A list of dictionaries which provide information related to the axes of an array property, which is relevant for slicing. + Each item, in order, represents the array axis as declared in the corresponding property definition. Each item MUST be a dictionary with the following REQUIRED fields: - - :field:`name`: String. - The dimension name of the corresponding array dimension. + - :field:`dimension_name`: String. + The dimension name of the corresponding array axis. - If the request specifies the :query-param:`property_slices` query parameter for any of the array dimensions of this array property the following key MUST be present: + If the request specifies the :query-param:`property_slices` query parameter for any of the array axes of this array property the following key MUST be present, otherwise it MUST be omitted or equal to :val:`null`: - :field:`requested_slice`: Dictionary. A field that describes the requested slice that was provided via the query parameter :query-param:`property_slices`. @@ -665,12 +665,15 @@ Information relating to the ability of the server to handle this query parameter The dictionary MAY contain the following fields: - - :field:`size`: Integer. - The length of this dimension for the specific entry that this metadata entry pertains to. + - :field:`length`: Integer. + The length of this array axis which MUST be the same as the length of the declared dimension for this axis in the corresponding property definition. + Note that the length of a dimension can be different for different entries if the length is not explicitly declared by the property definition. + For example, the number of frames, i.e., the length of the dimension named ``dim_frames``, is generally different for different trajectories. - :field:`sliceable`: Boolean. - If true, the server MUST handle slices for that dimension. - If false (which is the default), the server MAY handle slices for that dimension, or MAY return the error :http-error:`501 Not Implemented` when a client requests a slice involving this dimension. + If :val:`true`, the server MUST handle slices for that dimension. + If :val:`false`, the server MAY handle slices for that dimension, or MAY return the error :http-error:`501 Not Implemented` when a client requests a slice involving this dimension. + If the field is omitted or :val:`null`, it means the same thing as :val:`false`. - :field:`available_slice`: Dictionary or :val:`null`. This field describes a `slice object`_ where there MAY be non-null values in the data. @@ -682,40 +685,73 @@ Information relating to the ability of the server to handle this query parameter - ``{"start": 3, "stop": 7, "step": 2}`` means the server certifies that values at indexes 0,1,2,4,6 and any index from 8 to the end of the array are :val:`null`. -Below follows an example of the :field:`data` and :field:`meta` parts of a response using the JSON response format. -It communicates that the property value has been omitted from the response and metadata that makes it possible to request only a part of the data for this property. +Below follows an example of the :field:`data` and :field:`meta` parts of a response using the JSON response format for a request to the trajectory endpoint with the query parameter :query-param:`property_slices=dim_frames:3:37:5` and :query-param:`response_fields=cartesian_site_positions,_exmpl_temperature` where the trajectory consists of 432934 frames (with indexes 0 to 432933) and where the :field:`cartisian_site_positions` contains 7 sites: .. code:: jsonc { // ... "data": { - "type": "structures", + "type": "trajectories", "id": "2345678", "attributes": { - "a": null - } + "cartesian_site_positions": null, + "_exmpl_temperature": null + }, "meta": { "property_metadata": { - "a": { - "range": { - "range_ids": ["frames", "particles"], - "indexable_dim": ["frames"], - "data_range": [ - { - "start": 1, - "step": 1, - "stop": 200, + "cartesian_site_positions": { + "array_axes": [ + { + "dimension_name": "dim_frames", + "requested_slice": { + "start": 3, + "stop": 37, + "step": 5 }, - { - "start": 1, - "step": 1, - "stop": 3, - } - ], - "layout": "dense", - "nvalues": 600 - } + "available_slice": { + "start": 0, + "stop": 432933, + "step": 1 + }, + "sliceable": true, + "length": 432934, + } + { + "dimension_name": "dim_sites", + "requested_slice": { + "start": 3, + "stop": 37, + "step": 5 + }, + "available_slice": { + "start": 0, + "stop": 432933, + "step": 1 + }, + "sliceable": true, + "length": 432934, + } + ], + }, + "_exmpl_temperature": { + "array_axes": [ + { + "dimension_name": "dim_frames", + "requested_slice": { + "start": 3, + "stop": 37, + "step": 5 + }, + "available_slice": { + "start": 1000, + "stop": 4000, + "step": 30 + }, + "sliceable": true, + "length": 432934, + } + ] } }, "partial_data_links": { From f86330818bda0774c205eecad713c95a0830ff62 Mon Sep 17 00:00:00 2001 From: Rickard Armiento Date: Thu, 13 Jun 2024 22:49:14 +0200 Subject: [PATCH 17/34] Edit property slice definition [WIP] --- optimade.rst | 37 +++++++++++++++++++------------------ 1 file changed, 19 insertions(+), 18 deletions(-) diff --git a/optimade.rst b/optimade.rst index 7e70658bd..be1eb7711 100644 --- a/optimade.rst +++ b/optimade.rst @@ -539,8 +539,10 @@ Database providers are allowed to define their own metadata properties in :field For example, the metadata property definition of the field :field:`_exmpl_example_field` MUST NOT define a metadata field named, e.g., :field:`accuracy`; the field rather needs to be named, e.g., :field:`_exmpl_accuracy`. The reason for this limitation is to avoid name collisions with metadata fields defined by the OPTIMADE standard in the future that apply also to database-specific data fields. -Implementation of the :field:`meta` field is OPTIONAL. -However, when an implementation supports the :field:`property_metadata` field, it SHOULD include metadata fields for all properties which have metadata and are present in the data part of the response. +Implementation of the :field:`meta` field is OPTIONAL, unless the server implements slicing, in which case it is MANDATORY (see `Slices of array properties`_). +When an implementation supports the :field:`property_metadata` field, it SHOULD include metadata fields for all properties which have metadata and are present in the data part of the response. +If the client includes the string ``property_metadata`` in the query parameter :query-param:`response_fields`, the server MUST include metadata fields for all properties which have metadata. +Furthermore, if the server returns a metadata property, it must be included in its entirety, i.e., including all non-null fields. Example of a response in the JSON response format with two structure entries that each include a metadata property for the attribute field :field:`elements_ratios` and the database-specific per entry metadata field :field:`_exmpl_originates_from_project`: @@ -607,8 +609,6 @@ Example of the corresponding metadata property definition contained in the field Slices of array properties -------------------------- -# response_fields=property_metadata means MUST RETURN ALL METADATA. - The OPTIMADE standard defines a way for a client to request only a subset of the items of an array, referred to as a slice. The protocol for this functionality allows the server to allow slicing independently per entry, per array, and per array axis. This functionality is separate from the protocol described in `Transmission of large property values`_. @@ -616,7 +616,9 @@ Slices are used for a client to ask the server to only provide a subset of items In contrast, the protocol for large property values is used by the server implementation to transmit a set of items that it deems too large to provide inside the normal OPTIMADE response. The main mechanism is provided through the query parameter :query-param:`property_slices` defined in section `Single Entry URL Query Parameters`_. -Information relating to the ability of the server to handle this query parameter and the relevant ranges of indexes is provided using metadata property fields (see `Metadata properties`_) of the array property as defined below: +Information relating to the ability of the server to handle this query parameter and the relevant ranges of indexes is provided using metadata property field :field:`array_axes` (see `Metadata properties`_). +When the client request includes the query parameter :query-param:`property_slices`, the server MUST provide metadata for all properties for which including the subfield :field:`requested_slice` of the :field:`array_axes` is MANDATORY (see below). +The field :field:`array_axes` is defined as follows: - :field:`array_axes`: List of Dictionary. A list of dictionaries which provide information related to the axes of an array property, which is relevant for slicing. @@ -684,8 +686,7 @@ Information relating to the ability of the server to handle this query parameter - ``{"start": 3, "stop": 7, "step": 2}`` means the server certifies that values at indexes 0,1,2,4,6 and any index from 8 to the end of the array are :val:`null`. - -Below follows an example of the :field:`data` and :field:`meta` parts of a response using the JSON response format for a request to the trajectory endpoint with the query parameter :query-param:`property_slices=dim_frames:3:37:5` and :query-param:`response_fields=cartesian_site_positions,_exmpl_temperature` where the trajectory consists of 432934 frames (with indexes 0 to 432933) and where the :field:`cartisian_site_positions` contains 7 sites: +Below follows an example of the :field:`data` and :field:`meta` parts of a response using the JSON response format for a request to the trajectory endpoint with the query parameter :query-param:`property_slices=dim_frames:3:37:5` and :query-param:`response_fields=cartesian_site_positions,_exmpl_temperature` where the trajectory consists of 432934 frames (with indexes 0 to 432933) and where the :field:`cartisian_site_positions` contains 7 sites. Furthermore, the :field:`_exmpl_temperature` contains only :val:`null` values except for items with indexes 1000, 1030, 1060, ..., 4000 (where the values can be either numeric or :val:`null`). .. code:: jsonc @@ -716,21 +717,20 @@ Below follows an example of the :field:`data` and :field:`meta` parts of a respo }, "sliceable": true, "length": 432934, - } + }, { "dimension_name": "dim_sites", - "requested_slice": { - "start": 3, - "stop": 37, - "step": 5 - }, "available_slice": { "start": 0, - "stop": 432933, + "stop": 6, "step": 1 }, - "sliceable": true, - "length": 432934, + "sliceable": false, + "length": 7, + }, + { + "dimension_name": "dim_spatial", + "length": 3, } ], }, @@ -1277,11 +1277,12 @@ Standard OPTIONAL URL query parameters not in the JSON:API specification: ********* - + Additional OPTIONAL URL query parameters not described above are not considered to be part of this standard, and are instead considered to be "custom URL query parameters". These custom URL query parameters MUST be of the format "". These names adhere to the requirements on implementation-specific query parameters of `JSON:API v1.1 `__ since the database-provider-specific prefixes contain at least two underscores (a LOW LINE character '\_'). +If the server receives an unrecognized query parameter that does not adhere to this format it MUST return the error :http-error:`400 Bad Request`. Example uses of custom URL query parameters include providing an access token for the request, to tell the database to increase verbosity in error output, or providing a database-specific extended searching format. @@ -2440,7 +2441,7 @@ A Property Definition MUST be composed according to the combination of the requi The value :val:`TRUE` means the implementation includes the property in responses by default, i.e., when not specifically requested. The value :val:`FALSE` means that the property is only included when requested. Omitting the field or :val:`null` means the implementation does not declare if the property will be included in responses by default or not. - + - :field:`dimensions_supporting_ranges`: List of String or :val:`null`. A list of names of dimensions that appears in :field:`x-optimade-dimensions` fields within the property definition. A name appearing in the list means this dimension can be used in the :query-param:`property_ranges` query paramter to request subsets of the data along that dimension. From a73029a553d19fed344d5637626055e2e8c1be58 Mon Sep 17 00:00:00 2001 From: Rickard Armiento Date: Thu, 13 Jun 2024 23:04:37 +0200 Subject: [PATCH 18/34] Edit property slice definition [WIP] --- optimade.rst | 5 ----- 1 file changed, 5 deletions(-) diff --git a/optimade.rst b/optimade.rst index be1eb7711..55b73eef3 100644 --- a/optimade.rst +++ b/optimade.rst @@ -526,7 +526,6 @@ It communicates that the property value has been omitted from the response and i // ... } - Metadata properties ------------------- @@ -1275,10 +1274,6 @@ Standard OPTIONAL URL query parameters not in the JSON:API specification: Other OPTIONAL fields MUST NOT be returned when this parameter is present. Example: :query-url:`http://example.com/optimade/v1/structures?response_fields=last_modified,nsites` - -********* - - Additional OPTIONAL URL query parameters not described above are not considered to be part of this standard, and are instead considered to be "custom URL query parameters". These custom URL query parameters MUST be of the format "". These names adhere to the requirements on implementation-specific query parameters of `JSON:API v1.1 `__ since the database-provider-specific prefixes contain at least two underscores (a LOW LINE character '\_'). From c098e4782b93c5e9091931ea897d2fa5df95898b Mon Sep 17 00:00:00 2001 From: Rickard Armiento Date: Fri, 14 Jun 2024 09:11:47 +0200 Subject: [PATCH 19/34] Improve handing of Python dependencies --- optimade.rst | 37 +++++++++++-------------------------- 1 file changed, 11 insertions(+), 26 deletions(-) diff --git a/optimade.rst b/optimade.rst index 55b73eef3..bfbf3e16d 100644 --- a/optimade.rst +++ b/optimade.rst @@ -541,7 +541,7 @@ The reason for this limitation is to avoid name collisions with metadata fields Implementation of the :field:`meta` field is OPTIONAL, unless the server implements slicing, in which case it is MANDATORY (see `Slices of array properties`_). When an implementation supports the :field:`property_metadata` field, it SHOULD include metadata fields for all properties which have metadata and are present in the data part of the response. If the client includes the string ``property_metadata`` in the query parameter :query-param:`response_fields`, the server MUST include metadata fields for all properties which have metadata. -Furthermore, if the server returns a metadata property, it must be included in its entirety, i.e., including all non-null fields. +Furthermore, if the server returns metadata for a property, it must be included in its entirety, i.e., including all non-null fields. Example of a response in the JSON response format with two structure entries that each include a metadata property for the attribute field :field:`elements_ratios` and the database-specific per entry metadata field :field:`_exmpl_originates_from_project`: @@ -1373,34 +1373,27 @@ The meaning of these URL query parameters are as defined above in section `Entry One additional query parameter :query-param:`property_slices` MUST be handled by the API implementation either as defined below or by returning the error :http-error:`501 Not Implemented`: -- **property\_slices**: A set of slice objects which specify which sub-range of a property should be returned. - - The property to which this sub-range should be applied can be specified via the :query-param:`response_fields` query parameter. - Each slice consists of the name of a dimension directly followed by the requested slice in this dimension. - The dimension name and the start, stop and step values of the slice are separated by colons (":", ASCII 58(0x3A)) - The slice is defined in a similar manner as the `slice object`_. - If no value is placed between the colons for a component of the slice then the default value is used. +- **property\_slices**: A number of slice specifications to specify parts of array properties for the functionality described in `Slices of array properties`_. + The query parameter contains a comma-separated (",", ASCII 44(0x2C)) list of slice specification. + Each slice specification consists of a dimension name and the start, stop and step values of the slice separated by the colon character (":", ASCII 58(0x3A)) + Omitting the value between two colons specifies a default value. + The components of the slice are defined in the same way as for a `slice object`_. The first integer specifies the start of the slice, i.e. the first index in that dimension for which values should be returned. The default is the index of the first value. The second integer specifies the end/stop of the slice, i.e. the last index for which values should be returned. The default is the index of the last value of the property. The third integer specifies the step size in that dimension. The default is :val:`1` + The slices are 0-based, i.e. the first value has index 0, and inclusive i.e. for the dimension :val:`dim_frames` the range :val:`dim_frames:10:20:1` the last value returned belongs to index 20. - Multidimensional slices can be defined by specifying a range for each dimension. - These ranges are separated by a comma (",", ASCII 44(0x2C)). - The slices are 0-based, i.e. the first value has index 0, and inclusive i.e. for the dimension :val:`dim` the range :val:`dim:10:20:1` the last value returned belongs to index 20. - - If an array dimension is indicated as slicable, the response MUST include the items in the requested slice. - - If an array dimension is not indicated as slicable, the response MUST be one of the following: (i) An error is returned; or (ii) the items in the requested slice is returned. + Requirements and conventions for the response when this query parameter is used are described in `Slices of array properties`_. - If a dimension is not specified, it is assumed that all list items in that dimension is requested. + Example: - Example: In the following example, items from the Cartesian site positions array is requested only for the 31st to 71st sites (i.e., with indexes 30 through 70 inclusive) for 1 out of every 10 frames of the first 1000 frames (i.e., taking steps of 10 over indexes 0 through 999 inclusive, which requests the frames with index 0, 10, 20, 30, ..., 990) of a trajectory with ID :val:`id_12345`. + - :query-url:`http://optimade.example.com/v1/trajectories/id_12345?response_fields=cartesian_site_positions&property_ranges=dim_frames::999:10,dim_sites:30:70::` - :query-url:`http://example.com/optimade/v1/trajectories/id_12345?response_fields=cartesian_site_positions&property_ranges=dim_frames::999:10,dim_sites:30:70::` + This query URL requests items from the array :field:`cartesian_site_positions` is requested only for the 31st to 71st sites (i.e., with indexes 30 through 70 inclusive) for 1 out of every 10 frames of the first 1000 frames (i.e., taking steps of 10 over indexes 0 through 999 inclusive, which requests the frames with index 0, 10, 20, 30, ..., 990) of a trajectory with ID :val:`id_12345`. Single Entry JSON Response Schema ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -2437,10 +2430,6 @@ A Property Definition MUST be composed according to the combination of the requi The value :val:`FALSE` means that the property is only included when requested. Omitting the field or :val:`null` means the implementation does not declare if the property will be included in responses by default or not. - - :field:`dimensions_supporting_ranges`: List of String or :val:`null`. - A list of names of dimensions that appears in :field:`x-optimade-dimensions` fields within the property definition. - A name appearing in the list means this dimension can be used in the :query-param:`property_ranges` query paramter to request subsets of the data along that dimension. - - :field:`x-optimade-requirements`: Dictionary. A dictionary describing the level of OPTIMADE API functionality required by all implementations of this property. Omitting this field means the corresponding functionality is OPTIONAL. @@ -2478,10 +2467,6 @@ A Property Definition MUST be composed according to the combination of the requi Omitting the field is equivalent to :val:`may`. - - :field:`dimensions_supporting_ranges`: List of String or :val:`null`. - A list of names of dimensions that appears in :field:`x-optimade-dimensions` fields within the property definition. - Expresses if an implementation of this property is required to support using this dimension in the :query-param:`property_ranges` query paramter to request subsets of the data along that dimension. - Property Definition keys from JSON Schema ----------------------------------------- From 4f704c161dd8d6010d6b9523ea8c28953fcc0e21 Mon Sep 17 00:00:00 2001 From: Rickard Armiento Date: Fri, 14 Jun 2024 09:12:55 +0200 Subject: [PATCH 20/34] Improve handing of Python dependencies --- optimade.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/optimade.rst b/optimade.rst index bfbf3e16d..35c37ea7d 100644 --- a/optimade.rst +++ b/optimade.rst @@ -648,7 +648,7 @@ The field :field:`array_axes` is defined as follows: Examples: - - For :?:`property_slices=dim_frames:::` some of the equivalent valid representations are: + - For :query-string:`property_slices=dim_frames:::` some of the equivalent valid representations are: - ``{}`` - ``{"start": null, "stop": null, "step": null}`` @@ -658,9 +658,9 @@ The field :field:`array_axes` is defined as follows: Whereas, e.g., the following representation is not valid: - ``{"start": 0}`` (despite that :val:`0` is the default value of the :field:`start` field.) - But it is instead a valid representation if the query parameter was :?:`property_slices=dim_frames:0::`. + But it is instead a valid representation if the query parameter was :query-string:`property_slices=dim_frames:0::`. - - :?:`property_slices=dim_frames:3:5:2` the representation MUST be: + - :query-string:`property_slices=dim_frames:3:5:2` the representation MUST be: - ``{"start": 3, "stop": 5, "step": 2}`` From 266545b3c0279c20eae243c0990120c667f8f8e9 Mon Sep 17 00:00:00 2001 From: Rickard Armiento Date: Fri, 14 Jun 2024 10:16:41 +0200 Subject: [PATCH 21/34] Improve handing of Python dependencies --- optimade.rst | 801 ++++++++++++++++++++++++++------------------------- 1 file changed, 401 insertions(+), 400 deletions(-) diff --git a/optimade.rst b/optimade.rst index 35c37ea7d..4e688f709 100644 --- a/optimade.rst +++ b/optimade.rst @@ -660,106 +660,106 @@ The field :field:`array_axes` is defined as follows: - ``{"start": 0}`` (despite that :val:`0` is the default value of the :field:`start` field.) But it is instead a valid representation if the query parameter was :query-string:`property_slices=dim_frames:0::`. - - :query-string:`property_slices=dim_frames:3:5:2` the representation MUST be: +- :query-string:`property_slices=dim_frames:3:5:2` the representation MUST be: - - ``{"start": 3, "stop": 5, "step": 2}`` +- ``{"start": 3, "stop": 5, "step": 2}`` - The dictionary MAY contain the following fields: +The dictionary MAY contain the following fields: - - :field:`length`: Integer. - The length of this array axis which MUST be the same as the length of the declared dimension for this axis in the corresponding property definition. - Note that the length of a dimension can be different for different entries if the length is not explicitly declared by the property definition. - For example, the number of frames, i.e., the length of the dimension named ``dim_frames``, is generally different for different trajectories. +- :field:`length`: Integer. +The length of this array axis which MUST be the same as the length of the declared dimension for this axis in the corresponding property definition. +Note that the length of a dimension can be different for different entries if the length is not explicitly declared by the property definition. +For example, the number of frames, i.e., the length of the dimension named ``dim_frames``, is generally different for different trajectories. - - :field:`sliceable`: Boolean. - If :val:`true`, the server MUST handle slices for that dimension. - If :val:`false`, the server MAY handle slices for that dimension, or MAY return the error :http-error:`501 Not Implemented` when a client requests a slice involving this dimension. - If the field is omitted or :val:`null`, it means the same thing as :val:`false`. +- :field:`sliceable`: Boolean. +If :val:`true`, the server MUST handle slices for that dimension. +If :val:`false`, the server MAY handle slices for that dimension, or MAY return the error :http-error:`501 Not Implemented` when a client requests a slice involving this dimension. +If the field is omitted or :val:`null`, it means the same thing as :val:`false`. - - :field:`available_slice`: Dictionary or :val:`null`. - This field describes a `slice object`_ where there MAY be non-null values in the data. - By including this field, the server certifies that there are only :val:`null` values outside this slice. - If not provided, or equal to :val:`null` or an empty dictionary, the client cannot make any assumptions about what part of the array contains :val:`null` values. +- :field:`available_slice`: Dictionary or :val:`null`. +This field describes a `slice object`_ where there MAY be non-null values in the data. +By including this field, the server certifies that there are only :val:`null` values outside this slice. +If not provided, or equal to :val:`null` or an empty dictionary, the client cannot make any assumptions about what part of the array contains :val:`null` values. - Examples: +Examples: - - ``{"start": 3, "stop": 7, "step": 2}`` means the server certifies that values at indexes 0,1,2,4,6 and any index from 8 to the end of the array are :val:`null`. +- ``{"start": 3, "stop": 7, "step": 2}`` means the server certifies that values at indexes 0,1,2,4,6 and any index from 8 to the end of the array are :val:`null`. Below follows an example of the :field:`data` and :field:`meta` parts of a response using the JSON response format for a request to the trajectory endpoint with the query parameter :query-param:`property_slices=dim_frames:3:37:5` and :query-param:`response_fields=cartesian_site_positions,_exmpl_temperature` where the trajectory consists of 432934 frames (with indexes 0 to 432933) and where the :field:`cartisian_site_positions` contains 7 sites. Furthermore, the :field:`_exmpl_temperature` contains only :val:`null` values except for items with indexes 1000, 1030, 1060, ..., 4000 (where the values can be either numeric or :val:`null`). .. code:: jsonc - { - // ... - "data": { - "type": "trajectories", - "id": "2345678", - "attributes": { - "cartesian_site_positions": null, - "_exmpl_temperature": null - }, - "meta": { - "property_metadata": { - "cartesian_site_positions": { - "array_axes": [ - { - "dimension_name": "dim_frames", - "requested_slice": { - "start": 3, - "stop": 37, - "step": 5 - }, - "available_slice": { - "start": 0, - "stop": 432933, - "step": 1 - }, - "sliceable": true, - "length": 432934, - }, - { - "dimension_name": "dim_sites", - "available_slice": { - "start": 0, - "stop": 6, - "step": 1 - }, - "sliceable": false, - "length": 7, - }, - { - "dimension_name": "dim_spatial", - "length": 3, - } - ], - }, - "_exmpl_temperature": { - "array_axes": [ - { - "dimension_name": "dim_frames", - "requested_slice": { - "start": 3, - "stop": 37, - "step": 5 - }, - "available_slice": { - "start": 1000, - "stop": 4000, - "step": 30 - }, - "sliceable": true, - "length": 432934, - } - ] - } - }, - "partial_data_links": { - //... - } - } - } - // ... - } +{ +// ... +"data": { +"type": "trajectories", +"id": "2345678", +"attributes": { +"cartesian_site_positions": null, +"_exmpl_temperature": null +}, +"meta": { +"property_metadata": { +"cartesian_site_positions": { +"array_axes": [ + { + "dimension_name": "dim_frames", + "requested_slice": { + "start": 3, + "stop": 37, + "step": 5 + }, + "available_slice": { + "start": 0, + "stop": 432933, + "step": 1 + }, + "sliceable": true, + "length": 432934, + }, + { + "dimension_name": "dim_sites", + "available_slice": { + "start": 0, + "stop": 6, + "step": 1 + }, + "sliceable": false, + "length": 7, + }, + { + "dimension_name": "dim_spatial", + "length": 3, + } +], +}, +"_exmpl_temperature": { +"array_axes": [ + { + "dimension_name": "dim_frames", + "requested_slice": { + "start": 3, + "stop": 37, + "step": 5 + }, + "available_slice": { + "start": 1000, + "stop": 4000, + "step": 30 + }, + "sliceable": true, + "length": 432934, + } +] +} +}, +"partial_data_links": { +//... +} +} +} +// ... +} Responses ========= @@ -790,305 +790,305 @@ In the JSON response format, property types translate as follows: Every response SHOULD contain the following fields, and MUST contain at least :field:`meta`: - **meta**: a `JSON:API meta member `__ that contains JSON:API meta objects of non-standard meta-information. - It MUST be a dictionary with these fields: +It MUST be a dictionary with these fields: - - **api\_version**: a string containing the full version of the API implementation. - The version number string MUST NOT be prefixed by, e.g., "v". - Examples: :field-val:`1.0.0`, :field-val:`1.0.0-rc.2`. +- **api\_version**: a string containing the full version of the API implementation. +The version number string MUST NOT be prefixed by, e.g., "v". +Examples: :field-val:`1.0.0`, :field-val:`1.0.0-rc.2`. - - **query**: information on the query that was requested. - It MUST be a dictionary with this field: +- **query**: information on the query that was requested. +It MUST be a dictionary with this field: - - **representation**: a string with the part of the URL following the versioned or unversioned base URL that serves the API. - Query parameters that have not been used in processing the request MAY be omitted. - In particular, if no query parameters have been involved in processing the request, the query part of the URL MAY be excluded. - Example: :field-val:`/structures?filter=nelements=2`. +- **representation**: a string with the part of the URL following the versioned or unversioned base URL that serves the API. +Query parameters that have not been used in processing the request MAY be omitted. +In particular, if no query parameters have been involved in processing the request, the query part of the URL MAY be excluded. +Example: :field-val:`/structures?filter=nelements=2`. - - **more\_data\_available**: :field-val:`false` if the response contains all data for the request (e.g., a request issued to a single entry endpoint, or a :query-param:`filter` query at the last page of a paginated response) and :field-val:`true` if the response is incomplete in the sense that multiple objects match the request, and not all of them have been included in the response (e.g., a query with multiple pages that is not at the last page). +- **more\_data\_available**: :field-val:`false` if the response contains all data for the request (e.g., a request issued to a single entry endpoint, or a :query-param:`filter` query at the last page of a paginated response) and :field-val:`true` if the response is incomplete in the sense that multiple objects match the request, and not all of them have been included in the response (e.g., a query with multiple pages that is not at the last page). - :field:`meta` SHOULD also include these fields: +:field:`meta` SHOULD also include these fields: - - **time\_stamp**: a timestamp containing the date and time at which the query was executed. - - **data\_returned**: an integer containing the total number of data resource objects returned for the current :query-param:`filter` query, independent of pagination. - - **provider**: information on the database provider of the implementation. - It MUST be a dictionary with these fields: +- **time\_stamp**: a timestamp containing the date and time at which the query was executed. +- **data\_returned**: an integer containing the total number of data resource objects returned for the current :query-param:`filter` query, independent of pagination. +- **provider**: information on the database provider of the implementation. +It MUST be a dictionary with these fields: - - **name**: a short name for the database provider. - - **description**: a longer description of the database provider. - - **prefix**: database-provider-specific prefix (see section `Database-Provider-Specific Namespace Prefixes`_). +- **name**: a short name for the database provider. +- **description**: a longer description of the database provider. +- **prefix**: database-provider-specific prefix (see section `Database-Provider-Specific Namespace Prefixes`_). - :field:`provider` MAY include these fields: +:field:`provider` MAY include these fields: - - **homepage**: a `JSON API link `__, pointing to the homepage of the database provider, either directly as a string, or as an object which can contain the following fields: +- **homepage**: a `JSON API link `__, pointing to the homepage of the database provider, either directly as a string, or as an object which can contain the following fields: - - **href**: a string containing the homepage URL. - - **meta**: a meta object containing non-standard meta-information about the database provider's homepage. +- **href**: a string containing the homepage URL. +- **meta**: a meta object containing non-standard meta-information about the database provider's homepage. - :field:`meta` MAY also include these fields: +:field:`meta` MAY also include these fields: - - **data\_available**: an integer containing the total number of data resource objects available in the database for the endpoint. - - **last\_id**: a string containing the last ID returned. - - **response\_message**: response string from the server. - - **request\_delay**: a non-negative float giving time in seconds that the client is suggested to wait before issuing a subsequent request. +- **data\_available**: an integer containing the total number of data resource objects available in the database for the endpoint. +- **last\_id**: a string containing the last ID returned. +- **response\_message**: response string from the server. +- **request\_delay**: a non-negative float giving time in seconds that the client is suggested to wait before issuing a subsequent request. - Implementation note: the functionality of this field overlaps to some degree with features provided by the HTTP error :http-error:`429 Too Many Requests` and the `Retry-After HTTP header `__. - Implementations are suggested to provide consistent handling of request overload through both mechanisms. +Implementation note: the functionality of this field overlaps to some degree with features provided by the HTTP error :http-error:`429 Too Many Requests` and the `Retry-After HTTP header `__. +Implementations are suggested to provide consistent handling of request overload through both mechanisms. - - **database**: a dictionary describing the specific database accessible at this OPTIMADE API. - If provided, the dictionary fields SHOULD match those provided in the corresponding links entry for the database in the provider's index meta-database, outlined in `Links Endpoint JSON Response Schema`_. - The dictionary can contain the following OPTIONAL fields: +- **database**: a dictionary describing the specific database accessible at this OPTIMADE API. +If provided, the dictionary fields SHOULD match those provided in the corresponding links entry for the database in the provider's index meta-database, outlined in `Links Endpoint JSON Response Schema`_. +The dictionary can contain the following OPTIONAL fields: - - **id**: the identifier of this database within those served by this provider, i.e., the ID under which this database is served in this provider's index meta-database. - - **name**: a human-readable name for the database, e.g., for use in clients. - - **version**: a string describing the version of the database. - - **description**: a human-readable description of the database, e.g., for use in clients. - - **homepage**: a `JSON API link `__, pointing to a homepage for the particular database. - - **maintainer**: a dictionary providing details about the maintainer of the database, which MUST contain the single field: +- **id**: the identifier of this database within those served by this provider, i.e., the ID under which this database is served in this provider's index meta-database. +- **name**: a human-readable name for the database, e.g., for use in clients. +- **version**: a string describing the version of the database. +- **description**: a human-readable description of the database, e.g., for use in clients. +- **homepage**: a `JSON API link `__, pointing to a homepage for the particular database. +- **maintainer**: a dictionary providing details about the maintainer of the database, which MUST contain the single field: - - **email** with the maintainer's email address. +- **email** with the maintainer's email address. - - **implementation**: a dictionary describing the server implementation, containing the OPTIONAL fields: +- **implementation**: a dictionary describing the server implementation, containing the OPTIONAL fields: - - **name**: name of the implementation. - - **version**: version string of the current implementation. - - **homepage**: a `JSON API link `__, pointing to the homepage of the implementation. - - **source\_url**: a `JSON API link `__ pointing to the implementation source, either downloadable archive or version control system. +- **name**: name of the implementation. +- **version**: version string of the current implementation. +- **homepage**: a `JSON API link `__, pointing to the homepage of the implementation. +- **source\_url**: a `JSON API link `__ pointing to the implementation source, either downloadable archive or version control system. - - **maintainer**: a dictionary providing details about the maintainer of the implementation, MUST contain the single field: +- **maintainer**: a dictionary providing details about the maintainer of the implementation, MUST contain the single field: - - **email** with the maintainer's email address. +- **email** with the maintainer's email address. - - **issue\_tracker**: a `JSON API link `__ pointing to the implementation's issue tracker. +- **issue\_tracker**: a `JSON API link `__ pointing to the implementation's issue tracker. - - **warnings**: a list of warning resource objects representing non-critical errors or warnings. - A warning resource object is defined similarly to a `JSON:API error object `__, but MUST also include the field :field:`type`, which MUST have the value :field-val:`"warning"`. - The field :field:`detail` MUST be present and SHOULD contain a non-critical message, e.g., reporting unrecognized search attributes or deprecated features. - The field :field:`status`, representing an HTTP response status code, MUST NOT be present for a warning resource object. - This is an exclusive field for error resource objects. +- **warnings**: a list of warning resource objects representing non-critical errors or warnings. +A warning resource object is defined similarly to a `JSON:API error object `__, but MUST also include the field :field:`type`, which MUST have the value :field-val:`"warning"`. +The field :field:`detail` MUST be present and SHOULD contain a non-critical message, e.g., reporting unrecognized search attributes or deprecated features. +The field :field:`status`, representing an HTTP response status code, MUST NOT be present for a warning resource object. +This is an exclusive field for error resource objects. - Example for a deprecation warning: +Example for a deprecation warning: - .. code:: jsonc +.. code:: jsonc - { - "id": "dep_chemical_formula_01", - "type": "warning", - "code": "_exmpl_dep_chemical_formula", - "title": "Deprecation Warning", - "detail": "chemical_formula is deprecated, use instead chemical_formula_hill" - } +{ +"id": "dep_chemical_formula_01", +"type": "warning", +"code": "_exmpl_dep_chemical_formula", +"title": "Deprecation Warning", +"detail": "chemical_formula is deprecated, use instead chemical_formula_hill" +} - **Note**: warning :field:`id`\ s MUST NOT be trusted to identify the exceptional situations (i.e., they are not error codes), use instead the field :field:`code` for this. - Warning :field:`id`\ s can *only* be trusted to be unique in the list of warning resource objects, i.e., together with the :field:`type`. +**Note**: warning :field:`id`\ s MUST NOT be trusted to identify the exceptional situations (i.e., they are not error codes), use instead the field :field:`code` for this. +Warning :field:`id`\ s can *only* be trusted to be unique in the list of warning resource objects, i.e., together with the :field:`type`. - General OPTIMADE warning codes are specified in section `Warnings`_. +General OPTIMADE warning codes are specified in section `Warnings`_. - - Other OPTIONAL additional information *global to the query* that is not specified in this document, MUST start with a database-provider-specific prefix (see section `Database-Provider-Specific Namespace Prefixes`_). +- Other OPTIONAL additional information *global to the query* that is not specified in this document, MUST start with a database-provider-specific prefix (see section `Database-Provider-Specific Namespace Prefixes`_). - - Example for a request made to :query-url:`http://example.com/optimade/v1/structures/?filter=a=1 AND b=2`: +- Example for a request made to :query-url:`http://example.com/optimade/v1/structures/?filter=a=1 AND b=2`: - .. code:: jsonc - - { - "meta": { - "query": { - "representation": "/structures/?filter=a=1 AND b=2" - }, - "api_version": "1.0.0", - "schema": "http://schemas.optimade.org/openapi/v1/optimade.json", - "time_stamp": "2007-04-05T14:30:20Z", - "data_returned": 10, - "data_available": 10, - "more_data_available": false, - "provider": { - "name": "Example provider", - "description": "Provider used for examples, not to be assigned to a real database", - "prefix": "exmpl", - "homepage": "http://example.com" - }, - "implementation": { - "name": "exmpl-optimade", - "version": "0.1.0", - "source_url": "http://git.example.com/exmpl-optimade", - "maintainer": { - "email": "admin@example.com" - }, - "issue_tracker": "http://tracker.example.com/exmpl-optimade" - }, - "database": { - "id": "example_db", - "name": "Example database 1 (of many)", - "description": "The first example database in a series hosted by the Example Provider.", - "homepage": "http://database_one.example.com", - "maintainer": { - "email": "science_lead@example.com" - } - } - } - // ... - } - - - **schema**: a `JSON:API links object `__ that points to a schema for the response. - If it is a string, or a dictionary containing no :field:`meta` field, the provided URL MUST point at an `OpenAPI `__ schema. - It is possible that future versions of this specification allow for alternative schema types. - Hence, if the :field:`meta` field of the JSON:API links object is provided and contains a field :field:`schema_type` that is not equal to the string :field-val:`OpenAPI` the client MUST NOT handle failures to parse the schema or to validate the response against the schema as errors. +.. code:: jsonc - **Note**: The :field:`schema` field was previously RECOMMENDED in all responses, but is now demoted to being OPTIONAL since there now is a standard way of specifying a response schema in JSON:API through the :field:`describedby` subfield of the top-level :field:`links` field. +{ +"meta": { +"query": { +"representation": "/structures/?filter=a=1 AND b=2" +}, +"api_version": "1.0.0", +"schema": "http://schemas.optimade.org/openapi/v1/optimade.json", +"time_stamp": "2007-04-05T14:30:20Z", +"data_returned": 10, +"data_available": 10, +"more_data_available": false, +"provider": { +"name": "Example provider", +"description": "Provider used for examples, not to be assigned to a real database", +"prefix": "exmpl", +"homepage": "http://example.com" +}, +"implementation": { +"name": "exmpl-optimade", +"version": "0.1.0", +"source_url": "http://git.example.com/exmpl-optimade", +"maintainer": { +"email": "admin@example.com" +}, +"issue_tracker": "http://tracker.example.com/exmpl-optimade" +}, +"database": { +"id": "example_db", +"name": "Example database 1 (of many)", +"description": "The first example database in a series hosted by the Example Provider.", +"homepage": "http://database_one.example.com", +"maintainer": { +"email": "science_lead@example.com" +} +} +} +// ... +} + +- **schema**: a `JSON:API links object `__ that points to a schema for the response. +If it is a string, or a dictionary containing no :field:`meta` field, the provided URL MUST point at an `OpenAPI `__ schema. +It is possible that future versions of this specification allow for alternative schema types. +Hence, if the :field:`meta` field of the JSON:API links object is provided and contains a field :field:`schema_type` that is not equal to the string :field-val:`OpenAPI` the client MUST NOT handle failures to parse the schema or to validate the response against the schema as errors. + +**Note**: The :field:`schema` field was previously RECOMMENDED in all responses, but is now demoted to being OPTIONAL since there now is a standard way of specifying a response schema in JSON:API through the :field:`describedby` subfield of the top-level :field:`links` field. - **data**: The schema of this value varies by endpoint, it can be either a *single* `JSON:API resource object `__ or a *list* of JSON:API resource objects. - Every resource object needs the :field:`type` and :field:`id` fields, and its attributes (described in section `API Endpoints`_) need to be in a dictionary corresponding to the :field:`attributes` field. +Every resource object needs the :field:`type` and :field:`id` fields, and its attributes (described in section `API Endpoints`_) need to be in a dictionary corresponding to the :field:`attributes` field. - Every resource object MAY also contain a :field:`meta` field which MAY contain the following keys: +Every resource object MAY also contain a :field:`meta` field which MAY contain the following keys: - - **property_metadata**: an object containing per-entry and per-property metadata. - The keys are the names of the fields in :field:`attributes` for which metadata is available. - The values belonging to these keys are dictionaries containing the relevant metadata fields. - See also `Metadata properties`_ +- **property_metadata**: an object containing per-entry and per-property metadata. +The keys are the names of the fields in :field:`attributes` for which metadata is available. +The values belonging to these keys are dictionaries containing the relevant metadata fields. +See also `Metadata properties`_ - - **partial_data_links**: an object used to list links which can be used to fetch data that has been omitted from the :field:`data` part of the response. - The keys are the names of the fields in :field:`attributes` for which partial data links are available. - Each value is a list of objects that MUST have the following keys: +- **partial_data_links**: an object used to list links which can be used to fetch data that has been omitted from the :field:`data` part of the response. +The keys are the names of the fields in :field:`attributes` for which partial data links are available. +Each value is a list of objects that MUST have the following keys: - - **format**: String. - The name of the format provided via this link. - For one of the objects this :field:`format` field SHOULD have the value "jsonlines", which refers to the format in `OPTIMADE JSON lines partial data format`_. +- **format**: String. +The name of the format provided via this link. +For one of the objects this :field:`format` field SHOULD have the value "jsonlines", which refers to the format in `OPTIMADE JSON lines partial data format`_. - - **link**: String. - A `JSON API link `__ that points to a location from which the omitted data can be fetched. - There is no requirement on the syntax or format for the link URL. +- **link**: String. +A `JSON API link `__ that points to a location from which the omitted data can be fetched. +There is no requirement on the syntax or format for the link URL. - For more information about the mechanism to transmit large property values, including an example of the format of :field:`partial_data_links`, see `Transmission of large property values`_. +For more information about the mechanism to transmit large property values, including an example of the format of :field:`partial_data_links`, see `Transmission of large property values`_. The response MAY also return resources related to the primary data in the field: - **links**: a `JSON API links object `__ is REQUIRED for implementing pagination. - (see section `Entry Listing URL Query Parameters`_.) - Each field of a links object, i.e., a "link", MUST be one of: +(see section `Entry Listing URL Query Parameters`_.) +Each field of a links object, i.e., a "link", MUST be one of: - - :field-val:`null` - - a string representing a URI, or - - a dictionary ("link object") with fields +- :field-val:`null` +- a string representing a URI, or +- a dictionary ("link object") with fields - - **href**: a string representing a URI - - **meta**: (OPTIONAL) a meta object containing non-standard meta-information about the link +- **href**: a string representing a URI +- **meta**: (OPTIONAL) a meta object containing non-standard meta-information about the link - Example links objects: +Example links objects: - - **base\_url**: a links object representing the base URL of the implementation. Example: +- **base\_url**: a links object representing the base URL of the implementation. Example: - .. code:: jsonc +.. code:: jsonc - { - "links": { - "base_url": { - "href": "http://example.com/optimade", - "meta": { - "_exmpl_db_version": "3.2.1" - } - } - // ... - } - // ... - } +{ +"links": { +"base_url": { +"href": "http://example.com/optimade", +"meta": { +"_exmpl_db_version": "3.2.1" +} +} +// ... +} +// ... +} - The :field:`links` field SHOULD include the following links objects: +The :field:`links` field SHOULD include the following links objects: - - **describedby**: a links object giving the URL for a schema that describes the response. - The URL SHOULD resolve into a JSON formatted response returning a JSON object with top level :field:`$schema` and/or :field:`$id` fields that can be used by the client to identify the schema format. +- **describedby**: a links object giving the URL for a schema that describes the response. +The URL SHOULD resolve into a JSON formatted response returning a JSON object with top level :field:`$schema` and/or :field:`$id` fields that can be used by the client to identify the schema format. - **Note**: This field is the standard facility in JSON:API to communicate a response schema. - It overlaps in function with the field :field:`schema` in the top level :field:`meta` field. +**Note**: This field is the standard facility in JSON:API to communicate a response schema. +It overlaps in function with the field :field:`schema` in the top level :field:`meta` field. - The following fields are REQUIRED for implementing pagination: +The following fields are REQUIRED for implementing pagination: - - **next**: represents a link to fetch the next set of results. - When the current response is the last page of data, this field MUST be either omitted or :field-val:`null`\ -valued. +- **next**: represents a link to fetch the next set of results. +When the current response is the last page of data, this field MUST be either omitted or :field-val:`null`\ -valued. - An implementation MAY also use the following reserved fields for pagination. - They represent links in a similar way as for :field:`next`. +An implementation MAY also use the following reserved fields for pagination. +They represent links in a similar way as for :field:`next`. - - **prev**: the previous page of data. :field-val:`null` or omitted when the current response is the first page of data. - - **last**: the last page of data. - - **first**: the first page of data. +- **prev**: the previous page of data. :field-val:`null` or omitted when the current response is the first page of data. +- **last**: the last page of data. +- **first**: the first page of data. - Finally, the :field:`links` field MAY also include the following links object: +Finally, the :field:`links` field MAY also include the following links object: - - **self**: a links object giving the URL from which the response was obtained. +- **self**: a links object giving the URL from which the response was obtained. - **included**: a list of `JSON:API resource objects `__ related to the primary data contained in :field:`data`. - Responses that contain related resources under :field:`included` are known as `compound documents `__ in the JSON:API. +Responses that contain related resources under :field:`included` are known as `compound documents `__ in the JSON:API. - The definition of this field is found in the `JSON:API specification `__. - Specifically, if the query parameter :query-param:`include` is included in the request, :field:`included` MUST NOT include unrequested resource objects. - For further information on the parameter :query-param:`include`, see section `Entry Listing URL Query Parameters`_. +The definition of this field is found in the `JSON:API specification `__. +Specifically, if the query parameter :query-param:`include` is included in the request, :field:`included` MUST NOT include unrequested resource objects. +For further information on the parameter :query-param:`include`, see section `Entry Listing URL Query Parameters`_. - This value MUST be either an empty array or an array of related resource objects. +This value MUST be either an empty array or an array of related resource objects. If there were errors in producing the response all other fields MAY be present, but the top-level :field:`data` field MUST be skipped, and the following field MUST be present: - **errors**: a list of `JSON:API error objects `__, where the field :field:`detail` MUST be present. - All other fields are OPTIONAL. +All other fields are OPTIONAL. An example of a full response: .. code:: jsonc - { - "links": { - "next": null, - "base_url": { - "href": "http://example.com/optimade", - "meta": { - "_exmpl_db_version": "3.2.1" - } - } - }, - "meta": { - "query": { - "representation": "/structures?filter=a=1 AND b=2" - }, - "api_version": "1.0.0", - "time_stamp": "2007-04-05T14:30:20Z", - "data_returned": 10, - "data_available": 10, - "last_id": "xy10", - "more_data_available": false, - "provider": { - "name": "Example provider", - "description": "Provider used for examples, not to be assigned to a real database", - "prefix": "exmpl", - "homepage": { - "href": "http://example.com", - "meta": { - "_exmpl_title": "This is an example site" - } - } - }, - // - }, - "data": [ - // ... - ], - "included": [ - // ... - ] - } +{ +"links": { +"next": null, +"base_url": { +"href": "http://example.com/optimade", +"meta": { +"_exmpl_db_version": "3.2.1" +} +} +}, +"meta": { +"query": { +"representation": "/structures?filter=a=1 AND b=2" +}, +"api_version": "1.0.0", +"time_stamp": "2007-04-05T14:30:20Z", +"data_returned": 10, +"data_available": 10, +"last_id": "xy10", +"more_data_available": false, +"provider": { +"name": "Example provider", +"description": "Provider used for examples, not to be assigned to a real database", +"prefix": "exmpl", +"homepage": { +"href": "http://example.com", +"meta": { +"_exmpl_title": "This is an example site" +} +} +}, +// +}, +"data": [ +// ... +], +"included": [ +// ... +] +} - **@context**: A JSON-LD context that enables interpretation of data in the response as linked data. - If provided, it SHOULD be one of the following: +If provided, it SHOULD be one of the following: - - An object conforming to a JSON-LD standard, which includes a :field:`@version` field specifying the version of the standard. - - A string containing a URL that resolves to such an object. +- An object conforming to a JSON-LD standard, which includes a :field:`@version` field specifying the version of the standard. +- A string containing a URL that resolves to such an object. - **jsonapi**: A `JSON:API object `__. - The :field:`version` subfield SHOULD be :field-val:`"1.1"`. - The :field:`meta` subfield SHOULD be included and contain the following subfields: +The :field:`version` subfield SHOULD be :field-val:`"1.1"`. +The :field:`meta` subfield SHOULD be included and contain the following subfields: - - **api**: A string with the value "OPTIMADE". - - **api-version**: A string with the full version of the OPTIMADE standard that the processing and response adheres to. - This MAY be the version indicated at the top of this document, but MAY also be another version if the client, e.g., has used the query parameter :query-param:`api_hint` to request processing according to another version. +- **api**: A string with the value "OPTIMADE". +- **api-version**: A string with the full version of the OPTIMADE standard that the processing and response adheres to. +This MAY be the version indicated at the top of this document, but MAY also be another version if the client, e.g., has used the query parameter :query-param:`api_hint` to request processing according to another version. - If the server is able to handle serialization in such a way that it can dictate the order of the top level object members in the response, it is RECOMMENDED to put the :field:`jsonapi` as the first top level member to simplify identification of the response. +If the server is able to handle serialization in such a way that it can dictate the order of the top level object members in the response, it is RECOMMENDED to put the :field:`jsonapi` as the first top level member to simplify identification of the response. HTTP Response Status Codes -------------------------- @@ -1160,9 +1160,9 @@ Example response: .. code:: CSV - version - 1 - 0 +version +1 +0 The above response means that the API versions 1 and 0 are served under the versioned base URLs :query-url:`/v1` and :query-url:`/v0`, respectively. The order of the versions indicates that the API implementation regards version 1 as preferred over version 0. @@ -1178,14 +1178,14 @@ Each entry in the list includes a set of properties and their corresponding valu The section `Entry list`_ specifies properties as belonging to one of three categories: 1. Properties marked as REQUIRED in the response. - These properties MUST always be present for all entries in the response. +These properties MUST always be present for all entries in the response. 2. Properties marked as REQUIRED only if the query parameter :query-param:`response_fields` is not part of the request, or if they are explicitly requested in :query-param:`response_fields`. - Otherwise they MUST NOT be included. - One can think of these properties as constituting a default value for :query-param:`response_fields` when that parameter is omitted. +Otherwise they MUST NOT be included. +One can think of these properties as constituting a default value for :query-param:`response_fields` when that parameter is omitted. 3. Properties not marked as REQUIRED in any case, MUST be included only if explicitly requested in the query parameter :query-param:`response_fields`. - Otherwise they SHOULD NOT be included. +Otherwise they SHOULD NOT be included. Examples of valid entry listing endpoint URLs: @@ -1214,65 +1214,65 @@ Standard OPTIONAL URL query parameters standardized by the JSON:API specificatio - **filter**: a filter string, in the format described below in section `API Filtering Format Specification`_. - **page\_limit**: sets a numerical limit on the number of entries returned. - See `JSON:API 1.1 `__. - The API implementation MUST return no more than the number specified. - It MAY return fewer. - The database MAY have a maximum limit and not accept larger numbers (in which case the :http-error:`403 Forbidden` error code MUST be returned). - The default limit value is up to the API implementation to decide. - Example: :query-url:`http://example.com/optimade/v1/structures?page_limit=100` +See `JSON:API 1.1 `__. +The API implementation MUST return no more than the number specified. +It MAY return fewer. +The database MAY have a maximum limit and not accept larger numbers (in which case the :http-error:`403 Forbidden` error code MUST be returned). +The default limit value is up to the API implementation to decide. +Example: :query-url:`http://example.com/optimade/v1/structures?page_limit=100` - **page\_{offset, number, cursor, above, below}**: A server MUST implement pagination in the case of no user-specified :query-param:`sort` parameter (via the :field:`links` response field, see section `JSON Response Schema: Common Fields`_). - A server MAY implement pagination in concert with :query-param:`sort`. - The following parameters, all prefixed by "page\_", are RECOMMENDED for use with pagination. - If an implementation chooses +A server MAY implement pagination in concert with :query-param:`sort`. +The following parameters, all prefixed by "page\_", are RECOMMENDED for use with pagination. +If an implementation chooses - - *offset-based pagination*: using :field:`page_offset` and :field:`page_limit` is RECOMMENDED. - - *cursor-based pagination*: using :field:`page_cursor` and :field:`page_limit` is RECOMMENDED. - - *page-based pagination*: using :field:`page_number` and :field:`page_limit` is RECOMMENDED. It is RECOMMENDED that the first page has number 1, i.e., that :field:`page_number` is 1-based. - - *value-based pagination*: using :field:`page_above`/:field:`page_below` and :field:`page_limit` is RECOMMENDED. +- *offset-based pagination*: using :field:`page_offset` and :field:`page_limit` is RECOMMENDED. +- *cursor-based pagination*: using :field:`page_cursor` and :field:`page_limit` is RECOMMENDED. +- *page-based pagination*: using :field:`page_number` and :field:`page_limit` is RECOMMENDED. It is RECOMMENDED that the first page has number 1, i.e., that :field:`page_number` is 1-based. +- *value-based pagination*: using :field:`page_above`/:field:`page_below` and :field:`page_limit` is RECOMMENDED. - Examples (all OPTIONAL behavior a server MAY implement): +Examples (all OPTIONAL behavior a server MAY implement): - - skip 50 structures and fetch up to 100: :query-url:`/structures?page_offset=50&page_limit=100`. - - fetch page 2 of up to 50 structures per page: :query-url:`/structures?page_number=2&page_limit=50`. - - fetch up to 100 structures above sort-field value 4000 (in this example, server chooses to fetch results sorted by increasing :field:`id`, so :field:`page_above` value refers to an :field:`id` value): :query-url:`/structures?page_above=4000&page_limit=100`. +- skip 50 structures and fetch up to 100: :query-url:`/structures?page_offset=50&page_limit=100`. +- fetch page 2 of up to 50 structures per page: :query-url:`/structures?page_number=2&page_limit=50`. +- fetch up to 100 structures above sort-field value 4000 (in this example, server chooses to fetch results sorted by increasing :field:`id`, so :field:`page_above` value refers to an :field:`id` value): :query-url:`/structures?page_above=4000&page_limit=100`. - **sort**: If supporting sortable queries, an implementation MUST use the :query-param:`sort` query parameter with format as specified by `JSON:API 1.1 `__. - An implementation MAY support multiple sort fields for a single query. - If it does, it again MUST conform to the JSON:API 1.1 specification. +An implementation MAY support multiple sort fields for a single query. +If it does, it again MUST conform to the JSON:API 1.1 specification. - If an implementation supports sorting for an `entry listing endpoint `_, then the :endpoint:`/info/` endpoint MUST include, for each field name :field:`` in its :field:`data.properties.` response value that can be used for sorting, the key :field:`sortable` with value :field-val:`true`. - If a field name under an entry listing endpoint supporting sorting cannot be used for sorting, the server MUST either leave out the :field:`sortable` key or set it equal to :field-val:`false` for the specific field name. - The set of field names, with :field:`sortable` equal to :field-val:`true` are allowed to be used in the "sort fields" list according to its definition in the JSON:API 1.1 specification. - The field :field:`sortable` is in addition to each property description and other OPTIONAL fields. - An example is shown in section `Entry Listing Info Endpoints`_. +If an implementation supports sorting for an `entry listing endpoint `_, then the :endpoint:`/info/` endpoint MUST include, for each field name :field:`` in its :field:`data.properties.` response value that can be used for sorting, the key :field:`sortable` with value :field-val:`true`. +If a field name under an entry listing endpoint supporting sorting cannot be used for sorting, the server MUST either leave out the :field:`sortable` key or set it equal to :field-val:`false` for the specific field name. +The set of field names, with :field:`sortable` equal to :field-val:`true` are allowed to be used in the "sort fields" list according to its definition in the JSON:API 1.1 specification. +The field :field:`sortable` is in addition to each property description and other OPTIONAL fields. +An example is shown in section `Entry Listing Info Endpoints`_. - **include**: A server MAY implement the JSON:API concept of returning `compound documents `__ by utilizing the :query-param:`include` query parameter as specified by `JSON:API 1.0 `__. - All related resource objects MUST be returned as part of an array value for the top-level :field:`included` field, see section `JSON Response Schema: Common Fields`_. +All related resource objects MUST be returned as part of an array value for the top-level :field:`included` field, see section `JSON Response Schema: Common Fields`_. - The value of :query-param:`include` MUST be a comma-separated list of "relationship paths", as defined in the `JSON:API `__. - If relationship paths are not supported, or a server is unable to identify a relationship path a :http-error:`400 Bad Request` response MUST be made. +The value of :query-param:`include` MUST be a comma-separated list of "relationship paths", as defined in the `JSON:API `__. +If relationship paths are not supported, or a server is unable to identify a relationship path a :http-error:`400 Bad Request` response MUST be made. - The **default value** for :query-param:`include` is :query-val:`references`. - This means :entry:`references` entries MUST always be included under the top-level field :field:`included` as default, since a server assumes if :query-param:`include` is not specified by a client in the request, it is still specified as :query-string:`include=references`. - Note, if a client explicitly specifies :query-param:`include` and leaves out :query-val:`references`, :entry:`references` resource objects MUST NOT be included under the top-level field :field:`included`, as per the definition of :field:`included`, see section `JSON Response Schema: Common Fields`_. +The **default value** for :query-param:`include` is :query-val:`references`. +This means :entry:`references` entries MUST always be included under the top-level field :field:`included` as default, since a server assumes if :query-param:`include` is not specified by a client in the request, it is still specified as :query-string:`include=references`. +Note, if a client explicitly specifies :query-param:`include` and leaves out :query-val:`references`, :entry:`references` resource objects MUST NOT be included under the top-level field :field:`included`, as per the definition of :field:`included`, see section `JSON Response Schema: Common Fields`_. - **Note**: A query with the parameter :query-param:`include` set to the empty string means no related resource objects are to be returned under the top-level field :field:`included`. +**Note**: A query with the parameter :query-param:`include` set to the empty string means no related resource objects are to be returned under the top-level field :field:`included`. Standard OPTIONAL URL query parameters not in the JSON:API specification: - **response\_format**: the output format requested (see section `Response Format`_). - Defaults to the format string 'json', which specifies the standard output format described in this specification. - Example: :query-url:`http://example.com/optimade/v1/structures?response_format=xml` +Defaults to the format string 'json', which specifies the standard output format described in this specification. +Example: :query-url:`http://example.com/optimade/v1/structures?response_format=xml` - **email\_address**: an email address of the user making the request. - The email SHOULD be that of a person and not an automatic system. - Example: :query-url:`http://example.com/optimade/v1/structures?email_address=user@example.com` +The email SHOULD be that of a person and not an automatic system. +Example: :query-url:`http://example.com/optimade/v1/structures?email_address=user@example.com` - **response\_fields**: a comma-delimited set of fields to be provided in the output. - If provided, these fields MUST be returned along with the REQUIRED fields. - Other OPTIONAL fields MUST NOT be returned when this parameter is present. - Example: :query-url:`http://example.com/optimade/v1/structures?response_fields=last_modified,nsites` +If provided, these fields MUST be returned along with the REQUIRED fields. +Other OPTIONAL fields MUST NOT be returned when this parameter is present. +Example: :query-url:`http://example.com/optimade/v1/structures?response_fields=last_modified,nsites` Additional OPTIONAL URL query parameters not described above are not considered to be part of this standard, and are instead considered to be "custom URL query parameters". These custom URL query parameters MUST be of the format "". @@ -1287,8 +1287,8 @@ Examples: - :query-url:`http://example.com/optimade/v1/structures?_exmpl_warning_verbosity=10` - :query-url:`http://example.com/optimade/v1/structures?_exmpl_filter="elements all in [Al, Si, Ga]"` - **Note**: the specification presently makes no attempt to standardize access control mechanisms. - There are security concerns with access control based on URL tokens, and the above example is not to be taken as a recommendation for such a mechanism. +**Note**: the specification presently makes no attempt to standardize access control mechanisms. +There are security concerns with access control based on URL tokens, and the above example is not to be taken as a recommendation for such a mechanism. Entry Listing JSON Response Schema ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -1301,51 +1301,51 @@ In the default JSON response format every dictionary (`resource object `__ can OPTIONALLY contain the field - - **self**: the entry's URL +- **self**: the entry's URL - **meta**: a `JSON API meta object `__ that is used to communicate metadata. - See `JSON Response Schema: Common Fields`_ for more information about this field. +See `JSON Response Schema: Common Fields`_ for more information about this field. - **relationships**: a dictionary containing references to other entries according to the description in section `Relationships`_ encoded as `JSON:API Relationships `__. - The OPTIONAL human-readable description of the relationship MAY be provided in the :field:`description` field inside the :field:`meta` dictionary of the JSON:API resource identifier object. - All relationships to entries of the same entry type MUST be grouped into the same JSON:API relationship object and placed in the relationships dictionary with the entry type name as key (e.g., :entry:`structures`). +The OPTIONAL human-readable description of the relationship MAY be provided in the :field:`description` field inside the :field:`meta` dictionary of the JSON:API resource identifier object. +All relationships to entries of the same entry type MUST be grouped into the same JSON:API relationship object and placed in the relationships dictionary with the entry type name as key (e.g., :entry:`structures`). Example: .. code:: jsonc - { - "data": [ - { - "type": "structures", - "id": "example.db:structs:0001", - "attributes": { - "chemical_formula_descriptive": "Es2 O3", - "url": "http://example.db/structs/0001", - "immutable_id": "http://example.db/structs/0001@123", - "last_modified": "2007-04-05T14:30:20Z" - } - }, - { - "type": "structures", - "id": "example.db:structs:1234", - "attributes": { - "chemical_formula_descriptive": "Es2", - "url": "http://example.db/structs/1234", - "immutable_id": "http://example.db/structs/1234@123", - "last_modified": "2007-04-07T12:02:20Z" - } - } - // ... - ] - // ... - } +{ +"data": [ +{ +"type": "structures", +"id": "example.db:structs:0001", +"attributes": { +"chemical_formula_descriptive": "Es2 O3", +"url": "http://example.db/structs/0001", +"immutable_id": "http://example.db/structs/0001@123", +"last_modified": "2007-04-05T14:30:20Z" +} +}, +{ +"type": "structures", +"id": "example.db:structs:1234", +"attributes": { +"chemical_formula_descriptive": "Es2", +"url": "http://example.db/structs/1234", +"immutable_id": "http://example.db/structs/1234@123", +"last_modified": "2007-04-07T12:02:20Z" +} +} +// ... +] +// ... +} Single Entry Endpoints ---------------------- @@ -1373,25 +1373,26 @@ The meaning of these URL query parameters are as defined above in section `Entry One additional query parameter :query-param:`property_slices` MUST be handled by the API implementation either as defined below or by returning the error :http-error:`501 Not Implemented`: -- **property\_slices**: A number of slice specifications to specify parts of array properties for the functionality described in `Slices of array properties`_. +- **property\_slices**: A number of slice specifications to request only parts of array properties for the functionality described in `Slices of array properties`_. + + The query parameter contains a comma-separated (",", ASCII 44(0x2C)) list of slice specifications. + Each slice specification consists of an ordered sequence of four elements separated by the colon character (":", ASCII 58(0x3A)). + The elements in the sequence are the dimension name and the three components of the slice, i.e., the start, stop, and step values defined in the same way as for a `slice object`_. + Omitting the value for any of the components of the slice specifies a default value (however, all the colon separators MUST be included). + The start value specifies the first index in that dimension for which values should be returned (which is 0-based and inclusive). + The default is :val:`0`. + The stop value specifies the last index for which values should be returned (inclusive). + The default is the last index of the array along the specified dimension. + The step value specifies the step size in that dimension. + The default is :val:`1`. - The query parameter contains a comma-separated (",", ASCII 44(0x2C)) list of slice specification. - Each slice specification consists of a dimension name and the start, stop and step values of the slice separated by the colon character (":", ASCII 58(0x3A)) - Omitting the value between two colons specifies a default value. - The components of the slice are defined in the same way as for a `slice object`_. - The first integer specifies the start of the slice, i.e. the first index in that dimension for which values should be returned. - The default is the index of the first value. - The second integer specifies the end/stop of the slice, i.e. the last index for which values should be returned. - The default is the index of the last value of the property. - The third integer specifies the step size in that dimension. - The default is :val:`1` - The slices are 0-based, i.e. the first value has index 0, and inclusive i.e. for the dimension :val:`dim_frames` the range :val:`dim_frames:10:20:1` the last value returned belongs to index 20. + An empty value of the :query-param:`property_slices` query parameter MUST be interpreted as equivalent to the query parameter not being included in the request. Requirements and conventions for the response when this query parameter is used are described in `Slices of array properties`_. Example: - - :query-url:`http://optimade.example.com/v1/trajectories/id_12345?response_fields=cartesian_site_positions&property_ranges=dim_frames::999:10,dim_sites:30:70::` + - :query-url:`http://optimade.example.com/v1/trajectories/id_12345?response_fields=cartesian_site_positions&property_ranges=dim_frames::999:10,dim_sites:30:70:` This query URL requests items from the array :field:`cartesian_site_positions` is requested only for the 31st to 71st sites (i.e., with indexes 30 through 70 inclusive) for 1 out of every 10 frames of the first 1000 frames (i.e., taking steps of 10 over indexes 0 through 999 inclusive, which requests the frames with index 0, 10, 20, 30, ..., 990) of a trajectory with ID :val:`id_12345`. From e562f4922ffbfb5652815c5fa356359fb182d171 Mon Sep 17 00:00:00 2001 From: Rickard Armiento Date: Fri, 14 Jun 2024 10:20:32 +0200 Subject: [PATCH 22/34] Revert "Improve handing of Python dependencies" This reverts commit 266545b3c0279c20eae243c0990120c667f8f8e9. --- optimade.rst | 801 +++++++++++++++++++++++++-------------------------- 1 file changed, 400 insertions(+), 401 deletions(-) diff --git a/optimade.rst b/optimade.rst index 4e688f709..35c37ea7d 100644 --- a/optimade.rst +++ b/optimade.rst @@ -660,106 +660,106 @@ The field :field:`array_axes` is defined as follows: - ``{"start": 0}`` (despite that :val:`0` is the default value of the :field:`start` field.) But it is instead a valid representation if the query parameter was :query-string:`property_slices=dim_frames:0::`. -- :query-string:`property_slices=dim_frames:3:5:2` the representation MUST be: + - :query-string:`property_slices=dim_frames:3:5:2` the representation MUST be: -- ``{"start": 3, "stop": 5, "step": 2}`` + - ``{"start": 3, "stop": 5, "step": 2}`` -The dictionary MAY contain the following fields: + The dictionary MAY contain the following fields: -- :field:`length`: Integer. -The length of this array axis which MUST be the same as the length of the declared dimension for this axis in the corresponding property definition. -Note that the length of a dimension can be different for different entries if the length is not explicitly declared by the property definition. -For example, the number of frames, i.e., the length of the dimension named ``dim_frames``, is generally different for different trajectories. + - :field:`length`: Integer. + The length of this array axis which MUST be the same as the length of the declared dimension for this axis in the corresponding property definition. + Note that the length of a dimension can be different for different entries if the length is not explicitly declared by the property definition. + For example, the number of frames, i.e., the length of the dimension named ``dim_frames``, is generally different for different trajectories. -- :field:`sliceable`: Boolean. -If :val:`true`, the server MUST handle slices for that dimension. -If :val:`false`, the server MAY handle slices for that dimension, or MAY return the error :http-error:`501 Not Implemented` when a client requests a slice involving this dimension. -If the field is omitted or :val:`null`, it means the same thing as :val:`false`. + - :field:`sliceable`: Boolean. + If :val:`true`, the server MUST handle slices for that dimension. + If :val:`false`, the server MAY handle slices for that dimension, or MAY return the error :http-error:`501 Not Implemented` when a client requests a slice involving this dimension. + If the field is omitted or :val:`null`, it means the same thing as :val:`false`. -- :field:`available_slice`: Dictionary or :val:`null`. -This field describes a `slice object`_ where there MAY be non-null values in the data. -By including this field, the server certifies that there are only :val:`null` values outside this slice. -If not provided, or equal to :val:`null` or an empty dictionary, the client cannot make any assumptions about what part of the array contains :val:`null` values. + - :field:`available_slice`: Dictionary or :val:`null`. + This field describes a `slice object`_ where there MAY be non-null values in the data. + By including this field, the server certifies that there are only :val:`null` values outside this slice. + If not provided, or equal to :val:`null` or an empty dictionary, the client cannot make any assumptions about what part of the array contains :val:`null` values. -Examples: + Examples: -- ``{"start": 3, "stop": 7, "step": 2}`` means the server certifies that values at indexes 0,1,2,4,6 and any index from 8 to the end of the array are :val:`null`. + - ``{"start": 3, "stop": 7, "step": 2}`` means the server certifies that values at indexes 0,1,2,4,6 and any index from 8 to the end of the array are :val:`null`. Below follows an example of the :field:`data` and :field:`meta` parts of a response using the JSON response format for a request to the trajectory endpoint with the query parameter :query-param:`property_slices=dim_frames:3:37:5` and :query-param:`response_fields=cartesian_site_positions,_exmpl_temperature` where the trajectory consists of 432934 frames (with indexes 0 to 432933) and where the :field:`cartisian_site_positions` contains 7 sites. Furthermore, the :field:`_exmpl_temperature` contains only :val:`null` values except for items with indexes 1000, 1030, 1060, ..., 4000 (where the values can be either numeric or :val:`null`). .. code:: jsonc -{ -// ... -"data": { -"type": "trajectories", -"id": "2345678", -"attributes": { -"cartesian_site_positions": null, -"_exmpl_temperature": null -}, -"meta": { -"property_metadata": { -"cartesian_site_positions": { -"array_axes": [ - { - "dimension_name": "dim_frames", - "requested_slice": { - "start": 3, - "stop": 37, - "step": 5 - }, - "available_slice": { - "start": 0, - "stop": 432933, - "step": 1 - }, - "sliceable": true, - "length": 432934, - }, - { - "dimension_name": "dim_sites", - "available_slice": { - "start": 0, - "stop": 6, - "step": 1 - }, - "sliceable": false, - "length": 7, - }, - { - "dimension_name": "dim_spatial", - "length": 3, - } -], -}, -"_exmpl_temperature": { -"array_axes": [ - { - "dimension_name": "dim_frames", - "requested_slice": { - "start": 3, - "stop": 37, - "step": 5 - }, - "available_slice": { - "start": 1000, - "stop": 4000, - "step": 30 - }, - "sliceable": true, - "length": 432934, - } -] -} -}, -"partial_data_links": { -//... -} -} -} -// ... -} + { + // ... + "data": { + "type": "trajectories", + "id": "2345678", + "attributes": { + "cartesian_site_positions": null, + "_exmpl_temperature": null + }, + "meta": { + "property_metadata": { + "cartesian_site_positions": { + "array_axes": [ + { + "dimension_name": "dim_frames", + "requested_slice": { + "start": 3, + "stop": 37, + "step": 5 + }, + "available_slice": { + "start": 0, + "stop": 432933, + "step": 1 + }, + "sliceable": true, + "length": 432934, + }, + { + "dimension_name": "dim_sites", + "available_slice": { + "start": 0, + "stop": 6, + "step": 1 + }, + "sliceable": false, + "length": 7, + }, + { + "dimension_name": "dim_spatial", + "length": 3, + } + ], + }, + "_exmpl_temperature": { + "array_axes": [ + { + "dimension_name": "dim_frames", + "requested_slice": { + "start": 3, + "stop": 37, + "step": 5 + }, + "available_slice": { + "start": 1000, + "stop": 4000, + "step": 30 + }, + "sliceable": true, + "length": 432934, + } + ] + } + }, + "partial_data_links": { + //... + } + } + } + // ... + } Responses ========= @@ -790,305 +790,305 @@ In the JSON response format, property types translate as follows: Every response SHOULD contain the following fields, and MUST contain at least :field:`meta`: - **meta**: a `JSON:API meta member `__ that contains JSON:API meta objects of non-standard meta-information. -It MUST be a dictionary with these fields: + It MUST be a dictionary with these fields: -- **api\_version**: a string containing the full version of the API implementation. -The version number string MUST NOT be prefixed by, e.g., "v". -Examples: :field-val:`1.0.0`, :field-val:`1.0.0-rc.2`. + - **api\_version**: a string containing the full version of the API implementation. + The version number string MUST NOT be prefixed by, e.g., "v". + Examples: :field-val:`1.0.0`, :field-val:`1.0.0-rc.2`. -- **query**: information on the query that was requested. -It MUST be a dictionary with this field: + - **query**: information on the query that was requested. + It MUST be a dictionary with this field: -- **representation**: a string with the part of the URL following the versioned or unversioned base URL that serves the API. -Query parameters that have not been used in processing the request MAY be omitted. -In particular, if no query parameters have been involved in processing the request, the query part of the URL MAY be excluded. -Example: :field-val:`/structures?filter=nelements=2`. + - **representation**: a string with the part of the URL following the versioned or unversioned base URL that serves the API. + Query parameters that have not been used in processing the request MAY be omitted. + In particular, if no query parameters have been involved in processing the request, the query part of the URL MAY be excluded. + Example: :field-val:`/structures?filter=nelements=2`. -- **more\_data\_available**: :field-val:`false` if the response contains all data for the request (e.g., a request issued to a single entry endpoint, or a :query-param:`filter` query at the last page of a paginated response) and :field-val:`true` if the response is incomplete in the sense that multiple objects match the request, and not all of them have been included in the response (e.g., a query with multiple pages that is not at the last page). + - **more\_data\_available**: :field-val:`false` if the response contains all data for the request (e.g., a request issued to a single entry endpoint, or a :query-param:`filter` query at the last page of a paginated response) and :field-val:`true` if the response is incomplete in the sense that multiple objects match the request, and not all of them have been included in the response (e.g., a query with multiple pages that is not at the last page). -:field:`meta` SHOULD also include these fields: + :field:`meta` SHOULD also include these fields: -- **time\_stamp**: a timestamp containing the date and time at which the query was executed. -- **data\_returned**: an integer containing the total number of data resource objects returned for the current :query-param:`filter` query, independent of pagination. -- **provider**: information on the database provider of the implementation. -It MUST be a dictionary with these fields: + - **time\_stamp**: a timestamp containing the date and time at which the query was executed. + - **data\_returned**: an integer containing the total number of data resource objects returned for the current :query-param:`filter` query, independent of pagination. + - **provider**: information on the database provider of the implementation. + It MUST be a dictionary with these fields: -- **name**: a short name for the database provider. -- **description**: a longer description of the database provider. -- **prefix**: database-provider-specific prefix (see section `Database-Provider-Specific Namespace Prefixes`_). + - **name**: a short name for the database provider. + - **description**: a longer description of the database provider. + - **prefix**: database-provider-specific prefix (see section `Database-Provider-Specific Namespace Prefixes`_). -:field:`provider` MAY include these fields: + :field:`provider` MAY include these fields: -- **homepage**: a `JSON API link `__, pointing to the homepage of the database provider, either directly as a string, or as an object which can contain the following fields: + - **homepage**: a `JSON API link `__, pointing to the homepage of the database provider, either directly as a string, or as an object which can contain the following fields: -- **href**: a string containing the homepage URL. -- **meta**: a meta object containing non-standard meta-information about the database provider's homepage. + - **href**: a string containing the homepage URL. + - **meta**: a meta object containing non-standard meta-information about the database provider's homepage. -:field:`meta` MAY also include these fields: + :field:`meta` MAY also include these fields: -- **data\_available**: an integer containing the total number of data resource objects available in the database for the endpoint. -- **last\_id**: a string containing the last ID returned. -- **response\_message**: response string from the server. -- **request\_delay**: a non-negative float giving time in seconds that the client is suggested to wait before issuing a subsequent request. + - **data\_available**: an integer containing the total number of data resource objects available in the database for the endpoint. + - **last\_id**: a string containing the last ID returned. + - **response\_message**: response string from the server. + - **request\_delay**: a non-negative float giving time in seconds that the client is suggested to wait before issuing a subsequent request. -Implementation note: the functionality of this field overlaps to some degree with features provided by the HTTP error :http-error:`429 Too Many Requests` and the `Retry-After HTTP header `__. -Implementations are suggested to provide consistent handling of request overload through both mechanisms. + Implementation note: the functionality of this field overlaps to some degree with features provided by the HTTP error :http-error:`429 Too Many Requests` and the `Retry-After HTTP header `__. + Implementations are suggested to provide consistent handling of request overload through both mechanisms. -- **database**: a dictionary describing the specific database accessible at this OPTIMADE API. -If provided, the dictionary fields SHOULD match those provided in the corresponding links entry for the database in the provider's index meta-database, outlined in `Links Endpoint JSON Response Schema`_. -The dictionary can contain the following OPTIONAL fields: + - **database**: a dictionary describing the specific database accessible at this OPTIMADE API. + If provided, the dictionary fields SHOULD match those provided in the corresponding links entry for the database in the provider's index meta-database, outlined in `Links Endpoint JSON Response Schema`_. + The dictionary can contain the following OPTIONAL fields: -- **id**: the identifier of this database within those served by this provider, i.e., the ID under which this database is served in this provider's index meta-database. -- **name**: a human-readable name for the database, e.g., for use in clients. -- **version**: a string describing the version of the database. -- **description**: a human-readable description of the database, e.g., for use in clients. -- **homepage**: a `JSON API link `__, pointing to a homepage for the particular database. -- **maintainer**: a dictionary providing details about the maintainer of the database, which MUST contain the single field: + - **id**: the identifier of this database within those served by this provider, i.e., the ID under which this database is served in this provider's index meta-database. + - **name**: a human-readable name for the database, e.g., for use in clients. + - **version**: a string describing the version of the database. + - **description**: a human-readable description of the database, e.g., for use in clients. + - **homepage**: a `JSON API link `__, pointing to a homepage for the particular database. + - **maintainer**: a dictionary providing details about the maintainer of the database, which MUST contain the single field: -- **email** with the maintainer's email address. + - **email** with the maintainer's email address. -- **implementation**: a dictionary describing the server implementation, containing the OPTIONAL fields: + - **implementation**: a dictionary describing the server implementation, containing the OPTIONAL fields: -- **name**: name of the implementation. -- **version**: version string of the current implementation. -- **homepage**: a `JSON API link `__, pointing to the homepage of the implementation. -- **source\_url**: a `JSON API link `__ pointing to the implementation source, either downloadable archive or version control system. + - **name**: name of the implementation. + - **version**: version string of the current implementation. + - **homepage**: a `JSON API link `__, pointing to the homepage of the implementation. + - **source\_url**: a `JSON API link `__ pointing to the implementation source, either downloadable archive or version control system. -- **maintainer**: a dictionary providing details about the maintainer of the implementation, MUST contain the single field: + - **maintainer**: a dictionary providing details about the maintainer of the implementation, MUST contain the single field: -- **email** with the maintainer's email address. + - **email** with the maintainer's email address. -- **issue\_tracker**: a `JSON API link `__ pointing to the implementation's issue tracker. + - **issue\_tracker**: a `JSON API link `__ pointing to the implementation's issue tracker. -- **warnings**: a list of warning resource objects representing non-critical errors or warnings. -A warning resource object is defined similarly to a `JSON:API error object `__, but MUST also include the field :field:`type`, which MUST have the value :field-val:`"warning"`. -The field :field:`detail` MUST be present and SHOULD contain a non-critical message, e.g., reporting unrecognized search attributes or deprecated features. -The field :field:`status`, representing an HTTP response status code, MUST NOT be present for a warning resource object. -This is an exclusive field for error resource objects. + - **warnings**: a list of warning resource objects representing non-critical errors or warnings. + A warning resource object is defined similarly to a `JSON:API error object `__, but MUST also include the field :field:`type`, which MUST have the value :field-val:`"warning"`. + The field :field:`detail` MUST be present and SHOULD contain a non-critical message, e.g., reporting unrecognized search attributes or deprecated features. + The field :field:`status`, representing an HTTP response status code, MUST NOT be present for a warning resource object. + This is an exclusive field for error resource objects. -Example for a deprecation warning: + Example for a deprecation warning: -.. code:: jsonc + .. code:: jsonc -{ -"id": "dep_chemical_formula_01", -"type": "warning", -"code": "_exmpl_dep_chemical_formula", -"title": "Deprecation Warning", -"detail": "chemical_formula is deprecated, use instead chemical_formula_hill" -} + { + "id": "dep_chemical_formula_01", + "type": "warning", + "code": "_exmpl_dep_chemical_formula", + "title": "Deprecation Warning", + "detail": "chemical_formula is deprecated, use instead chemical_formula_hill" + } -**Note**: warning :field:`id`\ s MUST NOT be trusted to identify the exceptional situations (i.e., they are not error codes), use instead the field :field:`code` for this. -Warning :field:`id`\ s can *only* be trusted to be unique in the list of warning resource objects, i.e., together with the :field:`type`. + **Note**: warning :field:`id`\ s MUST NOT be trusted to identify the exceptional situations (i.e., they are not error codes), use instead the field :field:`code` for this. + Warning :field:`id`\ s can *only* be trusted to be unique in the list of warning resource objects, i.e., together with the :field:`type`. -General OPTIMADE warning codes are specified in section `Warnings`_. + General OPTIMADE warning codes are specified in section `Warnings`_. -- Other OPTIONAL additional information *global to the query* that is not specified in this document, MUST start with a database-provider-specific prefix (see section `Database-Provider-Specific Namespace Prefixes`_). + - Other OPTIONAL additional information *global to the query* that is not specified in this document, MUST start with a database-provider-specific prefix (see section `Database-Provider-Specific Namespace Prefixes`_). -- Example for a request made to :query-url:`http://example.com/optimade/v1/structures/?filter=a=1 AND b=2`: + - Example for a request made to :query-url:`http://example.com/optimade/v1/structures/?filter=a=1 AND b=2`: -.. code:: jsonc + .. code:: jsonc + + { + "meta": { + "query": { + "representation": "/structures/?filter=a=1 AND b=2" + }, + "api_version": "1.0.0", + "schema": "http://schemas.optimade.org/openapi/v1/optimade.json", + "time_stamp": "2007-04-05T14:30:20Z", + "data_returned": 10, + "data_available": 10, + "more_data_available": false, + "provider": { + "name": "Example provider", + "description": "Provider used for examples, not to be assigned to a real database", + "prefix": "exmpl", + "homepage": "http://example.com" + }, + "implementation": { + "name": "exmpl-optimade", + "version": "0.1.0", + "source_url": "http://git.example.com/exmpl-optimade", + "maintainer": { + "email": "admin@example.com" + }, + "issue_tracker": "http://tracker.example.com/exmpl-optimade" + }, + "database": { + "id": "example_db", + "name": "Example database 1 (of many)", + "description": "The first example database in a series hosted by the Example Provider.", + "homepage": "http://database_one.example.com", + "maintainer": { + "email": "science_lead@example.com" + } + } + } + // ... + } + + - **schema**: a `JSON:API links object `__ that points to a schema for the response. + If it is a string, or a dictionary containing no :field:`meta` field, the provided URL MUST point at an `OpenAPI `__ schema. + It is possible that future versions of this specification allow for alternative schema types. + Hence, if the :field:`meta` field of the JSON:API links object is provided and contains a field :field:`schema_type` that is not equal to the string :field-val:`OpenAPI` the client MUST NOT handle failures to parse the schema or to validate the response against the schema as errors. -{ -"meta": { -"query": { -"representation": "/structures/?filter=a=1 AND b=2" -}, -"api_version": "1.0.0", -"schema": "http://schemas.optimade.org/openapi/v1/optimade.json", -"time_stamp": "2007-04-05T14:30:20Z", -"data_returned": 10, -"data_available": 10, -"more_data_available": false, -"provider": { -"name": "Example provider", -"description": "Provider used for examples, not to be assigned to a real database", -"prefix": "exmpl", -"homepage": "http://example.com" -}, -"implementation": { -"name": "exmpl-optimade", -"version": "0.1.0", -"source_url": "http://git.example.com/exmpl-optimade", -"maintainer": { -"email": "admin@example.com" -}, -"issue_tracker": "http://tracker.example.com/exmpl-optimade" -}, -"database": { -"id": "example_db", -"name": "Example database 1 (of many)", -"description": "The first example database in a series hosted by the Example Provider.", -"homepage": "http://database_one.example.com", -"maintainer": { -"email": "science_lead@example.com" -} -} -} -// ... -} - -- **schema**: a `JSON:API links object `__ that points to a schema for the response. -If it is a string, or a dictionary containing no :field:`meta` field, the provided URL MUST point at an `OpenAPI `__ schema. -It is possible that future versions of this specification allow for alternative schema types. -Hence, if the :field:`meta` field of the JSON:API links object is provided and contains a field :field:`schema_type` that is not equal to the string :field-val:`OpenAPI` the client MUST NOT handle failures to parse the schema or to validate the response against the schema as errors. - -**Note**: The :field:`schema` field was previously RECOMMENDED in all responses, but is now demoted to being OPTIONAL since there now is a standard way of specifying a response schema in JSON:API through the :field:`describedby` subfield of the top-level :field:`links` field. + **Note**: The :field:`schema` field was previously RECOMMENDED in all responses, but is now demoted to being OPTIONAL since there now is a standard way of specifying a response schema in JSON:API through the :field:`describedby` subfield of the top-level :field:`links` field. - **data**: The schema of this value varies by endpoint, it can be either a *single* `JSON:API resource object `__ or a *list* of JSON:API resource objects. -Every resource object needs the :field:`type` and :field:`id` fields, and its attributes (described in section `API Endpoints`_) need to be in a dictionary corresponding to the :field:`attributes` field. + Every resource object needs the :field:`type` and :field:`id` fields, and its attributes (described in section `API Endpoints`_) need to be in a dictionary corresponding to the :field:`attributes` field. -Every resource object MAY also contain a :field:`meta` field which MAY contain the following keys: + Every resource object MAY also contain a :field:`meta` field which MAY contain the following keys: -- **property_metadata**: an object containing per-entry and per-property metadata. -The keys are the names of the fields in :field:`attributes` for which metadata is available. -The values belonging to these keys are dictionaries containing the relevant metadata fields. -See also `Metadata properties`_ + - **property_metadata**: an object containing per-entry and per-property metadata. + The keys are the names of the fields in :field:`attributes` for which metadata is available. + The values belonging to these keys are dictionaries containing the relevant metadata fields. + See also `Metadata properties`_ -- **partial_data_links**: an object used to list links which can be used to fetch data that has been omitted from the :field:`data` part of the response. -The keys are the names of the fields in :field:`attributes` for which partial data links are available. -Each value is a list of objects that MUST have the following keys: + - **partial_data_links**: an object used to list links which can be used to fetch data that has been omitted from the :field:`data` part of the response. + The keys are the names of the fields in :field:`attributes` for which partial data links are available. + Each value is a list of objects that MUST have the following keys: -- **format**: String. -The name of the format provided via this link. -For one of the objects this :field:`format` field SHOULD have the value "jsonlines", which refers to the format in `OPTIMADE JSON lines partial data format`_. + - **format**: String. + The name of the format provided via this link. + For one of the objects this :field:`format` field SHOULD have the value "jsonlines", which refers to the format in `OPTIMADE JSON lines partial data format`_. -- **link**: String. -A `JSON API link `__ that points to a location from which the omitted data can be fetched. -There is no requirement on the syntax or format for the link URL. + - **link**: String. + A `JSON API link `__ that points to a location from which the omitted data can be fetched. + There is no requirement on the syntax or format for the link URL. -For more information about the mechanism to transmit large property values, including an example of the format of :field:`partial_data_links`, see `Transmission of large property values`_. + For more information about the mechanism to transmit large property values, including an example of the format of :field:`partial_data_links`, see `Transmission of large property values`_. The response MAY also return resources related to the primary data in the field: - **links**: a `JSON API links object `__ is REQUIRED for implementing pagination. -(see section `Entry Listing URL Query Parameters`_.) -Each field of a links object, i.e., a "link", MUST be one of: + (see section `Entry Listing URL Query Parameters`_.) + Each field of a links object, i.e., a "link", MUST be one of: -- :field-val:`null` -- a string representing a URI, or -- a dictionary ("link object") with fields + - :field-val:`null` + - a string representing a URI, or + - a dictionary ("link object") with fields -- **href**: a string representing a URI -- **meta**: (OPTIONAL) a meta object containing non-standard meta-information about the link + - **href**: a string representing a URI + - **meta**: (OPTIONAL) a meta object containing non-standard meta-information about the link -Example links objects: + Example links objects: -- **base\_url**: a links object representing the base URL of the implementation. Example: + - **base\_url**: a links object representing the base URL of the implementation. Example: -.. code:: jsonc + .. code:: jsonc -{ -"links": { -"base_url": { -"href": "http://example.com/optimade", -"meta": { -"_exmpl_db_version": "3.2.1" -} -} -// ... -} -// ... -} + { + "links": { + "base_url": { + "href": "http://example.com/optimade", + "meta": { + "_exmpl_db_version": "3.2.1" + } + } + // ... + } + // ... + } -The :field:`links` field SHOULD include the following links objects: + The :field:`links` field SHOULD include the following links objects: -- **describedby**: a links object giving the URL for a schema that describes the response. -The URL SHOULD resolve into a JSON formatted response returning a JSON object with top level :field:`$schema` and/or :field:`$id` fields that can be used by the client to identify the schema format. + - **describedby**: a links object giving the URL for a schema that describes the response. + The URL SHOULD resolve into a JSON formatted response returning a JSON object with top level :field:`$schema` and/or :field:`$id` fields that can be used by the client to identify the schema format. -**Note**: This field is the standard facility in JSON:API to communicate a response schema. -It overlaps in function with the field :field:`schema` in the top level :field:`meta` field. + **Note**: This field is the standard facility in JSON:API to communicate a response schema. + It overlaps in function with the field :field:`schema` in the top level :field:`meta` field. -The following fields are REQUIRED for implementing pagination: + The following fields are REQUIRED for implementing pagination: -- **next**: represents a link to fetch the next set of results. -When the current response is the last page of data, this field MUST be either omitted or :field-val:`null`\ -valued. + - **next**: represents a link to fetch the next set of results. + When the current response is the last page of data, this field MUST be either omitted or :field-val:`null`\ -valued. -An implementation MAY also use the following reserved fields for pagination. -They represent links in a similar way as for :field:`next`. + An implementation MAY also use the following reserved fields for pagination. + They represent links in a similar way as for :field:`next`. -- **prev**: the previous page of data. :field-val:`null` or omitted when the current response is the first page of data. -- **last**: the last page of data. -- **first**: the first page of data. + - **prev**: the previous page of data. :field-val:`null` or omitted when the current response is the first page of data. + - **last**: the last page of data. + - **first**: the first page of data. -Finally, the :field:`links` field MAY also include the following links object: + Finally, the :field:`links` field MAY also include the following links object: -- **self**: a links object giving the URL from which the response was obtained. + - **self**: a links object giving the URL from which the response was obtained. - **included**: a list of `JSON:API resource objects `__ related to the primary data contained in :field:`data`. -Responses that contain related resources under :field:`included` are known as `compound documents `__ in the JSON:API. + Responses that contain related resources under :field:`included` are known as `compound documents `__ in the JSON:API. -The definition of this field is found in the `JSON:API specification `__. -Specifically, if the query parameter :query-param:`include` is included in the request, :field:`included` MUST NOT include unrequested resource objects. -For further information on the parameter :query-param:`include`, see section `Entry Listing URL Query Parameters`_. + The definition of this field is found in the `JSON:API specification `__. + Specifically, if the query parameter :query-param:`include` is included in the request, :field:`included` MUST NOT include unrequested resource objects. + For further information on the parameter :query-param:`include`, see section `Entry Listing URL Query Parameters`_. -This value MUST be either an empty array or an array of related resource objects. + This value MUST be either an empty array or an array of related resource objects. If there were errors in producing the response all other fields MAY be present, but the top-level :field:`data` field MUST be skipped, and the following field MUST be present: - **errors**: a list of `JSON:API error objects `__, where the field :field:`detail` MUST be present. -All other fields are OPTIONAL. + All other fields are OPTIONAL. An example of a full response: .. code:: jsonc -{ -"links": { -"next": null, -"base_url": { -"href": "http://example.com/optimade", -"meta": { -"_exmpl_db_version": "3.2.1" -} -} -}, -"meta": { -"query": { -"representation": "/structures?filter=a=1 AND b=2" -}, -"api_version": "1.0.0", -"time_stamp": "2007-04-05T14:30:20Z", -"data_returned": 10, -"data_available": 10, -"last_id": "xy10", -"more_data_available": false, -"provider": { -"name": "Example provider", -"description": "Provider used for examples, not to be assigned to a real database", -"prefix": "exmpl", -"homepage": { -"href": "http://example.com", -"meta": { -"_exmpl_title": "This is an example site" -} -} -}, -// -}, -"data": [ -// ... -], -"included": [ -// ... -] -} + { + "links": { + "next": null, + "base_url": { + "href": "http://example.com/optimade", + "meta": { + "_exmpl_db_version": "3.2.1" + } + } + }, + "meta": { + "query": { + "representation": "/structures?filter=a=1 AND b=2" + }, + "api_version": "1.0.0", + "time_stamp": "2007-04-05T14:30:20Z", + "data_returned": 10, + "data_available": 10, + "last_id": "xy10", + "more_data_available": false, + "provider": { + "name": "Example provider", + "description": "Provider used for examples, not to be assigned to a real database", + "prefix": "exmpl", + "homepage": { + "href": "http://example.com", + "meta": { + "_exmpl_title": "This is an example site" + } + } + }, + // + }, + "data": [ + // ... + ], + "included": [ + // ... + ] + } - **@context**: A JSON-LD context that enables interpretation of data in the response as linked data. -If provided, it SHOULD be one of the following: + If provided, it SHOULD be one of the following: -- An object conforming to a JSON-LD standard, which includes a :field:`@version` field specifying the version of the standard. -- A string containing a URL that resolves to such an object. + - An object conforming to a JSON-LD standard, which includes a :field:`@version` field specifying the version of the standard. + - A string containing a URL that resolves to such an object. - **jsonapi**: A `JSON:API object `__. -The :field:`version` subfield SHOULD be :field-val:`"1.1"`. -The :field:`meta` subfield SHOULD be included and contain the following subfields: + The :field:`version` subfield SHOULD be :field-val:`"1.1"`. + The :field:`meta` subfield SHOULD be included and contain the following subfields: -- **api**: A string with the value "OPTIMADE". -- **api-version**: A string with the full version of the OPTIMADE standard that the processing and response adheres to. -This MAY be the version indicated at the top of this document, but MAY also be another version if the client, e.g., has used the query parameter :query-param:`api_hint` to request processing according to another version. + - **api**: A string with the value "OPTIMADE". + - **api-version**: A string with the full version of the OPTIMADE standard that the processing and response adheres to. + This MAY be the version indicated at the top of this document, but MAY also be another version if the client, e.g., has used the query parameter :query-param:`api_hint` to request processing according to another version. -If the server is able to handle serialization in such a way that it can dictate the order of the top level object members in the response, it is RECOMMENDED to put the :field:`jsonapi` as the first top level member to simplify identification of the response. + If the server is able to handle serialization in such a way that it can dictate the order of the top level object members in the response, it is RECOMMENDED to put the :field:`jsonapi` as the first top level member to simplify identification of the response. HTTP Response Status Codes -------------------------- @@ -1160,9 +1160,9 @@ Example response: .. code:: CSV -version -1 -0 + version + 1 + 0 The above response means that the API versions 1 and 0 are served under the versioned base URLs :query-url:`/v1` and :query-url:`/v0`, respectively. The order of the versions indicates that the API implementation regards version 1 as preferred over version 0. @@ -1178,14 +1178,14 @@ Each entry in the list includes a set of properties and their corresponding valu The section `Entry list`_ specifies properties as belonging to one of three categories: 1. Properties marked as REQUIRED in the response. -These properties MUST always be present for all entries in the response. + These properties MUST always be present for all entries in the response. 2. Properties marked as REQUIRED only if the query parameter :query-param:`response_fields` is not part of the request, or if they are explicitly requested in :query-param:`response_fields`. -Otherwise they MUST NOT be included. -One can think of these properties as constituting a default value for :query-param:`response_fields` when that parameter is omitted. + Otherwise they MUST NOT be included. + One can think of these properties as constituting a default value for :query-param:`response_fields` when that parameter is omitted. 3. Properties not marked as REQUIRED in any case, MUST be included only if explicitly requested in the query parameter :query-param:`response_fields`. -Otherwise they SHOULD NOT be included. + Otherwise they SHOULD NOT be included. Examples of valid entry listing endpoint URLs: @@ -1214,65 +1214,65 @@ Standard OPTIONAL URL query parameters standardized by the JSON:API specificatio - **filter**: a filter string, in the format described below in section `API Filtering Format Specification`_. - **page\_limit**: sets a numerical limit on the number of entries returned. -See `JSON:API 1.1 `__. -The API implementation MUST return no more than the number specified. -It MAY return fewer. -The database MAY have a maximum limit and not accept larger numbers (in which case the :http-error:`403 Forbidden` error code MUST be returned). -The default limit value is up to the API implementation to decide. -Example: :query-url:`http://example.com/optimade/v1/structures?page_limit=100` + See `JSON:API 1.1 `__. + The API implementation MUST return no more than the number specified. + It MAY return fewer. + The database MAY have a maximum limit and not accept larger numbers (in which case the :http-error:`403 Forbidden` error code MUST be returned). + The default limit value is up to the API implementation to decide. + Example: :query-url:`http://example.com/optimade/v1/structures?page_limit=100` - **page\_{offset, number, cursor, above, below}**: A server MUST implement pagination in the case of no user-specified :query-param:`sort` parameter (via the :field:`links` response field, see section `JSON Response Schema: Common Fields`_). -A server MAY implement pagination in concert with :query-param:`sort`. -The following parameters, all prefixed by "page\_", are RECOMMENDED for use with pagination. -If an implementation chooses + A server MAY implement pagination in concert with :query-param:`sort`. + The following parameters, all prefixed by "page\_", are RECOMMENDED for use with pagination. + If an implementation chooses -- *offset-based pagination*: using :field:`page_offset` and :field:`page_limit` is RECOMMENDED. -- *cursor-based pagination*: using :field:`page_cursor` and :field:`page_limit` is RECOMMENDED. -- *page-based pagination*: using :field:`page_number` and :field:`page_limit` is RECOMMENDED. It is RECOMMENDED that the first page has number 1, i.e., that :field:`page_number` is 1-based. -- *value-based pagination*: using :field:`page_above`/:field:`page_below` and :field:`page_limit` is RECOMMENDED. + - *offset-based pagination*: using :field:`page_offset` and :field:`page_limit` is RECOMMENDED. + - *cursor-based pagination*: using :field:`page_cursor` and :field:`page_limit` is RECOMMENDED. + - *page-based pagination*: using :field:`page_number` and :field:`page_limit` is RECOMMENDED. It is RECOMMENDED that the first page has number 1, i.e., that :field:`page_number` is 1-based. + - *value-based pagination*: using :field:`page_above`/:field:`page_below` and :field:`page_limit` is RECOMMENDED. -Examples (all OPTIONAL behavior a server MAY implement): + Examples (all OPTIONAL behavior a server MAY implement): -- skip 50 structures and fetch up to 100: :query-url:`/structures?page_offset=50&page_limit=100`. -- fetch page 2 of up to 50 structures per page: :query-url:`/structures?page_number=2&page_limit=50`. -- fetch up to 100 structures above sort-field value 4000 (in this example, server chooses to fetch results sorted by increasing :field:`id`, so :field:`page_above` value refers to an :field:`id` value): :query-url:`/structures?page_above=4000&page_limit=100`. + - skip 50 structures and fetch up to 100: :query-url:`/structures?page_offset=50&page_limit=100`. + - fetch page 2 of up to 50 structures per page: :query-url:`/structures?page_number=2&page_limit=50`. + - fetch up to 100 structures above sort-field value 4000 (in this example, server chooses to fetch results sorted by increasing :field:`id`, so :field:`page_above` value refers to an :field:`id` value): :query-url:`/structures?page_above=4000&page_limit=100`. - **sort**: If supporting sortable queries, an implementation MUST use the :query-param:`sort` query parameter with format as specified by `JSON:API 1.1 `__. -An implementation MAY support multiple sort fields for a single query. -If it does, it again MUST conform to the JSON:API 1.1 specification. + An implementation MAY support multiple sort fields for a single query. + If it does, it again MUST conform to the JSON:API 1.1 specification. -If an implementation supports sorting for an `entry listing endpoint `_, then the :endpoint:`/info/` endpoint MUST include, for each field name :field:`` in its :field:`data.properties.` response value that can be used for sorting, the key :field:`sortable` with value :field-val:`true`. -If a field name under an entry listing endpoint supporting sorting cannot be used for sorting, the server MUST either leave out the :field:`sortable` key or set it equal to :field-val:`false` for the specific field name. -The set of field names, with :field:`sortable` equal to :field-val:`true` are allowed to be used in the "sort fields" list according to its definition in the JSON:API 1.1 specification. -The field :field:`sortable` is in addition to each property description and other OPTIONAL fields. -An example is shown in section `Entry Listing Info Endpoints`_. + If an implementation supports sorting for an `entry listing endpoint `_, then the :endpoint:`/info/` endpoint MUST include, for each field name :field:`` in its :field:`data.properties.` response value that can be used for sorting, the key :field:`sortable` with value :field-val:`true`. + If a field name under an entry listing endpoint supporting sorting cannot be used for sorting, the server MUST either leave out the :field:`sortable` key or set it equal to :field-val:`false` for the specific field name. + The set of field names, with :field:`sortable` equal to :field-val:`true` are allowed to be used in the "sort fields" list according to its definition in the JSON:API 1.1 specification. + The field :field:`sortable` is in addition to each property description and other OPTIONAL fields. + An example is shown in section `Entry Listing Info Endpoints`_. - **include**: A server MAY implement the JSON:API concept of returning `compound documents `__ by utilizing the :query-param:`include` query parameter as specified by `JSON:API 1.0 `__. -All related resource objects MUST be returned as part of an array value for the top-level :field:`included` field, see section `JSON Response Schema: Common Fields`_. + All related resource objects MUST be returned as part of an array value for the top-level :field:`included` field, see section `JSON Response Schema: Common Fields`_. -The value of :query-param:`include` MUST be a comma-separated list of "relationship paths", as defined in the `JSON:API `__. -If relationship paths are not supported, or a server is unable to identify a relationship path a :http-error:`400 Bad Request` response MUST be made. + The value of :query-param:`include` MUST be a comma-separated list of "relationship paths", as defined in the `JSON:API `__. + If relationship paths are not supported, or a server is unable to identify a relationship path a :http-error:`400 Bad Request` response MUST be made. -The **default value** for :query-param:`include` is :query-val:`references`. -This means :entry:`references` entries MUST always be included under the top-level field :field:`included` as default, since a server assumes if :query-param:`include` is not specified by a client in the request, it is still specified as :query-string:`include=references`. -Note, if a client explicitly specifies :query-param:`include` and leaves out :query-val:`references`, :entry:`references` resource objects MUST NOT be included under the top-level field :field:`included`, as per the definition of :field:`included`, see section `JSON Response Schema: Common Fields`_. + The **default value** for :query-param:`include` is :query-val:`references`. + This means :entry:`references` entries MUST always be included under the top-level field :field:`included` as default, since a server assumes if :query-param:`include` is not specified by a client in the request, it is still specified as :query-string:`include=references`. + Note, if a client explicitly specifies :query-param:`include` and leaves out :query-val:`references`, :entry:`references` resource objects MUST NOT be included under the top-level field :field:`included`, as per the definition of :field:`included`, see section `JSON Response Schema: Common Fields`_. -**Note**: A query with the parameter :query-param:`include` set to the empty string means no related resource objects are to be returned under the top-level field :field:`included`. + **Note**: A query with the parameter :query-param:`include` set to the empty string means no related resource objects are to be returned under the top-level field :field:`included`. Standard OPTIONAL URL query parameters not in the JSON:API specification: - **response\_format**: the output format requested (see section `Response Format`_). -Defaults to the format string 'json', which specifies the standard output format described in this specification. -Example: :query-url:`http://example.com/optimade/v1/structures?response_format=xml` + Defaults to the format string 'json', which specifies the standard output format described in this specification. + Example: :query-url:`http://example.com/optimade/v1/structures?response_format=xml` - **email\_address**: an email address of the user making the request. -The email SHOULD be that of a person and not an automatic system. -Example: :query-url:`http://example.com/optimade/v1/structures?email_address=user@example.com` + The email SHOULD be that of a person and not an automatic system. + Example: :query-url:`http://example.com/optimade/v1/structures?email_address=user@example.com` - **response\_fields**: a comma-delimited set of fields to be provided in the output. -If provided, these fields MUST be returned along with the REQUIRED fields. -Other OPTIONAL fields MUST NOT be returned when this parameter is present. -Example: :query-url:`http://example.com/optimade/v1/structures?response_fields=last_modified,nsites` + If provided, these fields MUST be returned along with the REQUIRED fields. + Other OPTIONAL fields MUST NOT be returned when this parameter is present. + Example: :query-url:`http://example.com/optimade/v1/structures?response_fields=last_modified,nsites` Additional OPTIONAL URL query parameters not described above are not considered to be part of this standard, and are instead considered to be "custom URL query parameters". These custom URL query parameters MUST be of the format "". @@ -1287,8 +1287,8 @@ Examples: - :query-url:`http://example.com/optimade/v1/structures?_exmpl_warning_verbosity=10` - :query-url:`http://example.com/optimade/v1/structures?_exmpl_filter="elements all in [Al, Si, Ga]"` -**Note**: the specification presently makes no attempt to standardize access control mechanisms. -There are security concerns with access control based on URL tokens, and the above example is not to be taken as a recommendation for such a mechanism. + **Note**: the specification presently makes no attempt to standardize access control mechanisms. + There are security concerns with access control based on URL tokens, and the above example is not to be taken as a recommendation for such a mechanism. Entry Listing JSON Response Schema ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -1301,51 +1301,51 @@ In the default JSON response format every dictionary (`resource object `__ can OPTIONALLY contain the field -- **self**: the entry's URL + - **self**: the entry's URL - **meta**: a `JSON API meta object `__ that is used to communicate metadata. -See `JSON Response Schema: Common Fields`_ for more information about this field. + See `JSON Response Schema: Common Fields`_ for more information about this field. - **relationships**: a dictionary containing references to other entries according to the description in section `Relationships`_ encoded as `JSON:API Relationships `__. -The OPTIONAL human-readable description of the relationship MAY be provided in the :field:`description` field inside the :field:`meta` dictionary of the JSON:API resource identifier object. -All relationships to entries of the same entry type MUST be grouped into the same JSON:API relationship object and placed in the relationships dictionary with the entry type name as key (e.g., :entry:`structures`). + The OPTIONAL human-readable description of the relationship MAY be provided in the :field:`description` field inside the :field:`meta` dictionary of the JSON:API resource identifier object. + All relationships to entries of the same entry type MUST be grouped into the same JSON:API relationship object and placed in the relationships dictionary with the entry type name as key (e.g., :entry:`structures`). Example: .. code:: jsonc -{ -"data": [ -{ -"type": "structures", -"id": "example.db:structs:0001", -"attributes": { -"chemical_formula_descriptive": "Es2 O3", -"url": "http://example.db/structs/0001", -"immutable_id": "http://example.db/structs/0001@123", -"last_modified": "2007-04-05T14:30:20Z" -} -}, -{ -"type": "structures", -"id": "example.db:structs:1234", -"attributes": { -"chemical_formula_descriptive": "Es2", -"url": "http://example.db/structs/1234", -"immutable_id": "http://example.db/structs/1234@123", -"last_modified": "2007-04-07T12:02:20Z" -} -} -// ... -] -// ... -} + { + "data": [ + { + "type": "structures", + "id": "example.db:structs:0001", + "attributes": { + "chemical_formula_descriptive": "Es2 O3", + "url": "http://example.db/structs/0001", + "immutable_id": "http://example.db/structs/0001@123", + "last_modified": "2007-04-05T14:30:20Z" + } + }, + { + "type": "structures", + "id": "example.db:structs:1234", + "attributes": { + "chemical_formula_descriptive": "Es2", + "url": "http://example.db/structs/1234", + "immutable_id": "http://example.db/structs/1234@123", + "last_modified": "2007-04-07T12:02:20Z" + } + } + // ... + ] + // ... + } Single Entry Endpoints ---------------------- @@ -1373,26 +1373,25 @@ The meaning of these URL query parameters are as defined above in section `Entry One additional query parameter :query-param:`property_slices` MUST be handled by the API implementation either as defined below or by returning the error :http-error:`501 Not Implemented`: -- **property\_slices**: A number of slice specifications to request only parts of array properties for the functionality described in `Slices of array properties`_. - - The query parameter contains a comma-separated (",", ASCII 44(0x2C)) list of slice specifications. - Each slice specification consists of an ordered sequence of four elements separated by the colon character (":", ASCII 58(0x3A)). - The elements in the sequence are the dimension name and the three components of the slice, i.e., the start, stop, and step values defined in the same way as for a `slice object`_. - Omitting the value for any of the components of the slice specifies a default value (however, all the colon separators MUST be included). - The start value specifies the first index in that dimension for which values should be returned (which is 0-based and inclusive). - The default is :val:`0`. - The stop value specifies the last index for which values should be returned (inclusive). - The default is the last index of the array along the specified dimension. - The step value specifies the step size in that dimension. - The default is :val:`1`. +- **property\_slices**: A number of slice specifications to specify parts of array properties for the functionality described in `Slices of array properties`_. - An empty value of the :query-param:`property_slices` query parameter MUST be interpreted as equivalent to the query parameter not being included in the request. + The query parameter contains a comma-separated (",", ASCII 44(0x2C)) list of slice specification. + Each slice specification consists of a dimension name and the start, stop and step values of the slice separated by the colon character (":", ASCII 58(0x3A)) + Omitting the value between two colons specifies a default value. + The components of the slice are defined in the same way as for a `slice object`_. + The first integer specifies the start of the slice, i.e. the first index in that dimension for which values should be returned. + The default is the index of the first value. + The second integer specifies the end/stop of the slice, i.e. the last index for which values should be returned. + The default is the index of the last value of the property. + The third integer specifies the step size in that dimension. + The default is :val:`1` + The slices are 0-based, i.e. the first value has index 0, and inclusive i.e. for the dimension :val:`dim_frames` the range :val:`dim_frames:10:20:1` the last value returned belongs to index 20. Requirements and conventions for the response when this query parameter is used are described in `Slices of array properties`_. Example: - - :query-url:`http://optimade.example.com/v1/trajectories/id_12345?response_fields=cartesian_site_positions&property_ranges=dim_frames::999:10,dim_sites:30:70:` + - :query-url:`http://optimade.example.com/v1/trajectories/id_12345?response_fields=cartesian_site_positions&property_ranges=dim_frames::999:10,dim_sites:30:70::` This query URL requests items from the array :field:`cartesian_site_positions` is requested only for the 31st to 71st sites (i.e., with indexes 30 through 70 inclusive) for 1 out of every 10 frames of the first 1000 frames (i.e., taking steps of 10 over indexes 0 through 999 inclusive, which requests the frames with index 0, 10, 20, 30, ..., 990) of a trajectory with ID :val:`id_12345`. From 315546a291920ff4f6c10aa464ea7f750f93fcb8 Mon Sep 17 00:00:00 2001 From: Rickard Armiento Date: Fri, 14 Jun 2024 10:24:40 +0200 Subject: [PATCH 23/34] Edit property slice definition [WIP] --- optimade.rst | 29 +++++++++++++++-------------- 1 file changed, 15 insertions(+), 14 deletions(-) diff --git a/optimade.rst b/optimade.rst index 35c37ea7d..2de298162 100644 --- a/optimade.rst +++ b/optimade.rst @@ -1373,25 +1373,26 @@ The meaning of these URL query parameters are as defined above in section `Entry One additional query parameter :query-param:`property_slices` MUST be handled by the API implementation either as defined below or by returning the error :http-error:`501 Not Implemented`: -- **property\_slices**: A number of slice specifications to specify parts of array properties for the functionality described in `Slices of array properties`_. - - The query parameter contains a comma-separated (",", ASCII 44(0x2C)) list of slice specification. - Each slice specification consists of a dimension name and the start, stop and step values of the slice separated by the colon character (":", ASCII 58(0x3A)) - Omitting the value between two colons specifies a default value. - The components of the slice are defined in the same way as for a `slice object`_. - The first integer specifies the start of the slice, i.e. the first index in that dimension for which values should be returned. - The default is the index of the first value. - The second integer specifies the end/stop of the slice, i.e. the last index for which values should be returned. - The default is the index of the last value of the property. - The third integer specifies the step size in that dimension. - The default is :val:`1` - The slices are 0-based, i.e. the first value has index 0, and inclusive i.e. for the dimension :val:`dim_frames` the range :val:`dim_frames:10:20:1` the last value returned belongs to index 20. +- **property\_slices**: A number of slice specifications to request only parts of array properties for the functionality described in `Slices of array properties`_. + + The query parameter contains a comma-separated (",", ASCII 44(0x2C)) list of slice specifications. + Each slice specification consists of an ordered sequence of four elements separated by the colon character (":", ASCII 58(0x3A)). + The elements in the sequence are the dimension name and the three components of the slice, i.e., the start, stop, and step values defined in the same way as for a `slice object`_. + Omitting the value for any of the components of the slice specifies a default value (however, all the colon separators MUST be included). + The start value specifies the first index in that dimension for which values should be returned (which is 0-based and inclusive). + The default is :val:`0`. + The stop value specifies the last index for which values should be returned (inclusive). + The default is the last index of the array along the specified dimension. + The step value specifies the step size in that dimension. + The default is :val:`1`. + + An empty value of the :query-param:`property_slices` query parameter MUST be interpreted as equivalent to the query parameter not being included in the request. Requirements and conventions for the response when this query parameter is used are described in `Slices of array properties`_. Example: - - :query-url:`http://optimade.example.com/v1/trajectories/id_12345?response_fields=cartesian_site_positions&property_ranges=dim_frames::999:10,dim_sites:30:70::` + - :query-url:`http://optimade.example.com/v1/trajectories/id_12345?response_fields=cartesian_site_positions&property_ranges=dim_frames::999:10,dim_sites:30:70:` This query URL requests items from the array :field:`cartesian_site_positions` is requested only for the 31st to 71st sites (i.e., with indexes 30 through 70 inclusive) for 1 out of every 10 frames of the first 1000 frames (i.e., taking steps of 10 over indexes 0 through 999 inclusive, which requests the frames with index 0, 10, 20, 30, ..., 990) of a trajectory with ID :val:`id_12345`. From ce9661b629381fe198a9184cbbc0c6b10ef0ad72 Mon Sep 17 00:00:00 2001 From: Giovanni Pizzi Date: Fri, 14 Jun 2024 10:37:22 +0200 Subject: [PATCH 24/34] Small sentence fix --- optimade.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/optimade.rst b/optimade.rst index 2de298162..03625b7e2 100644 --- a/optimade.rst +++ b/optimade.rst @@ -1394,7 +1394,7 @@ One additional query parameter :query-param:`property_slices` MUST be handled by - :query-url:`http://optimade.example.com/v1/trajectories/id_12345?response_fields=cartesian_site_positions&property_ranges=dim_frames::999:10,dim_sites:30:70:` - This query URL requests items from the array :field:`cartesian_site_positions` is requested only for the 31st to 71st sites (i.e., with indexes 30 through 70 inclusive) for 1 out of every 10 frames of the first 1000 frames (i.e., taking steps of 10 over indexes 0 through 999 inclusive, which requests the frames with index 0, 10, 20, 30, ..., 990) of a trajectory with ID :val:`id_12345`. + This query URL requests items from the array :field:`cartesian_site_positions` only for the 31st to 71st sites (i.e., with indexes 30 through 70 inclusive) for 1 out of every 10 frames of the first 1000 frames (i.e., taking steps of 10 over indexes 0 through 999 inclusive, which requests the frames with index 0, 10, 20, 30, ..., 990) of a trajectory with ID :val:`id_12345`. Single Entry JSON Response Schema ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ From 358157f8f82ed445e1eb94e7f6e9e12795cfff71 Mon Sep 17 00:00:00 2001 From: Giovanni Pizzi Date: Fri, 14 Jun 2024 10:49:41 +0200 Subject: [PATCH 25/34] Clarifying the part on dimension names. --- optimade.rst | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/optimade.rst b/optimade.rst index 03625b7e2..db3920b4a 100644 --- a/optimade.rst +++ b/optimade.rst @@ -2367,13 +2367,13 @@ A Property Definition MUST be composed according to the combination of the requi **REQUIRED keys:** - :field:`names`: List of Strings. - A list of names of the dimensions of the underlying one or multi-dimensionsional data represented as mutiple levels of lists. - The order is that the the first name applies to the outermost list, the next name to the lists embedded in that list, etc. + A list of names of the dimensions of the underlying one or multi-dimensionsional data represented as multiple levels of lists. + The order is such that the first name applies to the outermost list, the next name to the lists embedded in that list, etc. Dimension names defined by the OPTIMADE standard are prefixed by ``dim_``. - Dimension names defined by database or definition providers MUST be prefixed by the corresponding namespace prefix and SHOULD also be prefixed with ``dim_``, e.g., ``_exmpl_dim_particles``. - If, within one entry, two or more array dimensions in one or more properties share the same :field:`name`, those represent the same dimension. - For example, in a trajectory entry there is a property :property:`cartesian_site_positions` where one of the dimension names is :val:`dim_frames`. - If there is another one-dimensional array property :property:`_exmpl_energy` that specifies the same dimension name in its property definition, then this declares that the energy and site positions at index *i* pertain to the same frame. + Dimension names defined by database or definition providers MUST be prefixed by the corresponding database or namespace prefix, and SHOULD also be prefixed with ``dim_``, e.g., ``_exmpl_dim_particles``. + If, within one entry, two or more array axes in one or more properties share the same dimension :field:`name`, those represent the same dimension. + For example, let us consider the property :property:`cartesian_site_positions` of the trajectory entry, where the first dimension name is :val:`dim_frames`. + If there is another one-dimensional (i.e., with a single axis) array property :property:`_exmpl_energy` of the same trajectory entry that specifies in its property definition the same dimension name :val:`dim_frames` for its axis, then the values of :property:`_exmpl_energy` and of :property:`cartesian_site_positions` at index *i* pertain to the same frame. - :field:`sizes`: List of Integers or :val:`null`. A list of fixed length requirements on the underlying one or multi-dimensionsional data represented as mutiple levels of lists. From 7e29652ecf5a5503dbe5cfd9c571496b183ee621 Mon Sep 17 00:00:00 2001 From: Rickard Armiento Date: Sat, 15 Jun 2024 15:30:46 +0200 Subject: [PATCH 26/34] Apply suggestions from review Co-authored-by: Andrius Merkys Co-authored-by: Antanas Vaitkus --- optimade.rst | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/optimade.rst b/optimade.rst index db3920b4a..04daee596 100644 --- a/optimade.rst +++ b/optimade.rst @@ -623,12 +623,12 @@ The field :field:`array_axes` is defined as follows: A list of dictionaries which provide information related to the axes of an array property, which is relevant for slicing. Each item, in order, represents the array axis as declared in the corresponding property definition. - Each item MUST be a dictionary with the following REQUIRED fields: + Each item MUST be a dictionary with the following REQUIRED field: - :field:`dimension_name`: String. The dimension name of the corresponding array axis. - If the request specifies the :query-param:`property_slices` query parameter for any of the array axes of this array property the following key MUST be present, otherwise it MUST be omitted or equal to :val:`null`: + If the request specifies the :query-param:`property_slices` query parameter for any of the array axes of this array property the following key MUST be present, otherwise the key MUST be omitted or equal to :val:`null`: - :field:`requested_slice`: Dictionary. A field that describes the requested slice that was provided via the query parameter :query-param:`property_slices`. @@ -685,7 +685,7 @@ The field :field:`array_axes` is defined as follows: - ``{"start": 3, "stop": 7, "step": 2}`` means the server certifies that values at indexes 0,1,2,4,6 and any index from 8 to the end of the array are :val:`null`. -Below follows an example of the :field:`data` and :field:`meta` parts of a response using the JSON response format for a request to the trajectory endpoint with the query parameter :query-param:`property_slices=dim_frames:3:37:5` and :query-param:`response_fields=cartesian_site_positions,_exmpl_temperature` where the trajectory consists of 432934 frames (with indexes 0 to 432933) and where the :field:`cartisian_site_positions` contains 7 sites. Furthermore, the :field:`_exmpl_temperature` contains only :val:`null` values except for items with indexes 1000, 1030, 1060, ..., 4000 (where the values can be either numeric or :val:`null`). +Below follows an example of the :field:`data` and :field:`meta` parts of a response using the JSON response format for a request to the trajectory endpoint with the query parameter :query-param:`property_slices=dim_frames:3:37:5` and :query-param:`response_fields=cartesian_site_positions,_exmpl_temperature` where the trajectory consists of 432934 frames (with indexes 0 to 432933) and where the :field:`cartesian_site_positions` contains 7 sites. Furthermore, the :field:`_exmpl_temperature` contains only :val:`null` values except for items with indexes 1000, 1030, 1060, ..., 4000 (where the values can be either numeric or :val:`null`). .. code:: jsonc @@ -715,7 +715,7 @@ Below follows an example of the :field:`data` and :field:`meta` parts of a respo "step": 1 }, "sliceable": true, - "length": 432934, + "length": 432934 }, { "dimension_name": "dim_sites", @@ -725,13 +725,13 @@ Below follows an example of the :field:`data` and :field:`meta` parts of a respo "step": 1 }, "sliceable": false, - "length": 7, + "length": 7 }, { "dimension_name": "dim_spatial", - "length": 3, + "length": 3 } - ], + ] }, "_exmpl_temperature": { "array_axes": [ @@ -748,7 +748,7 @@ Below follows an example of the :field:`data` and :field:`meta` parts of a respo "step": 30 }, "sliceable": true, - "length": 432934, + "length": 432934 } ] } @@ -1394,7 +1394,7 @@ One additional query parameter :query-param:`property_slices` MUST be handled by - :query-url:`http://optimade.example.com/v1/trajectories/id_12345?response_fields=cartesian_site_positions&property_ranges=dim_frames::999:10,dim_sites:30:70:` - This query URL requests items from the array :field:`cartesian_site_positions` only for the 31st to 71st sites (i.e., with indexes 30 through 70 inclusive) for 1 out of every 10 frames of the first 1000 frames (i.e., taking steps of 10 over indexes 0 through 999 inclusive, which requests the frames with index 0, 10, 20, 30, ..., 990) of a trajectory with ID :val:`id_12345`. + This query URL requests items from the array :field:`cartesian_site_positions` only for the 31st to 71st sites (i.e., with indexes 30 through 70 inclusive) for 1 out of every 10 frames of the first 1000 frames (i.e., taking steps of 10 over indexes 0 through 999 inclusive, which requests the frames with indexes 0, 10, 20, 30, ..., 990) of a trajectory with ID :val:`id_12345`. Single Entry JSON Response Schema ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -2370,7 +2370,7 @@ A Property Definition MUST be composed according to the combination of the requi A list of names of the dimensions of the underlying one or multi-dimensionsional data represented as multiple levels of lists. The order is such that the first name applies to the outermost list, the next name to the lists embedded in that list, etc. Dimension names defined by the OPTIMADE standard are prefixed by ``dim_``. - Dimension names defined by database or definition providers MUST be prefixed by the corresponding database or namespace prefix, and SHOULD also be prefixed with ``dim_``, e.g., ``_exmpl_dim_particles``. + Dimension names defined by database or definition providers MUST be prefixed by the corresponding database or namespace prefix, and SHOULD also be prefixed by ``dim_``, e.g., ``_exmpl_dim_particles``. If, within one entry, two or more array axes in one or more properties share the same dimension :field:`name`, those represent the same dimension. For example, let us consider the property :property:`cartesian_site_positions` of the trajectory entry, where the first dimension name is :val:`dim_frames`. If there is another one-dimensional (i.e., with a single axis) array property :property:`_exmpl_energy` of the same trajectory entry that specifies in its property definition the same dimension name :val:`dim_frames` for its axis, then the values of :property:`_exmpl_energy` and of :property:`cartesian_site_positions` at index *i* pertain to the same frame. From 085030d2cba15e35b7ce8c7a7f9883530984d25d Mon Sep 17 00:00:00 2001 From: Rickard Armiento Date: Sat, 15 Jun 2024 16:06:08 +0200 Subject: [PATCH 27/34] Define the term array for OPTMADE data types --- optimade.rst | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/optimade.rst b/optimade.rst index 04daee596..e19681e0b 100644 --- a/optimade.rst +++ b/optimade.rst @@ -218,7 +218,10 @@ representation in all contexts. They are as follows: - Basic types: **string**, **integer**, **float**, **boolean**, **timestamp**. - **list**: an ordered collection of items, where all items are of the same type, unless they are unknown. A list can be empty, i.e., contain no items. -- **dictionary**: an associative array of **keys** and **values**, where **keys** are pre-determined strings, i.e., for the same entry property, the **keys** remain the same among different entries whereas the **values** change. + Multidimensional collections of items are represented as nested lists. + The specification uses **array** as a more general term for structures of nested lists representing single or multidimensional data, and the term **array axes** for the levels of nesting. + Note that arrays are represented using lists and not as a separate data type. +- **dictionary**: an associative set of **keys** and **values**, where **keys** are pre-determined strings, i.e., for the same entry property, the **keys** remain the same among different entries whereas the **values** change. The **values** of a dictionary can be any basic type, list, dictionary, or unknown. An entry property value that is not present in the database is **unknown**. @@ -683,7 +686,7 @@ The field :field:`array_axes` is defined as follows: Examples: - - ``{"start": 3, "stop": 7, "step": 2}`` means the server certifies that values at indexes 0,1,2,4,6 and any index from 8 to the end of the array are :val:`null`. + - ``{"start": 3, "stop": 7, "step": 2}`` means the server certifies that values at indexes 0, 1, 2, 4, 6 and any index from 8 to the end of the array are :val:`null`. Below follows an example of the :field:`data` and :field:`meta` parts of a response using the JSON response format for a request to the trajectory endpoint with the query parameter :query-param:`property_slices=dim_frames:3:37:5` and :query-param:`response_fields=cartesian_site_positions,_exmpl_temperature` where the trajectory consists of 432934 frames (with indexes 0 to 432933) and where the :field:`cartesian_site_positions` contains 7 sites. Furthermore, the :field:`_exmpl_temperature` contains only :val:`null` values except for items with indexes 1000, 1030, 1060, ..., 4000 (where the values can be either numeric or :val:`null`). @@ -1392,9 +1395,9 @@ One additional query parameter :query-param:`property_slices` MUST be handled by Example: - - :query-url:`http://optimade.example.com/v1/trajectories/id_12345?response_fields=cartesian_site_positions&property_ranges=dim_frames::999:10,dim_sites:30:70:` + - :query-url:`http://optimade.example.com/v1/trajectories/id_12345?response_fields=frame_cartesian_site_positions&property_ranges=dim_frames::999:10,dim_sites:30:70:` - This query URL requests items from the array :field:`cartesian_site_positions` only for the 31st to 71st sites (i.e., with indexes 30 through 70 inclusive) for 1 out of every 10 frames of the first 1000 frames (i.e., taking steps of 10 over indexes 0 through 999 inclusive, which requests the frames with indexes 0, 10, 20, 30, ..., 990) of a trajectory with ID :val:`id_12345`. + This query URL requests items from the array :field:`frame_cartesian_site_positions` only for the 31st to 71st sites (i.e., with indexes 30 through 70 inclusive) for 1 out of every 10 frames of the first 1000 frames (i.e., taking steps of 10 over indexes 0 through 999 inclusive, which requests the frames with indexes 0, 10, 20, 30, ..., 990) of a trajectory with ID :val:`id_12345`. Single Entry JSON Response Schema ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ From b3a96a0ac424324355b8b1f9373c97ae265de135 Mon Sep 17 00:00:00 2001 From: Rickard Armiento Date: Sat, 15 Jun 2024 16:16:36 +0200 Subject: [PATCH 28/34] Fix examples for slicing assuming a trajectory cartesian_site_coordinates with different format than the structure one --- optimade.rst | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/optimade.rst b/optimade.rst index e19681e0b..97f97723c 100644 --- a/optimade.rst +++ b/optimade.rst @@ -688,7 +688,7 @@ The field :field:`array_axes` is defined as follows: - ``{"start": 3, "stop": 7, "step": 2}`` means the server certifies that values at indexes 0, 1, 2, 4, 6 and any index from 8 to the end of the array are :val:`null`. -Below follows an example of the :field:`data` and :field:`meta` parts of a response using the JSON response format for a request to the trajectory endpoint with the query parameter :query-param:`property_slices=dim_frames:3:37:5` and :query-param:`response_fields=cartesian_site_positions,_exmpl_temperature` where the trajectory consists of 432934 frames (with indexes 0 to 432933) and where the :field:`cartesian_site_positions` contains 7 sites. Furthermore, the :field:`_exmpl_temperature` contains only :val:`null` values except for items with indexes 1000, 1030, 1060, ..., 4000 (where the values can be either numeric or :val:`null`). +Below follows an example of the :field:`data` and :field:`meta` parts of a response using the JSON response format for a request to the trajectory endpoint with the query parameter :query-param:`property_slices=dim_frames:3:37:5` and :query-param:`response_fields=frame_cartesian_site_positions,_exmpl_temperature` where the trajectory consists of 432934 frames (with indexes 0 to 432933) and where the :field:`frame_cartesian_site_positions` contains 7 sites. Furthermore, the :field:`_exmpl_temperature` contains only :val:`null` values except for items with indexes 1000, 1030, 1060, ..., 4000 (where the values can be either numeric or :val:`null`). .. code:: jsonc @@ -698,12 +698,12 @@ Below follows an example of the :field:`data` and :field:`meta` parts of a respo "type": "trajectories", "id": "2345678", "attributes": { - "cartesian_site_positions": null, + "frame_cartesian_site_positions": null, "_exmpl_temperature": null }, "meta": { "property_metadata": { - "cartesian_site_positions": { + "frame_cartesian_site_positions": { "array_axes": [ { "dimension_name": "dim_frames", @@ -2375,8 +2375,8 @@ A Property Definition MUST be composed according to the combination of the requi Dimension names defined by the OPTIMADE standard are prefixed by ``dim_``. Dimension names defined by database or definition providers MUST be prefixed by the corresponding database or namespace prefix, and SHOULD also be prefixed by ``dim_``, e.g., ``_exmpl_dim_particles``. If, within one entry, two or more array axes in one or more properties share the same dimension :field:`name`, those represent the same dimension. - For example, let us consider the property :property:`cartesian_site_positions` of the trajectory entry, where the first dimension name is :val:`dim_frames`. - If there is another one-dimensional (i.e., with a single axis) array property :property:`_exmpl_energy` of the same trajectory entry that specifies in its property definition the same dimension name :val:`dim_frames` for its axis, then the values of :property:`_exmpl_energy` and of :property:`cartesian_site_positions` at index *i* pertain to the same frame. + For example, let us consider the property :property:`frame_cartesian_site_positions` of the trajectory entry, where the first dimension name is :val:`dim_frames`. + If there is another one-dimensional (i.e., with a single axis) array property :property:`_exmpl_energy` of the same trajectory entry that specifies in its property definition the same dimension name :val:`dim_frames` for its axis, then the values of :property:`_exmpl_energy` and of :property:`frame_cartesian_site_positions` at index *i* pertain to the same frame. - :field:`sizes`: List of Integers or :val:`null`. A list of fixed length requirements on the underlying one or multi-dimensionsional data represented as mutiple levels of lists. From 6231d2a408b0140cbf7c82e2454f120e22e83c49 Mon Sep 17 00:00:00 2001 From: Rickard Armiento Date: Sat, 15 Jun 2024 16:37:33 +0200 Subject: [PATCH 29/34] Adjust confusing phrasing about slicing based on review comment --- optimade.rst | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/optimade.rst b/optimade.rst index 97f97723c..2e512801d 100644 --- a/optimade.rst +++ b/optimade.rst @@ -688,7 +688,8 @@ The field :field:`array_axes` is defined as follows: - ``{"start": 3, "stop": 7, "step": 2}`` means the server certifies that values at indexes 0, 1, 2, 4, 6 and any index from 8 to the end of the array are :val:`null`. -Below follows an example of the :field:`data` and :field:`meta` parts of a response using the JSON response format for a request to the trajectory endpoint with the query parameter :query-param:`property_slices=dim_frames:3:37:5` and :query-param:`response_fields=frame_cartesian_site_positions,_exmpl_temperature` where the trajectory consists of 432934 frames (with indexes 0 to 432933) and where the :field:`frame_cartesian_site_positions` contains 7 sites. Furthermore, the :field:`_exmpl_temperature` contains only :val:`null` values except for items with indexes 1000, 1030, 1060, ..., 4000 (where the values can be either numeric or :val:`null`). +Below follows an example of the :field:`data` and :field:`meta` parts of a response using the JSON response format for a request to the trajectory endpoint with the query parameter :query-param:`property_slices=dim_frames:3:37:5` and :query-param:`response_fields=frame_cartesian_site_positions,_exmpl_temperature` where the trajectory consists of 432934 frames (with indexes 0 to 432933) and where the :field:`frame_cartesian_site_positions` contains 7 sites. +Furthermore, the metadata subfield :field:`available_slice` in :field:`array_axes` in :field:`_exmpl_temperature` certifies that all values of the data field :field:`_exmpl_temperature` are :val:`null` except at indexes 1000, 1030, 1060, ..., 4000, where values can be either numeric or :val:`null`). .. code:: jsonc From 9f639d18577af27b9f47e8753f4f193b44a4ddb4 Mon Sep 17 00:00:00 2001 From: Giovanni Pizzi Date: Thu, 4 Jul 2024 09:13:03 +0200 Subject: [PATCH 30/34] Apply suggestions from code review Co-authored-by: Antanas Vaitkus --- optimade.rst | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/optimade.rst b/optimade.rst index 2e512801d..064628bb6 100644 --- a/optimade.rst +++ b/optimade.rst @@ -639,11 +639,11 @@ The field :field:`array_axes` is defined as follows: The implementation MUST preserve the values as given in the query parameter, including the distinction between specific values and default values even when they are equivalent. It MAY contain the following subfields that are defined according to the specification of a `slice object`_. - - :field:`start`: Integer or :val:`null`. + - :field:`start`: Non-negative integer or :val:`null`. - - :field:`stop`: Integer or :val:`null`. + - :field:`stop`: Non-negative integer or :val:`null`. - - :field:`step`: Integer or :val:`null`. + - :field:`step`: Positive integer or :val:`null`. A :val:`null` value for any of these fields means it has the default value defined for a `slice object`_. A missing value is equivalent to a :val:`null` value. @@ -1379,8 +1379,8 @@ One additional query parameter :query-param:`property_slices` MUST be handled by - **property\_slices**: A number of slice specifications to request only parts of array properties for the functionality described in `Slices of array properties`_. - The query parameter contains a comma-separated (",", ASCII 44(0x2C)) list of slice specifications. - Each slice specification consists of an ordered sequence of four elements separated by the colon character (":", ASCII 58(0x3A)). + The query parameter contains a comma-separated (",", ASCII symbol 44 dec, 0x2C hex) list of slice specifications. + Each slice specification consists of an ordered sequence of four elements separated by the colon character (":", ASCII symbol 58 dec, 0x3A hex). The elements in the sequence are the dimension name and the three components of the slice, i.e., the start, stop, and step values defined in the same way as for a `slice object`_. Omitting the value for any of the components of the slice specifies a default value (however, all the colon separators MUST be included). The start value specifies the first index in that dimension for which values should be returned (which is 0-based and inclusive). From 46eafb55dc3f584270cad6a0cd990cc247b0cab7 Mon Sep 17 00:00:00 2001 From: Giovanni Pizzi Date: Thu, 4 Jul 2024 09:20:25 +0200 Subject: [PATCH 31/34] Apply suggestions from code review Co-authored-by: Antanas Vaitkus --- optimade.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/optimade.rst b/optimade.rst index 064628bb6..46aa40eaa 100644 --- a/optimade.rst +++ b/optimade.rst @@ -221,7 +221,7 @@ representation in all contexts. They are as follows: Multidimensional collections of items are represented as nested lists. The specification uses **array** as a more general term for structures of nested lists representing single or multidimensional data, and the term **array axes** for the levels of nesting. Note that arrays are represented using lists and not as a separate data type. -- **dictionary**: an associative set of **keys** and **values**, where **keys** are pre-determined strings, i.e., for the same entry property, the **keys** remain the same among different entries whereas the **values** change. +- **dictionary**: a collection of **key**-**value** pairs, where **keys** are pre-determined strings, i.e., for the same entry property the **keys** remain the same among different entries whereas the **values** change. The **values** of a dictionary can be any basic type, list, dictionary, or unknown. An entry property value that is not present in the database is **unknown**. From 657a696526f0f050be16d111d0cab402d6a362bb Mon Sep 17 00:00:00 2001 From: Rickard Armiento Date: Tue, 18 Jun 2024 16:01:53 +0200 Subject: [PATCH 32/34] Change the format of property_slices to the syntax dim_xxx[start:stop:step] --- optimade.rst | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/optimade.rst b/optimade.rst index 46aa40eaa..59a0e240c 100644 --- a/optimade.rst +++ b/optimade.rst @@ -1379,8 +1379,9 @@ One additional query parameter :query-param:`property_slices` MUST be handled by - **property\_slices**: A number of slice specifications to request only parts of array properties for the functionality described in `Slices of array properties`_. - The query parameter contains a comma-separated (",", ASCII symbol 44 dec, 0x2C hex) list of slice specifications. - Each slice specification consists of an ordered sequence of four elements separated by the colon character (":", ASCII symbol 58 dec, 0x3A hex). + The query parameter contains a comma-separated (",", ASCII symbol 44 dec) list of slice specifications. + Each slice specification consists of a dimension name followed by the left square bracket character ("[", ASCII symbol 91 dec), a slice, and the right square bracket character ("]", ASCII symbol 93 dec). + The slice is given as an ordered sequence of three components separated by the colon character (":", ASCII symbol 58 dec). The elements in the sequence are the dimension name and the three components of the slice, i.e., the start, stop, and step values defined in the same way as for a `slice object`_. Omitting the value for any of the components of the slice specifies a default value (however, all the colon separators MUST be included). The start value specifies the first index in that dimension for which values should be returned (which is 0-based and inclusive). From 919b83d3e9cdcd3d07942aeef1ff5a2a0ef0c3c3 Mon Sep 17 00:00:00 2001 From: Rickard Armiento Date: Fri, 18 Oct 2024 11:15:25 +0200 Subject: [PATCH 33/34] Fix in formulation for property_slices format dim_xxx[start:stop:step] --- optimade.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/optimade.rst b/optimade.rst index 59a0e240c..b2e8a45f8 100644 --- a/optimade.rst +++ b/optimade.rst @@ -1382,7 +1382,7 @@ One additional query parameter :query-param:`property_slices` MUST be handled by The query parameter contains a comma-separated (",", ASCII symbol 44 dec) list of slice specifications. Each slice specification consists of a dimension name followed by the left square bracket character ("[", ASCII symbol 91 dec), a slice, and the right square bracket character ("]", ASCII symbol 93 dec). The slice is given as an ordered sequence of three components separated by the colon character (":", ASCII symbol 58 dec). - The elements in the sequence are the dimension name and the three components of the slice, i.e., the start, stop, and step values defined in the same way as for a `slice object`_. + The elements in the sequence are the start, stop, and step values defined in the same way as for a `slice object`_. Omitting the value for any of the components of the slice specifies a default value (however, all the colon separators MUST be included). The start value specifies the first index in that dimension for which values should be returned (which is 0-based and inclusive). The default is :val:`0`. From a623390c6b4073382eda72d2230f69c35b271570 Mon Sep 17 00:00:00 2001 From: Rickard Armiento Date: Fri, 18 Oct 2024 16:47:43 +0200 Subject: [PATCH 34/34] Apply suggestions from review Co-authored-by: ndaelman-hu <107392603+ndaelman-hu@users.noreply.github.com> --- optimade.rst | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/optimade.rst b/optimade.rst index b2e8a45f8..66fea6426 100644 --- a/optimade.rst +++ b/optimade.rst @@ -612,8 +612,8 @@ Slices of array properties -------------------------- The OPTIMADE standard defines a way for a client to request only a subset of the items of an array, referred to as a slice. -The protocol for this functionality allows the server to allow slicing independently per entry, per array, and per array axis. -This functionality is separate from the protocol described in `Transmission of large property values`_. +The protocol for this functionality specifies how a server MAY support slicing arrays independently per entry, per array, and per array axis. +This functionality is separate from (but compatible with) the protocol described in `Transmission of large property values`_. Slices are used for a client to ask the server to only provide a subset of items of an array, which can result in a small or large set of items. In contrast, the protocol for large property values is used by the server implementation to transmit a set of items that it deems too large to provide inside the normal OPTIMADE response. @@ -623,7 +623,7 @@ When the client request includes the query parameter :query-param:`property_slic The field :field:`array_axes` is defined as follows: - :field:`array_axes`: List of Dictionary. - A list of dictionaries which provide information related to the axes of an array property, which is relevant for slicing. + A list of dictionaries which provide information related to the axes of an array property, including sliceable axes. Each item, in order, represents the array axis as declared in the corresponding property definition. Each item MUST be a dictionary with the following REQUIRED field: @@ -680,8 +680,9 @@ The field :field:`array_axes` is defined as follows: If the field is omitted or :val:`null`, it means the same thing as :val:`false`. - :field:`available_slice`: Dictionary or :val:`null`. - This field describes a `slice object`_ where there MAY be non-null values in the data. - By including this field, the server certifies that there are only :val:`null` values outside this slice. + This field describes a `slice object`_. + By including this field, the server certifies that there MUST only be :val:`null` values outside this slice. + Inside the slice there MAY be any combination of null and non-null values. If not provided, or equal to :val:`null` or an empty dictionary, the client cannot make any assumptions about what part of the array contains :val:`null` values. Examples: @@ -2375,7 +2376,7 @@ A Property Definition MUST be composed according to the combination of the requi A list of names of the dimensions of the underlying one or multi-dimensionsional data represented as multiple levels of lists. The order is such that the first name applies to the outermost list, the next name to the lists embedded in that list, etc. Dimension names defined by the OPTIMADE standard are prefixed by ``dim_``. - Dimension names defined by database or definition providers MUST be prefixed by the corresponding database or namespace prefix, and SHOULD also be prefixed by ``dim_``, e.g., ``_exmpl_dim_particles``. + Dimension names defined by database or definition providers MUST be prefixed by the corresponding database or namespace prefix, which SHOULD then be immediately followed by ``dim_``, e.g., ``_exmpl_dim_particles``. If, within one entry, two or more array axes in one or more properties share the same dimension :field:`name`, those represent the same dimension. For example, let us consider the property :property:`frame_cartesian_site_positions` of the trajectory entry, where the first dimension name is :val:`dim_frames`. If there is another one-dimensional (i.e., with a single axis) array property :property:`_exmpl_energy` of the same trajectory entry that specifies in its property definition the same dimension name :val:`dim_frames` for its axis, then the values of :property:`_exmpl_energy` and of :property:`frame_cartesian_site_positions` at index *i* pertain to the same frame.