Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding the property_ranges query parameter and associated metadata #481

Open
wants to merge 39 commits into
base: develop
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
39 commits
Select commit Hold shift + click to select a range
3b9cbc5
Intermediate progress writing property ranges parameter.
JPBergsma Jun 26, 2023
eae1679
Added metadata fields to be able to use property ranges query parameter.
JPBergsma Jun 27, 2023
f00b52d
Small corrections.
JPBergsma Jun 28, 2023
2a0bef8
moved property ranges to single_entryendpoint query param + further r…
JPBergsma Jun 29, 2023
4a00af4
Some small changes after proof reading.
JPBergsma Jun 29, 2023
6905d7f
added that fields should be placed in the range dictionary.
JPBergsma Jun 29, 2023
d579c6e
Some more sentences that had to be moved to a new line.
JPBergsma Jun 30, 2023
6792962
Update optimade.rst
JPBergsma Jun 30, 2023
93ecd0b
removed unintentional changes in appendix.
JPBergsma Jun 30, 2023
a84c72e
Seperated property_ranges query parameter from the partial data.
JPBergsma Jul 4, 2023
dc11574
Merge branch 'develop' into JPBergsma/property_ranges
ml-evs Dec 19, 2023
17beeef
Apply formatting suggestions from code review
ml-evs Jan 18, 2024
cc89029
Merge branch 'develop' into JPBergsma/property_ranges
ml-evs Jan 18, 2024
6d8e24f
Apply suggestions from review
rartino Mar 22, 2024
680317b
Merge branch 'develop' into JPBergsma/property_ranges
rartino Mar 22, 2024
f2a2799
Merge branch 'develop' into JPBergsma/property_ranges
rartino Mar 22, 2024
63e3821
Minor corrections from review
rartino Mar 22, 2024
0efb5e7
Merge branch 'develop' into JPBergsma/property_ranges
rartino Jun 13, 2024
bf7e5b4
Edit property slice definition [WIP]
rartino Jun 13, 2024
d029756
Edit property slice definition [WIP]
rartino Jun 13, 2024
63f122f
Edit property slice definition [WIP]
rartino Jun 13, 2024
f863308
Edit property slice definition [WIP]
rartino Jun 13, 2024
a73029a
Edit property slice definition [WIP]
rartino Jun 13, 2024
c098e47
Improve handing of Python dependencies
rartino Jun 14, 2024
4f704c1
Improve handing of Python dependencies
rartino Jun 14, 2024
266545b
Improve handing of Python dependencies
rartino Jun 14, 2024
e562f49
Revert "Improve handing of Python dependencies"
rartino Jun 14, 2024
315546a
Edit property slice definition [WIP]
rartino Jun 14, 2024
ce9661b
Small sentence fix
giovannipizzi Jun 14, 2024
358157f
Clarifying the part on dimension names.
giovannipizzi Jun 14, 2024
7e29652
Apply suggestions from review
rartino Jun 15, 2024
085030d
Define the term array for OPTMADE data types
rartino Jun 15, 2024
b3a96a0
Fix examples for slicing assuming a trajectory cartesian_site_coordin…
rartino Jun 15, 2024
6231d2a
Adjust confusing phrasing about slicing based on review comment
rartino Jun 15, 2024
9f639d1
Apply suggestions from code review
giovannipizzi Jul 4, 2024
46eafb5
Apply suggestions from code review
giovannipizzi Jul 4, 2024
657a696
Change the format of property_slices to the syntax dim_xxx[start:stop…
rartino Jun 18, 2024
919b83d
Fix in formulation for property_slices format dim_xxx[start:stop:step]
rartino Oct 18, 2024
a623390
Apply suggestions from review
rartino Oct 18, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Edit property slice definition [WIP]
  • Loading branch information
rartino committed Jun 13, 2024
commit 63f122f635792be59525cb1956499524702f62f3
104 changes: 70 additions & 34 deletions optimade.rst
Original file line number Diff line number Diff line change
Expand Up @@ -610,24 +610,24 @@ Slices of array properties
# response_fields=property_metadata means MUST RETURN ALL METADATA.

The OPTIMADE standard defines a way for a client to request only a subset of the items of an array, referred to as a slice.
The protocol for this functionality allows the server to allow slicing independently per entry, per array, and per array dimension.
The protocol for this functionality allows the server to allow slicing independently per entry, per array, and per array axis.
This functionality is separate from the protocol described in `Transmission of large property values`_.
Slices are used for a client to ask the server to only provide a subset of items of an array, which can result in a small or large set of items.
In contrast, the protocol for large property values is used by the server implementation to transmit a set of items that it deems too large to provide inside the normal OPTIMADE response.

The main mechanism is provided through the query parameter :query-param:`property_slices` defined in section `Single Entry URL Query Parameters`_.
Information relating to the ability of the server to handle this query parameter and the relevant ranges of indexes is provided using metadata property fields (see `Metadata properties`_) of the array property as defined below:

- :field:`dimensions`: List of Dictionary.
A list of dictionaries which provide information related to the extents of an array property, which is relevant for slicing.
Each item, in order, represents the array dimension as declared in the corresponding property definition.
- :field:`array_axes`: List of Dictionary.
A list of dictionaries which provide information related to the axes of an array property, which is relevant for slicing.
Each item, in order, represents the array axis as declared in the corresponding property definition.

Each item MUST be a dictionary with the following REQUIRED fields:

- :field:`name`: String.
The dimension name of the corresponding array dimension.
- :field:`dimension_name`: String.
The dimension name of the corresponding array axis.

If the request specifies the :query-param:`property_slices` query parameter for any of the array dimensions of this array property the following key MUST be present:
If the request specifies the :query-param:`property_slices` query parameter for any of the array axes of this array property the following key MUST be present, otherwise it MUST be omitted or equal to :val:`null`:

- :field:`requested_slice`: Dictionary.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So, these are very different modes of querying. Typically, a client will have to query them in separate stages.

  1. I'd think it would be good to explicitely state this, either between this and specification above, or at the very beginning of the enumeration.
  2. Fun edge case: which response should a server provide when both fields are used in a query? Kind of silly, but I could see clients make this mistake.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure what you mean here: these fields are included as information from the server to the client to provide information related to slicing; so I'm not sure what the two different "modes" you refer to, or how you from what is written here or what stages you mean. Maybe you can clarify a bit?

Copy link

@ndaelman-hu ndaelman-hu Oct 18, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With 2 query modes, I meant:

  1. requesting metadata about the schema (explicitly demanding metadata on sliceable properties). This section explains the responses format.
  2. actually querying for data, sliced to the desired format.

Perhaps mode 1 could be split into 1.1 (sliceable metadata implicit requested) and 1.2 (sliceable metadata explicitly requested).

Anyhow, I was just asking for a clearer distinction between which query mode is covered where. Subtitles could work? Maybe I highlighted the wrong lines, but that would underscore my point.

Lastly, my 2nd question was about the responses when a query asks for both schema metadata and regular data.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I found the source of my confusion, let me add a suggested modification

A field that describes the requested slice that was provided via the query parameter :query-param:`property_slices`.
Expand Down Expand Up @@ -665,12 +665,15 @@ Information relating to the ability of the server to handle this query parameter

The dictionary MAY contain the following fields:

- :field:`size`: Integer.
The length of this dimension for the specific entry that this metadata entry pertains to.
- :field:`length`: Integer.
The length of this array axis which MUST be the same as the length of the declared dimension for this axis in the corresponding property definition.
Note that the length of a dimension can be different for different entries if the length is not explicitly declared by the property definition.
For example, the number of frames, i.e., the length of the dimension named ``dim_frames``, is generally different for different trajectories.

- :field:`sliceable`: Boolean.
If true, the server MUST handle slices for that dimension.
If false (which is the default), the server MAY handle slices for that dimension, or MAY return the error :http-error:`501 Not Implemented` when a client requests a slice involving this dimension.
If :val:`true`, the server MUST handle slices for that dimension.
If :val:`false`, the server MAY handle slices for that dimension, or MAY return the error :http-error:`501 Not Implemented` when a client requests a slice involving this dimension.
If the field is omitted or :val:`null`, it means the same thing as :val:`false`.

- :field:`available_slice`: Dictionary or :val:`null`.
This field describes a `slice object`_ where there MAY be non-null values in the data.
Expand All @@ -682,40 +685,73 @@ Information relating to the ability of the server to handle this query parameter
- ``{"start": 3, "stop": 7, "step": 2}`` means the server certifies that values at indexes 0,1,2,4,6 and any index from 8 to the end of the array are :val:`null`.


Below follows an example of the :field:`data` and :field:`meta` parts of a response using the JSON response format.
It communicates that the property value has been omitted from the response and metadata that makes it possible to request only a part of the data for this property.
Below follows an example of the :field:`data` and :field:`meta` parts of a response using the JSON response format for a request to the trajectory endpoint with the query parameter :query-param:`property_slices=dim_frames:3:37:5` and :query-param:`response_fields=cartesian_site_positions,_exmpl_temperature` where the trajectory consists of 432934 frames (with indexes 0 to 432933) and where the :field:`cartisian_site_positions` contains 7 sites:

.. code:: jsonc

{
// ...
"data": {
"type": "structures",
"type": "trajectories",
"id": "2345678",
"attributes": {
"a": null
}
"cartesian_site_positions": null,
"_exmpl_temperature": null
},
"meta": {
"property_metadata": {
"a": {
"range": {
"range_ids": ["frames", "particles"],
"indexable_dim": ["frames"],
"data_range": [
{
"start": 1,
"step": 1,
"stop": 200,
"cartesian_site_positions": {
"array_axes": [
{
"dimension_name": "dim_frames",
"requested_slice": {
"start": 3,
"stop": 37,
"step": 5
},
{
"start": 1,
"step": 1,
"stop": 3,
}
],
"layout": "dense",
"nvalues": 600
}
"available_slice": {
"start": 0,
"stop": 432933,
"step": 1
},
"sliceable": true,
"length": 432934,
}
{
"dimension_name": "dim_sites",
"requested_slice": {
"start": 3,
"stop": 37,
"step": 5
},
"available_slice": {
"start": 0,
"stop": 432933,
"step": 1
},
"sliceable": true,
"length": 432934,
}
],
},
"_exmpl_temperature": {
"array_axes": [
{
"dimension_name": "dim_frames",
"requested_slice": {
"start": 3,
"stop": 37,
"step": 5
},
"available_slice": {
"start": 1000,
"stop": 4000,
"step": 30
},
"sliceable": true,
"length": 432934,
}
]
}
},
"partial_data_links": {
Expand Down