[DRAFT] Array Support POC

[forestmvey]

Description

Add support for indexing arrays using square parenthesis. This implementation gives users the responsibility on how to handle their data. If array indexing parenthesis are supplied then the plugin assumes an array is to be found with supplied field name. In the event of non-array values or invalid indexes used by the user, an empty value will be returned.

The goal of adding the syntax option for returning and indexing arrays is to provide users the ability to utilize arrays without the need for introducing a breaking change with the SQL plugin. By offloading the responsibility for how to handle stored array data we can avoid the need for additional data mapping.

Syntax

• array[:]
• array[index]<[index]>

Examples

Dataset:

"array": [[1], [2, [3, 4]], 5]

---

Query:

SELECT array FROM arrays;

Result:
1

---

Query:

SELECT array[:] FROM arrays;

Result:
[[1], [2, [3, 4]], 5]

---

Query:

SELECT array[0] FROM arrays;

Result:
[1]

---

Query:

SELECT array[1] FROM arrays;

Result:
[2, [3, 4]]

---

Query:

SELECT array[1][1] FROM arrays;

Result:
[3, 4]

---

Query:

SELECT array[1][1][0] FROM arrays;

Result:
3

---

Query:

SELECT array[2] FROM arrays;

Result:
5

---

Query:

SELECT array[3] FROM arrays;

Result:
null

---

Object Arrays

The V2 engine has the current functionality for object arrays.
Dataset:

"objectArray": [{"id": [1, 2]}, {"id": 2}],
"nestedArray": [{"id": [1, 2]}, {"id": 2}]

---

Query:

SELECT objectArray FROM arrays;

Result
{"id": 1}

---

Query:

SELECT nestedArray FROM arrays;

Result
[{"id": [1, 2]}, {"id": 2}]

---

Added Functionality

---

Query:

SELECT objectArray[:] FROM arrays;
SELECT nestedArray[:] FROM arrays;

Result
[{"id": [1, 2]}, {"id": 2}]

---

Query:

SELECT objectArray[1] FROM arrays;
SELECT nestedArray[1] FROM arrays;

Result
{"id": 2}

---

Query:

SELECT objectArray[1].id FROM arrays;
SELECT nestedArray[1].id FROM arrays;

Result
[1, 2]

---

Query:

SELECT objectArray[1].id[1] FROM arrays;
SELECT nestedArray[1].id[1] FROM arrays;

Result
2

---

Edge Cases

• Index out of bounds returns empty
• Indexing non-array value returns empty

Additional Functionality

• Add support for ranges: array[<firstIndex>:<lastIndex>]

Limitations

Array support is limited to the support that OpenSearch has for arrays. Arrays are not mapped as array type but the type defined within the arrays. As well arrays are stored as a single document and will return entire arrays if a portion of the array matches a filter. For example an array consisting of [1, 2] would return the whole array when filtering for values of 1 or 2.

Clause Support

• ✅ SELECT Clause
  • ❌ Function Support
• ❌ ORDER BY
• ❌ WHERE
• ❌ GROUP BY
• ❌ HAVING

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[codecov]

Codecov Report

Merging #282 (41d7bf5) into integ-array-support-poc (e186cf7) will decrease coverage by 0.69%.
The diff coverage is 49.36%.

@@                      Coverage Diff                      @@
##             integ-array-support-poc     #282      +/-   ##
=============================================================
- Coverage                      99.98%   99.30%   -0.69%     
- Complexity                      2624     2637      +13     
=============================================================
  Files                            205      206       +1     
  Lines                           5955     6025      +70     
  Branches                         378      392      +14     
=============================================================
+ Hits                            5954     5983      +29     
- Misses                             1       38      +37     
- Partials                           0        4       +4     

Flag	Coverage Δ
sql-engine	99.30% <49.36%> (-0.69%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...ensearch/sql/expression/ExpressionNodeVisitor.java	95.45% <0.00%> (-4.55%)	⬇️
...org/opensearch/sql/analysis/QualifierAnalyzer.java	69.23% <11.11%> (-30.77%)	⬇️
...rg/opensearch/sql/analysis/ExpressionAnalyzer.java	92.09% <36.36%> (-7.91%)	⬇️
...earch/sql/expression/ArrayReferenceExpression.java	43.33% <43.33%> (ø)
...nsearch/sql/analysis/SelectExpressionAnalyzer.java	100.00% <100.00%> (ø)
...opensearch/sql/expression/HighlightExpression.java	96.29% <100.00%> (-3.71%)	⬇️
...opensearch/sql/expression/ReferenceExpression.java	100.00% <100.00%> (ø)
...h/sql/expression/function/OpenSearchFunctions.java	100.00% <100.00%> (ø)
...nsearch/sql/storage/bindingtuple/BindingTuple.java	100.00% <100.00%> (ø)

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[GumpacG]

Regarding future support for features like geo_shape, consider cases like:
Data:

{
    "borders": [
        {
            "border": {
                "type" : "linestring",
                "coordinates" : [[74.0060, 40.7128], [71.0589, 42.3601]]
            }
        },
        {
            "border": {
                "type" : "linestring",
                "coordinates" : [[74.0060, 40.7128], [71.0589, 42.3601]]
            }
        },
        {
            "border": {
                "type" : "linestring",
                "coordinates" : [[74.0060, 40.7128], [71.0589, 42.3601]]
            }
        },
        {
            "border": {
                "type" : "linestring",
                "coordinates" : [[74.0060, 40.7128], [71.0589, 42.3601]]
            }
        }
    ]  
}

• accessing coordinates with borders[0].border.coordinates
• accessing starting points of border with borders[0].border.coordinates[0]

Note: this example may be an edge case and may not represent typical real world use cases.

[Yury-Fridlyand]

1.

Please, add more tests:

• Array of objects
• Array in object
• Array in nested

Consider different tricky combinations, like this for the first test:

[
  {
    "id" : 1,
    "name" : "one"
  },
  {
    "id" : 2,
    "name" : "two"
  },
  null
]

2.

What if I do?

select int0[0] from calcs

Please, add a test for this.

3.

Please add a design doc.

4.

Please, fix column headers/names.

[Yury-Fridlyand]

Isn't a breaking change? Please, compare this with nested behavior in V1 and V2 @ 2.8.

[forestmvey]

This is V1 behaviour, but my updates make this consistent with V2 behaviour. I believe this should be considered a bug and not a breaking change because if we were to do a select on this value without the nested function, a 3 would be returned. Very weird case here but this new change aligns with whats expected with V2.

[Yury-Fridlyand]

Could decimalLiteral be negative?

[Yury-Fridlyand]

Make decimalLiteral named value in ANTLR grammar, so you can reference the name here.

[GumpacG]

Could you add an example in the PR description for SELECT *? Will this return the full array for a column when SELECT * is queried? Since SELECT array still returns just the first value, is this still the case for SELECT * as well?

[forestmvey]

Yes the functionality for SELECT * will stay the same, otherwise would be a breaking change.

[Yury-Fridlyand]

Is it a regex? Do you want to match : or \d+:\d+ (or even \d*:\d*)?
Add a comment for this function please.