Merge branch 'koff_small_fix' of https://github.com/nfarabullini/gt4py …

…into koff_small_fix

.github/pull_request_template.md

@@ -15,7 +15,7 @@ Delete this comment and add a proper description of the changes contained in thi
         - test: Adding missing tests or correcting existing tests
 
     <scope>: cartesian | eve | next | storage
-    # ONLY if changes are limited to a specific subsytem
+    # ONLY if changes are limited to a specific subsystem
 
 - PR Description:
 
@@ -27,7 +27,7 @@ Delete this comment and add a proper description of the changes contained in thi
 ## Requirements
 
 - [ ] All fixes and/or new features come with corresponding tests.
-- [ ] Important design decisions have been documented in the approriate ADR inside the [docs/development/ADRs/](docs/development/ADRs/Index.md) folder.
+- [ ] Important design decisions have been documented in the appropriate ADR inside the [docs/development/ADRs/](docs/development/ADRs/Index.md) folder.
 
 If this PR contains code authored by new contributors please make sure:
 

.gitpod/.vscode/launch.json

@@ -6,19 +6,28 @@
   "configurations": [
     {
       "name": "Python: Current File (just my code)",
-      "type": "python",
+      "type": "debugpy",
       "request": "launch",
       "program": "${file}",
       "console": "integratedTerminal",
       "justMyCode": true
     },
     {
       "name": "Python: Current File (all)",
-      "type": "python",
+      "type": "debugpy",
       "request": "launch",
       "program": "${file}",
       "console": "integratedTerminal",
       "justMyCode": false
+    },
+    {
+      "name": "Python: Debug Tests",
+      "type": "debugpy",
+      "request": "launch",
+      "program": "${file}",
+      "purpose": ["debug-test"],
+      "console": "integratedTerminal",
+      "justMyCode": true
     }
   ]
 }

docs/development/ADRs/0008-Mapping_Domain_to_Cpp-Backend.md

@@ -20,7 +20,7 @@ The Python embedded execution for Iterator IR keeps track of the current locatio
 
 ### Python side
 
-On the Python side, we label dimensions of fields with the location type, e.g. `Edge` or `Vertex`. The domain uses `named_ranges` that uses the same location types to express _where_ to iterate, e.g. `named_range(Vertex, range(0, 100))` is an iteration over the `Vertex` dimension, no order in the domain is required. Additionally, the `Connectivity` (aka `NeighborTableOffsetProvider` in the current implementation) describes the mapping between location types.
+On the Python side, we label dimensions of fields with the location type, e.g. `Edge` or `Vertex`. The domain uses `named_ranges` that uses the same location types to express _where_ to iterate, e.g. `named_range(Vertex, range(0, 100))` is an iteration over the `Vertex` dimension, no order in the domain is required. Additionally, the `Connectivity` describes the mapping between location types.
 
 ### C++ side
 

docs/development/ADRs/0019-Connectivities.md

@@ -0,0 +1,55 @@
+---
+tags: []
+---
+
+# [Connectivities]
+
+- **Status**: valid
+- **Authors**: Hannes Vogt (@havogt)
+- **Created**: 2024-11-08
+- **Updated**: 2024-11-08
+
+The representation of Connectivities (neighbor tables, `NeighborTableOffsetProvider`) and their identifier (offset tag, `FieldOffset`, etc.) was extended and modified based on the needs of different parts of the toolchain. Here we outline the ideas for consolidating the different closely-related concepts.
+
+## History
+
+In the early days of Iterator IR (ITIR), an `offset` was a literal in the IR. Its meaning was only provided at execution time by a mapping from `offset` tag to an entity that we labelled `OffsetProvider`. We had mainly 2 kinds of `OffsetProvider`: a `Dimension` representing a Cartesian shift and a `NeighborTableOffsetProvider` for unstructured shifts. Since the type of `offset` needs to be known for compilation (strided for Cartesian, lookup-table for unstructured), this prevents a clean interface for ahead-of-time compilation.
+For the frontend type-checking we later introduce a `FieldOffset` which contained type information of the mapped dimensions.
+For (field-view) embedded we introduced a `ConnectivityField` (now `Connectivity`) which could be generated from the OffsetProvider information.
+
+These different concepts had overlap but were not 1-to-1 replacements.
+
+## Decision
+
+We update and introduce the following concepts
+
+### Conceptual definitions
+
+**Connectivity** is a mapping from index (or product of indices) to index. It covers 1-to-1 mappings, e.g. Cartesian shifts, NeighborTables (2D mappings) and dynamic Cartesian shifts.
+
+**NeighborConnectivity** is a 2D mapping of the N neighbors of a Location A to a Location B.
+
+**NeighborTable** is a _NeighborConnectivity_ backed by a buffer.
+
+**ConnectivityType**, **NeighborConnectivityType** contains all information that is needed for compilation.
+
+### Full definitions
+
+See `next.common` module
+
+Note: Currently, the compiled backends supports only `NeighborConnectivity`s that are `NeighborTable`s. We do not yet encode this in the type and postpone discussion to the point where we support alternative implementations (e.g. `StridedNeighborConnectivity`).
+
+## Which parts of the toolchain use which concept?
+
+### Embedded
+
+Embedded execution of field-view supports any kind of `Connectivity`.
+Embedded execution of iterator (local) view supports only `NeighborConnectivity`s.
+
+### IR transformations and compiled backends
+
+All transformations and code-generation should use `ConnectivityType`, not the `Connectivity` which contains the runtime mapping.
+
+Note, currently the `global_tmps` pass uses runtime information, therefore this is not strictly enforced.
+
+The only supported `Connectivity`s in compiled backends (currently) are `NeighborTable`s.

docs/user/next/QuickstartGuide.md

@@ -155,8 +155,6 @@ This section approaches the pseudo-laplacian by introducing the required APIs pr
 - [Using reductions on connected mesh elements](#Using-reductions-on-connected-mesh-elements)
 - [Implementing the actual pseudo-laplacian](#Implementing-the-pseudo-laplacian)
 
-+++
-
 #### Defining the mesh and its connectivities
 
 The examples related to unstructured meshes use the mesh below. The edges (in blue) and the cells (in red) are numbered with zero-based indices.
@@ -237,7 +235,7 @@ E2C = gtx.FieldOffset("E2C", source=CellDim, target=(EdgeDim,E2CDim))
 Note that the field offset does not contain the actual connectivity table, that's provided through an _offset provider_:
 
 ```{code-cell} ipython3
-E2C_offset_provider = gtx.NeighborTableOffsetProvider(edge_to_cell_table, EdgeDim, CellDim, 2)
+E2C_offset_provider = gtx.as_connectivity([EdgeDim, E2CDim], codomain=CellDim, data=edge_to_cell_table, skip_value=-1)
 ```
 
 The field operator `nearest_cell_to_edge` below shows an example of applying this transform. There is a little twist though: the subscript in `E2C[0]` means that only the value of the first connected cell is taken, the second (if exists) is ignored.
@@ -385,7 +383,7 @@ As explained in the section outline, the pseudo-laplacian needs the cell-to-edge
 C2EDim = gtx.Dimension("C2E", kind=gtx.DimensionKind.LOCAL)
 C2E = gtx.FieldOffset("C2E", source=EdgeDim, target=(CellDim, C2EDim))
 
-C2E_offset_provider = gtx.NeighborTableOffsetProvider(cell_to_edge_table, CellDim, EdgeDim, 3)
+C2E_offset_provider = gtx.as_connectivity([CellDim, C2EDim], codomain=EdgeDim, data=cell_to_edge_table, skip_value=-1)
 ```
 
 **Weights of edge differences:**

docs/user/next/workshop/exercises/2_divergence_exercise.ipynb

@@ -81,7 +81,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 9,
+   "execution_count": null,
    "id": "5dbd2f62",
    "metadata": {},
    "outputs": [],
@@ -113,7 +113,7 @@
     "        edge_orientation.asnumpy(),\n",
     "    )\n",
     "\n",
-    "    c2e_connectivity = gtx.NeighborTableOffsetProvider(c2e_table, C, E, 3, has_skip_values=False)\n",
+    "    c2e_connectivity = gtx.as_connectivity([C, C2EDim], codomain=E, data=c2e_table)\n",
     "\n",
     "    divergence_gt4py = gtx.zeros(cell_domain, allocator=backend)\n",
     "\n",

docs/user/next/workshop/exercises/2_divergence_exercise_solution.ipynb

@@ -86,7 +86,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 12,
+   "execution_count": null,
    "id": "5dbd2f62",
    "metadata": {},
    "outputs": [],
@@ -118,7 +118,7 @@
     "        edge_orientation.asnumpy(),\n",
     "    )\n",
     "\n",
-    "    c2e_connectivity = gtx.NeighborTableOffsetProvider(c2e_table, C, E, 3, has_skip_values=False)\n",
+    "    c2e_connectivity = gtx.as_connectivity([C, C2EDim], codomain=E, data=c2e_table)\n",
     "\n",
     "    divergence_gt4py = gtx.zeros(cell_domain, allocator=backend)\n",
     "\n",

docs/user/next/workshop/exercises/3_gradient_exercise.ipynb

@@ -80,7 +80,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 11,
+   "execution_count": null,
    "id": "84b02762",
    "metadata": {},
    "outputs": [],
@@ -110,7 +110,7 @@
     "        edge_orientation.asnumpy(),\n",
     "    )\n",
     "\n",
-    "    c2e_connectivity = gtx.NeighborTableOffsetProvider(c2e_table, C, E, 3, has_skip_values=False)\n",
+    "    c2e_connectivity = gtx.as_connectivity([C, C2EDim], codomain=E, data=c2e_table)\n",
     "\n",
     "    gradient_gt4py_x = gtx.zeros(cell_domain, allocator=backend)\n",
     "    gradient_gt4py_y = gtx.zeros(cell_domain, allocator=backend)\n",

docs/user/next/workshop/exercises/3_gradient_exercise_solution.ipynb

@@ -93,7 +93,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 4,
+   "execution_count": null,
    "id": "84b02762",
    "metadata": {},
    "outputs": [],
@@ -123,7 +123,7 @@
     "        edge_orientation.asnumpy(),\n",
     "    )\n",
     "\n",
-    "    c2e_connectivity = gtx.NeighborTableOffsetProvider(c2e_table, C, E, 3, has_skip_values=False)\n",
+    "    c2e_connectivity = gtx.as_connectivity([C, C2EDim], codomain=E, data=c2e_table)\n",
     "\n",
     "    gradient_gt4py_x = gtx.zeros(cell_domain, allocator=backend)\n",
     "    gradient_gt4py_y = gtx.zeros(cell_domain, allocator=backend)\n",

docs/user/next/workshop/exercises/4_curl_exercise.ipynb

@@ -102,7 +102,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 4,
+   "execution_count": null,
    "id": "5b6ffc9e",
    "metadata": {},
    "outputs": [],
@@ -134,7 +134,7 @@
     "        edge_orientation.asnumpy(),\n",
     "    )\n",
     "\n",
-    "    v2e_connectivity = gtx.NeighborTableOffsetProvider(v2e_table, V, E, 6, has_skip_values=False)\n",
+    "    v2e_connectivity = gtx.as_connectivity([V, V2EDim], codomain=E, data=v2e_table)\n",
     "\n",
     "    curl_gt4py = gtx.zeros(vertex_domain, allocator=backend)\n",
     "\n",

docs/user/next/workshop/exercises/4_curl_exercise_solution.ipynb

@@ -107,7 +107,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 4,
+   "execution_count": null,
    "id": "5b6ffc9e",
    "metadata": {},
    "outputs": [],
@@ -139,7 +139,7 @@
     "        edge_orientation.asnumpy(),\n",
     "    )\n",
     "\n",
-    "    v2e_connectivity = gtx.NeighborTableOffsetProvider(v2e_table, V, E, 6, has_skip_values=False)\n",
+    "    v2e_connectivity = gtx.as_connectivity([V, V2EDim], codomain=E, data=v2e_table)\n",
     "\n",
     "    curl_gt4py = gtx.zeros(vertex_domain, allocator=backend)\n",
     "\n",

docs/user/next/workshop/exercises/5_vector_laplace_exercise.ipynb

@@ -228,7 +228,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 8,
+   "execution_count": null,
    "id": "f9cfc097",
    "metadata": {},
    "outputs": [],
@@ -272,10 +272,10 @@
     "        edge_orientation_cell.asnumpy(),\n",
     "    )\n",
     "\n",
-    "    c2e_connectivity = gtx.NeighborTableOffsetProvider(c2e_table, C, E, 3, has_skip_values=False)\n",
-    "    v2e_connectivity = gtx.NeighborTableOffsetProvider(v2e_table, V, E, 6, has_skip_values=False)\n",
-    "    e2v_connectivity = gtx.NeighborTableOffsetProvider(e2v_table, E, V, 2, has_skip_values=False)\n",
-    "    e2c_connectivity = gtx.NeighborTableOffsetProvider(e2c_table, E, C, 2, has_skip_values=False)\n",
+    "    c2e_connectivity = gtx.as_connectivity([C, C2EDim], codomain=E, data=c2e_table)\n",
+    "    v2e_connectivity = gtx.as_connectivity([V, V2EDim], codomain=E, data=v2e_table)\n",
+    "    e2v_connectivity = gtx.as_connectivity([E, E2VDim], codomain=V, data=e2v_table)\n",
+    "    e2c_connectivity = gtx.as_connectivity([E, E2CDim], codomain=C, data=e2c_table)\n",
     "\n",
     "    laplacian_gt4py = gtx.zeros(edge_domain, allocator=backend)\n",
     "\n",

docs/user/next/workshop/exercises/5_vector_laplace_exercise_solution.ipynb

@@ -249,7 +249,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 8,
+   "execution_count": null,
    "id": "f9cfc097",
    "metadata": {},
    "outputs": [],
@@ -293,10 +293,10 @@
     "        edge_orientation_cell.asnumpy(),\n",
     "    )\n",
     "\n",
-    "    c2e_connectivity = gtx.NeighborTableOffsetProvider(c2e_table, C, E, 3, has_skip_values=False)\n",
-    "    v2e_connectivity = gtx.NeighborTableOffsetProvider(v2e_table, V, E, 6, has_skip_values=False)\n",
-    "    e2v_connectivity = gtx.NeighborTableOffsetProvider(e2v_table, E, V, 2, has_skip_values=False)\n",
-    "    e2c_connectivity = gtx.NeighborTableOffsetProvider(e2c_table, E, C, 2, has_skip_values=False)\n",
+    "    c2e_connectivity = gtx.as_connectivity([C, C2EDim], codomain=E, data=c2e_table)\n",
+    "    v2e_connectivity = gtx.as_connectivity([V, V2EDim], codomain=E, data=v2e_table)\n",
+    "    e2v_connectivity = gtx.as_connectivity([E, E2VDim], codomain=V, data=e2v_table)\n",
+    "    e2c_connectivity = gtx.as_connectivity([E, E2CDim], codomain=C, data=e2c_table)\n",
     "\n",
     "    laplacian_gt4py = gtx.zeros(edge_domain, allocator=backend)\n",
     "\n",

docs/user/next/workshop/exercises/8_diffusion_exercise_solution.ipynb

@@ -118,7 +118,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 127,
+   "execution_count": null,
    "id": "f9cfc097",
    "metadata": {},
    "outputs": [],
@@ -156,10 +156,8 @@
     "        dt,\n",
     "    )\n",
     "\n",
-    "    e2c2v_connectivity = gtx.NeighborTableOffsetProvider(\n",
-    "        e2c2v_table, E, V, 4, has_skip_values=False\n",
-    "    )\n",
-    "    v2e_connectivity = gtx.NeighborTableOffsetProvider(v2e_table, V, E, 6, has_skip_values=False)\n",
+    "    e2c2v_connectivity = gtx.as_connectivity([E, E2C2VDim], codomain=V, data=e2c2v_table)\n",
+    "    v2e_connectivity = gtx.as_connectivity([V, V2EDim], codomain=E, data=v2e_table)\n",
     "\n",
     "    diffusion_step(\n",
     "        u,\n",

docs/user/next/workshop/slides/slides_2.ipynb

@@ -281,17 +281,19 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 10,
+   "execution_count": null,
    "id": "6d30a5e1",
    "metadata": {},
    "outputs": [],
    "source": [
-    "E2C_offset_provider = gtx.NeighborTableOffsetProvider(e2c_table, Edge, Cell, 2)"
+    "E2C_offset_provider = gtx.as_connectivity(\n",
+    "    [Edge, E2CDim], codomain=Cell, data=e2c_table, skip_value=-1\n",
+    ")"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 12,
+   "execution_count": null,
    "id": "d62f6c98",
    "metadata": {},
    "outputs": [
@@ -311,7 +313,7 @@
     "    return cell_field(E2C[0])  # 0th index to isolate edge dimension\n",
     "\n",
     "\n",
-    "@gtx.program  # uses skip_values, therefore we cannot use embedded\n",
+    "@gtx.program\n",
     "def run_nearest_cell_to_edge(\n",
     "    cell_field: gtx.Field[Dims[Cell], float64], edge_field: gtx.Field[Dims[Edge], float64]\n",
     "):\n",

src/gt4py/_core/definitions.py

@@ -439,13 +439,21 @@ def ndim(self) -> int: ...
     @property
     def shape(self) -> tuple[int, ...]: ...
 
+    @property
+    def strides(self) -> tuple[int, ...]: ...
+
     @property
     def dtype(self) -> Any: ...
 
+    @property
+    def itemsize(self) -> int: ...
+
     def item(self) -> Any: ...
 
     def astype(self, dtype: npt.DTypeLike) -> NDArrayObject: ...
 
+    def any(self) -> bool: ...
+
     def __getitem__(self, item: Any) -> NDArrayObject: ...
 
     def __abs__(self) -> NDArrayObject: ...
@@ -496,4 +504,4 @@ def __and__(self, other: NDArrayObject | Scalar) -> NDArrayObject: ...
 
     def __or__(self, other: NDArrayObject | Scalar) -> NDArrayObject: ...
 
-    def __xor(self, other: NDArrayObject | Scalar) -> NDArrayObject: ...
+    def __xor__(self, other: NDArrayObject | Scalar) -> NDArrayObject: ...

src/gt4py/cartesian/frontend/gtscript_frontend.py

@@ -1460,6 +1460,13 @@ def visit_Assign(self, node: ast.Assign) -> list:
                                 loc=nodes.Location.from_ast_node(t),
                             )
 
+                    if self.backend_name in ["gt:gpu"]:
+                        raise GTScriptSyntaxError(
+                            message=f"Assignment to non-zero offsets in K is not available in {self.backend_name} as an unsolved bug remains."
+                            "Please refer to https://github.com/GridTools/gt4py/issues/1754.",
+                            loc=nodes.Location.from_ast_node(t),
+                        )
+
             if not self._is_known(name):
                 if name in self.temp_decls:
                     field_decl = self.temp_decls[name]