-
Notifications
You must be signed in to change notification settings - Fork 49
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge branch 'main' into field_storage_interface
- Loading branch information
Showing
47 changed files
with
1,020 additions
and
420 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -117,3 +117,4 @@ test py310: | |
| - SUBPACKAGE: eve | ||
| - SUBPACKAGE: next | ||
| VARIANT: [-nomesh, -atlas] | ||
| SUBVARIANT: [-cuda11x, -cpu] | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
118 changes: 118 additions & 0 deletions
118
docs/development/ADRs/0016-Multiple-Backends-and-Build-Systems.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,118 @@ | ||
| --- | ||
| tags: [backend, gridtools, bindings, libraries, otf] | ||
| --- | ||
|
|
||
| # Support for Multiple Backends, Build Systems and Libraries | ||
|
|
||
| - **Status**: valid | ||
| - **Authors**: Rico Häuselmann (@DropD) | ||
| - **Created**: 2023-10-11 | ||
| - **Updated**: 2023-10-11 | ||
|
|
||
| In the process of enabling CUDA for the GTFN backend, we encountered a potential support matrix of build systems x target language libraries. The current design requires build systems about all the libraries they can be used with. We decided that the matrix is too small for now and to not revisit the existing design yet. | ||
|
|
||
| ## Context | ||
|
|
||
| ADRs [0009](0009-Compiled_Backend_Integration.md), [0011](0011-On_The_Fly_Compilation.md) and [0012](0012-GridTools_Cpp_OTF_Steps.md) detail the design decisions around what is loosely referred as "gt4py.next backends". In summary the goals are: | ||
|
|
||
| - extensibility | ||
| - adding backends should not require changing existing code | ||
| - adding / modifying backend modules like build systems / compilers should not be blocked by assumptions in other modules. | ||
| - modularity | ||
| - increase the chance that two different backends (for example GTFN and another C++ backend) can share code. | ||
|
|
||
| Therefore the concerns of generating code in the target language, generating python bindings in the target language and of building (compiling) the generated code are separated it code generator, bindings generator and compile step / build system. The compile step is written to be build system agnostic. | ||
|
|
||
| There is one category that connects all these concerns: libraries written in the target language and used in generated / bindings code. | ||
|
|
||
| Current design: | ||
|
|
||
| ```mermaid | ||
| graph LR | ||
| gtgen("GTFN code generator (C++/Cuda)") --> |GridTools::fn_naive| Compiler | ||
| gtgen("GTFN code generator (C++/Cuda)") --> |GridTools::fn_gpu| Compiler | ||
| nb("nanobind bindings generator") --> |nanobind| Compiler | ||
| Compiler --> CMakeProject --> CMakeListsGenerator | ||
| Compiler --> CompiledbProject --> CMakeListsGenerator | ||
| ``` | ||
|
|
||
| The current design contains two mappings: | ||
|
|
||
| - library name -> CMake `find_package()` call | ||
| - library name -> CMake target name | ||
|
|
||
| and the gridtools cpu/gpu link targets are differentiated by internally separating between two fictitious "gridtools_cpu" and "gridtools_gpu" libraries. | ||
|
|
||
| ## concerns | ||
|
|
||
| ### Usage | ||
|
|
||
| The "gridtools_cpu" and "gridtools_gpu" fake library names add to the learning curve for this part of the code. Reuse of the existing components might require this knowledge. | ||
|
|
||
| ### Scalability | ||
|
|
||
| Adding a new backend using the existing build systems but relying on different libraries has to modify existing build system components (at the very least CMakeListsGenerator). | ||
|
|
||
| ### Separation of concerns | ||
|
|
||
| It makes more sense to separate the concerns of how to generate a valid build system configuration and how to use a particular library in a particular build system than to mix the two. | ||
|
|
||
| ## Decision | ||
|
|
||
| Currently the code overhead is in the tens of lines, and there are no concrete plans to add more compiled backends or different build systems. Therefore we decide to keep the current design for now but to redesign as soon as the matrix grows. | ||
| To this end ToDo comments are added in the relevant places | ||
|
|
||
| ## Consequences | ||
|
|
||
| Initial GTFN gpu support will not be blocked by design work. | ||
|
|
||
| ## Alternatives Considered | ||
|
|
||
| ### Push build system support to the LibraryDependency instance | ||
|
|
||
| ``` | ||
| #src/gt4py/next/otf/binding/interface.py | ||
| ... | ||
| class LibraryDependency: | ||
| name: str | ||
| version: str | ||
| link_targets: list[str] | ||
| include_headers: list[str] | ||
| ``` | ||
|
|
||
| - Simple, choice is made at code generator level, where the knowledge should be | ||
| - Interface might not suit every build system | ||
| - Up to the implementer to make the logic for choosing reusable (or not) | ||
|
|
||
| ### Create additional data structures to properly separate concerns | ||
|
|
||
| ``` | ||
| class BuildSystemConfig: | ||
| device_type: core_defs.DeviceType | ||
| ... | ||
| class LibraryAdaptor: | ||
| library: LibraryDependency | ||
| build_system: CMakeProject | ||
| def config_phase(self, config: BuildSystemConfig) -> str: | ||
| import gridtools_cpp | ||
| cmake_dir = gridtools_cpp.get_cmake_dir() | ||
| return f"find_package(... {cmake_dir} ... )" | ||
| def build_phase(self, config: BuildSystemConfig) -> str: | ||
| return "" # header only library | ||
| def link_phase(self, main_target_name: str, config: BuildSystemConfig) -> str: | ||
| return f"target_link_libraries({main_target_name} ...)" | ||
| ``` | ||
|
|
||
| - More general and fully extensible, adaptors can be added for any required library / build system combination without touching existing code (depending on the registering mechanism). | ||
| - More likely to be reusable as choices are explicit and can be overridden separately by sub classing. | ||
| - More design work required. Open questions: | ||
| - Design the interface to work with any build system | ||
| - How to register adaptors? entry points? global dictionary? |
105 changes: 105 additions & 0 deletions
105
src/gt4py/next/ffront/foast_passes/type_alias_replacement.py
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,105 @@ | ||
| # GT4Py - GridTools Framework | ||
| # | ||
| # Copyright (c) 2014-2023, ETH Zurich | ||
| # All rights reserved. | ||
| # | ||
| # This file is part of the GT4Py project and the GridTools framework. | ||
| # GT4Py is free software: you can redistribute it and/or modify it under | ||
| # the terms of the GNU General Public License as published by the | ||
| # Free Software Foundation, either version 3 of the License, or any later | ||
| # version. See the LICENSE.txt file at the top-level directory of this | ||
| # distribution for a copy of the license or check <https://www.gnu.org/licenses/>. | ||
| # | ||
| # SPDX-License-Identifier: GPL-3.0-or-later | ||
|
|
||
| from dataclasses import dataclass | ||
| from typing import Any, cast | ||
|
|
||
| import gt4py.next.ffront.field_operator_ast as foast | ||
| from gt4py.eve import NodeTranslator, traits | ||
| from gt4py.eve.concepts import SourceLocation, SymbolName, SymbolRef | ||
| from gt4py.next.ffront import dialect_ast_enums | ||
| from gt4py.next.ffront.fbuiltins import TYPE_BUILTIN_NAMES | ||
| from gt4py.next.type_system import type_specifications as ts | ||
| from gt4py.next.type_system.type_translation import from_type_hint | ||
|
|
||
|
|
||
| @dataclass | ||
| class TypeAliasReplacement(NodeTranslator, traits.VisitorWithSymbolTableTrait): | ||
| """ | ||
| Replace Type Aliases with their actual type. | ||
| After this pass, the type aliases used for explicit construction of literal | ||
| values and for casting field values are replaced by their actual types. | ||
| """ | ||
|
|
||
| closure_vars: dict[str, Any] | ||
|
|
||
| @classmethod | ||
| def apply( | ||
| cls, node: foast.FunctionDefinition | foast.FieldOperator, closure_vars: dict[str, Any] | ||
| ) -> tuple[foast.FunctionDefinition, dict[str, Any]]: | ||
| foast_node = cls(closure_vars=closure_vars).visit(node) | ||
| new_closure_vars = closure_vars.copy() | ||
| for key, value in closure_vars.items(): | ||
| if isinstance(value, type) and key not in TYPE_BUILTIN_NAMES: | ||
| new_closure_vars[value.__name__] = closure_vars[key] | ||
| return foast_node, new_closure_vars | ||
|
|
||
| def is_type_alias(self, node_id: SymbolName | SymbolRef) -> bool: | ||
| return ( | ||
| node_id in self.closure_vars | ||
| and isinstance(self.closure_vars[node_id], type) | ||
| and node_id not in TYPE_BUILTIN_NAMES | ||
| ) | ||
|
|
||
| def visit_Name(self, node: foast.Name, **kwargs) -> foast.Name: | ||
| if self.is_type_alias(node.id): | ||
| return foast.Name( | ||
| id=self.closure_vars[node.id].__name__, location=node.location, type=node.type | ||
| ) | ||
| return node | ||
|
|
||
| def _update_closure_var_symbols( | ||
| self, closure_vars: list[foast.Symbol], location: SourceLocation | ||
| ) -> list[foast.Symbol]: | ||
| new_closure_vars: list[foast.Symbol] = [] | ||
| existing_type_names: set[str] = set() | ||
|
|
||
| for var in closure_vars: | ||
| if self.is_type_alias(var.id): | ||
| actual_type_name = self.closure_vars[var.id].__name__ | ||
| # Avoid multiple definitions of a type in closure_vars | ||
| if actual_type_name not in existing_type_names: | ||
| new_closure_vars.append( | ||
| foast.Symbol( | ||
| id=actual_type_name, | ||
| type=ts.FunctionType( | ||
| pos_or_kw_args={}, | ||
| kw_only_args={}, | ||
| pos_only_args=[ts.DeferredType(constraint=ts.ScalarType)], | ||
| returns=cast( | ||
| ts.DataType, from_type_hint(self.closure_vars[var.id]) | ||
| ), | ||
| ), | ||
| namespace=dialect_ast_enums.Namespace.CLOSURE, | ||
| location=location, | ||
| ) | ||
| ) | ||
| existing_type_names.add(actual_type_name) | ||
| elif var.id not in existing_type_names: | ||
| new_closure_vars.append(var) | ||
| existing_type_names.add(var.id) | ||
|
|
||
| return new_closure_vars | ||
|
|
||
| def visit_FunctionDefinition( | ||
| self, node: foast.FunctionDefinition, **kwargs | ||
| ) -> foast.FunctionDefinition: | ||
| return foast.FunctionDefinition( | ||
| id=node.id, | ||
| params=node.params, | ||
| body=self.visit(node.body, **kwargs), | ||
| closure_vars=self._update_closure_var_symbols(node.closure_vars, node.location), | ||
| location=node.location, | ||
| ) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.