Skip to content

Commit

Permalink
Add icon-liskov DSL Preprocessor (#117)
Browse files Browse the repository at this point in the history
Introduces icon_liskov a preprocessor which enables the generation of integration code between ICON and Gt4Py stencils in Icon4Py.
  • Loading branch information
samkellerhals authored Feb 24, 2023
1 parent 686e5aa commit 5624c37
Show file tree
Hide file tree
Showing 41 changed files with 4,129 additions and 17 deletions.
40 changes: 23 additions & 17 deletions .flake8
Original file line number Diff line number Diff line change
Expand Up @@ -3,25 +3,31 @@
max-line-length = 100
max-complexity = 15
doctests = true
ignore =
B008 # Do not perform function calls in argument defaults
D1 # Public code object needs docstring
DAR # Disable dargling errors by default
E203 # Whitespace before ':' (black formatter breaks this sometimes)
E501 # Line too long (using Bugbear's B950 warning)
W503 # Line break occurred before a binary operator
extend-ignore =
# Do not perform function calls in argument defaults
B008,
# Public code object needs docstring
D1,
# Disable dargling errors by default
DAR,
# Whitespace before ':' (black formatter breaks this sometimes)
E203,
# Line too long (using Bugbear's B950 warning)
E501,
# Line break occurred before a binary operator
W503

exclude =
.eggs
.gt_cache
.ipynb_checkpoints
.tox
_local_
build
dist
docs
_external_src
tests/_disabled
.eggs,
.gt_cache,
.ipynb_checkpoints,
.tox,
_local_,
build,
dist,
docs,
_external_src,
tests/_disabled,
setup.py

rst-roles =
Expand Down
1 change: 1 addition & 0 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,7 @@ repos:
# args: [--remove]
- id: name-tests-test
args: [--pytest-test-first]
exclude: ^liskov/tests/samples

- repo: https://gitlab.com/bmares/check-json5
rev: v1.0.0
Expand Down
126 changes: 126 additions & 0 deletions liskov/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,126 @@
# icon4py-liskov

A preprocessor that facilitates integration of gt4py code into the ICON model.

## Installation

To install the icon4py-liskov package, follow the instructions in the `README.md` file located in the root of the repository.

## Description

The icon4py-liskov package includes the `icon_liskov` CLI tool which takes a fortran file as input and processes it with the ICON-Liskov DSL Preprocessor. This preprocessor adds the necessary `USE` statements and generates OpenACC `DATA CREATE` statements and declares DSL input/output fields based on directives in the input file. The preprocessor also processes stencils defined in the input file using the `START STENCIL` and `END STENCIL` directives, inserting the necessary code to run the stencils and adding nvtx profile statements if specified with the `--profile` flag.

### Usage

To use the `icon_liskov` tool, run the following command:

```bash
icon_liskov <input_filepath> <output_filepath> [--profile]
```

Where `input_filepath` is the path to the input file to be processed, and `output_filepath` is the path to the output file. The optional `--profile` flag adds nvtx profile statements to the stencils.

### Preprocessor directives

The ICON-Liskov DSL Preprocessor supports the following directives:

#### `!$DSL IMPORTS()`

This directive generates the necessary `USE` statements to import the Fortran to C interfaces.

#### `!$DSL START CREATE()`

This directive generates an OpenACC `DATA CREATE` statement for all output fields used in each DSL (icon4py) stencil.

#### `!$DSL END CREATE()`

This directive generates an OpenACC `END DATA` statement which is neccessary to close the OpenACC data region.

#### `!$DSL DECLARE()`

This directive is used to declare all DSL input/output fields. The required arguments are the field name and its associated dimensions. For example:

```fortran
!$DSL DECLARE(vn=(nproma, p_patch%nlev, p_patch%nblks_e))
```

will generate the following code:

```fortran
! DSL INPUT / OUTPUT FIELDS
REAL(wp), DIMENSION((nproma, p_patch%nlev, p_patch%nblks_e)) :: vn_before
```

Furthermore, this directive also takes two optional keyword arguments. `type` takes a string which will be used to fill in the type of the declared field, for example `type=LOGICAL`. `suffix` takes a string which will be used as the suffix of the field e.g. `suffix=dsl`, by default the suffix is `before`.

#### `!$DSL START STENCIL()`

This directive denotes the start of a stencil. Required arguments are `name`, `vertical_lower`, `vertical_upper`, `horizontal_lower`, `horizontal_upper`. The value for `name` must correspond to a stencil found in one of the stencil modules inside `icon4py`, and all fields defined in the directive must correspond to the fields defined in the respective icon4py stencil. Optionally, absolute and relative tolerances for the output fields can also be set using the `_tol` or `_abs` suffixes respectively. An example call looks like this:

```fortran
!$DSL START STENCIL(name=mo_nh_diffusion_stencil_06; &
!$DSL z_nabla2_e=z_nabla2_e(:,:,1); area_edge=p_patch%edges%area_edge(:,1); &
!$DSL fac_bdydiff_v=fac_bdydiff_v; vn=p_nh_prog%vn(:,:,1); vn_abs_tol=1e-21_wp; &
!$DSL vertical_lower=1; vertical_upper=nlev; &
!$DSL horizontal_lower=i_startidx; horizontal_upper=i_endidx)
```

In addition, other optional keyword arguments are the following:

- `accpresent`: Takes a boolean string input, and controls the default data-sharing behavior for variables used in the OpenACC parallel region. Setting the flag to true will cause all variables to be assumed present on the device by default (`DEFAULT(PRESENT)`), and no explicit data-sharing attributes need to be specified. Setting it to false will require explicit data-sharing attributes for every variable used in the parallel region (`DEFAULT(NONE)`). By default it is set to false.<br><br>

- `mergecopy`: Takes a boolean string input. When set to True consecutive before field copy regions of stencils that have the mergecopy flag set to True are combined into a single before field copy region with a new name created by concatenating the names of the merged stencil regions. This is useful when there are consecutive stencils. By default it is set to false.<br><br>

- `copies`: Takes a boolean string input, and controls whether before field copies should be made or not. If set to False only the `#ifdef __DSL_VERIFY` directive is generated. Defaults to true.<br><br>

#### `!$DSL END STENCIL()`

This directive denotes the end of a stencil. The required argument is `name`, which must match the name of the preceding `START STENCIL` directive.

Together, the `START STENCIL` and `END STENCIL` directives result in the following generated code at the start and end of a stencil respectively.

```fortran
#ifdef __DSL_VERIFY
!$ACC PARALLEL IF( i_am_accel_node .AND. acc_on ) DEFAULT(NONE) ASYNC(1)
vn_before(:, :, :) = vn(:, :, :)
!$ACC END PARALLEL
```

```fortran
call nvtxEndRange()
#endif
call wrap_run_mo_nh_diffusion_stencil_06( &
z_nabla2_e=z_nabla2_e(:, :, 1), &
area_edge=p_patch%edges%area_edge(:, 1), &
fac_bdydiff_v=fac_bdydiff_v, &
vn=p_nh_prog%vn(:, :, 1), &
vn_before=vn_before(:, :, 1), &
vn_abs_tol=1e-21_wp, &
vertical_lower=1, &
vertical_upper=nlev, &
horizontal_lower=i_startidx, &
horizontal_upper=i_endidx
)
```

Additionally, there are the following keyword arguments:

- `noendif`: Takes a boolean string input and controls whether an `#endif` is generated or not. Defaults to false.<br><br>

- `noprofile`: Takes a boolean string input and controls whether a nvtx end profile directive is generated or not. Defaults to false.<br><br>

#### `!$DSL INSERT()`

This directive allows the user to generate any text that is placed between the parentheses. This is useful for situations where custom code generation is necessary.

#### `!$DSL START PROFILE()`

This directive allows generating an nvtx start profile data statement, and takes the stencil `name` as an argument.

#### `!$DSL END PROFILE()`

This directive allows generating an nvtx end profile statement.

#### `!$DSL ENDIF()`

This directive generates an `#endif` statement.
3 changes: 3 additions & 0 deletions liskov/requirements-dev.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
-r ../base-requirements-dev.txt
-e ../common
-e .
3 changes: 3 additions & 0 deletions liskov/requirements.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
-r ../base-requirements.txt
../common
.
55 changes: 55 additions & 0 deletions liskov/setup.cfg
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
# This file is mainly used to configure package creation with setuptools.
# Documentation:
# http://setuptools.readthedocs.io/en/latest/setuptools.html#configuring-setup-using-setup-cfg-files
#
[metadata]
name = icon4py_liskov
description = ICON preprocessor to integrate Gt4Py code.
long_description = file: README.md
long_description_content_type = text/markdown
url = https://github.com/C2SM/icon4py
author = ETH Zurich
author_email = [email protected]
license = gpl3
license_files = LICENSE
platforms = Linux, Mac
classifiers =
Development Status :: 3 - Alpha
Intended Audience :: Science/Research
License :: OSI Approved :: GNU General Public License v3 or later (GPLv3+)
Operating System :: POSIX
Programming Language :: Python
Programming Language :: Python :: 3
Programming Language :: Python :: 3 :: Only
Programming Language :: Python :: 3.10
Programming Language :: Python :: Implementation :: CPython
Topic :: Scientific/Engineering :: Atmospheric Science
Topic :: Scientific/Engineering :: Mathematics
Topic :: Scientific/Engineering :: Physics
project_urls =
Source Code = https://github.com/GridTools/gt4py

[options]
packages = find_namespace:
install_requires =
icon4py-common

python_requires = >=3.10
package_dir =
= src
zip_safe = False

[options.package_data]
# References:
# https://setuptools.pypa.io/en/latest/userguide/datafiles.html
# https://github.com/abravalheri/experiment-setuptools-package-data
* = *.md, *.rst, *.toml, *.txt, py.typed

[options.packages.find]
where = src
exclude =
tests

[options.entry_points]
console_scripts =
icon_liskov = icon4py.liskov.cli:main
18 changes: 18 additions & 0 deletions liskov/setup.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
# ICON4Py - ICON inspired code in Python and GT4Py
#
# Copyright (c) 2022, ETH Zurich and MeteoSwiss
# All rights reserved.
#
# This file is free software: you can redistribute it and/or modify it under
# the terms of the GNU General Public License as published by the
# Free Software Foundation, either version 3 of the License, or any later
# version. See the LICENSE.txt file at the top-level directory of this
# distribution for a copy of the license or check <https://www.gnu.org/licenses/>.
#
# SPDX-License-Identifier: GPL-3.0-or-later

from setuptools import setup


if __name__ == "__main__":
setup()
12 changes: 12 additions & 0 deletions liskov/src/icon4py/liskov/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
# ICON4Py - ICON inspired code in Python and GT4Py
#
# Copyright (c) 2022, ETH Zurich and MeteoSwiss
# All rights reserved.
#
# This file is free software: you can redistribute it and/or modify it under
# the terms of the GNU General Public License as published by the
# Free Software Foundation, either version 3 of the License, or any later
# version. See the LICENSE.txt file at the top-level directory of this
# distribution for a copy of the license or check <https://www.gnu.org/licenses/>.
#
# SPDX-License-Identifier: GPL-3.0-or-later
64 changes: 64 additions & 0 deletions liskov/src/icon4py/liskov/cli.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
# ICON4Py - ICON inspired code in Python and GT4Py
#
# Copyright (c) 2022, ETH Zurich and MeteoSwiss
# All rights reserved.
#
# This file is free software: you can redistribute it and/or modify it under
# the terms of the GNU General Public License as published by the
# Free Software Foundation, either version 3 of the License, or any later
# version. See the LICENSE.txt file at the top-level directory of this
# distribution for a copy of the license or check <https://www.gnu.org/licenses/>.
#
# SPDX-License-Identifier: GPL-3.0-or-later

import pathlib

import click

from icon4py.liskov.logger import setup_logger
from icon4py.liskov.pipeline import (
load_gt4py_stencils,
parse_fortran_file,
run_code_generation,
)


logger = setup_logger(__name__)


@click.command("icon_liskov")
@click.argument(
"input_filepath",
type=click.Path(
exists=True, dir_okay=False, resolve_path=True, path_type=pathlib.Path
),
)
@click.argument(
"output_filepath",
type=click.Path(dir_okay=False, resolve_path=True, path_type=pathlib.Path),
)
@click.option(
"--profile", "-p", is_flag=True, help="Add nvtx profile statements to stencils."
)
def main(
input_filepath: pathlib.Path, output_filepath: pathlib.Path, profile: bool
) -> None:
"""Command line interface for interacting with the ICON-Liskov DSL Preprocessor.
Usage:
icon_liskov <input_filepath> <output_filepath> [--profile]
Options:
-p --profile Add nvtx profile statements to stencils.
Arguments:
input_filepath Path to the input file to process.
output_filepath Path to the output file to generate.
"""
parsed = parse_fortran_file(input_filepath)
parsed_checked = load_gt4py_stencils(parsed)
run_code_generation(parsed_checked, input_filepath, output_filepath, profile)


if __name__ == "__main__":
main()
15 changes: 15 additions & 0 deletions liskov/src/icon4py/liskov/codegen/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
# ICON4Py - ICON inspired code in Python and GT4Py
#
# Copyright (c) 2022, ETH Zurich and MeteoSwiss
# All rights reserved.
#
# This file is free software: you can redistribute it and/or modify it under
# the terms of the GNU General Public License as published by the
# Free Software Foundation, either version 3 of the License, or any later
# version. See the LICENSE.txt file at the top-level directory of this
# distribution for a copy of the license or check <https://www.gnu.org/licenses/>.
#
# SPDX-License-Identifier: GPL-3.0-or-later

# required import for CheckForDirectiveClasses metaclass
import icon4py.liskov.parsing.types as ts # noqa: F401
Loading

0 comments on commit 5624c37

Please sign in to comment.