Skip to content

Commit

Permalink
Merge pull request #45 from EcoExtreML/add_octave
Browse files Browse the repository at this point in the history 
Add octave and matlab to run module
  • Loading branch information
@SarahAlidoost
SarahAlidoost authored Nov 14, 2022
2 parents d34f8f9 + 6a0d024 commit beb341f
Show file tree
Hide file tree
Showing 12 changed files with 649 additions and 738 deletions.
95 changes: 7 additions & 88 deletions CONTRIBUTING.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,8 @@ This repository includes the python package `PyStemmusScope` for running the STE

## Configure the python package for development and testing

To contribute to development of the python package, we recommend installing the package in development mode.
To contribute to the development of the python package, we recommend installing
the package in development mode.


### Installation
Expand All @@ -15,26 +16,20 @@ First, clone this repository:
git clone https://github.com/EcoExtreML/STEMMUS_SCOPE_Processing.git
```

Then install the package:
Then install the package (On Windows, use `python` instead of `python3`):

```sh
cd STEMMUS_SCOPE_Processing
pip install -e .
```

or

```sh
python setup.py develop
python3 -m install -e .[dev]
```

### Run tests

The testing framework used here is [PyTest](https://pytest.org). You can run
tests as:
tests as (On Windows, use `python` instead of `python3`):

```sh
pytest
python3 -m pytest
```

### Build documentation
Expand Down Expand Up @@ -65,80 +60,4 @@ isort
## Development of STEMMUS_SCOPE model

<!-- markdown-link-check-disable-next-line -->
To contribute to the STEMMUS_SCOPE model, you need access to the model source code that is stored in the repository [STEMMUS_SCOPE](https://github.com/EcoExtreML/STEMMUS_SCOPE). You also need a MATLAB license.

### Development on Snellius using MATLAB

[Snellius](https://servicedesk.surfsara.nl/wiki/display/WIKI/Snellius) is the
Dutch National supercomputer hosted at SURF. MATLAB `2021a` is installed on
Snellius, see the script
[`run_jupyter_lab_snellius_dev.sh`](https://github.com/EcoExtreML/STEMMUS_SCOPE_Processing/blob/main/run_jupyter_lab_snellius_dev.sh)
on how to load the module.

The script `run_jupyter_lab_snellius_dev.sh` activates the conda environment `pystemmusscope` and creates a jupyter lab server on Snellius for running the notebook
interactively. Make sure that you create the `pystemmusscope` conda environment before submitting the the bash script. See **Create pystemmusscope environment** below.

<details>
<summary>Create pystemmusscope environment</summary>

Run the commands below in a terminal:

```sh
# Download and install Mamba on linux
wget https://github.com/conda-forge/miniforge/releases/latest/download/Mambaforge-pypy3-Linux-x86_64.sh
bash Mambaforge-pypy3-Linux-x86_64.sh -b -p ~/mamba

# Update base environment
. ~/mamba/bin/activate
mamba update --name base mamba

# Download environment file
wget https://raw.githubusercontent.com/EcoExtreML/STEMMUS_SCOPE_Processing/main/environment.yml

# Create a conda environment called 'pystemmusscope' with all required dependencies
mamba env create -f environment.yml

# The environment can be activated with
. ~/mamba/bin/activate pystemmusscope

```
</details>


### Development on CRIB using MATLAB

[CRIB](https://crib.utwente.nl/) is the ITC Geospatial Computing Platform.

MATLAB `2021a` is installed on CRIB, and supports Python `3.8`, see [Versions of Python Compatible with MATLAB Products](https://www.mathworks.com/content/dam/mathworks/mathworks-dot-com/support/sysreq/files/python-compatibility.pdf).

1. Log in CRIB with your username and password and select a proper compute unit.
2. Install `PyStemmusScope` package. This step needs to be done once.
<details>
<summary>Install pystemmusscope with python 3.8</summary>

Run the commands below in a terminal:

```sh
# Download and install Mamba on linux
wget https://github.com/conda-forge/miniforge/releases/latest/download/Mambaforge-pypy3-Linux-x86_64.sh
bash Mambaforge-pypy3-Linux-x86_64.sh -b -p ~/mamba

# Update base environment
. ~/mamba/bin/activate
mamba update --name base mamba

# Download environment file
wget https://raw.githubusercontent.com/EcoExtreML/STEMMUS_SCOPE_Processing/main/environment_3.8.yml

# Create a conda environment called 'pystemmusscope' with all required dependencies
mamba env create -f environment_3.8.yml
```
</details>

3. click on the `Remote Desktop` in the
Launcher. Click on the `Applications`. You will find the 'MATLAB' software under
the `Research`.
4. After clicking on 'MATLAB', it asks for your account information that is
connected to a MATLAB license.
5. Open the file `STEMMUS_SCOPE_run.m` and set the path of `config_file` to `../config_file_crib.txt` and change `WorkDir` and other configurations in `model.setup()`.
6. Then, run the main script `STEMMUS_SCOPE_run.m`.
To contribute to the STEMMUS_SCOPE model, you need access to the model source code that is stored in the repository [STEMMUS_SCOPE](https://github.com/EcoExtreML/STEMMUS_SCOPE).
143 changes: 113 additions & 30 deletions PyStemmusScope/stemmus_scope.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,9 @@

import logging
import os
import shlex
import subprocess
from pathlib import Path
from typing import Dict
from . import config_io
from . import forcing_io
Expand All @@ -12,32 +14,99 @@

logger = logging.getLogger(__name__)

def _is_model_src_exe(model_src_path: Path):
"""Check if input exists. Returns True if input is a file and False if it is
a directory.
Args:
model_src_path(Path): path to Stemmus_Scope executable file or to a
directory containing model source codes.
"""
if model_src_path.is_file():
msg = ("The model executable file can be used on a Unix system "
"where MCR is installed, see the "
"`documentaion<https://pystemmusscope.readthedocs.io/>`_.")
logger.info("%s", msg)
return True
if model_src_path.is_dir():
return False
msg = (
"Provide a valid path to an executable file or "
"to a directory containing model source codes, "
"see the `documentaion<https://pystemmusscope.readthedocs.io/>`_.")
raise ValueError(msg)


def _check_interpreter(interpreter: str):
if interpreter not in {"Octave" , "Matlab"}:
msg = (
"Set `interpreter` as Octave or Matlab to run the model using source codes."
"Otherwise set `model_src_path` to the model executable file, "
"see the `documentaion<https://pystemmusscope.readthedocs.io/>`_.")
raise ValueError(msg)


def _run_sub_process(args: list, cwd):
# pylint: disable=consider-using-with
result = subprocess.Popen(
args, cwd=cwd,
stdout=subprocess.PIPE, stderr=subprocess.PIPE,
shell=True,
)
exit_code = result.wait()
stdout, stderr = result.communicate()
#TODO handle stderr properly
# when using octave, exit_code might be 139
# see issue STEMMUS_SCOPE_Processing/issues/46
if exit_code not in [0, 139]:
raise subprocess.CalledProcessError(
returncode=exit_code, cmd=args, stderr=stderr, output=stdout
)
if exit_code == 139:
logger.warning(stderr)

# TODO return log info line by line!
logger.info(stdout)
return stdout.decode('utf-8')


class StemmusScope():
"""PyStemmusScope wrapper around Stemmus_Scope model.
see https://gmd.copernicus.org/articles/14/1379/2021/
It sets the model with a configuration file and executable file.
It also prepares forcing and soil data for model run.
Configures the model and prepares forcing and soil data for the model run.
Args:
config_file(str): path to Stemmus_Scope configuration file. An example
config_file can be found in tests/test_data in `STEMMUS_SCOPE_Processing
repository <https://github.com/EcoExtreML/STEMMUS_SCOPE_Processing>`_
exe_file(str): path to Stemmus_Scope executable file.
model_src_path(str): path to Stemmus_Scope executable file or to a
directory containing model source codes.
interpreter(str, optional): use `Matlab` or `Octave`. Only required if
`model_src_path` is a path to model source codes.
Example:
See notebooks/run_model_in_notebook.ipynb in `STEMMUS_SCOPE_Processing
repository <https://github.com/EcoExtreML/STEMMUS_SCOPE_Processing>`_
"""

def __init__(self, config_file: str, exe_file: str):
def __init__(self, config_file: str, model_src_path: str, interpreter: str = None):
# make sure paths are abolute and path objects
config_file = utils.to_absolute_path(config_file)
self.exe_file = utils.to_absolute_path(exe_file)
model_src_path = utils.to_absolute_path(model_src_path)

# check the path to model source
self.exe_file = None
if _is_model_src_exe(model_src_path):
self.exe_file = model_src_path
else:
_check_interpreter(interpreter)

self.model_src = model_src_path
self.interpreter = interpreter

# read config template
self._configs = config_io.read_config(config_file)
self._config = config_io.read_config(config_file)

def setup(
self,
Expand All @@ -61,30 +130,27 @@ def setup(
"""
# update config template if needed
if WorkDir:
self._configs["WorkDir"] = WorkDir
self._config["WorkDir"] = WorkDir

if ForcingFileName:
self._configs["ForcingFileName"] = ForcingFileName
self._config["ForcingFileName"] = ForcingFileName

if NumberOfTimeSteps:
self._configs["NumberOfTimeSteps"] = NumberOfTimeSteps
self._config["NumberOfTimeSteps"] = NumberOfTimeSteps

# create customized config file and input/output directories for model run
_, _, self.cfg_file = config_io.create_io_dir(
self._configs["ForcingFileName"], self._configs
self._config["ForcingFileName"], self._config
)

# read the run config file
self._configs = config_io.read_config(self.cfg_file)
self._config = config_io.read_config(self.cfg_file)

# prepare forcing data
forcing_io.prepare_forcing(self._configs)
forcing_io.prepare_forcing(self._config)

# prepare soil data
soil_io.prepare_soil_data(self._configs)

# set matlab log dir
os.environ['MATLAB_LOG_DIR'] = str(self._configs["InputPath"])
soil_io.prepare_soil_data(self._config)

return str(self.cfg_file)

Expand All @@ -94,23 +160,40 @@ def run(self) -> str:
Args:
Returns:
Tuple with stdout and stderr
string, the model log
"""

# run the model
args = [f"{self.exe_file} {self.cfg_file}"]
result = subprocess.run(
args, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True, check=True,
)
stdout = result.stdout

# TODO return log info line by line!
logger.info("%s", stdout)

return stdout
if self.exe_file:
# run using MCR
args = [f"{self.exe_file} {self.cfg_file}"]
# set matlab log dir
os.environ['MATLAB_LOG_DIR'] = str(self._config["InputPath"])
result = _run_sub_process(args, None)
if self.interpreter=="Matlab":
# set Matlab arguments
path_to_config = f"'{self.cfg_file}'"
eval_code= f'STEMMUS_SCOPE_exe({path_to_config});exit;'
args = ["matlab", "-r", eval_code, "-nodisplay", "-nosplash", "-nodesktop"]
# seperate args dont work on linux!
if utils.os_name() !="nt":
args = shlex.join(args)
result = _run_sub_process(args, self.model_src)
if self.interpreter=="Octave":
# set Octave arguments
# use subprocess instead of oct2py,
# see issue STEMMUS_SCOPE_Processing/issues/46
path_to_config = f"'{self.cfg_file}'"
# fix for windows
path_to_config = path_to_config.replace("\\", "/")
eval_code = f'STEMMUS_SCOPE_exe({path_to_config});exit;'
args = ["octave", "--eval", eval_code, "--no-gui", "--silent"]
# seperate args dont work on linux!
if utils.os_name() !="nt":
args = shlex.join(args)
result = _run_sub_process(args, self.model_src)
return result


@property
def config(self) -> Dict:
"""Return the configurations for this model."""
return self._configs
return self._config
Loading

0 comments on commit beb341f

Please sign in to comment.