Wflow ZMQ Server (#356)

* Start with Wflow (BMI) ZMQ server
initialization of Wflow model

* improve error handling and check request client

* add `get_start_time` function
also changed the `ModelHandler` so the same `wflow_bmi` function can be used for initialization and other BMI functions.

* add `get_end_time` function

* add functions: `finalize` and `update_until`

* fix shutdown of server

* add remaining BMI functions and tests

* run tests from runtests.jl

* renamed bmi_api.jl to bmi_service.jl
should have been part of previous 2 commits...

* set states during initialization
Move set states functionality from initialization function to a separate `set_states` function for each Model type. It also makes coupling to OpenDA more convenient (load and save states from OpenDA).

* add functions `load_state` and `save_state`

* improve and fix bmi
- fix list of exchange variables, if a model does not include reservoirs or lakes, these variables should be left out.
- add grid types "scalar" (for uniform/constant variables) and "none" (for variables not related to a model grid)
- add "flextopo" model

* split struct `SurfaceFlow`
SurfaceFlow is used for both kinematic wave river and overland flow. This is now split into structs SurfaceFlowRiver and SurfaceFlowLand. It simplifies the orginal update function (split into two, logic easier to follow), and dispatching on these types is easier (e.g. for BMI to extract relevant properties as grid type).

* fix tests and code because of update BMI
- use simulation starttime as t = 0 (not 1970).
- BMI.get_time_units: return "s" (instead of "seconds since 1970-01-01T00:00:00").
- BMI.get_time_units now gets called on the model rather than the type.

* add `get_start_unix_time` function
to get a reference datetime. Model start time (BMI) returns always 0.0.

* Flexible port number
ZMQ port number can be provided as command line argument (default = 5555).

* update test
change of `starttime` definition

* Fix bmi.jl
File was not up-to-date with Master.

* Align with BMI fixes Master branch

* Update tests
Scalar variables are not exposed through BMI.

* Add doc-strings functions, README.md, comments
Also update docs Wflow.jl (mention possibilty to run Wflow as a server).

* Rename module and add start function server

* Use relative paths in README

* Update README

* Update docs (typo OpenDA)

Co-authored-by: Martijn Visser <[email protected]>

* Update docs

Co-authored-by: Martijn Visser <[email protected]>

* Update server/README.md

Co-authored-by: Martijn Visser <[email protected]>

* Update server/README.md

Co-authored-by: Martijn Visser <[email protected]>

* Do not export `start`

Co-authored-by: Martijn Visser <[email protected]>

* Update README.md

Co-authored-by: Martijn Visser <[email protected]>

* Update flow.jl (remove trailing spaces)

Co-authored-by: Martijn Visser <[email protected]>

* Address review comments

* Fix import for tests

* Make testing work from `Pkg`
- Project.toml needs uuid
- Logging dep was missing

* Run CI for Wflow Server

* Fix missing yml extension

* Delete CIWflowServer

* Fix syntax `workflow_run`

* Update CIWflowServer.yml

* Remove `workflow_run` (only works on the main branch)
CI workflow for Wflow ZMQ Server now also triggered by PR (and push to Master) as CI workflow for Wflow. Wflow test model (Moselle) for Wflow ZMQ Server is downloaded to Wflow\server\test\.

* Fix paths

* Add `julia-runtest@v1`

* Replace `julia-runtest@v1` (runs Wflow tests)

* Add `sbm_config.toml` for tests

* Local dev install of Wflow

* Update test WflowServer (merge master)

* Remove exit from CIWflowServer

* Remove threads from name CI WflowServer

* Update codecov-action

* Add dependabot githubactions

* Specify directory processcoverage CI WflowServer

---------

Co-authored-by: Martijn Visser <[email protected]>

.github/dependabot.yml

@@ -0,0 +1,7 @@
+# https://docs.github.com/github/administering-a-repository/configuration-options-for-dependency-updates
+version: 2
+updates:
+  - package-ecosystem: "github-actions"
+    directory: "/" # Location of package manifests
+    schedule:
+      interval: "weekly"

.github/workflows/CI.yml

@@ -44,6 +44,7 @@ jobs:
         env:
           JULIA_NUM_THREADS: ${{ matrix.threads }}
       - uses: julia-actions/julia-processcoverage@v1
-      - uses: codecov/codecov-action@v3
+      - uses: codecov/codecov-action@v4
         with:
-          file: lcov.info
+          files: lcov.info
+          token: ${{ secrets.CODECOV_TOKEN }}

.github/workflows/CIWflowServer.yml

@@ -0,0 +1,56 @@
+name: Wflow Server CI
+on:
+  pull_request:
+  push:
+    branches:
+      - master
+    tags: '*'
+jobs:
+  test:
+    name: WflowServer Julia ${{ matrix.version }} - ${{ matrix.os }} - ${{ matrix.arch }} - ${{ github.event_name }}
+    runs-on: ${{ matrix.os }}
+    strategy:
+      fail-fast: false
+      matrix:
+        version:
+          - '1.6'
+          - '1'
+        os:
+          - ubuntu-latest
+          - windows-latest
+        arch:
+          - x64
+    steps:
+      - uses: actions/checkout@v2
+      - uses: julia-actions/setup-julia@v1
+        with:
+          version: ${{ matrix.version }}
+          arch: ${{ matrix.arch }}
+      - uses: actions/cache@v1
+        env:
+          cache-name: cache-artifacts
+        with:
+          path: ~/.julia/artifacts
+          key: ${{ runner.os }}-test-${{ env.cache-name }}-${{ hashFiles('**/Project.toml') }}
+          restore-keys: |
+            ${{ runner.os }}-test-${{ env.cache-name }}-
+            ${{ runner.os }}-test-
+            ${{ runner.os }}-
+      - name: Install Julia dependencies
+        shell: julia --project=server {0}
+        run: |
+          using Pkg;
+          # dev install Wflow
+          pkg"dev ."
+      - name: Run the Wflow Server tests
+        continue-on-error: true
+        run: >
+          julia --color=yes --project=server -e 'using Pkg; Pkg.test("WflowServer", coverage=true)'
+          shell: bash
+      - uses: julia-actions/julia-processcoverage@v1
+        with:
+          directories: server/src
+      - uses: codecov/codecov-action@v4
+        with:
+          files: lcov.info
+          token: ${{ secrets.CODECOV_TOKEN }}

docs/src/user_guide/additional_options.md

@@ -289,3 +289,9 @@ expectations, which then can get parsed with these Delft-FEWS log parsing settin
     <debugLinePattern >* [Debug] *</debugLinePattern >
 </logFile>
 ```
+
+## [Run Wflow as a ZMQ Server]
+It is possible to run Wflow as a ZMQ Server, for example for the coupling to the
+[OpenDA](https://openda.org/) software for data-assimilation. The code for the Wflow ZMQ
+Server is not part of the Wflow.jl package, and is located
+[here](https://github.com/Deltares/Wflow.jl/tree/zmq_server/server).

server/Project.toml

@@ -0,0 +1,25 @@
+name = "WflowServer"
+uuid = "6f3aa2dc-f527-41f4-85fc-b3cbabe74cf6"
+authors = ["Deltares and contributors"]
+version = "0.1.0"
+
+[deps]
+JSON3 = "0f8b85d8-7281-11e9-16c2-39a750bddbf1"
+Logging = "56ddb016-857b-54e1-b83d-db4d58db5568"
+StructTypes = "856f2bd8-1eba-4b0a-8007-ebc267875bd4"
+Wflow = "d48b7d99-76e7-47ae-b1d5-ff0c1cf9a818"
+ZMQ = "c2297ded-f4af-51ae-bb23-16f91089e4e1"
+
+[compat]
+JSON3 = "1.14"
+StructTypes = "1.10"
+ZMQ = "1.2"
+julia = "1.6"
+
+[extras]
+Downloads = "f43a241f-c20a-4ad4-852c-f6b1247861c6"
+Statistics = "10745b16-79ce-11e8-11f9-7d13ad32a3b2"
+Test = "8dfed614-e22c-5e08-85e1-65c5234f0b40"
+
+[targets]
+test = ["Downloads", "Statistics", "Test"]

server/README.md

@@ -0,0 +1,51 @@
+# Wflow ZMQ Server
+Call [Wflow](https://github.com/Deltares/Wflow.jl) functions exposed through the Basic Model
+Interface (BMI) implementation of Wflow and three additonal Wflow functions related to model
+states and start time of the model in Unix time, using [ZeroMQ](https://zeromq.org/) with
+TCP data transport. The Wflow ZMQ Server allows users to interact with a Wflow model from
+many other programming languages with ZeroMQ bindings. An example is the use of
+[OpenDA](https://openda.org/) Java software for data-assimilation. The coupling of OpenDA
+and Wflow through a black box model approach is too slow because it requires a restart of
+Wflow (multiple times), while this is not required with a client-server approach.
+
+The Wflow ZMQ Server can be started through an interactive Julia version, using different
+functions. First, start the `WflowServer` environment on startup, in the current directory:
+```
+julia --project=.
+```
+then start the Wflow ZMQ Server, listening on port 5556 with the `start` function as
+follows:
+```julia-repl
+julia> using WflowServer
+julia> WflowServer.start(5556)
+```
+or start the Wflow ZMQ Server, listening on port 5556 with the `main` function:
+```julia-repl
+julia> using WflowServer
+julia> WflowServer.main(["port=5556"])
+```
+or start the Wflow ZMQ Server with the `main` function using the default port number 5555:
+```julia-repl
+julia> using WflowServer
+julia> WflowServer.main()
+```
+Finally, it is also possible to start the Wflow ZMQ server directly from the command line
+with the Julia script `run_server.jl` in the current directory, as follows:
+```
+julia --project=. run_server.jl --port 5556
+```
+
+## JSON
+JSON is used for data serialization. The Wflow ZMQ Server maps a JSON message to a Julia
+structure directly. Below examples of two messages that provide a function name `fn` and
+other arguments required by the exposed Wflow functions (mostly through BMI):
+
+```
+# initialize a Wflow model through the configuration file `sbm_config.toml`
+"""{"fn": "initialize", "config_file": "sbm_config.toml"}"""
+# update model until time 86400.0
+"""{"fn": "update_until", "time": 86400.0}"""
+```
+
+The [tests](/server/test/) provide further examples of the interface to the Wflow (BMI) and
+server functions.

server/run_server.jl

@@ -0,0 +1,3 @@
+using WflowServer
+
+WflowServer.main()

server/src/WflowServer.jl

@@ -0,0 +1,9 @@
+module WflowServer
+import ZMQ
+import JSON3
+import StructTypes
+import Wflow
+
+include("bmi_service.jl")
+include("server.jl")
+end