Improve modelbuilder notebook

[veenstrajelmer]

Notebook was improved in #463

Further improvements:

• now using tpxo which requires large buffer size for plibnd, consider using EOT20 online dataset (not implemented) or tidemodel='GTSMv4.1_opendap' >> buffer can be reduced anyway since we also use nearest interp now
• prevent opendap gebco cutout link in modelbuilder notebook, no API is available, maybe via gebco ddb opendap. Or maybe add coarse dataset to opendap?
• prevent modelbuilder import, expose via dfm_tools directly
• support for waq in modelbuilder example script #504
• Minor modelbuilder improvements #602
• Fill nans in ini sal/tem netcdf #616
• Add initial fields for tracers #618
• CMEMS phyc not available in reanalysis #622
• Improve robustness of CMEMS/CDS authentication #603
• dsd improvements modelbuilder notebook #631
• modelbuilder print modeltree from dimr instead of mdu #725
• simplify cdsapi.Client() call after new cdsapi release #822
• Update to copernicus-marine-client #630
• Improvements in copernicus-marine-client related code #694
• Increase minimal copernicusmarine version #692
• Make conversion_dict an input parameter for the function cmems_nc_to_bc #838
• Add example with ETOPO online dataset #836
• remove os.listdir command after Show_tree output is not printed for old ext files HYDROLIB-core#581 is fixed
• connect to myint cmems reanalyis dataset also #861
• cmems_nc_to_ini produces netcdf file incoherent depth variable #944
• fix paths in dfmt.create_model_exec_files batfile #1029
• add fes2022 tidal source #924
• snap observation points to nearest cell #898
• contextily is still instable, maybe remove it from notebook
• add ssc stations to map instead of arbitrary points (coverage is not super good, but at least it makes sense)
• mdu.geometry.layertype in modelbuilder example script should be 2, also mdu.physics.temperature was defined twice
• add function to overwrite several mdu keywords with coastal 2D/3D values #951
• if a polyline does not result in any plipoints, consider not writing it in extfile (FM raises an error in case of an ext entry for which the polygon does not have any forcing/bc blocks associated, happened in DSO project)
• add 3D to notebook and discontinue py example script (also transfer othogonality visualization somewhere, linked p-drive in that script is now archived anyway)
• Disruptions and update to new cdsapi #739
• Consider time correction for ERA5 rainfall and more #899
• suppress CMEMS and CDS logging in notebook (not in dfmt functions) with import logging; logging.getLogger("copernicus_marine_root_logger").setLevel(level="CRITICAL")
• EDITO: Improve dfm modelbuilder workflow Deltares-research/EditoServices#1
• prevent print of 2023-11-03 14:24:35,649 INFO Sending request to https://cds.climate.copernicus.eu/api/v2/resources/dummy >> quiet argument for Client class?
• consider alternative bathymetry from https://help.marine.copernicus.eu/en/articles/4733481-how-to-access-bathymetry-dataset-used-in-copernicus-marine-model-products
• also support initial values in polygon?
• nans in ini sal/tem are filled per 1d dim (first lat, then lon), consider 2d interpolation
• consider filling source netcdfs for bc files first over lat/lon, instead of only filling bc over depth.
• prevent pydap connection if cmems files are already available, is difficult since we check for available times in products. This is not relevant anymore when we use copernicus-marine-client
• make conversion_dict keys all lowercase and convert all input to lowercase with .lower()? Relevant for e.g. tracerbndNO3. All lowercase is also recognized by FM.
• make conversion_dict a global setting, so a custom conversion dict is automatically used everywhere in the modelbuilding process
• Add function to modelbuilder to generate observations along coastline #575 (or see below example)
• consider creating a function that combines open_dataset_extra, interp_regularnc_to_plipoints, plipointsDataset_to_ForcingModel and maybe more, and remove cmems_nc_to_bc. Is ForcingModel_object.save() required
• conversion of single plifile to multiplifile: Avoid duplicate polyline names in geodataframe_to_PolyFile #492
• test model in windows exe and DeltaShell GUI, not all will work (e.g. lowercase keywords in bc-file are not properly read): Incompatible with Precompiled DFM #710
• add note about importing in GUI with link to Incompatible with Precompiled DFM #710
• adding river discharges (preferrably from a global public datasource) as source/sinks: GRDC has no API and no coastal points. Swedish SMHI EHYPE dataset?
• add dry points (upon grid generation if still necessary)
• add structures, cross-sections, thindams etc (quite complex to to this in a generic way, bit still add support for adding manually)
• Support D-Water Quality input in ExtOldModel HYDROLIB-core#551
• waq: write timfiles for source/sinks (for river discharges in large scale models and WAQ sources), e.g. p:\11209231-003-bes-modellering\hydrodynamica-waq\preprocessing\hydrology_runoff\hydrology_nutrient_to_tim_waq.py (by YC) >> bc files will be implemented in delft3dfm. sourcesinks are also used for river discharges.
• also wastewater: p:\11209231-003-bes-modellering\hydrodynamica-waq\preprocessing\wastewater_20231129\scripts\ (by LB) (is also sourcesinks)
• waq: HYDROLIB-core: Reader of substance file, including comparison to timfile columns. Example model in p:\11209251-sito-veerse-meer\model\runs_2019-2020\Veerse_meer_Model_to_subversion\dflowfm3d_dwaq-veerse_meer-j19_6-v1a_LH. Related issue: If old external forcing file references a pli file, the accompanying .tim/.cmp should also be read and stored HYDROLIB-core#501 (also contains column order of timfiles, so not super important if sourcesinks are forced via bc)
• waq: HYDROLIB-core: ignore or support waq statistics file (is defined in the substance file)
• waq: HYDROLIB-core: support iniwaq* in ext file?
• more WAQ things necessary: contact MW, FD, AM, LvdH

Add sources for e.g. tide models, CMEMS and ERA5:

• UHSLC (public): https://uhslc.soest.hawaii.edu/
• IOC: https://www.ioc-sealevelmonitoring.org/
• GESLA3: https://gesla787883612.wordpress.com/
• FES2014 (not public, commercial paid): https://www.aviso.altimetry.fr/en/data/products/auxiliary-products/global-tide-fes/description-fes2014.html
• EOT20 (public): https://essd.copernicus.org/articles/13/3869/2021/
• GTSMv4.1 (public): https://www.deltares.nl/en/expertise/projects/global-modelling-of-tides-and-storm-surges

Add obspoints (or along coastline with #575):

obsfile = os.path.join(dir_output, f"{model_name}_obs.xyn")
obs_x = [-68.2240067, -68.2283020, -68.3852158, -68.1552582]
obs_y = [12.5409269, 12.2554617, 12.0076866, 12.0341187]
obs_n = ['Bonaire01', 'Bonaire02', 'Bonaire03', 'Bonaire04']
xynpoints = [hcdfm.XYNPoint(x=x,y=y,n=n) for x,y,n in zip(obs_x, obs_y, obs_n)]
xynmodel = hcdfm.XYNModel(points=xynpoints)
xynmodel.save(obsfile)

Running the model:

• generate sh file for snellius/h7/amazon, singularity? (bat for windows is handled, as well as generic dimr_config.xml)
• required: singularity, jupyter, maybe file browsing/editing (eg mobaxterm)
• download FM singularity via container harbour instead of download.py (OvD), currently from download portal: https://github.com/Deltares/dfm_tools/pull/488/files
• also make it possible to run the model without modelbuilder, so download+unpack zip with modelfiles?
• binder runs on Kubernetes
• C-scale used docker: https://github.com/c-scale-community/workflow-coastal-hydrowaq/tree/main/fm_model
• singularity on: https://download.deltares.nl/en/delft3dfm-2d3d-beta-singularity (direct download link available)
• add singularity container download #487
• singularity nor docker available on binder per default
• Alternative: singularity also runs on AWS (RdG)
• Alternative: snellius supports interactive jupyter notebook, this should also work on h7 if a node can be reserved for the course
• Alternative: C-SCALE uses https://replay.notebooks.egi.eu/, replay is EGI's implementation of mybinder
• alternatively, EDITO uses jupyter and has connection to HPC
• also possible to use binderhub on different cluster
• if all else fails: zip+download model and run on local machine with bat file

Interactive jupyter slurm:

#!/bin/bash
#SBATCH -t 1:00:00
#SBATCH -N 1

# Make sure the jupyter command is available, either by loading the appropriate modules, sourcing your own virtual environment, etc.
module load 2021
source /home/muiss/miniconda3/etc/profile.d/conda.sh
conda activate /home/muiss/miniconda3/envs/era5-e

# Choose random port and print instructions to connect
PORT=`shuf -i 5000-5999 -n 1`
LOGIN_HOST=${SLURM_SUBMIT_HOST}-pub.snellius.surf.nl
BATCH_HOST=$(hostname)

echo "To connect to the notebook type the following command from your local terminal:"
echo "ssh -J ${USER}@${LOGIN_HOST} ${USER}@${BATCH_HOST} -L ${PORT}:localhost:${PORT}"
echo
echo "After connection is established in your local browser go to the address:"
echo "http://localhost:${PORT}"

jupyter lab --no-browser --port ${PORT}