[3pt] PR: New unit inundation and run benchmark tests tool

.gitignore

@@ -89,6 +89,7 @@ config/**
 !config/r2f_config.env
 !config/*.lst
 !config/source_codes.csv
+!config/symbology/
 
 # Base data #
 #############

README.md

@@ -83,11 +83,14 @@ This download will include the following files / folders:
 3. National Water Model (NWM) Flowline Hydrofabric: nwm_flows.gpkg
 4. National Water Model to Watershed Boundary Lookup: nwm_wbd_lookup.nc
 5. National Water Model (NWM) Catchments file: nwm_catchments.gpkg
+6. You will need a HUC specific DEM which should exist in s3://noaa-nws-owp-fim/ras2fim/inputs/dems/ras_3dep_HUC8_10m/HUC8_{your huc number}_dem.tif. You can save it anywhere on your computer but it is suggested to save it at C:\ras2fim_data\inputs\dems\ras_3dep_HUC8_10m.
+<br>
+If that HUC8 DEM does not exist, you can create one using the tools/acquire_and_preprocess_3dep_dems.py. Include the argument of -skips3 so it does not attemp to upload it to an S3 bucket unless you have your own. ras2fim.py has an parameter to point to this file if not in the default path.
 <br><br>
 
 ### (2) Get AWS Folder - OWP_ras_models folder and OWP_ras_models_catalog.csv
 
-At this point, ras2fim.py needs a file named OWP_ras_models_catalog.csv, or similar, and we have loaded a sample for you. It has some meta data that is used in the final output files. While the file must exist with the correct schema, it will not fail if records in it do not match.  This file may become optional at a later point, but for now, please include it and also add the `-mc` argument to `ras2fim.py`. eg. `-mc c:\ras2fim_data\OWP_ras_models\OWP_ras_models_catalog.csv` (or pathing of your choice of course, as is with most arguments).
+At this point, ras2fim.py needs a file named OWP_ras_models_catalog.csv, or similar, and we have loaded a sample for you. It has some meta data that is used in the final output files. Thee file must have records that match models being processed. For the 'model_id' add any unique numbers you like, if you haven't used the tools/s3_get_models.py tool to download from an S3 bucket. Please include it and also add the `-mc` argument to `ras2fim.py`. eg. `-mc c:\ras2fim_data\OWP_ras_models\OWP_ras_models_catalog_{the huc8 number}_.csv` (or pathing of your choice of course, as is with most arguments).
 
 To download the `OWP_ras_models` folder, you AWS CLI command will be (adjusting for path overrides if you like):
 ```
@@ -103,14 +106,9 @@ The install package can be downloaded from [the USACE website](https://github.co
 ### (4) Clone the Git-hub repository
 <img src="https://github.com/NOAA-OWP/ras2fim/blob/main/doc/Git_logo.png" align="right" alt="git logo" height="80"> Install [git](https://git-scm.com/downloads) onto your Windows machine. Next, clone this ras2fim reporitory on to your Windows machine (see note below). Path to the windows folder of your choice, then type:
 ```
-git clone https://github.com/NOAA-OWP/ras2fim.git (see note below)
+git clone https://github.com/NOAA-OWP/ras2fim.git
 ```
-**Note: ras2fim V2 series has started development as of early November 2023. Initial check-ins for the default branch will fail for the short term future until an advanced version is ready.** When the CHANGELOG.md shows version check-ins starting with the phrase `V2.0.beta.x`, means the branch is not yet operational but you are still welcome to contibute to the new WIP branch. Once the usable versions have been merged, the CHANGELOG.md will start with the number of `v2.0.0.0`.
 <br>
-**If you want to view the last stable `V1` series, use the following git clone command which will come from the ras2fim `main (V1)` branch.**
-```
-git clone https://github.com/NOAA-OWP/ras2fim.git -b V1
-```
 
 ### (5) Building and Testing ras2fim
 

config/s3_unit_download_files.lst

@@ -0,0 +1,7 @@
+## Relative to an S3 'inputs' path  - use [] for huc subsitution.
+## Case sensitive for the S3 paths
+X-National_Datasets\nwm_flows.gpkg
+X-National_Datasets\nwm_wbd_lookup.nc
+X-National_Datasets\WBD_National.gpkg
+X-National_Datasets\WBD_HUC8\HUC8_[].gpkg
+dems\ras_3dep_HUC8_10m\HUC8_[]_dem.tif

config/source_codes.csv

@@ -1 +1,5 @@
 ble,FEMA Base Level Engineering (BLE)
+nws,National Weather Services
+ifc,Iowa Flood Services
+ras2fim,ras2fim
+usgs,United States Geological Survey

config/symbology/qgis/agreement_raster.qml

@@ -0,0 +1,39 @@
+<!DOCTYPE qgis PUBLIC 'http://mrcc.com/qgis.dtd' 'SYSTEM'>
+<qgis styleCategories="AllStyleCategories" version="3.10.5-A Coruña" minScale="1e+08" maxScale="0" hasScaleBasedVisibilityFlag="0">
+  <flags>
+    <Identifiable>1</Identifiable>
+    <Removable>1</Removable>
+    <Searchable>1</Searchable>
+  </flags>
+  <customproperties>
+    <property value="false" key="WMSBackgroundLayer"/>
+    <property value="false" key="WMSPublishDataSourceUrl"/>
+    <property value="0" key="embeddedWidgets/count"/>
+    <property value="Value" key="identify/format"/>
+  </customproperties>
+  <pipe>
+    <rasterrenderer type="paletted" opacity="1" alphaBand="-1" band="1">
+      <rasterTransparency/>
+      <minMaxOrigin>
+        <limits>None</limits>
+        <extent>WholeRaster</extent>
+        <statAccuracy>Estimated</statAccuracy>
+        <cumulativeCutLower>0.02</cumulativeCutLower>
+        <cumulativeCutUpper>0.98</cumulativeCutUpper>
+        <stdDevFactor>2</stdDevFactor>
+      </minMaxOrigin>
+      <colorPalette>
+        <paletteEntry label="TN" value="0" alpha="255" color="#000000"/>
+        <paletteEntry label="FN" value="1" alpha="255" color="#fdae61"/>
+        <paletteEntry label="FP" value="2" alpha="255" color="#d7191c"/>
+        <paletteEntry label="TP" value="3" alpha="255" color="#2c7bb6"/>
+        <paletteEntry label="Waterbody mask" value="4" alpha="255" color="#b2b2b2"/>
+      </colorPalette>
+      <colorramp type="randomcolors" name="[source]"/>
+    </rasterrenderer>
+    <brightnesscontrast brightness="0" contrast="0"/>
+    <huesaturation colorizeBlue="128" colorizeRed="255" grayscaleMode="0" colorizeStrength="100" colorizeOn="0" saturation="0" colorizeGreen="128"/>
+    <rasterresampler maxOversampling="2"/>
+  </pipe>
+  <blendMode>0</blendMode>
+</qgis>

config/symbology/qgis/inundation_raster.qml

@@ -0,0 +1,44 @@
+<!DOCTYPE qgis PUBLIC 'http://mrcc.com/qgis.dtd' 'SYSTEM'>
+<qgis styleCategories="AllStyleCategories" minScale="1e+8" hasScaleBasedVisibilityFlag="0" version="3.12.0-București" maxScale="0">
+  <flags>
+    <Identifiable>1</Identifiable>
+    <Removable>1</Removable>
+    <Searchable>1</Searchable>
+  </flags>
+  <customproperties>
+    <property key="WMSBackgroundLayer" value="false"/>
+    <property key="WMSPublishDataSourceUrl" value="false"/>
+    <property key="embeddedWidgets/count" value="0"/>
+    <property key="identify/format" value="Value"/>
+  </customproperties>
+  <pipe>
+    <rasterrenderer type="singlebandpseudocolor" opacity="1" classificationMin="0" alphaBand="-1" nodataColor="" classificationMax="0" band="1">
+      <rasterTransparency/>
+      <minMaxOrigin>
+        <limits>None</limits>
+        <extent>WholeRaster</extent>
+        <statAccuracy>Estimated</statAccuracy>
+        <cumulativeCutLower>0.02</cumulativeCutLower>
+        <cumulativeCutUpper>0.98</cumulativeCutUpper>
+        <stdDevFactor>2</stdDevFactor>
+      </minMaxOrigin>
+      <rastershader>
+        <colorrampshader classificationMode="1" colorRampType="INTERPOLATED" clip="0">
+          <colorramp name="[source]" type="gradient">
+            <prop v="166,97,26,255" k="color1"/>
+            <prop v="1,133,113,255" k="color2"/>
+            <prop v="0" k="discrete"/>
+            <prop v="gradient" k="rampType"/>
+            <prop v="0.25;223,194,125,255:0.5;245,245,245,255:0.75;128,205,193,255" k="stops"/>
+          </colorramp>
+          <item label="Dry" alpha="128" value="0" color="#dfc27d"/>
+          <item label="Wet" alpha="179" value="0" color="#0018f5"/>
+        </colorrampshader>
+      </rastershader>
+    </rasterrenderer>
+    <brightnesscontrast brightness="0" contrast="0"/>
+    <huesaturation colorizeGreen="128" grayscaleMode="0" colorizeOn="0" colorizeStrength="100" saturation="0" colorizeRed="255" colorizeBlue="128"/>
+    <rasterresampler maxOversampling="2"/>
+  </pipe>
+  <blendMode>0</blendMode>
+</qgis>

config/symbology/qgis/inundation_vector.qml

@@ -0,0 +1,184 @@
+<!DOCTYPE qgis PUBLIC 'http://mrcc.com/qgis.dtd' 'SYSTEM'>
+<qgis hasScaleBasedVisibilityFlag="0" readOnly="0" simplifyDrawingHints="1" labelsEnabled="0" simplifyDrawingTol="1" styleCategories="AllStyleCategories" maxScale="0" minScale="100000000" simplifyMaxScale="1" simplifyAlgorithm="0" simplifyLocal="1" version="3.12.0-București">
+  <flags>
+    <Identifiable>1</Identifiable>
+    <Removable>1</Removable>
+    <Searchable>1</Searchable>
+  </flags>
+  <renderer-v2 type="singleSymbol" enableorderby="0" forceraster="0" symbollevels="0">
+    <symbols>
+      <symbol type="fill" alpha="1" force_rhr="0" clip_to_extent="1" name="0">
+        <layer class="SimpleFill" enabled="1" pass="0" locked="0">
+          <prop k="border_width_map_unit_scale" v="3x:0,0,0,0,0,0"/>
+          <prop k="color" v="31,120,180,255"/>
+          <prop k="joinstyle" v="bevel"/>
+          <prop k="offset" v="0,0"/>
+          <prop k="offset_map_unit_scale" v="3x:0,0,0,0,0,0"/>
+          <prop k="offset_unit" v="MM"/>
+          <prop k="outline_color" v="31,120,180,255"/>
+          <prop k="outline_style" v="solid"/>
+          <prop k="outline_width" v="0.26"/>
+          <prop k="outline_width_unit" v="MM"/>
+          <prop k="style" v="solid"/>
+          <data_defined_properties>
+            <Option type="Map">
+              <Option type="QString" name="name" value=""/>
+              <Option name="properties"/>
+              <Option type="QString" name="type" value="collection"/>
+            </Option>
+          </data_defined_properties>
+        </layer>
+      </symbol>
+    </symbols>
+    <rotation/>
+    <sizescale/>
+  </renderer-v2>
+  <customproperties>
+    <property key="embeddedWidgets/count" value="0"/>
+    <property key="variableNames"/>
+    <property key="variableValues"/>
+  </customproperties>
+  <blendMode>0</blendMode>
+  <featureBlendMode>0</featureBlendMode>
+  <layerOpacity>1</layerOpacity>
+  <SingleCategoryDiagramRenderer attributeLegend="1" diagramType="Histogram">
+    <DiagramCategory labelPlacementMethod="XHeight" backgroundColor="#ffffff" spacingUnitScale="3x:0,0,0,0,0,0" opacity="1" enabled="0" showAxis="1" barWidth="5" direction="0" lineSizeScale="3x:0,0,0,0,0,0" height="15" minimumSize="0" minScaleDenominator="0" lineSizeType="MM" sizeType="MM" penAlpha="255" spacing="5" maxScaleDenominator="1e+8" rotationOffset="270" diagramOrientation="Up" scaleBasedVisibility="0" backgroundAlpha="255" spacingUnit="MM" scaleDependency="Area" width="15" penColor="#000000" penWidth="0" sizeScale="3x:0,0,0,0,0,0">
+      <fontProperties description="Sans Serif,9,-1,5,50,0,0,0,0,0" style=""/>
+      <axisSymbol>
+        <symbol type="line" alpha="1" force_rhr="0" clip_to_extent="1" name="">
+          <layer class="SimpleLine" enabled="1" pass="0" locked="0">
+            <prop k="capstyle" v="square"/>
+            <prop k="customdash" v="5;2"/>
+            <prop k="customdash_map_unit_scale" v="3x:0,0,0,0,0,0"/>
+            <prop k="customdash_unit" v="MM"/>
+            <prop k="draw_inside_polygon" v="0"/>
+            <prop k="joinstyle" v="bevel"/>
+            <prop k="line_color" v="35,35,35,255"/>
+            <prop k="line_style" v="solid"/>
+            <prop k="line_width" v="0.26"/>
+            <prop k="line_width_unit" v="MM"/>
+            <prop k="offset" v="0"/>
+            <prop k="offset_map_unit_scale" v="3x:0,0,0,0,0,0"/>
+            <prop k="offset_unit" v="MM"/>
+            <prop k="ring_filter" v="0"/>
+            <prop k="use_custom_dash" v="0"/>
+            <prop k="width_map_unit_scale" v="3x:0,0,0,0,0,0"/>
+            <data_defined_properties>
+              <Option type="Map">
+                <Option type="QString" name="name" value=""/>
+                <Option name="properties"/>
+                <Option type="QString" name="type" value="collection"/>
+              </Option>
+            </data_defined_properties>
+          </layer>
+        </symbol>
+      </axisSymbol>
+    </DiagramCategory>
+  </SingleCategoryDiagramRenderer>
+  <DiagramLayerSettings showAll="1" placement="1" linePlacementFlags="18" obstacle="0" dist="0" priority="0" zIndex="0">
+    <properties>
+      <Option type="Map">
+        <Option type="QString" name="name" value=""/>
+        <Option name="properties"/>
+        <Option type="QString" name="type" value="collection"/>
+      </Option>
+    </properties>
+  </DiagramLayerSettings>
+  <geometryOptions geometryPrecision="0" removeDuplicateNodes="0">
+    <activeChecks/>
+    <checkConfiguration type="Map">
+      <Option type="Map" name="QgsGeometryGapCheck">
+        <Option type="double" name="allowedGapsBuffer" value="0"/>
+        <Option type="bool" name="allowedGapsEnabled" value="false"/>
+        <Option type="QString" name="allowedGapsLayer" value=""/>
+      </Option>
+    </checkConfiguration>
+  </geometryOptions>
+  <referencedLayers/>
+  <referencingLayers/>
+  <fieldConfiguration>
+    <field name="fid">
+      <editWidget type="TextEdit">
+        <config>
+          <Option/>
+        </config>
+      </editWidget>
+    </field>
+    <field name="HydroID">
+      <editWidget type="Range">
+        <config>
+          <Option/>
+        </config>
+      </editWidget>
+    </field>
+  </fieldConfiguration>
+  <aliases>
+    <alias name="" index="0" field="fid"/>
+    <alias name="" index="1" field="HydroID"/>
+  </aliases>
+  <excludeAttributesWMS/>
+  <excludeAttributesWFS/>
+  <defaults>
+    <default applyOnUpdate="0" expression="" field="fid"/>
+    <default applyOnUpdate="0" expression="" field="HydroID"/>
+  </defaults>
+  <constraints>
+    <constraint notnull_strength="1" constraints="3" unique_strength="1" field="fid" exp_strength="0"/>
+    <constraint notnull_strength="0" constraints="0" unique_strength="0" field="HydroID" exp_strength="0"/>
+  </constraints>
+  <constraintExpressions>
+    <constraint exp="" field="fid" desc=""/>
+    <constraint exp="" field="HydroID" desc=""/>
+  </constraintExpressions>
+  <expressionfields/>
+  <attributeactions>
+    <defaultAction key="Canvas" value="{00000000-0000-0000-0000-000000000000}"/>
+  </attributeactions>
+  <attributetableconfig sortExpression="" actionWidgetStyle="dropDown" sortOrder="0">
+    <columns>
+      <column type="field" width="-1" hidden="0" name="fid"/>
+      <column type="field" width="-1" hidden="0" name="HydroID"/>
+      <column type="actions" width="-1" hidden="1"/>
+    </columns>
+  </attributetableconfig>
+  <conditionalstyles>
+    <rowstyles/>
+    <fieldstyles/>
+  </conditionalstyles>
+  <storedexpressions/>
+  <editform tolerant="1"></editform>
+  <editforminit/>
+  <editforminitcodesource>0</editforminitcodesource>
+  <editforminitfilepath></editforminitfilepath>
+  <editforminitcode><![CDATA[# -*- coding: utf-8 -*-
+"""
+QGIS forms can have a Python function that is called when the form is
+opened.
+
+Use this function to add extra logic to your forms.
+
+Enter the name of the function in the "Python Init function"
+field.
+An example follows:
+"""
+from qgis.PyQt.QtWidgets import QWidget
+
+def my_form_open(dialog, layer, feature):
+	geom = feature.geometry()
+	control = dialog.findChild(QWidget, "MyLineEdit")
+]]></editforminitcode>
+  <featformsuppress>0</featformsuppress>
+  <editorlayout>generatedlayout</editorlayout>
+  <editable>
+    <field editable="1" name="HydroID"/>
+    <field editable="1" name="fid"/>
+  </editable>
+  <labelOnTop>
+    <field labelOnTop="0" name="HydroID"/>
+    <field labelOnTop="0" name="fid"/>
+  </labelOnTop>
+  <widgets/>
+  <previewExpression>fid</previewExpression>
+  <mapTip></mapTip>
+  <layerGeometryType>2</layerGeometryType>
+</qgis>