Merge pull request #661 from akrherz/240202-2

Omnibus

cgi-bin/request/coop.py

@@ -443,7 +443,7 @@ def do_simple(cursor, ctx):
         SELECT station, high, low, precip, snow, snowd, narr_srad,
         era5land_srad, temp_estimated, precip_estimated,
         era5land_soilt4_avg, era5land_soilm4_avg, era5land_soilm1m_avg,
-        nldas_soilt4_avg, nldas_soilm4_avg,
+        nldas_soilt4_avg, nldas_soilm4_avg, nldas_soilm1m_avg,
         merra_srad, hrrr_srad,
         to_char(('{thisyear}-'||month||'-'||extract(day from day))::date,
         'YYYY/mm/dd') as day,
@@ -460,7 +460,7 @@ def do_simple(cursor, ctx):
         SELECT station, high, low, precip, snow, snowd, narr_srad,
         era5land_srad, temp_estimated, precip_estimated,
         era5land_soilt4_avg, era5land_soilm4_avg, era5land_soilm1m_avg,
-        nldas_soilt4_avg, nldas_soilm4_avg,
+        nldas_soilt4_avg, nldas_soilm4_avg, nldas_soilm1m_avg,
         merra_srad, hrrr_srad,
         to_char(day, 'YYYY/mm/dd') as day,
         extract(doy from day) as doy,

htdocs/info/iem.php

@@ -7,15 +7,17 @@
 
 <h3>Iowa Environmental Mesonet</h3>
 
+<p><strong>Last Updated:</strong> 2 February 2024</p>
+
 <br><div><h3>Background</h3>
 <p>The Iowa Environmental Mesonet [IEM] aims to gather, collect,
-compare, disseminate and archive observations made in Iowa.  Unlike other 
+compare, disseminate and archive observations made in Iowa and beyond.  Unlike other 
 mesonet projects, the IEM does not own or operate any of the automated stations.
-Rather, the IEM collects data from existing resources in the state.  The result
+Rather, the IEM collects data from existing public resources.  The result
 is a low-cost, high resolution mesonet for use in a wide 
-range of disciplines.
+range of disciplines.</p>
 
-<br><br>One of the first questions we are often asked is, 'What does
+<p>One of the first questions we are often asked is, 'What does
 Mesonet mean?' <i>Meso-net</i> is a combination of two meteorological
 terms.  <i>Meso</i> refers to a spatial scale on which Meteorologists
 define certain weather phenomena. In the context of an observing network,
@@ -31,22 +33,21 @@
 <ul>
  <li>Iowa Department of Transportation [IaDOT]</li>
  <li>Iowa State University & Department of Agronomy [ISU]</li>
- <li>KCCI-TV8 Des Moines, Iowa [KCCI]</li>
+ <li>Various broadcast TV stations</li>
  <li>National Weather Service [NWS]</li>
 </ul>
 
 <br><h3>Data Networks</h3>
-<p>As of 1 April 2002, the IEM is gathering information from 
-over 7 permanent observing networks in the State.  These networks include...
+<p>As of the year 2024, the IEM is gathering information from the following:
 <ul>
- <li>Automated Surface Observing System [<a href="/ASOS/">ASOS</a>]</li>
- <li>Automated Weather Observing System [<a href="/AWOS/">AWOS</a>]</li>
+ <li>Automated Surface Observing System [<a href="/ASOS/">ASOS/AWOS</a>]</li>
  <li>Cooperative Observer Program [<a href="/COOP/">COOP</a>]</li>
+ <li>Community Collaborative Rain, Hail and Snow Network [<a href="https://cocorahs.org">CoCoRaHS</a>]</li>
  <li>River Gauges / Data Collection Platforms [<a href="/DCP/">DCP</a>]</li>
- <li>Iowa State Agricultural Climate Network [<a href="/agclimate">ISUAG</a>]</li>
+ <li>Iowa State University Soil Moisture Network [<a href="/agclimate">ISUAG</a>]</li>
  <li>Roadway Weather Information System [<a href="/RWIS/">RWIS</a>]</li>
  <li>Soil Climate Analysis Network [<a href="/scan/">SCAN</a>]</li>
- <li>KCCI-TV8 School Network [<a href="/schoolnet/">SNET</a>]</li>
+ <li>US Climate Reference Network [<a href="/current/uscrn.phtml">USCRN</a>]</li>
 </ul></p>
 
 <p>Clearly, the aforementioned list provides a wide range of
@@ -57,14 +58,12 @@
 application for use in aviation.  The RWIS sites are located near major
 highways and provide pavement temperatures for frost forecasting and
 chemical application guidance.  The ISUAG sites primarily monitor soil
-temperatures and augment precipitation observations in the state. The
-schoolnet sites, while located in poor meteorological locations, are
-intended to give public visibility to the local station and serve as a
-learning tool for students.  The DCP network provides river gauging needed
+temperatures and augment precipitation observations in the state. 
+ The DCP network provides river gauging needed
 for flood prediction and observation.  The COOP provides a daily weather 
-record for climatological use.
+record for climatological use.</p>
 
-<br><br>If you put all of these networks together, you can see the value
+<p>If you put all of these networks together, you can see the value
 that each network brings.  Combining them into one product is very
 difficult, hence the need for the IEM.  Sites in different networks are 
 not always similar in reporting
@@ -74,24 +73,8 @@
 station reports in the same units.  These issues are important to consider
 before beginning any quality control work.</p>
 
-<br><h3>Future of the IEM</h3>
-<p>Public response to the IEM have and continue to be 
-very positive.  It would be unproductive for the IEM to work on 
-projects/products that the public has no interest in.  Feedback from 
-end-users of the data is always welcome.  Currently, we are moving forward 
-on these projects...
-<ul>
- <li>Back-filling the IEM data archive from times before the IEM came into 
-existance.</li>
- <li>Identifying locations in the state where new sensors/sites would be 
-most beneficially placed.</li>
- <li>Building climatologies of stations and networks in the state.</li>
- <li>Maintaining/enhancing a meta-database of site information.</li>
- <li>Creating data products in GIS-ready formats.</li>
- <li>Meeting the data needs of end-users.</li>
- <li>Quality control issues.</li>
-</ul>
-<br><br>If you have any questions or comments, please
+
+<p>If you have any questions or comments, please
 <a href="contacts.php">let us know</a>.
 
 EOF;

htdocs/request/coop/fe.phtml

@@ -113,6 +113,7 @@ for one day, set the start and end times to the same value.</i>
   <option value="era5land_soilm1m_avg">Estimate: ERA5 Land Soil Moisture 0-1m [m3/m3] (1950-)</option>
   <option value="nldas_soilt4_avg">Estimate: NLDASv2 NOAHv2 Soil Temperature 0-10cm [F] (1979-)</option>
   <option value="nldas_soilm4_avg">Estimate: NLDASv2 NOAHv2 Soil Moisture 0-10cm [m3/m3] (1979-)</option>
+  <option value="nldas_soilm1m_avg">Estimate: NLDASv2 NOAHv2 Soil Moisture 0-1m [m3/m3] (1979-)</option>
 </select>
 
 <p><h4>4. Specialized Data/Model Formats</h4>

scripts/climodat/nldas_extract.py

@@ -10,12 +10,11 @@
 import numpy as np
 import pandas as pd
 from affine import Affine
+from pyiem.database import get_dbconnc, get_sqlalchemy_conn
 from pyiem.grid.zs import CachingZonalStats
 from pyiem.iemre import hourly_offset
 from pyiem.util import (
     convert_value,
-    get_dbconn,
-    get_sqlalchemy_conn,
     logger,
     ncopen,
     utc,
@@ -59,7 +58,7 @@ def build_stations(dt) -> pd.DataFrame:
             params=(dt,),
             index_col="station",
         )
-    for col in ["nldas_soilt4_avg", "nldas_soilm4_avg"]:
+    for col in ["nldas_soilt4_avg", "nldas_soilm4_avg", "nldas_soilm1m_avg"]:
         df[col] = np.nan
     df["i"] = np.nan
     df["j"] = np.nan
@@ -79,26 +78,33 @@ def compute(df, sids, dt, do_regions=False):
     with ncopen(ncfn) as nc:
         lons = nc.variables["lon"][:]
         lats = nc.variables["lat"][:]
+        tsel = slice(idx0, idx1)
         if f"{dt:%m%d}" == "1231":
             # Close enough
-            soilm = np.mean(nc.variables["soilm"][idx0:, 0], 0)
-            soilt = np.mean(nc.variables["soilt"][idx0:, 0], 0)
-        else:
-            soilm = np.mean(nc.variables["soilm"][idx0:idx1, 0], 0)
-            soilt = np.mean(nc.variables["soilt"][idx0:idx1, 0], 0)
+            tsel = slice(idx0, None)
+        soilm = np.mean(nc.variables["soilm"][tsel, 0], 0)
+        soilm1m = (
+            np.mean(nc.variables["soilm"][tsel, 0], 0) * 10.0
+            + np.mean(nc.variables["soilm"][tsel, 1], 0) * 30.0
+            + np.mean(nc.variables["soilm"][tsel, 2], 0) * 60.0
+        ) / 100.0
+        soilt = np.mean(nc.variables["soilt"][tsel, 0], 0)
 
     df["i"] = np.digitize(df["lon"].values, lons)
     df["j"] = np.digitize(df["lat"].values, lats)
     soilm = soilm.filled(np.nan)
+    soilm1m = soilm1m.filled(np.nan)
     soilt = soilt.filled(np.nan)
 
     for sid, row in df.loc[sids].iterrows():
         df.at[sid, "nldas_soilt4_avg"] = soilt[int(row["j"]), int(row["i"])]
         df.at[sid, "nldas_soilm4_avg"] = soilm[int(row["j"]), int(row["i"])]
+        df.at[sid, "nldas_soilm1m_avg"] = soilm1m[int(row["j"]), int(row["i"])]
 
     if do_regions:
         compute_regions(soilt, "nldas_soilt4_avg", df)
         compute_regions(soilm, "nldas_soilm4_avg", df)
+        compute_regions(soilm, "nldas_soilm1m_avg", df)
 
     LOG.info("IA0200 %s", df.loc["IA0200"])
 
@@ -123,13 +129,13 @@ def do(dt):
 
     # prevent NaN from being inserted
     df = df.replace({np.nan: None})
-    pgconn = get_dbconn("coop")
-    cursor = pgconn.cursor()
+    pgconn, cursor = get_dbconnc("coop")
     cursor.executemany(
         """
         UPDATE alldata set
         nldas_soilt4_avg = %(nldas_soilt4_avg)s,
-        nldas_soilm4_avg = %(nldas_soilm4_avg)s
+        nldas_soilm4_avg = %(nldas_soilm4_avg)s,
+        nldas_soilm1m_avg = %(nldas_soilm1m_avg)s
         where station = %(station)s and day = %(day)s
         """,
         df.to_dict("records"),

scripts/ua/ingest_from_rucsoundings.py

@@ -70,11 +70,10 @@
 import click
 import pandas as pd
 import requests
+from pyiem.database import get_dbconn, get_sqlalchemy_conn
 from pyiem.network import Table as NetworkTable
 from pyiem.util import (
     exponential_backoff,
-    get_dbconn,
-    get_sqlalchemy_conn,
     logger,
     utc,
 )
@@ -105,7 +104,12 @@ def conv_hhmm(self, raw):
             return self.valid.replace(hour=0, minute=0) + datetime.timedelta(
                 minutes=int(raw)
             )
-        ts = self.valid.replace(hour=int(raw[:-2]), minute=int(raw[-2:]))
+        minute = int(raw[-2:])
+        hour = int(raw[:-2])
+        if minute > 59:
+            minute = minute - 60
+            hour += 1
+        ts = self.valid.replace(hour=hour, minute=minute)
         if ts.hour > 20 and self.valid.hour < 2:
             ts -= datetime.timedelta(days=1)
         return ts
@@ -153,13 +157,12 @@ def database_save(self, txn):
         )
         txn.execute("DELETE from raob_profile where fid = %s", (fid,))
         if txn.rowcount > 0 and self.valid.hour in [0, 12]:
-            if self.station != "KSYA":  # noisey
-                LOG.warning(
-                    "RAOB del %s rows for sid: %s valid: %s",
-                    txn.rowcount,
-                    self.station,
-                    self.valid.strftime("%Y-%m-%d %H"),
-                )
+            LOG.info(
+                "RAOB del %s rows for sid: %s valid: %s",
+                txn.rowcount,
+                self.station,
+                self.valid.strftime("%Y-%m-%d %H"),
+            )
         table = f"raob_profile_{self.valid.year}"
         for d in self.profile:
             txn.execute(
@@ -319,7 +322,7 @@ def main(valid, station):
                 rob.database_save(cursor)
         except Exception as exp:
             fn = f"/tmp/{sid}_{valid:%Y%m%d%H%M}_fail"
-            LOG.info("FAIL %s %s %s, check %s for data", sid, valid, exp, fn)
+            LOG.warning("FAIL %s %s %s, content at %s", sid, valid, exp, fn)
             with open(fn, "wb") as fh:
                 fh.write(req.content)
         finally: