docs update

docs/source/_static/aws-minimal.csv

@@ -0,0 +1,45 @@
+"Column","Units","Data type","Description"
+"wsi_series","","Integer","WIGOS identifier series"
+"wsi_issuer","","Integer","WIGOS issuer of identifier"
+"wsi_issue_number","","Integer","WIGOS issue number"
+"wsi_local","","Character","WIGOS local identifier"
+"wmo_block_number","","Integer","WMO block number"
+"wmo_station_number","","Integer","WMO station number"
+"station_type","","Integer","Type of observing station, encoding using code table 0 02 001 (set to 0, automatic)"
+"year","","Integer","Year (UTC), the time of observation (based on the actual time the barometer is read)"
+"month","","Integer","Month (UTC), the time of observation (based on the actual time the barometer is read)"
+"day","","Integer","Day (UTC), the time of observation (based on the actual time the barometer is read)"
+"hour","","Integer","Hour (UTC), the time of observation (based on the actual time the barometer is read)"
+"minute","","Integer","Minute (UTC), the time of observation (based on the actual time the barometer is read)"
+"latitude","degrees","Decimal","Latitude of the station (to 5 decimal places)"
+"longitude","degrees","Decimal","Longitude of the station (to 5 decimal places)"
+"station_height_above_msl","meters","Decimal","Height of the station ground above mean sea level (to 1 decimal place)"
+"barometer_height_above_msl","meters","Decimal","Height of the barometer above mean sea level (to 1 decimal place), typically height of station ground plus the height of the sensor above local ground"
+"station_pressure","Pa","Decimal","Pressure observed at the station level to the nearest 10 pascals"
+"msl_pressure","Pa","Decimal","Pressure reduced to mean sea level to the nearest 10 pascals"
+"geopotential_height","gpm","Integer","Geoptential height expressed in geopotential meters (gpm) to 0 decimal places"
+"thermometer_height","meters","Decimal","Height of thermometer or temperature sensor above the local ground to 2 decimal places"
+"air_temperature","Kelvin","Decimal","Instantaneous air temperature to 2 decimal places"
+"dewpoint_temperature","Kelvin","Decimal","Instantaneous dewpoint temperature to 2 decimal places"
+"relative_humidity","%","Integer","Instantaneous relative humidity to zero decimal places"
+"method_of_ground_state_measurement","code table","Integer","Method of observing the state of the ground, encoded using code table 0 02 176"
+"ground_state","code table","Integer","State of the ground encoded using code table 0 20 062"
+"method_of_snow_depth_measurement","code table","Integer","Method of observing the snow depth encoded using code table 0 02 177"
+"snow_depth","meters","Decimal","Snow depth at time of observation to 2 decimal places"
+"precipitation_intensity","kg m-2 h-1","Decimal","Intensity of precipitation at time of observation to 5 decimal places"
+"anemometer_height","meters","Decimal","Height of the anemometer above local ground  to 2 decimal place"
+"time_period_of_wind","minutes","Integer","Defines the time period over which the wind speed and direction have been averaged. Set to -10 to indicate a measurement period over the preceeding 10 minutes."
+"wind_direction","degrees","Integer","Wind direction (at anemometer height) averaged from the caterisan components over the indicated time period, 0 decimal places"
+"wind_speed","ms-1","Decimal","Wind speed (at anemometer height) averaged from the cartesian components over the indicated time period, 1 decimal place"
+"maximum_wind_gust_direction_10_minutes","degrees","Integer","Highest 3 second average over the preceeding 10 minutes, 0 decimal places"
+"maximum_wind_gust_speed_10_minutes","ms-1","Decimal","Highest 3 second average over the preceeding 10 minutes, 1 decimal place"
+"maximum_wind_gust_direction_1_hour","degrees","Integer","Highest 3 second average over the preceeding hour, 0 decimal places"
+"maximum_wind_gust_speed_1_hour","ms-1","Decimal","Highest 3 second average over the preceeding hour, 1 decimal place"
+"maximum_wind_gust_direction_3_hours","degrees","Integer","Highest 3 second average over the preceeding 3 hours, 0 decimal places"
+"maximum_wind_gust_speed_3_hours","ms-1","Decimal","Highest 3 second average over the preceeding 3 hours, 1 decimal place"
+"rain_sensor_height","meters","Decimal","Height of the rain gauge above local ground to 2 decimal place"
+"total_precipitation_1_hour","kg m-2","Decimal","Total precipitation over the past hour, 1 decimal place"
+"total_precipitation_3_hours","kg m-2","Decimal","Total precipitation over the past 3 hours, 1 decimal place"
+"total_precipitation_6_hours","kg m-2","Decimal","Total precipitation over the past 6 hours, 1 decimal place"
+"total_precipitation_12_hours","kg m-2","Decimal","Total precipitation over the past 12 hours, 1 decimal place"
+"total_precipitation_24_hours","kg m-2","Decimal","Total precipitation over the past 24 hours, 1 decimal place"

docs/source/index.rst

@@ -17,7 +17,7 @@ wis2box enables World Meteorological Organization (WMO) members to publish and d
 * Free and Open Source (FOSS)
 * containerized: use of Docker, enabling easy deployment to cloud or on-premises infrastructure
 
-Live demonstration instances of wis2box can be found at at https://demo.wis2box.wis.wmo.int.
+wis2box deployments currently sharing data on the WIS2 network can be found at https://demo.wis2box.wis.wmo.int.
 
 User guide
 ==========

docs/source/reference/running/station-metadata.rst

@@ -3,43 +3,77 @@
 Station metadata
 ================
 
-wis2box is designed to support data ingest and processing of any kind. For observations,
-processing workflow typically requires station metadata to be present at runtime.
+`OSCAR/Surface`_ is the key resource for station metadata and contains a global compilation of the metadata for stations
+that exchange data internationally.
+Prior to exchanging data on WIS2, stations must be registered in OSCAR/Surface and assigned a WIGOS station identifier (WSI).
 
-To manage your stations of interest, create a CSV file named ``metadata/station/station_list.csv`` in ``$WIS2BOX_HOST_DATADIR``,
-specifying one line per station as follows:
+The wis2box API can be used to retrieve station metadata from OSCAR/Surface and cache a subset of the station metadata in the wis2box backend.
 
-.. literalinclude:: ../../../../examples/config/station_list.csv
+The station metadata can be cached in two ways:
+
+* from the command-line in the wis2box-management container, after populating the station list CSV file
+* from the station editor in the wis2box-webapp, available at ``$WIS2BOX_URL/wis2box-webapp/station``
+
+The plugins for converting data to BUFR will use the cached station metadata as follows:
+
+* synop2bufr: The station metadata is used to derives the WSI by matching with the traditional station identifier in the FM-12 input data. The station metadata is also used to encode the location and barometer height above sea-level into the BUFR message
+* csv2bufr: The station metadata list is used to whitelist records to publish. If no match is found in the station list for an input record based on the WSI, the record is not published
+* bufr2bufr: If the WSI is missing in the input BUFR data, the WSI is looked up in the station metadata list using the traditional station identifier (block and station number, ship callsign etc.) and inserted into the BUFR message. Similarly, if the location or station elevation are missing in the input BUFR data, these are looked up in the station metadata list and inserted into the BUFR message as required before further processing
+
+Using the station editor in the wis2box-webapp
+----------------------------------------------
 
-This CSV file is used by wis2box data processing pipelines and is required before starting automated
-processing.
+When using the station editor in the wis2box-webapp, if a WIGOS Station Identifier (WSI) is provided, the associated station metadata will be populated from OSCAR/Surface.
 
-.. note:: run the command ``wis2box metadata station publish-collection`` to
-          publish your stations as a collection to the wis2box API
+After the data has been fetched you can populate any missing fields and associate the station with one or more topics.
 
+Using the command-line
+----------------------
 
-.. seealso:: :ref:`api-publishing`
+To cache station metadata from the command-line, edit the CSV file named ``metadata/station/station_list.csv`` in ``$WIS2BOX_HOST_DATADIR``,
+specifying one line per station as follows:
+
+.. literalinclude:: ../../../../examples/config/station_list.csv
 
-OSCAR/Surface
--------------
+Then login in to the wis2box-container and run the command ``wis2box metadata station publish-collection``
+to insert all stations in the station list into the backend.
 
-wis2box can derive station information from `OSCAR/Surface`_.  To verify station metadata from OSCAR/Surface:
+Within the wis2box-container you can fetch the required station metadata from OSCAR/Surface using the following command:
 
 .. code-block:: bash
 
    wis2box metadata station get WSI
 
-
 where ``WSI`` is the WIGOS Station Identifier.  This command will return the information required in the
 station list for wis2box data processing and publication.  To add the station information to the station list,
 copy and paste the output of the above command, or rerun the above command, writing to the station list
-automatically:
-
+file as follows:
 
 .. code-block:: bash
 
    wis2box metadata station get WSI >> ~/wis2box-data/metadata/station/station_list.csv
 
+After using the command-line to cache the station metadata, you will need to associate the stations with one or more topics to visualize the stations in the wis2box-ui.
+
+To associate all stations in your station metadata to one topic, you can use the following command:
+
+.. code-block:: bash
+
+   wis2box metadata station add-topic <topic-id>
+
+To add a topic to a single station, you can use the following command:
+
+.. code-block:: bash
+
+   python3 wis2box-ctl.py login
+   wis2box metadata station add-topic --wsi <station-id> <topic-id>
+
+To add a topic to all stations from a specific territory, for example Italy, you can use the following command:
+
+.. code-block:: bash
+
+   python3 wis2box-ctl.py login
+   wis2box metadata station add-topic --territory-name Italy <topic-id>
 
 Summary
 -------

docs/source/user/data-ingest.rst

@@ -31,6 +31,30 @@ Interactive data ingestion requires an execution token, which can be generated u
 
    Be sure to record the token value, as it will not be shown again. If you lose the token, you can generate a new one.
 
+data mappings plugins
+---------------------
+
+The plugins you have configured for your dataset mappings will determine the actions taken when data is received in the MinIO storage bucket.
+
+The wis2box provides 3 types of built-in plugins to publish data in BUFR format:
+
+* `bufr2bufr` : the input is received in BUFR format and split by subset, where each subset is published as a separate bufr message
+* `synop2bufr` : the input is received in FM-12 SYNOP format and converted to BUFR format. The year and month are extracted from the file pattern
+* `csv2bufr` : the input is received in csv format and converted to BUFR format
+
+To publish data for other data formats you can use the 'Universal' plugin, which will pass through the data without any conversion.
+Please note that you will need to ensure that the date timestamp can be extracted from the file pattern when using this plugin.
+
+The AWS template in csv2bufr plugin
+-----------------------------------
+
+When using the csv2bufr plugin, the columns are mapped to BUFR encoded values using a template as defined in the repository `csv2bufr-templates`_.
+
+An example of a CSV file that can be ingested using the 'AWS' mappings template can be downloaded here :download:`AWS-example <../_static/aws-minimal.csv>`
+
+The CSV columns description of the AWS template can be downloaded here :download:`AWS-reference <../_static/aws-minimal.csv>`
+
+
 MinIO user interface
 --------------------
 
@@ -129,6 +153,16 @@ See below a Python example to upload data using the MinIO package:
     If you are running the script on the same host as wis2box, you can use the endpoint ``http://localhost:9000`` as in the example. 
     Otherwise, replace localhost with the IP address of the host running wis2box. 
 
+.. note::
+
+    The MinIO package is required for running the script above.
+
+    To install the MinIO package, run the following command:
+
+    .. code-block:: bash
+
+        pip3 install minio
+
 wis2box-ftp
 -----------
 
@@ -209,3 +243,4 @@ Next: :ref:`public-services-setup`
 .. _`wis2box-ftp`: https://github.com/wmo-im/wis2box-ftp
 .. _`wis2box-data-subscriber`: https://github.com/wmo-im/wis2box-data-subscriber
 .. _`WIS2 topic hierarchy`: https://github.com/wmo-im/wis2-topic-hierarchy
+.. _`csv2bufr-templates`: https://github.com/wmo-im/csv2bufr-templates

docs/source/user/getting-started.rst

@@ -8,7 +8,17 @@ The recommended OS is Ubuntu 20.04 LTS.
 
 .. note::
 
-   wis2box may work on other operating systems (for example AlmaLinux), but the officially supported OS is Ubuntu.
+   wis2box may work on other operating systems (for example AlmaLinux or Windows), but the officially supported OS is Ubuntu.
+
+When choosing the environment for the wis2box, consider the following:
+
+* The wis2box instance should have suitable Internet connectivity, to download the required Docker images used in the wis2box stack
+* In order for the wis2box instance to function as a WIS2 Node, HTTP and MQTT ports on the instance need to be accessible over the public Internet
+* Ensure that the system providing input data can access the wis2box instance
+
+.. note::
+
+    Before exposing the wis2box to the Internet, please review the 'security considerations' section in the :ref:`public-services-setup` section. 
 
 Network requirements
 --------------------

docs/source/user/public-services-setup.rst

@@ -5,13 +5,31 @@ Public services setup
 
 To share your data with the WIS2 network, you need to expose some of the wis2box services to the Global Services:
 
-* The Global Cache needs to be able to access to your HTTP endpoint at port 80 to download data published by the wis2box instance
-* The Global Broker needs to be able to subscribe to your MQTT endpoint at port 1883 to receive WIS2 notifications published by the wis2box instance
+* The Global Cache needs to be able to access to your HTTP endpoint to download data published by the wis2box instance.  The web-proxy service in the wis2box stack will proxy the content of ``wis2box-public`` bucket at ``/data/`` on port 80, or on port 443 when using SSL
+* The Global Broker needs to be able to subscribe to your MQTT endpoint to receive WIS2 notifications published by the wis2box instance.  mosquitto is available on port 1883 on your host by default, or on port 8883 when using SSL
 
-Nginx (HTTP)
-^^^^^^^^^^^^
+Security considerations
+^^^^^^^^^^^^^^^^^^^^^^^
 
-wis2box runs a local nginx container allowing access to the following HTTP based services on port 80:
+When exposing your services to the public Internet, it is important to consider the security implications of doing so.
+
+Please ensure that you follow these best practices to ensure your wis2box-instance is secure:
+
+* Ensure that your wis2box instance runs behind a firewall and only exposes the necessary ports to the public internet (e.g. 80 or 443 for HTTP, 1883 or 8883 for MQTT)
+* MQTT subscribers should use ``everyone/everyone`` as the username/password to subscribe to the WIS2 notifications published by your wis2box instance
+* Never share the values of ``WIS2BOX_BROKER_PASSWORD`` and ``WIS2BOX_STORAGE_PASSWORD`` as they are only for internal use
+* Store the authentication tokens used in the wis2box-webapp securely and do not share them with unauthorized users
+* Use SSL/TLS encryption to secure your services
+* Consider customizing the default web configuration defined in ``nginx/nginx.conf`` to expose only the services to be shared with the public
+
+The wis2box development team is not responsible for the security of your wis2box-instance and it is your responsibility to ensure that your wis2box instance is secure.
+
+GitHub issues and discussions provide a resourece and forum to discuss general wis2box features, bugs and updates.  For specific security related questions, please write to ``wis2-support at wmo.int``.
+
+web-proxy (nginx)
+^^^^^^^^^^^^^^^^^
+
+wis2box runs a local nginx container allowing access to the following HTTP based services:
 
 .. csv-table::
    :header: Function, URL
@@ -24,7 +42,11 @@ wis2box runs a local nginx container allowing access to the following HTTP based
 
 You can edit ``nginx/nginx.conf`` to control which services are exposed through the nginx-container include in your stack.
 
-You can edit ``docker-compose.override.yml`` to change the port on which the ``web-proxy`` service exposes HTTP on the localhost.
+By default the web-proxy service is exposed on port 80 on the host running wis2box.
+
+SSL can be enabled by setting the ``WIS2BOX_SSL_CERT`` and ``WIS2BOX_SSL_KEY`` environment variables to the location of your SSL certificate and private key respectively.
+
+When SSL is enabled, the web-proxy service is exposed on port 443 on the host running wis2box and uses the configuration defined in ``nginx/nginx-ssl.conf``.
 
 .. note::
     The canonical link referenced in WIS2 notification messages by your wis2box will use the basepath ``WIS2BOX_URL/data``.

docs/source/user/setup.rst

@@ -210,7 +210,7 @@ The next step is to add station metadata to your wis2box. This can be done inter
 
 Please note only data for stations that have been added to wis2box will be ingested and result in WIS2 notifications being published.
 
-If you want to bulk insert station-data from a CSV file, please refer to the :ref:`bulk-insert-stations` section.
+If you want to bulk insert station metadata from a CSV file, please refer to the `Bulk inserting stations from CSV`_ section.
 
 The station editor can be accessed by visiting the URL you specified during the configuration step, and adding ``/wis2box-webapp/station`` to the URL.