From e7b7d1f62ce89897b6d7ffbd74ad1605552ff6aa Mon Sep 17 00:00:00 2001 From: Maaike Date: Wed, 24 Apr 2024 09:55:06 +0200 Subject: [PATCH 01/15] security section --- docs/source/index.rst | 2 +- docs/source/user/public-services-setup.rst | 34 ++++++++++++++++++---- 2 files changed, 29 insertions(+), 7 deletions(-) diff --git a/docs/source/index.rst b/docs/source/index.rst index 3e7972c5..00c3fb19 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -17,7 +17,7 @@ wis2box enables World Meteorological Organization (WMO) members to publish and d * Free and Open Source (FOSS) * containerized: use of Docker, enabling easy deployment to cloud or on-premises infrastructure -Live demonstration instances of wis2box can be found at at https://demo.wis2box.wis.wmo.int. +You can view wis2box deployments currently sharing data on the WIS2 network at https://demo.wis2box.wis.wmo.int. User guide ========== diff --git a/docs/source/user/public-services-setup.rst b/docs/source/user/public-services-setup.rst index 2c39d189..8f76889e 100644 --- a/docs/source/user/public-services-setup.rst +++ b/docs/source/user/public-services-setup.rst @@ -5,13 +5,33 @@ Public services setup To share your data with the WIS2 network, you need to expose some of the wis2box services to the Global Services: -* The Global Cache needs to be able to access to your HTTP endpoint at port 80 to download data published by the wis2box instance -* The Global Broker needs to be able to subscribe to your MQTT endpoint at port 1883 to receive WIS2 notifications published by the wis2box instance +* The Global Cache needs to be able to access to your HTTP endpoint to download data published by the wis2box instance, +by default web-proxy in the wis2box-stack will proxy the content of wis2box-public bucket at /data/ on port 80, or on port 443 when using SSL +* The Global Broker needs to be able to subscribe to your MQTT endpoint to receive WIS2 notifications published by the wis2box instance, +by default mosquitto is available on port 1883 on your host, or on port 8883 when using SSL -Nginx (HTTP) -^^^^^^^^^^^^ +Security considerations +^^^^^^^^^^^^^^^^^^^^^^^ -wis2box runs a local nginx container allowing access to the following HTTP based services on port 80: +When exposing your services to the public internet, it is important to consider the security implications of doing so. + +Please ensure that you follow these best practices to ensure your wis2box-instance is secure: + +* Ensure that your wis2box-instance runs behind a firewall and only exposes the necessary ports to the public internet (e.g. 80, 443, 1883, 8883) +* MQTT-subscribers should use everyone/everyone as the username/password to subscribe to the WIS2 notifications published by your wis2box-instance +* Never share the values of WIS2BOX_BROKER_PASSWORD and WIS2BOX_STORAGE_PASSWORD, they are for internal use only +* Store the authentication tokens used in the wis2box-webapp securely and do not share them with unauthorized users +* Use SSL/TLS encryption to secure your services +* Consider customizing the default web-configuration defined in nginx/nginx.conf to only expose the services you want to share with the public + +The wis2box development team is not responsible for the security of your wis2box-instance. +It is your responsibility to ensure that your wis2box-instance is secure. +For any questions on concerns you can write an e-mail to wis2-support@wmo.int + +web-proxy (nginx) +^^^^^^^^^^^^^^^^^ + +wis2box runs a local nginx container allowing access to the following HTTP based services: .. csv-table:: :header: Function, URL @@ -24,7 +44,9 @@ wis2box runs a local nginx container allowing access to the following HTTP based You can edit ``nginx/nginx.conf`` to control which services are exposed through the nginx-container include in your stack. -You can edit ``docker-compose.override.yml`` to change the port on which the ``web-proxy`` service exposes HTTP on the localhost. +By default the web-proxy service is exposed on port 80 on the host running wis2box. +You can enable SSL by setting the environment variable ``WIS2BOX_SSL_CERT`` and ``WIS2BOX_SSL_KEY`` to the location of your SSL certificate and private key respectively. +When SSL is enabled, the web-proxy service is exposed on port 443 on the host running wis2box and uses the configuration defined in ``nginx/nginx-ssl.conf``. .. note:: The canonical link referenced in WIS2 notification messages by your wis2box will use the basepath ``WIS2BOX_URL/data``. From 08f2f79f0b5446df932a9f8aa7a525745b694ca7 Mon Sep 17 00:00:00 2001 From: Maaike Date: Wed, 24 Apr 2024 12:01:20 +0200 Subject: [PATCH 02/15] reference security in getting-started --- docs/source/user/getting-started.rst | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/docs/source/user/getting-started.rst b/docs/source/user/getting-started.rst index 19e0cf3a..f83bfef1 100644 --- a/docs/source/user/getting-started.rst +++ b/docs/source/user/getting-started.rst @@ -8,7 +8,17 @@ The recommended OS is Ubuntu 20.04 LTS. .. note:: - wis2box may work on other operating systems (for example AlmaLinux), but the officially supported OS is Ubuntu. + wis2box may work on other operating systems (for example AlmaLinux or Windows), but the officially supported OS is Ubuntu. + +When choosing the environment for the wis2box, consider the following: + +* The wis2box-instance should have good Internet connectivity, to download the docker images used in the wis2box-stack +* In order for the wis2box-instance to function as a WIS2-node, you need to ensure that the HTTP and MQTT ports on the instance can be made accessible over the public Internet. +* Ensure that the system providing input data can access the wis2box-instance + +.. note:: + + Before exposing the wis2box to the Internet, please review the 'security considerations' section in the :ref:`public-services` section. Network requirements -------------------- From 2dbb139fffc983a83ec97de879d1d21bad72b725 Mon Sep 17 00:00:00 2001 From: Maaike Date: Wed, 24 Apr 2024 19:19:54 +0200 Subject: [PATCH 03/15] elaborate on station-metadata and OSCAR --- .../reference/running/station-metadata.rst | 58 ++++++++++++++----- 1 file changed, 42 insertions(+), 16 deletions(-) diff --git a/docs/source/reference/running/station-metadata.rst b/docs/source/reference/running/station-metadata.rst index 507004bd..ac1e347f 100644 --- a/docs/source/reference/running/station-metadata.rst +++ b/docs/source/reference/running/station-metadata.rst @@ -3,43 +3,69 @@ Station metadata ================ -wis2box is designed to support data ingest and processing of any kind. For observations, -processing workflow typically requires station metadata to be present at runtime. +The authoritative source of station metadata for international reporting surface based stations is :ref:`OSCAR/Surface`_. -To manage your stations of interest, create a CSV file named ``metadata/station/station_list.csv`` in ``$WIS2BOX_HOST_DATADIR``, -specifying one line per station as follows: +The wis2box API can be used to retrieve station metadata from OSCAR/Surface and cache a subset of this station metadata in the backend. -.. literalinclude:: ../../../../examples/config/station_list.csv +The station metadata can be cached in two ways: + +* from the command-line in the wis2box-container by populating the station list CSV file +* or using the station editor in the wis2box-webapp, available at ``WIS2BOX_URL/wis2box-webapp/station`` -This CSV file is used by wis2box data processing pipelines and is required before starting automated -processing. +Using the station editor in the wis2box-webapp +---------------------------------------------- -.. note:: run the command ``wis2box metadata station publish-collection`` to - publish your stations as a collection to the wis2box API +When using the station editor in the wis2box-webapp, you can provide a WIGOS Station Identifier (WSI) and the station metadata will be fetched from OSCAR/Surface. +After the data has been fetched you can populate any missing fields and associate the station with one or more topics. -.. seealso:: :ref:`api-publishing` +Using the command-line +---------------------- -OSCAR/Surface -------------- +To cache station metadata from the command-line, edit the CSV file named ``metadata/station/station_list.csv`` in ``$WIS2BOX_HOST_DATADIR``, +specifying one line per station as follows: + +.. literalinclude:: ../../../../examples/config/station_list.csv -wis2box can derive station information from `OSCAR/Surface`_. To verify station metadata from OSCAR/Surface: +Then login in to the wis2box-container and run the command ``wis2box metadata station publish-collection`` +to insert all the stations in the station list into the backend. + +Within the wis2box-container you can fetch the required station metadata from OSCAR/Surface using the following command: .. code-block:: bash wis2box metadata station get WSI - where ``WSI`` is the WIGOS Station Identifier. This command will return the information required in the station list for wis2box data processing and publication. To add the station information to the station list, copy and paste the output of the above command, or rerun the above command, writing to the station list -automatically: - +file as follows: .. code-block:: bash wis2box metadata station get WSI >> ~/wis2box-data/metadata/station/station_list.csv +After using the command-line to cache the station metadata, you will need to associate the stations with one or more topics to see the stations in the wis2box-ui. + +If you want to associate all stations in your station metadata to one topic, you can use the following command: + +.. code-block:: bash + + wis2box metadata station add-topic + +If you want to add a topic to a single station, you can use the following command: + +.. code-block:: bash + + python3 wis2box-ctl.py login + wis2box metadata station add-topic --wsi + +If you want to add a topic to all stations from a specific territory, for example Italy, you can use the following command: + +.. code-block:: bash + + python3 wis2box-ctl.py login + wis2box metadata station add-topic --territory-name Italy Summary ------- From 97609de468e721028839b93578a8f122fadd572f Mon Sep 17 00:00:00 2001 From: Maaike Date: Wed, 24 Apr 2024 19:52:08 +0200 Subject: [PATCH 04/15] add some additional documentation on plugins --- docs/source/_static/aws-minimal.csv | 45 +++++++++++++++++++++++++++++ docs/source/user/data-ingest.rst | 17 +++++++++++ 2 files changed, 62 insertions(+) create mode 100644 docs/source/_static/aws-minimal.csv diff --git a/docs/source/_static/aws-minimal.csv b/docs/source/_static/aws-minimal.csv new file mode 100644 index 00000000..4af5446d --- /dev/null +++ b/docs/source/_static/aws-minimal.csv @@ -0,0 +1,45 @@ +"Column","Units","Data type","Description" +"wsi_series","","Integer","WIGOS identifier series" +"wsi_issuer","","Integer","WIGOS issuer of identifier" +"wsi_issue_number","","Integer","WIGOS issue number" +"wsi_local","","Character","WIGOS local identifier" +"wmo_block_number","","Integer","WMO block number" +"wmo_station_number","","Integer","WMO station number" +"station_type","","Integer","Type of observing station, encoding using code table 0 02 001 (set to 0, automatic)" +"year","","Integer","Year (UTC), the time of observation (based on the actual time the barometer is read)" +"month","","Integer","Month (UTC), the time of observation (based on the actual time the barometer is read)" +"day","","Integer","Day (UTC), the time of observation (based on the actual time the barometer is read)" +"hour","","Integer","Hour (UTC), the time of observation (based on the actual time the barometer is read)" +"minute","","Integer","Minute (UTC), the time of observation (based on the actual time the barometer is read)" +"latitude","degrees","Decimal","Latitude of the station (to 5 decimal places)" +"longitude","degrees","Decimal","Longitude of the station (to 5 decimal places)" +"station_height_above_msl","meters","Decimal","Height of the station ground above mean sea level (to 1 decimal place)" +"barometer_height_above_msl","meters","Decimal","Height of the barometer above mean sea level (to 1 decimal place), typically height of station ground plus the height of the sensor above local ground" +"station_pressure","Pa","Decimal","Pressure observed at the station level to the nearest 10 pascals" +"msl_pressure","Pa","Decimal","Pressure reduced to mean sea level to the nearest 10 pascals" +"geopotential_height","gpm","Integer","Geoptential height expressed in geopotential meters (gpm) to 0 decimal places" +"thermometer_height","meters","Decimal","Height of thermometer or temperature sensor above the local ground to 2 decimal places" +"air_temperature","Kelvin","Decimal","Instantaneous air temperature to 2 decimal places" +"dewpoint_temperature","Kelvin","Decimal","Instantaneous dewpoint temperature to 2 decimal places" +"relative_humidity","%","Integer","Instantaneous relative humidity to zero decimal places" +"method_of_ground_state_measurement","code table","Integer","Method of observing the state of the ground, encoded using code table 0 02 176" +"ground_state","code table","Integer","State of the ground encoded using code table 0 20 062" +"method_of_snow_depth_measurement","code table","Integer","Method of observing the snow depth encoded using code table 0 02 177" +"snow_depth","meters","Decimal","Snow depth at time of observation to 2 decimal places" +"precipitation_intensity","kg m-2 h-1","Decimal","Intensity of precipitation at time of observation to 5 decimal places" +"anemometer_height","meters","Decimal","Height of the anemometer above local ground to 2 decimal place" +"time_period_of_wind","minutes","Integer","Defines the time period over which the wind speed and direction have been averaged. Set to -10 to indicate a measurement period over the preceeding 10 minutes." +"wind_direction","degrees","Integer","Wind direction (at anemometer height) averaged from the caterisan components over the indicated time period, 0 decimal places" +"wind_speed","ms-1","Decimal","Wind speed (at anemometer height) averaged from the cartesian components over the indicated time period, 1 decimal place" +"maximum_wind_gust_direction_10_minutes","degrees","Integer","Highest 3 second average over the preceeding 10 minutes, 0 decimal places" +"maximum_wind_gust_speed_10_minutes","ms-1","Decimal","Highest 3 second average over the preceeding 10 minutes, 1 decimal place" +"maximum_wind_gust_direction_1_hour","degrees","Integer","Highest 3 second average over the preceeding hour, 0 decimal places" +"maximum_wind_gust_speed_1_hour","ms-1","Decimal","Highest 3 second average over the preceeding hour, 1 decimal place" +"maximum_wind_gust_direction_3_hours","degrees","Integer","Highest 3 second average over the preceeding 3 hours, 0 decimal places" +"maximum_wind_gust_speed_3_hours","ms-1","Decimal","Highest 3 second average over the preceeding 3 hours, 1 decimal place" +"rain_sensor_height","meters","Decimal","Height of the rain gauge above local ground to 2 decimal place" +"total_precipitation_1_hour","kg m-2","Decimal","Total precipitation over the past hour, 1 decimal place" +"total_precipitation_3_hours","kg m-2","Decimal","Total precipitation over the past 3 hours, 1 decimal place" +"total_precipitation_6_hours","kg m-2","Decimal","Total precipitation over the past 6 hours, 1 decimal place" +"total_precipitation_12_hours","kg m-2","Decimal","Total precipitation over the past 12 hours, 1 decimal place" +"total_precipitation_24_hours","kg m-2","Decimal","Total precipitation over the past 24 hours, 1 decimal place" diff --git a/docs/source/user/data-ingest.rst b/docs/source/user/data-ingest.rst index 93215675..8ad30731 100644 --- a/docs/source/user/data-ingest.rst +++ b/docs/source/user/data-ingest.rst @@ -31,6 +31,23 @@ Interactive data ingestion requires an execution token, which can be generated u Be sure to record the token value, as it will not be shown again. If you lose the token, you can generate a new one. +data mappings plugins +--------------------- + +The plugins you have configured for your dataset mappings will determine the actions taken when data is received in the MinIO storage bucket. + +The wis2box provides 3 types of built-in plugins to publish data in BUFR format: + +* `bufr2bufr` : the input is received in bufr format and split by subset, where each subset is published as a separate bufr message +* `synop2bufr` : the input is received in FM-12 synop format and converted to bufr format. The year and month are extracted from the file-pattern +* `csv2bufr` : the input is received in csv format and converted to bufr format + +When using the csv2bufr plugin, the columns are mapped to bufr-encoded values using a mappings-template. +You can find the reference for the built-in 'AWS'-mappings template here [aws-full.csv](../_static/aws-full.csv) + +To publish data for other data-formats you can use the 'Universal' plugin, which will pass through the data without any conversion. +Please note that you will need to ensure that the date-timestamp can be extracted from the file-pattern when using this plugin. + MinIO user interface -------------------- From 751866b6e2c0b3de73e64a0285f6da6119db8676 Mon Sep 17 00:00:00 2001 From: Tom Kralidis Date: Wed, 24 Apr 2024 16:57:33 -0400 Subject: [PATCH 05/15] Update index.rst --- docs/source/index.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/index.rst b/docs/source/index.rst index 00c3fb19..c721268c 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -17,7 +17,7 @@ wis2box enables World Meteorological Organization (WMO) members to publish and d * Free and Open Source (FOSS) * containerized: use of Docker, enabling easy deployment to cloud or on-premises infrastructure -You can view wis2box deployments currently sharing data on the WIS2 network at https://demo.wis2box.wis.wmo.int. +wis2box deployments currently sharing data on the WIS2 network can be found at https://demo.wis2box.wis.wmo.int. User guide ========== From 384c78e257d9ed20ea05110daf6169ba13688975 Mon Sep 17 00:00:00 2001 From: Tom Kralidis Date: Wed, 24 Apr 2024 17:02:28 -0400 Subject: [PATCH 06/15] Update station-metadata.rst --- .../reference/running/station-metadata.rst | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/docs/source/reference/running/station-metadata.rst b/docs/source/reference/running/station-metadata.rst index ac1e347f..883d2127 100644 --- a/docs/source/reference/running/station-metadata.rst +++ b/docs/source/reference/running/station-metadata.rst @@ -9,13 +9,13 @@ The wis2box API can be used to retrieve station metadata from OSCAR/Surface and The station metadata can be cached in two ways: -* from the command-line in the wis2box-container by populating the station list CSV file -* or using the station editor in the wis2box-webapp, available at ``WIS2BOX_URL/wis2box-webapp/station`` +* from the command-line in the wis2box-management container by populating the station list CSV file +* from the station editor in the wis2box-webapp, available at ``$WIS2BOX_URL/wis2box-webapp/station`` Using the station editor in the wis2box-webapp ---------------------------------------------- -When using the station editor in the wis2box-webapp, you can provide a WIGOS Station Identifier (WSI) and the station metadata will be fetched from OSCAR/Surface. +When using the station editor in the wis2box-webapp, if a WIGOS Station Identifier (WSI) is provided, the associated station metadata will be populated from OSCAR/Surface. After the data has been fetched you can populate any missing fields and associate the station with one or more topics. @@ -28,7 +28,7 @@ specifying one line per station as follows: .. literalinclude:: ../../../../examples/config/station_list.csv Then login in to the wis2box-container and run the command ``wis2box metadata station publish-collection`` -to insert all the stations in the station list into the backend. +to insert all stations in the station list into the backend. Within the wis2box-container you can fetch the required station metadata from OSCAR/Surface using the following command: @@ -45,22 +45,22 @@ file as follows: wis2box metadata station get WSI >> ~/wis2box-data/metadata/station/station_list.csv -After using the command-line to cache the station metadata, you will need to associate the stations with one or more topics to see the stations in the wis2box-ui. +After using the command-line to cache the station metadata, you will need to associate the stations with one or more topics to visualize the stations in the wis2box-ui. -If you want to associate all stations in your station metadata to one topic, you can use the following command: +To associate all stations in your station metadata to one topic, you can use the following command: .. code-block:: bash wis2box metadata station add-topic -If you want to add a topic to a single station, you can use the following command: +To add a topic to a single station, you can use the following command: .. code-block:: bash python3 wis2box-ctl.py login wis2box metadata station add-topic --wsi -If you want to add a topic to all stations from a specific territory, for example Italy, you can use the following command: +To add a topic to all stations from a specific territory, for example Italy, you can use the following command: .. code-block:: bash From 2ac6ca749d616035db60c14b04309874808edbc7 Mon Sep 17 00:00:00 2001 From: Tom Kralidis Date: Wed, 24 Apr 2024 17:08:03 -0400 Subject: [PATCH 07/15] Update data-ingest.rst --- docs/source/user/data-ingest.rst | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/docs/source/user/data-ingest.rst b/docs/source/user/data-ingest.rst index 8ad30731..619eb3f1 100644 --- a/docs/source/user/data-ingest.rst +++ b/docs/source/user/data-ingest.rst @@ -38,15 +38,15 @@ The plugins you have configured for your dataset mappings will determine the act The wis2box provides 3 types of built-in plugins to publish data in BUFR format: -* `bufr2bufr` : the input is received in bufr format and split by subset, where each subset is published as a separate bufr message -* `synop2bufr` : the input is received in FM-12 synop format and converted to bufr format. The year and month are extracted from the file-pattern -* `csv2bufr` : the input is received in csv format and converted to bufr format +* `bufr2bufr` : the input is received in BUFR format and split by subset, where each subset is published as a separate bufr message +* `synop2bufr` : the input is received in FM-12 SYNOP format and converted to BUFR format. The year and month are extracted from the file pattern +* `csv2bufr` : the input is received in csv format and converted to BUFR format -When using the csv2bufr plugin, the columns are mapped to bufr-encoded values using a mappings-template. -You can find the reference for the built-in 'AWS'-mappings template here [aws-full.csv](../_static/aws-full.csv) +When using the csv2bufr plugin, the columns are mapped to BUFR encoded values using a template. +The reference for the built-in 'AWS' mappings template can be found at [aws-full.csv](../_static/aws-full.csv) -To publish data for other data-formats you can use the 'Universal' plugin, which will pass through the data without any conversion. -Please note that you will need to ensure that the date-timestamp can be extracted from the file-pattern when using this plugin. +To publish data for other data formats you can use the 'Universal' plugin, which will pass through the data without any conversion. +Please note that you will need to ensure that the date timestamp can be extracted from the file pattern when using this plugin. MinIO user interface -------------------- From a75ad11f0e5675bd0e173685df0f60d526c407c1 Mon Sep 17 00:00:00 2001 From: Tom Kralidis Date: Wed, 24 Apr 2024 17:10:40 -0400 Subject: [PATCH 08/15] Update getting-started.rst --- docs/source/user/getting-started.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/source/user/getting-started.rst b/docs/source/user/getting-started.rst index f83bfef1..a562be4c 100644 --- a/docs/source/user/getting-started.rst +++ b/docs/source/user/getting-started.rst @@ -12,9 +12,9 @@ The recommended OS is Ubuntu 20.04 LTS. When choosing the environment for the wis2box, consider the following: -* The wis2box-instance should have good Internet connectivity, to download the docker images used in the wis2box-stack -* In order for the wis2box-instance to function as a WIS2-node, you need to ensure that the HTTP and MQTT ports on the instance can be made accessible over the public Internet. -* Ensure that the system providing input data can access the wis2box-instance +* The wis2box instance should have suitable Internet connectivity, to download the required Docker images used in the wis2box stack +* In order for the wis2box instance to function as a WIS2 Node, HTTP and MQTT ports on the instance need to be accessible over the public Internet +* Ensure that the system providing input data can access the wis2box instance .. note:: From b4c344cdcabe102b82dc4ca640004a8a335ae8fa Mon Sep 17 00:00:00 2001 From: Tom Kralidis Date: Wed, 24 Apr 2024 17:17:25 -0400 Subject: [PATCH 09/15] Update public-services-setup.rst --- docs/source/user/public-services-setup.rst | 27 +++++++++++----------- 1 file changed, 14 insertions(+), 13 deletions(-) diff --git a/docs/source/user/public-services-setup.rst b/docs/source/user/public-services-setup.rst index 8f76889e..4b31c2be 100644 --- a/docs/source/user/public-services-setup.rst +++ b/docs/source/user/public-services-setup.rst @@ -5,28 +5,27 @@ Public services setup To share your data with the WIS2 network, you need to expose some of the wis2box services to the Global Services: -* The Global Cache needs to be able to access to your HTTP endpoint to download data published by the wis2box instance, -by default web-proxy in the wis2box-stack will proxy the content of wis2box-public bucket at /data/ on port 80, or on port 443 when using SSL -* The Global Broker needs to be able to subscribe to your MQTT endpoint to receive WIS2 notifications published by the wis2box instance, -by default mosquitto is available on port 1883 on your host, or on port 8883 when using SSL +* The Global Cache needs to be able to access to your HTTP endpoint to download data published by the wis2box instance. +The web-proxy service in the wis2box stack will proxy the content of ``wis2box-public`` bucket at ``/data/`` on port 80, or on port 443 when using SSL +* The Global Broker needs to be able to subscribe to your MQTT endpoint to receive WIS2 notifications published by the wis2box instance. mosquitto is available on port 1883 on your host by default, or on port 8883 when using SSL Security considerations ^^^^^^^^^^^^^^^^^^^^^^^ -When exposing your services to the public internet, it is important to consider the security implications of doing so. +When exposing your services to the public Internet, it is important to consider the security implications of doing so. Please ensure that you follow these best practices to ensure your wis2box-instance is secure: -* Ensure that your wis2box-instance runs behind a firewall and only exposes the necessary ports to the public internet (e.g. 80, 443, 1883, 8883) -* MQTT-subscribers should use everyone/everyone as the username/password to subscribe to the WIS2 notifications published by your wis2box-instance -* Never share the values of WIS2BOX_BROKER_PASSWORD and WIS2BOX_STORAGE_PASSWORD, they are for internal use only +* Ensure that your wis2box instance runs behind a firewall and only exposes the necessary ports to the public internet (e.g. 80 or 443 for HTTP, 1883 or 8883 for MQTT) +* MQTT subscribers should use ``everyone/everyone`` as the username/password to subscribe to the WIS2 notifications published by your wis2box instance +* Never share the values of ``WIS2BOX_BROKER_PASSWORD`` and ``WIS2BOX_STORAGE_PASSWORD`` as they are only for internal use * Store the authentication tokens used in the wis2box-webapp securely and do not share them with unauthorized users * Use SSL/TLS encryption to secure your services -* Consider customizing the default web-configuration defined in nginx/nginx.conf to only expose the services you want to share with the public +* Consider customizing the default web configuration defined in ``nginx/nginx.conf`` to expose only the services to be shared with the public -The wis2box development team is not responsible for the security of your wis2box-instance. -It is your responsibility to ensure that your wis2box-instance is secure. -For any questions on concerns you can write an e-mail to wis2-support@wmo.int +The wis2box development team is not responsible for the security of your wis2box-instance and it is your responsibility to ensure that your wis2box instance is secure. + +GitHub issues and discussions provide a resourece and forum to discuss general wis2box features, bugs and updates. For specific security related questions, please write to ``wis2-support at wmo.int``. web-proxy (nginx) ^^^^^^^^^^^^^^^^^ @@ -45,7 +44,9 @@ wis2box runs a local nginx container allowing access to the following HTTP based You can edit ``nginx/nginx.conf`` to control which services are exposed through the nginx-container include in your stack. By default the web-proxy service is exposed on port 80 on the host running wis2box. -You can enable SSL by setting the environment variable ``WIS2BOX_SSL_CERT`` and ``WIS2BOX_SSL_KEY`` to the location of your SSL certificate and private key respectively. + +SSL can be enabled by setting the ``WIS2BOX_SSL_CERT`` and ``WIS2BOX_SSL_KEY`` environment variables to the location of your SSL certificate and private key respectively. + When SSL is enabled, the web-proxy service is exposed on port 443 on the host running wis2box and uses the configuration defined in ``nginx/nginx-ssl.conf``. .. note:: From b1bf8de3d5987ad2d01a3b41f0759536c2d63168 Mon Sep 17 00:00:00 2001 From: Tom Kralidis Date: Wed, 24 Apr 2024 19:45:22 -0400 Subject: [PATCH 10/15] fix doc build --- docs/source/reference/running/station-metadata.rst | 2 +- docs/source/user/getting-started.rst | 2 +- docs/source/user/public-services-setup.rst | 3 +-- docs/source/user/setup.rst | 2 +- 4 files changed, 4 insertions(+), 5 deletions(-) diff --git a/docs/source/reference/running/station-metadata.rst b/docs/source/reference/running/station-metadata.rst index 883d2127..6ddd8aa9 100644 --- a/docs/source/reference/running/station-metadata.rst +++ b/docs/source/reference/running/station-metadata.rst @@ -3,7 +3,7 @@ Station metadata ================ -The authoritative source of station metadata for international reporting surface based stations is :ref:`OSCAR/Surface`_. +The authoritative source of station metadata for international reporting surface based stations is `OSCAR/Surface`_. The wis2box API can be used to retrieve station metadata from OSCAR/Surface and cache a subset of this station metadata in the backend. diff --git a/docs/source/user/getting-started.rst b/docs/source/user/getting-started.rst index a562be4c..4b4eb5ee 100644 --- a/docs/source/user/getting-started.rst +++ b/docs/source/user/getting-started.rst @@ -18,7 +18,7 @@ When choosing the environment for the wis2box, consider the following: .. note:: - Before exposing the wis2box to the Internet, please review the 'security considerations' section in the :ref:`public-services` section. + Before exposing the wis2box to the Internet, please review the 'security considerations' section in the :ref:`public-services-setup` section. Network requirements -------------------- diff --git a/docs/source/user/public-services-setup.rst b/docs/source/user/public-services-setup.rst index 4b31c2be..ee51347e 100644 --- a/docs/source/user/public-services-setup.rst +++ b/docs/source/user/public-services-setup.rst @@ -5,8 +5,7 @@ Public services setup To share your data with the WIS2 network, you need to expose some of the wis2box services to the Global Services: -* The Global Cache needs to be able to access to your HTTP endpoint to download data published by the wis2box instance. -The web-proxy service in the wis2box stack will proxy the content of ``wis2box-public`` bucket at ``/data/`` on port 80, or on port 443 when using SSL +* The Global Cache needs to be able to access to your HTTP endpoint to download data published by the wis2box instance. The web-proxy service in the wis2box stack will proxy the content of ``wis2box-public`` bucket at ``/data/`` on port 80, or on port 443 when using SSL * The Global Broker needs to be able to subscribe to your MQTT endpoint to receive WIS2 notifications published by the wis2box instance. mosquitto is available on port 1883 on your host by default, or on port 8883 when using SSL Security considerations diff --git a/docs/source/user/setup.rst b/docs/source/user/setup.rst index c42d89bf..14ce5bb7 100644 --- a/docs/source/user/setup.rst +++ b/docs/source/user/setup.rst @@ -210,7 +210,7 @@ The next step is to add station metadata to your wis2box. This can be done inter Please note only data for stations that have been added to wis2box will be ingested and result in WIS2 notifications being published. -If you want to bulk insert station-data from a CSV file, please refer to the :ref:`bulk-insert-stations` section. +If you want to bulk insert station metadata from a CSV file, please refer to the `Bulk inserting stations from CSV`_ section. The station editor can be accessed by visiting the URL you specified during the configuration step, and adding ``/wis2box-webapp/station`` to the URL. From 7d11d20567a3ea8997eb523af0b03ca01d90693f Mon Sep 17 00:00:00 2001 From: Maaike Date: Thu, 25 Apr 2024 19:11:39 +0200 Subject: [PATCH 11/15] describe relationship between plugins and station metadata --- docs/source/reference/running/station-metadata.rst | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/docs/source/reference/running/station-metadata.rst b/docs/source/reference/running/station-metadata.rst index 6ddd8aa9..e786f931 100644 --- a/docs/source/reference/running/station-metadata.rst +++ b/docs/source/reference/running/station-metadata.rst @@ -3,7 +3,7 @@ Station metadata ================ -The authoritative source of station metadata for international reporting surface based stations is `OSCAR/Surface`_. +The authoritative source of station metadata for internationally reporting surface based stations is `OSCAR/Surface`_. The wis2box API can be used to retrieve station metadata from OSCAR/Surface and cache a subset of this station metadata in the backend. @@ -12,6 +12,12 @@ The station metadata can be cached in two ways: * from the command-line in the wis2box-management container by populating the station list CSV file * from the station editor in the wis2box-webapp, available at ``$WIS2BOX_URL/wis2box-webapp/station`` +The plugins for converting data to bufr will use the station metadata in the following ways: + +* synop2bufr: The station metadata is used to derives the WIGOS-station-identifier by matching with the traditional station identifierin the FM-12 input data. The station metadata is also used to encode the location and barometer height above sea-level into the BUFR message. +* csv2bufr: The station metadata list is used to whitelist records to publish. If no match is found in the station list for an input record based on the WSI, the record is not published. +* bufr2bufr: If the WIGOS Station Identifier (WSI) is missing in the input BUFR data, the WSI is looked up in the station metadata list using the traditional station identifier (block and station number, ship callsign etc.) and inserted into the BUFR message. Similarly, if the location or station elevation are missing in the input BUFR data, these are looked up in the station metadata list and inserted into the BUFR message as required before further processing. + Using the station editor in the wis2box-webapp ---------------------------------------------- From a337c59074c680d473e0e0838d2fe13928097971 Mon Sep 17 00:00:00 2001 From: Maaike Date: Thu, 25 Apr 2024 19:30:26 +0200 Subject: [PATCH 12/15] additional docs on aws-template plus csv download-link --- docs/source/user/data-ingest.rst | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-) diff --git a/docs/source/user/data-ingest.rst b/docs/source/user/data-ingest.rst index 619eb3f1..aa2c1841 100644 --- a/docs/source/user/data-ingest.rst +++ b/docs/source/user/data-ingest.rst @@ -42,12 +42,19 @@ The wis2box provides 3 types of built-in plugins to publish data in BUFR format: * `synop2bufr` : the input is received in FM-12 SYNOP format and converted to BUFR format. The year and month are extracted from the file pattern * `csv2bufr` : the input is received in csv format and converted to BUFR format -When using the csv2bufr plugin, the columns are mapped to BUFR encoded values using a template. -The reference for the built-in 'AWS' mappings template can be found at [aws-full.csv](../_static/aws-full.csv) - To publish data for other data formats you can use the 'Universal' plugin, which will pass through the data without any conversion. Please note that you will need to ensure that the date timestamp can be extracted from the file pattern when using this plugin. +The AWS template in csv2bufr plugin +----------------------------------- + +When using the csv2bufr plugin, the columns are mapped to BUFR encoded values using a template as defined in the repository `csv2bufr-templates`_. + +An example of a CSV file that can be ingested using the 'AWS' mappings template can be downloaded here :download:`AWS-example <../_static/aws-example.csv>` + +The CSV-columns description of the AWS-template can be downloaded here :download:`AWS-reference <../_static/aws-minimal.csv>` + + MinIO user interface -------------------- @@ -226,3 +233,4 @@ Next: :ref:`public-services-setup` .. _`wis2box-ftp`: https://github.com/wmo-im/wis2box-ftp .. _`wis2box-data-subscriber`: https://github.com/wmo-im/wis2box-data-subscriber .. _`WIS2 topic hierarchy`: https://github.com/wmo-im/wis2-topic-hierarchy +.. _`csv2bufr-templates`: https://github.com/wmo-im/csv2bufr-templates \ No newline at end of file From 231e3705de7c4ecfe7aac6e58ee38d81ff6103cd Mon Sep 17 00:00:00 2001 From: Maaike Date: Thu, 25 Apr 2024 22:01:45 +0200 Subject: [PATCH 13/15] OSCAR is not authorative source, reword --- docs/source/reference/running/station-metadata.rst | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/docs/source/reference/running/station-metadata.rst b/docs/source/reference/running/station-metadata.rst index e786f931..009ef267 100644 --- a/docs/source/reference/running/station-metadata.rst +++ b/docs/source/reference/running/station-metadata.rst @@ -3,20 +3,22 @@ Station metadata ================ -The authoritative source of station metadata for internationally reporting surface based stations is `OSCAR/Surface`_. +`OSCAR/Surface`_ is the key resource for station metadata and contains a global compilation of the metadata for stations +that exchange data internationally. +Prior to exchanging data on the WIS2, stations must be registered in OSCAR/Surface and assigned a WIGOS-station-identifier (WSI). -The wis2box API can be used to retrieve station metadata from OSCAR/Surface and cache a subset of this station metadata in the backend. +The wis2box API can be used to retrieve station metadata from OSCAR/Surface and cache a subset of the station metadata in the wis2box backend. The station metadata can be cached in two ways: -* from the command-line in the wis2box-management container by populating the station list CSV file +* from the command-line in the wis2box-management container, after populating the station list CSV file * from the station editor in the wis2box-webapp, available at ``$WIS2BOX_URL/wis2box-webapp/station`` -The plugins for converting data to bufr will use the station metadata in the following ways: +The plugins for converting data to bufr will use the cached station metadata in the following ways: -* synop2bufr: The station metadata is used to derives the WIGOS-station-identifier by matching with the traditional station identifierin the FM-12 input data. The station metadata is also used to encode the location and barometer height above sea-level into the BUFR message. +* synop2bufr: The station metadata is used to derives the WSI by matching with the traditional station identifierin the FM-12 input data. The station metadata is also used to encode the location and barometer height above sea-level into the BUFR message. * csv2bufr: The station metadata list is used to whitelist records to publish. If no match is found in the station list for an input record based on the WSI, the record is not published. -* bufr2bufr: If the WIGOS Station Identifier (WSI) is missing in the input BUFR data, the WSI is looked up in the station metadata list using the traditional station identifier (block and station number, ship callsign etc.) and inserted into the BUFR message. Similarly, if the location or station elevation are missing in the input BUFR data, these are looked up in the station metadata list and inserted into the BUFR message as required before further processing. +* bufr2bufr: If the WSI is missing in the input BUFR data, the WSI is looked up in the station metadata list using the traditional station identifier (block and station number, ship callsign etc.) and inserted into the BUFR message. Similarly, if the location or station elevation are missing in the input BUFR data, these are looked up in the station metadata list and inserted into the BUFR message as required before further processing. Using the station editor in the wis2box-webapp ---------------------------------------------- From 4d1d151c7daea79c6c4888a1026d8631ef286629 Mon Sep 17 00:00:00 2001 From: Maaike Date: Fri, 26 Apr 2024 08:29:53 +0200 Subject: [PATCH 14/15] add a note on minio --- docs/source/user/data-ingest.rst | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/docs/source/user/data-ingest.rst b/docs/source/user/data-ingest.rst index aa2c1841..9b1c203d 100644 --- a/docs/source/user/data-ingest.rst +++ b/docs/source/user/data-ingest.rst @@ -153,6 +153,16 @@ See below a Python example to upload data using the MinIO package: If you are running the script on the same host as wis2box, you can use the endpoint ``http://localhost:9000`` as in the example. Otherwise, replace localhost with the IP address of the host running wis2box. +.. note:: + + You need to install the MinIO package for python otherwise you get the error ``ModuleNotFoundError: No module named 'minio'``. + + To install the MinIO package, run the following command: + + .. code-block:: bash + + pip3 install minio + wis2box-ftp ----------- From 890cd2ac226720c7a702a282437e4c504e774939 Mon Sep 17 00:00:00 2001 From: Tom Kralidis Date: Sun, 28 Apr 2024 07:58:48 -0400 Subject: [PATCH 15/15] doc fixes --- docs/source/reference/running/station-metadata.rst | 10 +++++----- docs/source/user/data-ingest.rst | 8 ++++---- 2 files changed, 9 insertions(+), 9 deletions(-) diff --git a/docs/source/reference/running/station-metadata.rst b/docs/source/reference/running/station-metadata.rst index 009ef267..59a12424 100644 --- a/docs/source/reference/running/station-metadata.rst +++ b/docs/source/reference/running/station-metadata.rst @@ -5,7 +5,7 @@ Station metadata `OSCAR/Surface`_ is the key resource for station metadata and contains a global compilation of the metadata for stations that exchange data internationally. -Prior to exchanging data on the WIS2, stations must be registered in OSCAR/Surface and assigned a WIGOS-station-identifier (WSI). +Prior to exchanging data on WIS2, stations must be registered in OSCAR/Surface and assigned a WIGOS station identifier (WSI). The wis2box API can be used to retrieve station metadata from OSCAR/Surface and cache a subset of the station metadata in the wis2box backend. @@ -14,11 +14,11 @@ The station metadata can be cached in two ways: * from the command-line in the wis2box-management container, after populating the station list CSV file * from the station editor in the wis2box-webapp, available at ``$WIS2BOX_URL/wis2box-webapp/station`` -The plugins for converting data to bufr will use the cached station metadata in the following ways: +The plugins for converting data to BUFR will use the cached station metadata as follows: -* synop2bufr: The station metadata is used to derives the WSI by matching with the traditional station identifierin the FM-12 input data. The station metadata is also used to encode the location and barometer height above sea-level into the BUFR message. -* csv2bufr: The station metadata list is used to whitelist records to publish. If no match is found in the station list for an input record based on the WSI, the record is not published. -* bufr2bufr: If the WSI is missing in the input BUFR data, the WSI is looked up in the station metadata list using the traditional station identifier (block and station number, ship callsign etc.) and inserted into the BUFR message. Similarly, if the location or station elevation are missing in the input BUFR data, these are looked up in the station metadata list and inserted into the BUFR message as required before further processing. +* synop2bufr: The station metadata is used to derives the WSI by matching with the traditional station identifier in the FM-12 input data. The station metadata is also used to encode the location and barometer height above sea-level into the BUFR message +* csv2bufr: The station metadata list is used to whitelist records to publish. If no match is found in the station list for an input record based on the WSI, the record is not published +* bufr2bufr: If the WSI is missing in the input BUFR data, the WSI is looked up in the station metadata list using the traditional station identifier (block and station number, ship callsign etc.) and inserted into the BUFR message. Similarly, if the location or station elevation are missing in the input BUFR data, these are looked up in the station metadata list and inserted into the BUFR message as required before further processing Using the station editor in the wis2box-webapp ---------------------------------------------- diff --git a/docs/source/user/data-ingest.rst b/docs/source/user/data-ingest.rst index 9b1c203d..f6cbaf83 100644 --- a/docs/source/user/data-ingest.rst +++ b/docs/source/user/data-ingest.rst @@ -50,9 +50,9 @@ The AWS template in csv2bufr plugin When using the csv2bufr plugin, the columns are mapped to BUFR encoded values using a template as defined in the repository `csv2bufr-templates`_. -An example of a CSV file that can be ingested using the 'AWS' mappings template can be downloaded here :download:`AWS-example <../_static/aws-example.csv>` +An example of a CSV file that can be ingested using the 'AWS' mappings template can be downloaded here :download:`AWS-example <../_static/aws-minimal.csv>` -The CSV-columns description of the AWS-template can be downloaded here :download:`AWS-reference <../_static/aws-minimal.csv>` +The CSV columns description of the AWS template can be downloaded here :download:`AWS-reference <../_static/aws-minimal.csv>` MinIO user interface @@ -155,7 +155,7 @@ See below a Python example to upload data using the MinIO package: .. note:: - You need to install the MinIO package for python otherwise you get the error ``ModuleNotFoundError: No module named 'minio'``. + The MinIO package is required for running the script above. To install the MinIO package, run the following command: @@ -243,4 +243,4 @@ Next: :ref:`public-services-setup` .. _`wis2box-ftp`: https://github.com/wmo-im/wis2box-ftp .. _`wis2box-data-subscriber`: https://github.com/wmo-im/wis2box-data-subscriber .. _`WIS2 topic hierarchy`: https://github.com/wmo-im/wis2-topic-hierarchy -.. _`csv2bufr-templates`: https://github.com/wmo-im/csv2bufr-templates \ No newline at end of file +.. _`csv2bufr-templates`: https://github.com/wmo-im/csv2bufr-templates