1.0b8

This is the eighth 1.0 beta release of wis2box. This release includes a new CAP XML data pipeline, added utilities for WIS2 data download, as well numerous bug fixes and updates for WIS2 compliance.

Users are strongly encouraged to update to this release for WIS2 compliance.

Enhancements/updates

• add CAP XML data pipeline
• add data download functionality (via wis2downloader)
• add support for GTS headers applied in WIS2 Notification message
• adds discovery metadata republish command in CLI (includes automated daily publishing job)
• add station association capability in CLI
• show all system warnings (previously set to errors only)
• dataset driven approach implemented in wis2box data automated ingest and CLI
• improved logging / logfile management
• improved disk usage monitoring and management
• update MinIO container version
• add UI vulnerability testing for wis2box-webapp and wis2box-ui
• updated help links for FM-12 and CSV to BUFR in wis2box-webapp
• add metadata_id (for all data) and security (for recommended data) elements to WIS2 Notifications
• store local copy of WIGOS station metadata lookups
• add new discovery metadata templates in wis2box-webapp (drifters and ship based surface weather observations)
• supporting documentation for configuring email alerts from Grafana dashboard
• improved management of API configuration in wis2box-api

Bug fixes

• fix WCMP2 output themes/concepts formatting
• store local copy of WIGOS station metadata lookups for stability
• ensure wis2box starts successfully on Docker host restart
• add test to ensure WIS2BOX_BROKER_PUBLIC is correctly configured

New installations

Please follow the instructions provided at https://docs.wis2box.wis.wmo.int/en/1.0b8/user/getting-started.html

Migrating existing installations

When updating an existing wis2box instance, please run the following commands from the current release directory:

# stop wis2box
python3 wis2box-ctl.py stop

# delete old Docker images
docker image prune -a

# copy wis2box.env from the old to the new release directory
cp ~/wis2box-1.0b7/wis2box.env ~/wis2box-1.0b8/wis2box.env

wis2box-1.0b8 includes new/added download functionality. To use this functionality, run the following commands:

# set the DOCKER_GID environment variable
echo "DOCKER_GID=$(getent group docker | cut -d: -f3)" >> ~/wis2box-1.0b8/wis2box.env

# and add the "/downloads" directory to the directory defined as your WIS2BOX_HOST_DATADIR and set the permission to the docker group:
export $(grep -v '^#' ~/wis2box-1.0b8/wis2box.env | xargs -d '\n')
mkdir -p ${WIS2BOX_HOST_DATADIR}/downloads
chown -R $(whoami):docker ${WIS2BOX_HOST_DATADIR}/downloads
chmod -R 775 ${WIS2BOX_HOST_DATADIR}/downloads

Go to the new wis2box release directory and start wis2box:

cd ~/wis2box-1.0b8
python3 wis2box-ctl.py start

Documentation

Documentation for this release is available at https://docs.wis2box.wis.wmo.int/en/1.0b8

Feedback

All contributions are always welcome. Feedback is welcome via GitHub issues or discussions.

The wis2box team

https://docs.wis2box.wis.wmo.int

1.0b7

This is the seventh 1.0 beta release of wis2box. This release includes a number of new features, enhancements and bugfixes. This release introduces a "dataset" driven approach to closer align to core WIS2 concepts. A "dataset" in wis2box is a single record containing both the metadata and the data mappings definition. The separate data-mappings.yml will not longer be used.

This release also introduces the "dataset" editor that supports the creation of WCMP2 discovery metadata and data mappings definitions without requiring the user to edit configuration files directly.

The release also includes the a new "universal" data plugin to support the publishing of data that is produced upstream of wi2box.

Users are strongly encouraged to update to this latest release for WIS2 compliance.

Enhancements/updates

• move data mappings to discovery metadata record, removing data-mappings.yml
• added dataset editor to wis2box-webapp to support WCMP2 publishing and data mappings configuration
• added basic authentication to wis2box-webapp
• add new universal data pipeline
• update station metadata columns to be consistent between station list CSV and station editor, using WIGOS metadata tables
• csv2bufr templates now referenced from external repository
• remove API links from WCMP2 records
• numerous documentation updates, including updates for station metadata as well as security recommendations
• reduced loading-times for wis2box-ui

Bug fixes

• fix incorrect originating centre and sub centre incorrect in BUFR generation
• fix links in WCMP2 metadata

New installation

Please follow the instructions provided at https://docs.wis2box.wis.wmo.int/en/latest/user/getting-started.html

Migrating existing installations

Migrating existing installations will require you to recreate your discovery metadata records.

First, login to the wis2box-management container and unpublish your current discovery metadata:

cd wis2box-1.0b6
python3 wis2box-ctl.py login
wis2box metadata discovery unpublish <metadata-id>

Repeat the wis2box metadata discovery unpublish command until there are no items remaining in https://YOUR_HOST/oapi/collections/discovery-metadata/items

Then stop wis2box, and download the new release and extract the archive:

python3 wis2box-ctl.py stop
cd ../
wget https://github.com/wmo-im/wis2box/releases/download/1.0b6/wis2box-setup-1.0b7.zip
unzip wis2box-setup-1.0b7.zip

Copy your old wis2box.env and delete the api-config volume:

cp ~/wis2box-1.0b6/wis2box.env ~/wis2box-1.0b7wis2box.env
docker volume rm wis2box_project_api-config

Optionally, add new entries to wis2box.env for environment variables for WIS2BOX_WEBAPP_USERNAME and WIS2BOX_WEBAPP_PASSWORD. If these are not defined, the default credentials are:

WIS2BOX_WEBAPP_USERNAME=wis2box-user
WIS2BOX_WEBAPP_PASSWORD=${WIS2BOX_STORAGE_PASSWORD}

Then start the wis2box from the new release-folder:

cd ~/wis2box-1.0b7
python3 wis2box-ctl.py start

And run the migration script to update your stations to match the new definitions as follows:

python3 wis2box-ctl.py execute python3 /app/migrations/v1_0b6_to_v1_0b7/update_station_definition_v1.0b7.py

Finally use the dataset editor in wis2box-webapp to recreate your datasets, see https://docs.wis2box.wis.wmo.int/en/1.0b7/user/setup.html#adding-datasets.

Note that you can also recreate your metadata by adding data mappings within your wis2box metadata configurations and use the wis2box dataset publish <file-name> command to publish

Documentation

Documentation for this release is available at https://docs.wis2box.wis.wmo.int/en/1.0b7

Feedback

All contributions are always welcome. Feedback is welcome via GitHub issues or discussions.

The wis2box team

https://docs.wis2box.wis.wmo.int

wis2box-1.0b7-dev

This is in the WIP-release for the upcoming wis2box-1.0b7 release, as reference by the 'latest' documentation.

1.0b6

This is the sixth 1.0 beta release of wis2box. This release includes a number of new features, enhancements and bugfixes. This release implements the required updates to the WIS2 Topic Hierarchy regarding dataset topics. Users are strongly encouraged to update to this latest release for WIS2 compliance.

Enhancements/updates:

• Implementation of the WIS2 Topic Hierarchy update to merge country and centre-id (see the section below)
• UI: add option/toggle to display station points as clusters #588
• duplicate-data-check: wis2box will not publish the same data twice and instead print an error #600
• updated-data-check: wis2box will detect updates and add a link to indicate data is updated #565
• WNM: provide data inline in notification messages when data size less than 4096 bytes
• Storage guidelines are now 24GB
• Station list: barometer height is no longer calculated if not defined (left null/blank)
• updating of BUFR tools to their latest versions
• collect NGINX logs in loki

Bug fixes:

• BUFR handling:
  • "Passed array is too small" fix (#559)
  • fix TSI matching (#581)
• WCMP2
  • fix update date calculation
  • fix contact information
  • fix metadata message publication
• WTH
  • remove pywcmp in lieu of pywis-topics for topic handling

Choosing a valid centre-id for the pre-operational phase

The WIS2 Topic Hierarchy merges country and centre-id in a new centre-id that uniquely defines a WIS2-node.

If you plan to use wis2box-1.0b6 to run a WIS2-node in the pre-operational phase, please ensure to use your new centre-id starting with your TLD during the configuration process.

If you are unsure how to define the centre-id for your WIS2-node, please contact WMO.

New installation

Please follow the instructions provided at https://docs.wis2box.wis.wmo.int/en/latest/user/getting-started.html

Migrating existing installations

The following steps are required when upgrading from wis2box 1.0b5 to 1.0b6:

# stop current wis2box
cd ~/wis2box-1.0b5 && python3 wis2box-ctl.py stop

# download new release in home-directory
cd ~/ && wget https://github.com/wmo-im/wis2box/releases/download/1.0b6/wis2box-setup-1.0b6.zip

# unzip and go to new directory
unzip wis2box-setup-1.0b6.zip && cd wis2box-1.0b6

IMPORTANT: before starting the new wis2box-release your configuration files need to be updated to use the new WTH. You can use the migration-script to update your configuration files

# run migration script to update configuration
python3 migration/update_configuration.py

The migration-script will ask you to answer a series of questions to replace country-code and old centre-id with your new centre-id, as per the following example:

Enter your 3-letter country-code: ita
3-letter country-code: ita ? (y/n) y
Enter your old centre-id: maaike-test
Old centre-id: maaike-test ? (y/n) y
Enter your new centre-id: it-maaike-test
New centre-id: it-maaike-test ? (y/n) y
Updated file 'metadata-synop.yml'
Updated file 'data-mappings.yml'

Check and verify that your data-mappings.yml file contains the new topic hierarchy. An example can be found below:

data:
    it-maaike-test.data.core.weather.surface-based-observations.synop:
        plugins:
            txt:
                - plugin: wis2box.data.synop2bufr.ObservationDataSYNOP2BUFR
                  notify: true

Continue with the update:

# start wis2box
python3 wis2box-ctl.py start  # wait until the new images are downloaded

# login to wis2box and execute commands
python3 wis2box-ctl.py login

# create new data collection, replace with your filename
wis2box data add-collection /data/wis2box/metadata/discovery/metadata-synop.yml

# publish new discovery metadata, replace with your filename
wis2box metadata discovery publish /data/wis2box/metadata/discovery/metadata-synop.yml

Verify the wis2box UI to ensure that there are 2 entries per dataset (one for old and one for new topic hierarchy):

Continue with the update:

# reindex your data collection items, replace with your old and new discovery metadata ID
wis2box data reindex-collection-items urn:x-wmo:md:ita:maaike-test:surface-based-observations.synop urn:x-wmo:md:it-maaike-test:surface-based-observations.synop

# unpublish your old dataset using the old discovery metadata ID
wis2box metadata discovery unpublish urn:x-wmo:md:ita:maaike-test:surface-based-observations.synop

# associate stations to the new topic hierarchy
wis2box metadata station add-topic origin/a/wis2/it-maaike-test/data/core/weather/surface-based-observations/synop

# exit the wis2box-management container
exit

Finally you can clean the unused Docker images to preserve diskspace:

docker image prune -a

(Or docker image prune without -a depending on your docker version)

As a final step, you will need to update any data ingestion processes to copy to new corresponding folder in the wis2box-incoming bucket

(Old)

(New)

Documentation

Documentation for this release is available at https://docs.wis2box.wis.wmo.int/en/1.0b6

Feedback

All contributions are always welcome. Feedback is welcome via GitHub issues or discussions.

The wis2box team

https://docs.wis2box.wis.wmo.int

1.0b5

This is the fifth 1.0 beta release of wis2box. This release includes a number of new features, enhancements and bugfixes.

Updates

wis2box-webapp (new container) provides an additional web application adding the following functionality:

• new station editor user interface with support to query OSCAR/Surface
• data entry via web-based form for SYNOP ASCII and CSV
• monitoring dashboard to via data notifications and download/inspect BUFR produced by wis2box

Updates to support wis2box configuration:

• new configuration creation script to allow the user to a create initial wis2box.env, data-mappings.yml and template discovery metadata files by answering a series of question on the command line
• environment files default.env and dev.env have been replaced by wis2box.env

Enhancements:

• add barometer height to station metadata management
• enable adding more user to default mosquitto broker
• fix to generate valid WNM (base64 integrity)
• fixes to generate valid WCMP2 (needs flushing of ES discovery-metadata)
• support for Docker Compose v2 (Note is it required or supported?)
• CSV publication (csv2bufr):
  • Improved logging and return of warnings and errors to the result object
  • BUFR templates now managed though the csv2bufr.templates module (note this will be moved to a stand alone module in a future release)
  • addition of minimal template for data from AWS weather stations

Bug fixes:

• fix dataset deletion workflow
• do not remove discovery metadata when running data cleaning
• BUFR publication: BUFR headers are now correctly set in extracted subsets. Previously the headers were not copied, leading to incorrect replication factors and data corruption
• GeoJSON API publication: code table entry description now correctly extracted and set in GeoJSON output. Previously only the first word of the description was extracted

important

• use the station editor in wis2box-webapp and ensure your stations are associated to a topic to ensure you can see the data in the "MAP" and "EXPLORE" option of wis2box-ui
• the cronjob running wis2box metadata station publish-collection in the wis2box-management every 10 minutes is now disabled

Migrating existing installations

The following steps are required when upgrading from wis2box 1.0b4 to 1.0b5:

# stop current wis2box
cd ~/wis2box-1.0b4 && python3 wis2box-ctl.py stop
# prune unused docker images
docker image prune -a
# delete api-config volume
docker volume rm wis2box_project_api-config
# download new release in home-directory
cd && wget https://github.com/wmo-im/wis2box/releases/download/1.0b5/wis2box-setup-1.0b5.zip
# unzip and go to new directory
unzip wis2box-setup-1.0b5.zip && cd wis2box-1.0b5
# run migration script to create wis2box.env
python3 migration/migration_create_wis2box_env.py
# check content of wis2box.env correctly copied your local updates in dev:env
cat wis2box.env
# run migration script to add barometer_height column to station_list.csv:
python3 migration/migration_update_station_list.py
# start wis2box
python3 wis2box-ctl.py start
# login to wis2box and execute commands
python3 wis2box-ctl.py login
# (inside wis2box-management container)
wis2box data add-collection /data/wis2box/surface-weather-observations.yml
wis2box metadata station publish-collection
# new images are downloaded (this may take awhile)

For installations older than 1.0b4, it is recommended to start from a clean/fresh installation.

To start a fresh installation, please clean your docker system after stopping your wis2box:

python3 wis2box-ctl.py stop
docker image prune -a
docker volume prune -a

...then delete the directory containing your current release and download the latest release and following the setup instructions.

Documentation

Documentation for this release is available at https://docs.wis2box.wis.wmo.int/en/1.0b5

Feedback

All contributions are always welcome. Feedback is welcome via GitHub issues or discussions.

The wis2box team

https://docs.wis2box.wis.wmo.int

1.0b4

This is the fourth beta release of wis2box. This release includes a number of bugfixes.

Updates

• discovery metadata publication: fix message publication for discovery metadata (#454)
• metrics: do not display failure descriptions: metrics (#455)
• update to latest stable versions of csv2bufr, synop2bufr and pymetdecoder (#457)
• safeguard station metadata geometry z (#458)

Migrating existing installations

There are no migrations required when upgrading from 1.0b3.

Documentation

Documentation for this release is available at https://docs.wis2box.wis.wmo.int/en/1.0b4

Feedback

All contributions are always welcome. Feedback is welcome via GitHub issues or discussions.

The wis2box team

https://docs.wis2box.wis.wmo.int

1.0b3

This is the third beta release of wis2box. This release includes a number of enhancements and bugfixes.

Updates

• new experimental metadata UI for station (#376) and discovery metadata (#154) management. This functionality is added to the release to facilitate initial stakeholder feedback, and is disabled by default. Future releases will provide the UI by default with detailed documentation on use
• wis2box-api enhancements
  • add metadata validation workflow
  • upgrade to Elasticsearch 8
• validation checks on topic matching for discovery metadata and API managements workflows (#431
• addition of data subscriber for integration of automated weather station (AWS) data (#430)
• improved user documentation for managing environment variables and configuration settings

Migrating existing installations

The upgrade to Elasticsearch 8 will update to your API data. If you are using wis2box 1.0b2, then you should be able to upgrade without issue. For versions previous to 1.0b2, a two-step upgrade is required as follows once 1.0b3 is installed:

python3 wis2box-ctl.py stop
# download wis2box 1.0b3
vi docker-compose.yml  # change elasticsearch version from 8.6.2 to 7.17.0 to ensure a clean upgrade from 7.16.2
python3 wis2box-ctl.py start
python3 wis2box-ctl.py stop
vi docker-compose.yml  # change elasticsearch version from 7.17.0 to 8.6.2
docker volume rm wis2box_project_api-config
python3 wis2box-ctl.py start

Documentation

Documentation for this release is available at https://docs.wis2box.wis.wmo.int/en/1.0b3

Feedback

All contributions are always welcome. Feedback is welcome via GitHub issues or discussions.

The wis2box team

https://docs.wis2box.wis.wmo.int

1.0b2

This is the second beta release of wis2box. This release includes a number of enhancements and bugfixes.

Updates

• force error handling in bufr2geojson (#417)
• add optional FTP forwarder service (#297)
• add improved monitoring features (#329)
• run station publishing on startup and every 10 minutes (#413)
• fix date handling for BUFR pipeline (#412)
• allow csv2bufr templates to be local or external (#405)
• add standard data pipeline reporting (#329)
• update to ecCodes 2.28.0 (#381)
• add SSL deployment option (#140)
• update WIS2 notification message generation (#391)
• align data mappings workflow to stations (#388)
• set broker default message queue to 1000 (#377)
• add public user to broker for WIS2 compliance (#386)
• add WIS2 topic hierarchy validation (#228)

Migrating existing installations

Given the changes in various configurations, it is recommended to install 1.0b2 from scratch:

• delete wis2box Docker volumes
• update discovery metadata YAML configuration examples
• update station_list.csv from examples provided
• repeat runtime configuration

Documentation

Documentation for this release is available at https://docs.wis2box.wis.wmo.int/en/1.0b2

Feedback

All contributions are always welcome. Feedback is welcome via GitHub issues or discussions.

The wis2box team

https://docs.wis2box.wis.wmo.int

1.0b1

This is the first beta release wis2box. This release includes a number of updates and bugfixes, including:

• new support for SYNOP to BUFR processing (#304)
• local station metadata management (#324)
• numerous documentation enhancements, including a user guide and reference documentation
• ensure API collections names are lower case
• streamlining of csv2bufr templates
• new WMO theme for documentation (#74)
• refactored and streamlined repository (#213)

User updates/changes (for existing installations):

• station metadata is managed in $WIS2BOX_DATADIR/metadata/station/station_list.csv and is managed by the user
• station metadata in the API needs to be flushed manually given model changes. From the wis2box-management container (i.e. python3 wis2box-ctl.py login):
  • curl -XDELETE http://elasticsearch:9200/stations
  • wis2box-ctl.py metadata station publish-collection
• existing data-mappings.yml files need to be updated to point the csv2bufr data pipeline to use template synop_bufr.json (replacing synop-bufr.json)

Documentation for 1.0.beta1 is available at https://docs.wis2box.wis.wmo.int/en/1.0b1

All contributions are always welcome. Feedback is welcome via GitHub issues or discussions.

The wis2box team

https://docs.wis2box.wis.wmo.int

0.5.1

This is the 0.5.1 working release of wis2box. This is a bugfix released; updates include:

• fix BUFR multiple message handling (#331)
• remove Elasticsearch security warnings (#317)
• fix topics for message publishing (#338)
• improve data failure handling / message

Documentation for 0.5.1 is available at https://docs.wis2box.wis.wmo.int/en/0.5.1

All contributions are always welcome. Feedback is welcome via GitHub issues or discussions.

The wis2box team

https://docs.wis2box.wis.wmo.int