various fixes (#119)

* remove reference to username.wis2.training

* use return instead of exit in setup_student_vm.sh

* make GDC exercise wording more generic, don't use local IP used in Fiji

* add explicit login and exit commands to avoid confusion

* updated wording regarding required stations update

documentation/docs/practical-sessions/automating-data-ingestion.en.md

@@ -198,7 +198,7 @@ cp ~/exercise-materials/data-ingest-exercises/wis2box-ftp.env ~/wis2box-1.0b8/
 
     - `FTP_USER`: the username for the ftp-endpoint (to be defined by the user)
     - `FTP_PASS`: the password for the ftp-endpoint (to be defined by the user)
-    - `FTP_HOST`: the hostname of your wis2box-instance (e.g. `username.wis2.training`)
+    - `FTP_HOST`: the hostname or host-IP of your wis2box-instance
     - `WIS2BOX_STORAGE_USERNAME`: the MinIO storage user (e.g. `wis2box`)
     - `WIS2BOX_STORAGE_PASSWORD`: the MinIO storage password (see your `wis2box.env` file)
     - `WIS2BOX_STORAGE_ENDPOINT`: the MinIO storage endpoint, you can leave this set to `http://minio:9000` when running the wis2box-ftp on the same docker network as the MinIO service.
@@ -241,7 +241,7 @@ The file was forwarded by the wis2box-ftp service to the 'wis2box-incoming' buck
 You can also use `ftp` from the command line:
 
 ```bash
-ftp username.wis2.training
+ftp <my-hostname-or-ip>
 ```
 Login using the credentials defined in `wis2box-ftp.env` for the `FTP_USER` and `FTP_PASS` environment variables, and then create a directory and upload a file as follows:
 

documentation/docs/practical-sessions/datasets-with-access-control.md

@@ -24,7 +24,7 @@ Make sure you are connected to the MQTT broker of your wis2box instance using MQ
 
 Ensure you have a web browser open with the wis2box-webapp for your instance by going to `http://<your-host>/wis2box-webapp`.
 
-## create a new dataset with data policy 'recommended'
+## Exercise 1: create a new dataset with data policy 'recommended'
 
 Go to the 'dataset editor' page in the wis2box-webapp and create a new dataset. Use the same centre-id as in the previous practical sessions and use the template='surface-weather-observations/synop'. 
 
@@ -36,39 +36,52 @@ Click 'OK' to proceed.
 
 In the dataset editor, set the data policy to 'recommended' (note that this will update the identifier and replace 'core' with 'reco') and fill all the required fields.
 
-Ensure the dataset is published and the WIS2 Notification Message announcing the new Discovery Metadata record is published.
+Submit the dataset and check that the new dataset is created in the wis2box-webapp.
 
-Add the following stations to your "recommended" dataset to ensure the test data can be ingested and published:
+Check MQTT-explorer to see that you receive the WIS2 Notification Message announcing the new Discovery Metadata record on the topic `origin/a/wis2/<your-centre-id>/metadata`.	
 
-- 0-20000-0-60351
-- 0-20000-0-60355
-- 0-20000-0-60360
-
-## add an access token to the dataset
+## Exercise 2: add an access token to the dataset
 
 Login to the wis2box-management container,
 
 ```bash
-docker exec -it wis2box-management bash
+cd ~/wis2box-1.0b8
+python3 wis2box-ctl.py login
 ```
 
 From command line inside the container you can secure a dataset using the `wis2box auth add-token` command, using the flag `-mdi` to specify the metadata-identifier of the dataset and the access token as an argument.
 
 For example, to add the access token `S3cr3tT0k3n` to the dataset with metadata-identifier `urn:wmo:md:not-my-centre:core.surface-based-observations.synop`:	
 
 ```bash
-wis2box auth add-token -mdi urn:wmo:md:not-my-centre:core.surface-based-observations.synop S3cr3tT0k3n
+wis2box auth add-token -mdi urn:wmo:md:not-my-centre:reco.surface-based-observations.synop S3cr3tT0k3n
+```
+
+Exit the wis2box-management container:
+
+```bash
+exit
 ```
 
-## publish some data to the dataset
+## Exercise 3: publish some data to the dataset
 
 Copy the file `exercise-materials/access-control-exercises/aws-example2.csv` to the directory defined by `WIS2BOX_HOST_DATADIR` in your `wis2box.env`:
 
 ```bash
 cp ~/exercise-materials/access-control-exercises/aws-example2.csv ~/wis2box-data
 ```
 
-Then login to the **wis2box-management** container:
+Then use WinSCP or a command line editor to edit the file `aws-example2.csv` and update the WIGOS-station-identifiers in the input-data to match the stations you have in your wis2box instance. 
+
+Next, go to the station-editor in the wis2box-webapp. For each station you used in `aws-example2.csv`, update the 'topic' field to match the 'topic' of the dataset you created in the previous exercise.
+
+This station will now be associated to 2 topics, one for the 'core' dataset and one for the 'recommended' dataset:
+
+<img alt="edit-stations-add-topics" src="../../assets/img/edit-stations-add-topics.png" width="600">
+
+You will need to use your token for `collections/stations` to save the updated station data.
+
+Next, login to the wis2box-management container:
 
 ```bash
 cd ~/wis2box-1.0b8
@@ -78,16 +91,16 @@ python3 wis2box-ctl.py login
 From the wis2box command line we can ingest the sample data file `aws-example2.csv` into a specific dataset as follows:
 
 ```bash
-wis2box data ingest -p /data/wis2box/aws-example2.csv --metadata-id urn:wmo:md:not-my-centre:core.surface-based-observations.synop
+wis2box data ingest -p /data/wis2box/aws-example2.csv --metadata-id urn:wmo:md:not-my-centre:reco.surface-based-observations.synop
 ```
 
-Make sure to provide the correct metadata-identifier for your dataset and check that you receive WIS2 data-notifications in MQTT Explorer.
+Make sure to provide the correct metadata-identifier for your dataset and **check that you receive WIS2 data-notifications in MQTT Explorer**, on the topic `origin/a/wis2/<your-centre-id>/data/recommended/surface-based-observations/synop`.
 
 Check the canonical link in the WIS2 Notification Message and copy/paste the link to the browser to try and download the data.
 
 You should see a 403 Forbidden error.
 
-## add the access token to HTTP headers to access the dataset
+## Exercise 5: add the access token to HTTP headers to access the dataset
 
 In order to demonstrate that the access token is required to access the dataset we will reproduce the error you saw in the browser using the command line function `wget`.
 
@@ -107,10 +120,6 @@ wget --header="Authorization: Bearer S3cr3tT0k3n" <canonical-link>
 
 Now the data should be downloaded successfully.
 
-## clean up
-
-Delete the dataset using the dataset editor.
-
 ## Conclusion
 
 !!! success "Congratulations!"

documentation/docs/practical-sessions/discovering-datasets-from-wis2-gdc.md

@@ -8,7 +8,7 @@ title: Discovering datasets from the WIS2 Global Discovery Catalogue
 
     By the end of this practical session, you will be able to:
 
-    - use pywiscat to discover datasets from the Global Discovery Catalogue
+    - use pywiscat to discover datasets from the Global Discovery Catalogue (GDC)
 
 ## Introduction
 
@@ -20,7 +20,8 @@ At the moment, the following GDCs are available:
 - China Meteorological Administration: <https://gdc.wis.cma.cn/api>
 - Deutscher Wetterdienst: <https://wis2.dwd.de/gdc>
 
-For this exercise, we will use the WIS2 GDC setup as part of the training, found at <http://192.168.7.153>.
+
+During local training sessions, a local GDC is set up to allow participants to query the GDC for the metadata they published from their wis2box-instances. In this case the trainers will provide the URL to the local GDC.
 
 ## Preparation
 
@@ -39,7 +40,7 @@ pip3 install pywiscat
     If you encounter the following error:
 
     ```
-    WARNING: The script pywiscat is installed in '/home/tkralidis/.local/bin' which is not on PATH.
+    WARNING: The script pywiscat is installed in '/home/username/.local/bin' which is not on PATH.
     Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
     ```
 
@@ -62,7 +63,7 @@ pywiscat --version
 By default, pywiscat connects to Canada's Global Discovery Catalogue.  Let's configure pywiscat to query the training GDC by setting the `PYWISCAT_GDC_URL` environment variable:
 
 ```bash
-export PYWISCAT_GDC_URL=http://192.168.7.153
+export PYWISCAT_GDC_URL=http://<local-gdc-host-or-ip>
 ```
 
 Let's use [pywiscat](https://github.com/wmo-im/pywiscat) to query the GDC setup as part of the training.
@@ -82,7 +83,7 @@ pywiscat search
     How many records are returned from the search?
 
 ??? success "Click to reveal answer"
-    There should be over 40 records returned
+    The number of records depends on the GDC you are querying. When using the local training GDC, you should see that the number of records is equal to the number of datasets that have been ingested into the GDC during the other practical sessions.
 
 Let's try querying the GDC with a keyword:
 

documentation/docs/practical-sessions/downloading-data-from-wis2.en.md

@@ -120,7 +120,11 @@ To add this subscription, use the following command:
 wis2downloader add-subscription --topic cache/a/wis2/de-dwd-gts-to-wis2/#
 ```
 
-Then exit the **wis2downloader** container by typing `exit`.
+Then exit the **wis2downloader** container by typing `exit`:
+
+```bash
+exit
+```
 
 Check the wis2downloader dashboard in Grafana to see the new subscription added. Wait a few minutes and you should see the first downloads starting. Go to he next exercise once you have confirmed that the downloads are starting.
 
@@ -136,12 +140,24 @@ Note that the downloaded data is stored in directories named after the topic the
 
 ## Exercise 5: removing subscriptions from the wis2downloader
 
-Next, log in to the wis2downloader container and remove the subscription you made from the wis2downloader, using the following command:
+Next, log back in to the wis2downloader container:
+
+```bash
+python3 wis2box-ctl.py login wis2downloader
+```
+
+and remove the subscription you made from the wis2downloader, using the following command:
 
 ```bash
 wis2downloader remove-subscription --topic cache/a/wis2/de-dwd-gts-to-wis2/#
 ```
 
+And exit the wis2downloader container by typing `exit`:
+
+```bash
+exit
+```
+
 Check the wis2downloader dashboard in Grafana to see the subscription removed. You should see the downloads stopping.
 
 ## Exercise 6: subscribe to the wis2training-broker and setup a new subscription

documentation/docs/practical-sessions/initializing-wis2box.en.md

@@ -69,7 +69,7 @@ The directory /home/username/wis2box-data has been created.
 
 Next, you will be asked to enter the URL for your wis2box. This is the URL that will be used to access the wis2box web application, API and UI.
 
-Please use `http://<your-hostname>` as the URL. Remember that your hostname is defined by your `username.wis2.training`
+Please use `http://<your-hostname-or-ip>` as the URL.
 
 ```{.copy}
 Please enter the URL of the wis2box:

environment/fake-global-services/README.md → environment/fake-global-broker-and-cache/README.md

@@ -1,24 +1,28 @@
+# fake-global-broker-and-cache
+
+To run a local MQTT broker and a local cache that can be used to simulate the global broker and cache in the WIS2 environment during WIS2 training sessions.
+
+# NOTE
+
+If you want to use a wis2box-instance to serve the purpose of the fake global cache and fake global broker, the following changes are required on the services included by the wis2box-stack:
+
+* Update acf.conf in the "mosquitto" container to allow the wis2box-user to publish on cache/a/wis2/# (restart the container after making changes)
+* create the bucket "cache" in the "wis2box-minio" container and set the access policy to public
+
 # Environment variables
 
+Define these in a local ``.env`` file or in the environment of the VM.
+
 * ``MINIO_HOST`` - MINIO host and port used to act as the fake global cache, e.g. ``instructor.wis2.training:9000`` 
-* ``MINIO_KEY``  - MINIO access key / user ID for writing to the fake cache, e.g. ``wis2box``
-* ``MINIO_SECRET`` - MINIO secret for writing to the fake cache 
-* ``GC_URL`` - Base URL used to access files on the fake cache, e.g.  ``http://instructor.wis2.training``
+* ``MINIO_KEY``  - MINIO access key / user ID for writing to the fake cache, e.g. ``wis2box`` when using wis2box
+* ``MINIO_SECRET`` - MINIO secret for writing to the fake cache, corresponds to WIS2BOX_STORAGE_PASSWORD when using wis2box
+* ``GC_URL`` - Base URL used to access files on the fake cache, e.g.  ``http://instructor.wis2.training:9000``
 * ``GB_HOST`` - Host name of the MQTT broker to republish to, e.g. ``instructor.wis2.training``
-* ``GB_UID`` - User ID used for publishing, e.g. ``wis2box``
-* ``GB_PWD`` - Password used for publishing
-
-You can define these in a local ``.env`` file or in the environment of the VM.
+* ``GB_UID`` - User ID used for publishing, e.g. ``wis2box`` when using wis2box
+* ``GB_PWD`` - Password used for publishing, corresponds to WIS2BOX_BRIDGE_PASSWORD when using wis2box
 
 # Usage
 
 In the VM run:
 
 ``python subscribe_and_republish.py``
-
-# NOTE
-
-If you want to use a wis2box-instance to serve the purpose of the fake global cache and fake global broker, the following changes are required on the services included by the wis2box-stack:
-
-* Update acf.conf in the "mosquitto" container to allow the wis2box-user to publish on cache/a/wis2/# (restart the container after making changes)
-* create the bucket "cache" in the "wis2box-minio" container and set the access policy to public

environment/fake-global-services/subscribe_and_republish.py → environment/fake-global-broker-and-cache/subscribe_and_republish.py

@@ -52,12 +52,6 @@ def on_message(client, userdata, msg):
     else:
         print(f"Not republishing data with topic {msg.topic}")
         return
-    pub.single(
-        topic = msg.topic,
-        payload = json.dumps(payload),
-        hostname = GB_HOST,
-        auth={"username": GB_UID, "password": GB_PWD}
-    )
     # if topic contains data/core or metadata, download to cache
     if 'data/core' in msg.topic:
         print("Downloading core data")
@@ -88,15 +82,15 @@ def on_message(client, userdata, msg):
         print("Downloading")
         http = urllib3.PoolManager()
         response = http.request("GET", canonical)
-        # save to temporaray file
+        # save to tempory file
         temp = tempfile.NamedTemporaryFile(mode="w")
         print(f"Saving to {temp.name}")
         with open(temp.name, "wb") as fh:
             fh.write(response.data)
         print("Data written, now putting to minio")
         # connect to MinIO
         try:
-            minio_client = Minio(MINIO_BUCKET, access_key=MINIO_KEY,
+            minio_client = Minio(MINIO_HOST, access_key=MINIO_KEY,
                            secret_key=MINIO_SECRET, secure=False)
         except Exception as e:
             print("Error connecting to Minio")

environment/fake-global-broker-and-cache/wis2nodes.json

@@ -0,0 +1,29 @@
+[
+    {"host": "192.168.7.122", "port": 1883, "pwd": "everyone", "uid": "everyone", "protocol": "tcp"},
+    {"host": "192.168.7.123", "port": 1883, "pwd": "everyone", "uid": "everyone", "protocol": "tcp"},
+    {"host": "192.168.7.124", "port": 1883, "pwd": "everyone", "uid": "everyone", "protocol": "tcp"},
+    {"host": "192.168.7.125", "port": 1883, "pwd": "everyone", "uid": "everyone", "protocol": "tcp"},
+    {"host": "192.168.7.126", "port": 1883, "pwd": "everyone", "uid": "everyone", "protocol": "tcp"},
+    {"host": "192.168.7.127", "port": 1883, "pwd": "everyone", "uid": "everyone", "protocol": "tcp"},
+    {"host": "192.168.7.128", "port": 1883, "pwd": "everyone", "uid": "everyone", "protocol": "tcp"},
+    {"host": "192.168.7.129", "port": 1883, "pwd": "everyone", "uid": "everyone", "protocol": "tcp"},
+    {"host": "192.168.7.130", "port": 1883, "pwd": "everyone", "uid": "everyone", "protocol": "tcp"},
+    {"host": "192.168.7.131", "port": 1883, "pwd": "everyone", "uid": "everyone", "protocol": "tcp"},
+    {"host": "192.168.7.132", "port": 1883, "pwd": "everyone", "uid": "everyone", "protocol": "tcp"},
+    {"host": "192.168.7.133", "port": 1883, "pwd": "everyone", "uid": "everyone", "protocol": "tcp"},
+    {"host": "192.168.7.134", "port": 1883, "pwd": "everyone", "uid": "everyone", "protocol": "tcp"},
+    {"host": "192.168.7.135", "port": 1883, "pwd": "everyone", "uid": "everyone", "protocol": "tcp"},
+    {"host": "192.168.7.137", "port": 1883, "pwd": "everyone", "uid": "everyone", "protocol": "tcp"},
+    {"host": "192.168.7.146", "port": 1883, "pwd": "everyone", "uid": "everyone", "protocol": "tcp"},
+    {"host": "192.168.7.139", "port": 1883, "pwd": "everyone", "uid": "everyone", "protocol": "tcp"},
+    {"host": "192.168.7.140", "port": 1883, "pwd": "everyone", "uid": "everyone", "protocol": "tcp"},
+    {"host": "192.168.7.141", "port": 1883, "pwd": "everyone", "uid": "everyone", "protocol": "tcp"},
+    {"host": "192.168.7.142", "port": 1883, "pwd": "everyone", "uid": "everyone", "protocol": "tcp"},
+    {"host": "192.168.7.143", "port": 1883, "pwd": "everyone", "uid": "everyone", "protocol": "tcp"},
+    {"host": "192.168.7.144", "port": 1883, "pwd": "everyone", "uid": "everyone", "protocol": "tcp"},
+    {"host": "192.168.7.145", "port": 1883, "pwd": "everyone", "uid": "everyone", "protocol": "tcp"},
+    {"host": "192.168.7.147", "port": 1883, "pwd": "everyone", "uid": "everyone", "protocol": "tcp"},
+    {"host": "192.168.7.148", "port": 1883, "pwd": "everyone", "uid": "everyone", "protocol": "tcp"},
+    {"host": "192.168.7.149", "port": 1883, "pwd": "everyone", "uid": "everyone", "protocol": "tcp"},
+    {"host": "192.168.7.151", "port": 1883, "pwd": "everyone", "uid": "everyone", "protocol": "tcp"}
+]

environment/setup_student_vm.sh

@@ -7,7 +7,7 @@ if [ $# -eq 0 ]
   then
     echo "No arguments supplied"
     echo "Usage: ./setup_student_vm.sh <USERNAME> <HOST_IP>"
-    exit 1
+    return 1
 fi
 
 # first argument of the script is the username
@@ -16,9 +16,9 @@ USERNAME=$1
 HOST_IP=$2
 
 echo "USERNAME=`echo $USERNAME`"
-read -p "Continue? (Y/N): " confirm && [[ $confirm == [yY] || $confirm == [yY][eE][sS] ]] || exit 1
+read -p "Continue? (Y/N): " confirm && [[ $confirm == [yY] || $confirm == [yY][eE][sS] ]] || return 1
 echo "HOST_IP=`echo $HOST_IP`"
-read -p "Continue? (Y/N): " confirm && [[ $confirm == [yY] || $confirm == [yY][eE][sS] ]] || exit 1
+read -p "Continue? (Y/N): " confirm && [[ $confirm == [yY] || $confirm == [yY][eE][sS] ]] || return 1
 
 # execute commands over ssh
 
@@ -37,7 +37,7 @@ ssh -o StrictHostKeyChecking=no wmo_admin@`echo $HOST_IP` "sudo unzip -o /tmp/wi
 ssh -o StrictHostKeyChecking=no wmo_admin@`echo $HOST_IP` "rm -rf /tmp/wis2box-setup-1.0b8.zip"
 
 # copy the latest exercise materials to the student-vm
-ssh -o StrictHostKeyChecking=no wmo_admin@`echo $HOST_IP` "wget https://training.wis2box.wis2dev.io/exercise-materials.zip -O /tmp/exercise-materials.zip"
+ssh -o StrictHostKeyChecking=no wmo_admin@`echo $HOST_IP` "wget https://training.wis2box.wis.wmo.int/exercise-materials.zip -O /tmp/exercise-materials.zip"
 # unzip the exercise-materials.zip
 ssh -o StrictHostKeyChecking=no wmo_admin@`echo $HOST_IP` "sudo unzip -o /tmp/exercise-materials.zip -d /home/`echo $USERNAME`/"
 # remove the exercise-materials.zip