update codelist generator/publisher implementation (#6)

.github/workflows/README.md

@@ -0,0 +1,11 @@
+# GitHub Actions
+
+The following GitHub Actions are in place for this repository:
+
+- on Pull Request: test the generation of WCMP2 Codelists to TTL files
+- on Commit/Push to `main` branch: generate WCMP2 Codelists to TTL files, and push to `publication` branch
+- on Commit/Push to `publication` branch: publish the TTL files to the WMO Codes Registry testing environment
+
+Edit `codelists/*.csv` files -> GitHub Pull Request (test generation) -> Merge Pull Request (generate and commit to `publication` branch) -> Publish to WMO Codes Registry testing environment
+
+Publication to the WMO Codes Registry operational environment is executed as a manual step.

.github/workflows/generate-and-commit-ttl.yml

@@ -0,0 +1,41 @@
+name: Generate WCMP2 Codelists as TTL files and commit
+
+on:
+  push:
+    branches:
+      - main
+    paths:
+      - '**.yml'
+      - 'codelists/**.csv'
+
+jobs:
+  main:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ['3.10']
+    steps:
+      - uses: actions/checkout@v2
+      - name: Set up Python ${{ matrix.python-version }}
+        uses: actions/setup-python@v2
+        with:
+          python-version: ${{ matrix.python-version }}
+      - name: Generate TTL files
+        run: |
+          python3 scripts/codelists2ttl.py
+      - name: checkout publication branch
+        uses: actions/checkout@v3
+        with:
+          ref: publication
+      - name: update publication branch and publish
+        run: |
+          mkdir /tmp/wis
+          mv wis/ tmp/
+          git checkout publication
+          git config --global user.email "[email protected]"
+          git config --global user.name "Tom Kralidis"
+          rm -rf wis/*
+          cp -rpf /tmp/wis/* .
+          git add .
+          git commit -am "update WMCP2 Codelists TTL files"
+          git push

.github/workflows/generate-ttl.yml

@@ -0,0 +1,23 @@
+name: Generate WCMP2 Codelists as TTL files
+
+on:
+  pull_request:
+    paths:
+      - '**.yml'
+      - 'codelists/**.csv'
+
+jobs:
+  main:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ['3.10']
+    steps:
+      - uses: actions/checkout@v2
+      - name: Set up Python ${{ matrix.python-version }}
+        uses: actions/setup-python@v2
+        with:
+          python-version: ${{ matrix.python-version }}
+      - name: Generate TTL files
+        run: |
+          python3 scripts/codelists2ttl.py

.github/workflows/publish-to-wmo-codes.yml

@@ -0,0 +1,41 @@
+name: Publish TTL files to WMO Codes Registry testing environment
+
+env:
+  WMO_CODES_TEST_USER_ID: ${{ secrets.WMO_CODES_TEST_USER_ID }}
+  WMO_CODES_TEST_API_KEY: ${{ secrets.WMO_CODES_TEST_API_KEY }}
+
+on:
+  push:
+    branches:
+      - publication
+    paths:
+      - 'wis/**.ttl'
+
+jobs:
+  main:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ['3.10']
+    steps:
+      - uses: actions/checkout@v2
+        with:
+          ref: publication 
+      - name: Set up Python ${{ matrix.python-version }}
+        uses: actions/setup-python@v2
+        with:
+          python-version: ${{ matrix.python-version }}
+      - name: Copy TTL files
+        run: |
+          mkdir /tmp/wis
+          cp -rp wis/ /tmp/
+      - uses: actions/checkout@v2
+        with:
+          ref: main
+      - name: Install dependencies
+        run: |
+          python3 -m pip install --upgrade pip
+          pip3 install -r scripts/requirements.txt
+      - name: update publication branch and publish
+        run: |
+          python3 scripts/upload_changes.py https://api.github.com/users/{WMO_CODES_TEST_USER_ID} {WMO_CODES_TEST_API_KEY} test /tmp/wis

codelists.csv

@@ -0,0 +1,5 @@
+Name,Description,Source
+contact-role,Contact role,https://github.com/radiantearth/stac-spec/blob/master/collection-spec/collection-spec.md#provider-object
+global-service-type,Global service type,
+link-type,Link type,
+resource-type,Resource type,

scripts/README.md

@@ -0,0 +1,86 @@
+# WMO Codes Registry management
+
+## Overview
+
+The scripts in this directory are used to manage the WCMP2 codelists publication to
+the WMO Codes Registry.
+
+## The WMO Codes Registry
+
+The [WMO Codes Registry](https://codes.wmo.int) is an authoritative service that
+provides a number of registers defining controlled vocabularies used in various
+WMO standards and systems.
+
+The service provides an API in support of automated workflow to manage codelist
+registers.  The API is available as follows:
+
+- https://ci.codes.wmo.int: testing
+- https://codes.wmo.int: production
+
+API usage requires an account and credentials.  Contact WMO Secretariat to be
+provided access to the WMO Codes Registry API (a GitHub user id is required).
+
+Once you receive access, an API Key is required to manage resources on the registry.
+To create an API Key, once logged into the registry, select _Admin / Create a temporary password (API Key)_,
+and click _Create password_ to generate an API key.
+
+## Mapping from WCMP2 codelists to the WMO Code Registry
+
+The overall setup of WCMP2 codelists publication to the WMO Codes Registry works as follows:
+
+`wis` / CSV filename (without file extension) / CSV row `Name`
+
+where:
+
+- `wis` is the root `reg:Register`
+- each WCMP2 codelist CSV file is a `reg:Register` itself, attached to the `wis` register as a sub-register
+- each row in a WCMP2 codelist CSV file is a `skos:Concept` tied to its sub-register
+
+## Publication workflows
+
+Managing WCMP2 codelists publication to the WMO Codes Registry involves the following steps:
+
+- creating the `wis` register
+- generating TTL files from CSV
+- publishing TTL files to the WMO Codes Registry
+
+### Creating the `wis` register
+
+TODO
+
+### Generating TTLs
+
+To generate TTL files, from the root of the repository, run the following command:
+
+```bash
+python3 scripts/codeslists2ttl.py
+```
+
+This will create all TTL files in a directory called `wis`.
+
+### Publishing TTLs
+
+To generate TTL files, from the root of the repository, run the following command:
+
+```bash
+python3 scripts/upload_changes.py https://api.github.com/users/{user_id} <password> <environment> <output-directory>
+```
+
+where:
+
+- `user_id` is your GitHub userid
+- `password` is the API Key (see the [#overview](Overview) with instructions on how to generate an API Key
+- `environment` is whether to upload change to the testing or production environment
+- `output-directory` is the resulting directly where TTL outputs should published from
+
+Examples:
+
+```bash
+# publish to test environment on https://ci.codes.wmo.int
+python3 scripts/upload_changes.py https://api.github.com/users/tomkralidis API_KEY test wis
+
+# publish to production environment on https://codes.wmo.int
+python3 scripts/upload_changes.py https://api.github.com/users/tomkralidis API_KEY prod wis
+```
+
+This will create/update all resources on the WMO Codes Registry.