implement publication workflow to codes.wmo.int #6

tomkralidis · 2024-02-01T15:10:56Z

update tables (@tomkralidis)
- add source column of a URL for values that are defined external to WMO
investigate workflow to convert CSVs to RDF/TTL (@antje-s @josusky)
update specification and examples to point to new/updated codelist values (@tomkralidis)

Examples:

josusky · 2024-02-05T12:45:13Z

I haven't fully figured out how the upload of TTL files to (ci).codes.wmo.int works, the script from "https://github.com/wmo-im/wmds" expects some JSON as input. On the other hand, I have figured out how to adjust makeWMDREntities.py to convert "our" CSVs. Problem is what parameters should we set in the TTL files. They are generated using a textual template that sets "identity", "notation", "label" and "description". So far I have used "Name" for all of them except "description", where I have used "Description" column. But I have no idea where to put our "Source" column.

josusky · 2024-02-05T12:47:47Z

Oh, only now have I noticed that @antje-s has created branch csv2registry, I am going to check what is new there.

josusky · 2024-02-05T12:54:49Z

OK, so she went further than me. Her version of the script puts "Source" into "skos:related" and she made custom versions of other WMDS scripts too.

tomkralidis · 2024-02-05T13:19:07Z

Great work here! Note that if the source column is not empty, this means that the codelist value is from an upstream controlled vocabulary. In which case we should use owl:sameAs instead.

tomkralidis · 2024-02-07T14:28:24Z

Notes from 2024-02-07 meeting with @antje-s / @josusky

TTL generation (@antje-s)
- CSV sources "Source" column: empty means local definition, else upstream/remote
- a non-empty "Source" column should produce a an owl:sameAs and skos:exactMatch or the URL in the concept template
- generated TTLs should not go into main branch, but a branch called publication
Upload to WMO Codes Registry (@josusky)
- determine interaction / payload specifics from registry API
@tomkralidis will review implementation thereafter

Next meeting is on 2024-02-09.

antje-s · 2024-02-09T13:32:15Z

Updated after our meeting today:

only add entry for source column if not empty or None
use "wis" as directory name in branch publication for TTLs
delete branch publication if exists, use always an new branch publication
deleted publish script files from branch "csv2registry"

I think we need for PR from csv2registry branch:
1).github/workflows/generateTTLandCommit.yml
2)scripts/makeWCMP2Entities.py
3)codelists/wcmp2-tables.csv

Many thanks for all the support, I have learned a lot again
@josusky: if you want me to do anything further, please let me know

tomkralidis · 2024-02-09T13:48:05Z

Next steps:

consolidated PR with scripts to 1. generate TTLs and 2. publish to codes.wmo.int
clarify publication workflow
- main branch: CI/CD to ci.codes.wmo.int
- release: CI/CD or manual process? Needs discussion with @amilan17

The script has been simplified, and a bit of documentation was added. It now uploads files from a directory, so it is not necessary to construct a JSON "uploads" dictionary.

* Create generateTTLandCommit.yml * Create makeWCMP2Entities.py toDo - adding column for soure * Create wcmp2-tables.csv * Update wcmp2-tables.csv * Update wcmp2-tables.csv * Create prodRegister * Create testRegister * Create check_urls.py * Create test-gen.py * Create __init__.py * Create generator.py * Create registry.py toDo - publish - userid - publish - post_file * Create generate-readme.py * Update makeWCMP2Entities.py * Update makeWCMP2Entities.py * Update check_urls.py * Update makeWCMP2Entities.py * Update makeWCMP2Entities.py * Update makeWCMP2Entities.py * Update contact-role.csv * Update global-service-type.csv * Update link-type.csv * Update resource-type.csv * Github Action Commit * Update makeWCMP2Entities.py skos:related for our Source links? * Github Action Commit * Update makeWCMP2Entities.py * Github Action Commit * first try to other branch * Update generateTTLandCommit.yml * Update makeWCMP2Entities.py * Update generateTTLandCommit.yml * Update makeWCMP2Entities.py * Update generateTTLandCommit.yml * Update generateTTLandCommit.yml * Update generateTTLandCommit.yml * Update generateTTLandCommit.yml * Update generateTTLandCommit.yml * Update generateTTLandCommit.yml * Update generateTTLandCommit.yml * Delete ttls_wcmp2 directory * Update test-gen.py read registry value from file * Create uploadChanges.py * Update generateTTLandCommit.yml ignore diffs to main in publiacation branch * Update registry.py wis2 - wis in registry-url * Delete scripts/test-gen.py * Delete scripts/registry_deployment directory * Delete scripts/check_urls.py * Delete scripts/generate-readme.py * Update generateTTLandCommit.yml * Update makeWCMP2Entities.py * Delete scripts/uploadChanges.py * Update makeWCMP2Entities.py * Update makeWCMP2Entities.py * Delete prodRegister * Delete testRegister * Add dry-run mode * Add dry run mode to uploadChanges.py * Refactor uploadChanges.py script #6 The script has been simplified, and a bit of documentation was added. It now uploads files from a directory, so it is not necessary to construct a JSON "uploads" dictionary. --------- Co-authored-by: antje-s <[email protected]> Co-authored-by: Jan Osusky <[email protected]>

tomkralidis · 2024-02-09T15:46:08Z

Implemented in #8. Excellent work @antje-s and @josusky ! Thank you for landing this so quickly.

amilan17 · 2024-02-09T15:52:32Z

The plan is to publish everything to the public codes.wmo.int registry with a status of experimental at the beginning of March. This will ensure that all of the links are active when members are reviewing the documents.

tomkralidis · 2024-02-11T20:17:26Z

PR update in #10 (see Notes section for required clarifications). Request for clarifications will be sent via email.

update codelist generator/publisher implementation (#6)

tomkralidis · 2024-02-12T12:02:00Z

Copying notes here for next steps:

GitHub Secrets for WMO Codes Registry publishing from GitHub Actions have not been set yet and need clarification/discussion
we need clarification on what/how to publish the "root" wis register. Doing so manually or automatically yields an "invalid" status

The codelists2ttl.py no longer generates wis.ttl - we are not going to change this top-level register once the WIS specification is published, so it can remain a manual process. Fixed checking of sub-register existence and a few other minor issues in upload_changes.py.

* fix status handling and a few typos * Fix codelists2ttl.py and upload_changes.py #6 The codelists2ttl.py no longer generates wis.ttl - we are not going to change this top-level register once the WIS specification is published, so it can remain a manual process. Fixed checking of sub-register existence and a few other minor issues in upload_changes.py. * fix merging issue * Update scripts/README.md #6

amilan17 · 2024-03-06T15:44:22Z

@tomkralidis

@david-i-berry suggests using something different than owl:sameAs when the code and definition are copied from another source.
two ideas:
https://www.w3.org/TR/rdf-schema/#ch_isdefinedby
or
https://www.dublincore.org/specifications/dublin-core/dcmi-terms/elements11/source/

tomkralidis · 2024-03-11T13:56:30Z

2024-03-11

implement category to enable visibility of codelists on codes.wmo.int homepage (UI)
CI to update ci.codes.wmo.int (status=experimental)
WMO to update codes.wmo.int (status=approved)

Actions:

@amilan17 @antje-s @josusky to meet with UK Met Office to discuss category lifecycle
@josusky @antje-s to change owl:sameAs to rdfs:isDefinedBy (https://github.com/wmo-im/wcmp2-codelists/blob/main/scripts/codelists2ttl.py)
@amilan17 should then be able to publish codes.wmo.int proper

tomkralidis · 2024-03-13T17:06:57Z

Update: this should be ready for publication to codes.wmo.int proper. @amilan17 let us know if this is successful, at which point we can close this issue.

david-i-berry · 2024-03-28T11:43:29Z

Note that the authenticate part of upload_changes needs to be modified to use port 443 at:

wcmp2-codelists/scripts/upload_changes.py

Line 46 in 6b9eab5

url = f'{base_url}/system/security/apilogin'

i.e.

url = f'{base_url}:443/system/security/apilogin'

tomkralidis · 2024-03-28T12:11:50Z

Thanks @david-i-berry. The result is the same whether 443 is explicitly set or not.

josusky · 2024-03-28T13:44:41Z

I agree with @tomkralidis, port 443 is default for HTTPS. Only if a non-standard port, let say 8181 is used, it needs to be explicitly set.

amilan17 · 2024-03-28T15:13:28Z

I was able to upload AFTER removing the global-service-type TTLs. For some reason, the software does not accept those, but the TTLS are identical to the accepted TTLS accept for the content referring to global-service-type... ?!

josusky · 2024-03-28T15:26:56Z

Hi Anna, I am checking the issue with global-service type sub-register, the server is responding with with an internal error when I try to GET (read) the resource: $ curl https://ci.codes.wmo.int/wis/global-service-type/ Cannot invoke "com.epimorphics.registry.core.Description.getRoot()" because the return value of "com.epimorphics.registry.store.StoreAPI.getCurrentVersion (String)" is null I will try to remove this sub-register and retry. JanO

…

On Thu, Mar 28, 2024 at 4:13 PM Anna Milan ***@***.***> wrote: I was able to upload AFTER removing the global-service-type TTLs. For some reason, the software does not accept those, but the TTLS are identical to the accepted TTLS accept for the content referring to global-service-type... ?! — Reply to this email directly, view it on GitHub <#6 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFS4LF6LXUEVD4SDF5SDHW3Y2QQR3AVCNFSM6AAAAABCVCN44OVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMRVGQ3DOOBWGE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

amilan17 · 2024-03-28T15:30:58Z

Yes, I get the same error when trying to upload

ValueError: Cannot upload to https://ci.codes.wmo.int/wis/global-service-type: 500 : Cannot invoke "com.epimorphics.registry.core.Description.getRoot()" because the return value of "com.epimorphics.registry.store.StoreAPI.getCurrentVersion(String)" is null

josusky · 2024-03-28T15:50:42Z

I am afraid that the removal of sub-register did not help, I am still getting the internal server error.

…

On Thu, Mar 28, 2024 at 4:31 PM Anna Milan ***@***.***> wrote: Yes, I get the same error when trying to upload ValueError: Cannot upload to https://ci.codes.wmo.int/wis/global-service-type: 500 : Cannot invoke "com.epimorphics.registry.core.Description.getRoot()" because the return value of "com.epimorphics.registry.store.StoreAPI.getCurrentVersion(String)" is null — Reply to this email directly, view it on GitHub <#6 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFS4LF76Q5O6JJJOZZDSYE3Y2QSTRAVCNFSM6AAAAABCVCN44OVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMRVGUYDMMBVGA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

josusky · 2024-04-02T06:46:00Z

The error message changed a bit over the weekend, now it reveals less of the internal information, but it is still an "Internal Server Error". I don't think we can fix this from our side. @amilan17 could you please contact the server administrator and request help?

BTW, I have created a register "wis2" (https://ci.codes.wmo.int/wis2) and successfully uploaded all codes to it.

amilan17 · 2024-04-03T13:33:02Z

@josusky how did you manage to successfully upload all codes? The error didn't prevent that on the wis2 registry?

josusky · 2024-04-03T14:37:37Z

It seems that the problem somehow related to the wis register.

amilan17 · 2024-04-05T07:42:43Z

Codelists are now published to https://codes.wmo.int/ with the status of Experimental.

tomkralidis assigned tomkralidis, amilan17, josusky and antje-s Feb 1, 2024

tomkralidis added the enhancement New feature or request label Feb 1, 2024

tomkralidis added a commit that referenced this issue Feb 1, 2024

add source column to codelist values (#6)

03af6eb

tomkralidis mentioned this issue Feb 1, 2024

add source column to codelist values (#6) #7

Merged

tomkralidis added a commit that referenced this issue Feb 1, 2024

add source column to codelist values (#6) (#7)

7b72341

josusky added a commit that referenced this issue Feb 9, 2024

Refactor uploadChanges.py script #6

8422e99

The script has been simplified, and a bit of documentation was added. It now uploads files from a directory, so it is not necessary to construct a JSON "uploads" dictionary.

josusky mentioned this issue Feb 9, 2024

Csv2ttl #9

Closed

josusky added a commit that referenced this issue Feb 9, 2024

Refactor uploadChanges.py script #6

ee6243e

The script has been simplified, and a bit of documentation was added. It now uploads files from a directory, so it is not necessary to construct a JSON "uploads" dictionary.

josusky added a commit that referenced this issue Feb 9, 2024

Refactor uploadChanges.py script #6

f72dcff

The script has been simplified, and a bit of documentation was added. It now uploads files from a directory, so it is not necessary to construct a JSON "uploads" dictionary.

tomkralidis added a commit that referenced this issue Feb 11, 2024

update codelist generator/publisher implementation (#6)

d6cef3d

tomkralidis added a commit that referenced this issue Feb 11, 2024

update codelist generator/publisher implementation (#6)

d6663ff

tomkralidis added a commit that referenced this issue Feb 11, 2024

update codelist generator/publisher implementation (#6)

3c6fabb

tomkralidis added a commit that referenced this issue Feb 11, 2024

update codelist generator/publisher implementation (#6)

cab7b95

tomkralidis mentioned this issue Feb 11, 2024

Determine publication approach in Codes Registry wmo-im/wis2-topic-hierarchy#26

Closed

josusky added a commit that referenced this issue Feb 12, 2024

Merge pull request #10 from wmo-im/tk-script-updates

1c98e8a

update codelist generator/publisher implementation (#6)

josusky added a commit that referenced this issue Feb 12, 2024

Update scripts/README.md #6

a1d582b

josusky added a commit that referenced this issue Feb 12, 2024

Fix stupid typo in scripts/README.md #6

383ccbf

josusky added a commit that referenced this issue Feb 12, 2024

Fix stupid typo in scripts/README.md #6

2f8507c

tomkralidis closed this as completed Apr 5, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

implement publication workflow to codes.wmo.int #6

implement publication workflow to codes.wmo.int #6

tomkralidis commented Feb 1, 2024 •

edited

Loading

josusky commented Feb 5, 2024

josusky commented Feb 5, 2024

josusky commented Feb 5, 2024

tomkralidis commented Feb 5, 2024

tomkralidis commented Feb 7, 2024

antje-s commented Feb 9, 2024

tomkralidis commented Feb 9, 2024

tomkralidis commented Feb 9, 2024

amilan17 commented Feb 9, 2024

tomkralidis commented Feb 11, 2024

tomkralidis commented Feb 12, 2024

amilan17 commented Mar 6, 2024

tomkralidis commented Mar 11, 2024

tomkralidis commented Mar 13, 2024

david-i-berry commented Mar 28, 2024 •

edited

Loading

tomkralidis commented Mar 28, 2024

josusky commented Mar 28, 2024

amilan17 commented Mar 28, 2024

josusky commented Mar 28, 2024 via email

amilan17 commented Mar 28, 2024

josusky commented Mar 28, 2024 via email

josusky commented Apr 2, 2024

amilan17 commented Apr 3, 2024

josusky commented Apr 3, 2024

amilan17 commented Apr 5, 2024

implement publication workflow to codes.wmo.int #6

implement publication workflow to codes.wmo.int #6

Comments

tomkralidis commented Feb 1, 2024 • edited Loading

josusky commented Feb 5, 2024

josusky commented Feb 5, 2024

josusky commented Feb 5, 2024

tomkralidis commented Feb 5, 2024

tomkralidis commented Feb 7, 2024

antje-s commented Feb 9, 2024

tomkralidis commented Feb 9, 2024

tomkralidis commented Feb 9, 2024

amilan17 commented Feb 9, 2024

tomkralidis commented Feb 11, 2024

tomkralidis commented Feb 12, 2024

amilan17 commented Mar 6, 2024

tomkralidis commented Mar 11, 2024

tomkralidis commented Mar 13, 2024

david-i-berry commented Mar 28, 2024 • edited Loading

tomkralidis commented Mar 28, 2024

josusky commented Mar 28, 2024

amilan17 commented Mar 28, 2024

josusky commented Mar 28, 2024 via email

amilan17 commented Mar 28, 2024

josusky commented Mar 28, 2024 via email

josusky commented Apr 2, 2024

amilan17 commented Apr 3, 2024

josusky commented Apr 3, 2024

amilan17 commented Apr 5, 2024

tomkralidis commented Feb 1, 2024 •

edited

Loading

david-i-berry commented Mar 28, 2024 •

edited

Loading