Nyaml2nxdl migration #1303
Open
Nyaml2nxdl migration #1303
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…#50) * Added a pip-compile locked requirements for devs * Reverted nomad-lab install back * Fixed missing astroid version * cleanup + ease on dev reqs * Added a relaxed requirements.txt for installation with Nomad * Unrelaxed pyproject.toml dev reqs and reverted xarray to older version * Allowed python3.7 and fixed dask[array] version * Merge adapt-nexus-to-metainfo-changes and CI updates * cleanup * Brought in requirements.txt * draft refactoring and reorganizing of nomad-parser-nexus, decide first how tests should finally before structured, then the tests and absolute paths in the ipynbs and code sections should be changed * renaming tools; adding and fixing unit tests * fix some linting issues * addig reference data for testing nexus checker * fixing linting issues; adding test reference file * bringing Example to front * tested ipynb notebooks, added jupyterlab_h5web to the environment * improved descriptions, and placed jupyterlab in pyproject toml, tested compiling a dev-requirements locally successfully * updating readme.md as a result of testing * Added nexus definitions to the module package * fixing README files and the ipynb for APM * use dataconverter alias from command line * harmonising EM and MPES notebooks * Updated YAML file * Updated jupyter notebook for ellipsometry + DAT and YAML file * Delete ELLIPSOMETRY.NeXus.READER.EXAMPLE.02.ipynb * Delete test.yaml * fix data file locations Co-authored-by: Adam Fekete <[email protected]> Co-authored-by: Adam Fekete <[email protected]> Co-authored-by: kuehbachm <[email protected]> Co-authored-by: sanbrock <[email protected]> Co-authored-by: sanbrock <[email protected]> Co-authored-by: Carola Emminger <[email protected]> Co-authored-by: cmmngr <[email protected]>
…st_fileline_error, test_symnols, test_attributes. test_extends
* Yaml comments are perfectly stored in comment_chain of CommentCollecotor class. * readging comment from yaml to xml (nexus field). * Apply comment for 'nexus symbols', 'nexus attributes' and modify code part for root level doc. * Apply comment for 'nexus link', 'nexus choice', 'nexus dimension' and 'nexus group'. * nxdl to yaml comment: root level comment is preserved * Collecting comments in both directions (nyaml to nxdl and nxdl to yaml) and adding an extra option --check-consistency for generating the same file version from a file given as input. * nyaml2nxdl yaml_comment test. * Tests are passed * Fix pylint, pydoc and mypy * All tests and pylint, mypy are passed * Fixing conversation in pull request. * Response to requested changes. --------- Co-authored-by: RubelMozumder <[email protected]>
* Renames to pynxtools geändert: .github/workflows/pylint.yml geändert: .gitmodules modified: .github/workflows/pylint.yml * Properly adds definitions * Rename left over nexusutils reference from rebase * Merges xps reader.py
* Renames to pynxtools geändert: .github/workflows/pylint.yml geändert: .gitmodules modified: .github/workflows/pylint.yml * Properly adds definitions * Rename left over nexusutils reference from rebase * Merges xps reader.py
* Changes for removing old errors related ':' in yaml and xml file. * To store nxdl file into yaml as comment and generate or retrieve nxdl from provided yaml. Wokring perfectly. * Bug: last line of yaml file was not counted * Passing all the tests, pylint, mypy and pydocstyle. * Fixing copy-right doc, new_line_char at the end of 'deprecated key', version from of nexus. * pylint.yml checked. --------- Co-authored-by: RubelMozumder <[email protected]>
* Changes for removing old errors related ':' in yaml and xml file. * To store nxdl file into yaml as comment and generate or retrieve nxdl from provided yaml. Wokring perfectly. * Bug: last line of yaml file was not counted * Passing all the tests, pylint, mypy and pydocstyle. * Fixing copy-right doc, new_line_char at the end of 'deprecated key', version from of nexus. * pylint.yml checked. --------- Co-authored-by: RubelMozumder <[email protected]>
Co-authored-by: RubelMozumder <[email protected]>
Co-authored-by: RubelMozumder <[email protected]>
* temporary changes. * Including some changes for optionality error. * Passing test successfully. Cleaning up some tests. * Removing autogenerated 'optional' attribute from nxdl.xml. * Removing autogenerated 'optional' attribute from nxdl.xml. * Removing autogenerated 'optional' attribute from nxdl.xml. * Removing autogenerated 'optional' attribute from nxdl.xml. * Comments or corrections are resolved. * Corrections from review.
* temporary changes. * Including some changes for optionality error. * Passing test successfully. Cleaning up some tests. * Removing autogenerated 'optional' attribute from nxdl.xml. * Removing autogenerated 'optional' attribute from nxdl.xml. * Removing autogenerated 'optional' attribute from nxdl.xml. * Removing autogenerated 'optional' attribute from nxdl.xml. * Comments or corrections are resolved. * Corrections from review.
1 task
PeterC-DLS
reviewed
Nov 3, 2023
prjemian
reviewed
Nov 7, 2023
This is not in the original statement of this PR except by mention of #1282. Can that be separated from this PR? I would like to review documentation changes separate from the proposition of new editing tools. |
|
Still, I do not see any justification to make NeXus definitions become an installable Python package. This is beyond the scope of NeXus definitions and belongs in its own repository. Such a separate Python package could be used here with the other I cannot approve this PR as-is. The intent is wrong. |
|
Hi @prjemian, there is no longer any configuration to make anything a built Python package in this PR. |
Language is cleaned as requested by prjamin in the PR.
prjemian
reviewed
Nov 9, 2023
cleaner definition for optionality as suggested by @prjamin in the PR.
|
I read a conversation somewhere asking about copy-right year ( It was hard coded by |
|
Do not change the starting date for NeXus Copyright lines. There is history in those dates, much of which precedes 2014. The earliest year shown is the year the NXDL was established. Annually, the ending copyright year has been updated by a script, run in response to an issue. The changes are reviewed as a PR, such as this one from 2022. (base) prjemian@arf:~/.../NeXus/definitions$ git grep -i copyright | grep -i "(c)" | grep NeXus
Makefile:# Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
applications/NXarchive.nxdl.xml:# Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
applications/NXarpes.nxdl.xml:# Copyright (C) 2012-2022 NeXus International Advisory Committee (NIAC)
applications/NXcanSAS.nxdl.xml:# Copyright (C) 2012-2022 NeXus International Advisory Committee (NIAC)
applications/NXdirecttof.nxdl.xml:# Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
applications/NXfluo.nxdl.xml:# Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
applications/NXindirecttof.nxdl.xml:# Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
applications/NXiqproc.nxdl.xml:# Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
applications/NXlauetof.nxdl.xml:# Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
applications/NXmonopd.nxdl.xml:# Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
applications/NXmx.nxdl.xml: # Copyright (C) 2013-2022 NeXus International Advisory Committee (NIAC)
applications/NXrefscan.nxdl.xml:# Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
applications/NXreftof.nxdl.xml:# Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
applications/NXsas.nxdl.xml:# Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
applications/NXsastof.nxdl.xml:# Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
applications/NXscan.nxdl.xml:# Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
applications/NXspe.nxdl.xml: # Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
applications/NXsqom.nxdl.xml:# Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
applications/NXstxm.nxdl.xml: # Copyright (C) 2014-2022 NeXus International Advisory Committee (NIAC)
applications/NXtas.nxdl.xml:# Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
applications/NXtofnpd.nxdl.xml:# Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
applications/NXtofraw.nxdl.xml:# Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
applications/NXtofsingle.nxdl.xml:# Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
applications/NXtomo.nxdl.xml:# Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
applications/NXtomophase.nxdl.xml:# Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
applications/NXtomoproc.nxdl.xml:# Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
applications/NXxas.nxdl.xml:# Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
applications/NXxasproc.nxdl.xml:# Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
applications/NXxbase.nxdl.xml:# Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
applications/NXxeuler.nxdl.xml:# Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
applications/NXxkappa.nxdl.xml:# Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
applications/NXxlaue.nxdl.xml:# Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
applications/NXxlaueplate.nxdl.xml:# Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
applications/NXxnb.nxdl.xml:# Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
applications/NXxrot.nxdl.xml:# Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
applications/nxdlformat.xsl: # Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
base_classes/NXaperture.nxdl.xml:# Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
base_classes/NXattenuator.nxdl.xml:# Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
base_classes/NXbeam.nxdl.xml:# Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
base_classes/NXbeam_stop.nxdl.xml:# Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
base_classes/NXbending_magnet.nxdl.xml:# Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
base_classes/NXcapillary.nxdl.xml:# Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
base_classes/NXcite.nxdl.xml: # Copyright (C) 2014-2022 NeXus International Advisory Committee (NIAC)
base_classes/NXcollection.nxdl.xml:# Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
base_classes/NXcollimator.nxdl.xml:# Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
base_classes/NXcrystal.nxdl.xml:# Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
base_classes/NXcylindrical_geometry.nxdl.xml:# Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
base_classes/NXdata.nxdl.xml:# Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
base_classes/NXdetector.nxdl.xml:# Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
base_classes/NXdetector_group.nxdl.xml:# Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
base_classes/NXdetector_module.nxdl.xml:# Copyright (C) 2013-2022 NeXus International Advisory Committee (NIAC)
base_classes/NXdisk_chopper.nxdl.xml:# Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
base_classes/NXentry.nxdl.xml:# Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
base_classes/NXenvironment.nxdl.xml:# Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
base_classes/NXevent_data.nxdl.xml:# Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
base_classes/NXfermi_chopper.nxdl.xml:# Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
base_classes/NXfilter.nxdl.xml:# Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
base_classes/NXflipper.nxdl.xml:# Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
base_classes/NXfresnel_zone_plate.nxdl.xml:# Copyright (C) 2014-2022 NeXus International Advisory Committee (NIAC)
base_classes/NXgeometry.nxdl.xml:# Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
base_classes/NXgrating.nxdl.xml:# Copyright (C) 2014-2022 NeXus International Advisory Committee (NIAC)
base_classes/NXguide.nxdl.xml:# Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
base_classes/NXinsertion_device.nxdl.xml:# Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
base_classes/NXinstrument.nxdl.xml:# Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
base_classes/NXlog.nxdl.xml:# Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
base_classes/NXmirror.nxdl.xml:# Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
base_classes/NXmoderator.nxdl.xml:# Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
base_classes/NXmonitor.nxdl.xml:# Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
base_classes/NXmonochromator.nxdl.xml:# Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
base_classes/NXnote.nxdl.xml:# Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
base_classes/NXobject.nxdl.xml:# Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
base_classes/NXoff_geometry.nxdl.xml:# Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
base_classes/NXorientation.nxdl.xml:# Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
base_classes/NXparameters.nxdl.xml:# Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
base_classes/NXpdb.nxdl.xml:# Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
base_classes/NXpinhole.nxdl.xml:# Copyright (C) 2014-2022 NeXus International Advisory Committee (NIAC)
base_classes/NXpolarizer.nxdl.xml:# Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
base_classes/NXpositioner.nxdl.xml:# Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
base_classes/NXprocess.nxdl.xml:# Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
base_classes/NXroot.nxdl.xml:# Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
base_classes/NXsample.nxdl.xml:# Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
base_classes/NXsample_component.nxdl.xml:# Copyright (C) 2016-2022 NeXus International Advisory Committee (NIAC)
base_classes/NXsensor.nxdl.xml:# Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
base_classes/NXshape.nxdl.xml:# Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
base_classes/NXslit.nxdl.xml:# Copyright (C) 2014-2022 NeXus International Advisory Committee (NIAC)
base_classes/NXsource.nxdl.xml:# Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
base_classes/NXsubentry.nxdl.xml:# Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
base_classes/NXtransformations.nxdl.xml:# Copyright (C) 2014-2022 NeXus International Advisory Committee (NIAC)
base_classes/NXtranslation.nxdl.xml:# Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
base_classes/NXuser.nxdl.xml:# Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
base_classes/NXvelocity_selector.nxdl.xml:# Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
base_classes/NXxraylens.nxdl.xml:# Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
base_classes/nxdlformat.xsl: # Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
contributed_definitions/NXcontainer.nxdl.xml:# Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
contributed_definitions/NXcsg.nxdl.xml:# Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
contributed_definitions/NXcxi_ptycho.nxdl.xml: # Copyright (C) 2014-2022 NeXus International Advisory Committee (NIAC)
contributed_definitions/NXelectrostatic_kicker.nxdl.xml: # Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
contributed_definitions/NXmagnetic_kicker.nxdl.xml: # Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
contributed_definitions/NXquadric.nxdl.xml:# Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
contributed_definitions/NXquadrupole_magnet.nxdl.xml: # Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
contributed_definitions/NXregion.nxdl.xml:# Copyright (C) 2013-2021 NeXus International Advisory Committee (NIAC)
contributed_definitions/NXseparator.nxdl.xml: # Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
contributed_definitions/NXsolenoid_magnet.nxdl.xml: # Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
contributed_definitions/NXsolid_geometry.nxdl.xml:# Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
contributed_definitions/NXspin_rotator.nxdl.xml: # Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
contributed_definitions/NXxpcs.nxdl.xml:# Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
contributed_definitions/nxdlformat.xsl: # Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
manual/source/copyright.rst:Copyright (C) 1996-2022 NeXus International Advisory Committee (NIAC)
manual/source/examples/NX__template__.nxdl.xml:# Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
manual/source/examples/NXwoni.nxdl.xml:# Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
nxdl.xsd: Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
nxdlTypes.xsd: Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
utils/create_release_notes.py:# Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC)
utils/update_copyright_date.py:# Copyright (C) 2008-2022 NeXus International Advisory Committee (NIAC) |
|
Please finish the work for the stated goals on this PR or withdraw it. It cannot be reviewed until the authors have finished with their intended changes. After review, no changes should be made that do not address the review comments. All reviewer comments should be addressed. It is the reviewer's responsibility to close a review conversation once the reviewer deems the conversation is complete. PR authors should not close these conversations. Still, this reviewer question remains unanswered:
|
The initial aim of this PR is summarised in its description: merge the suggestions from 1272 and 1282 into one PR, because both were dependent on a code from pynxtools which was requested to be moved into definitions repo in nexusformat rather then to use it as dependency. Hence, keeping the 2 PRs separate would have ment to make requested modifications of separate reviews on the same code parallel in two different PRs, in two different branches. To simplify the management of the reviews, it was suggested to close the original PRs and rather request a new one: This PR has been composed from PR #1272, and #1282 with also conserving micro-crediting along the development history. #1272 aimed enhancements for the documentation by providing an extra link if applicable to the first vocabulary item referenced (e.g. /NXxkappa/entry/sample-group should reference /NXxbase/entry/sample-group and not only provide a link to /NXsample) #1282 aimed to add a convenient function to the Makefile for converting definitions between nxdl.xml and nyaml and allow simpler editing of the definitions by humans in yaml format, while keeping the nxdl.xml files as the single source of truth. |
@prjemian,The files mentioned here are not part of this PR and no change has come from this PR. The copyright years you mentioned in those files are originally the main branch. |
|
Right. Those changes are PR #1322 |
|
Feedback from Telco:
This will greatly simplify this PR and it lets us keep the code more organized. Thanks for the work! |
|
as suggested the PR are split to two again:
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR has been composed from PR #1272, and #1282 with also conserving micro-crediting along the development history.
adding links to first references of the vocabulary items #1272 aimed enhancements for the documentation by providing an extra link if applicable to the first vocabulary item referenced (e.g. /NXxkappa/entry/sample-group should reference /NXxbase/entry/sample-group and not only provide a link to /NXsample)
Nyaml #1282 aimed to add a convenient function to the Makefile for converting definitions between nxdl.xml and nyaml and allow simpler editing of the definitions by humans in yaml format, while keeping the nxdl.xml files as the single source of truth.
As comments describe, the 2023 Code Camp participants suggested to
Since there was an intersection of the codes required for both PRs, it was easier to pull the code and its changes together into this new PR.