Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
* Refactored cg_primitive base classes for two key aspects: i) make all primitives inherit from new base class NXcg_primitive_set which makes a substantial number of repeated properties of specialized primitive base classes unnecessary, ii) introducing a rigorous mathematical set theoretical approach to describe the number type N, N0, Z, R, R+, R, iii) introduce short hand notation to reduce writing costs for shape/dimensions of arrays: now numpy syntax is used, for scalars (), (1, can be omitted, experience in FAIRmat has shown that in almost all cases the optional doc string to a dimension was almost never used and thus yaml based descriptions can be written more succinct, iv) proof-read docstrings and shortened them, introduced the suggestion to use info: as a keyword which should for now be just appended to the docstring as usual but in the future should be treated as a comment, i.e. only meant for human contextualization but ignored for semantic interpretation of concepts, a key difference that I hope this will bring is to reduce also the length of base classes and appdef as they show up in html pages, specifically for appdefs like NXem many of the conceptual ideas which these docstrings currently contain would be much better placed in a proper documentation instead of the actual class definition in the hope that the overall appearance of classes becomes easier and faster for people to power read; maybe this also works against some criticism that NeXus is considered complex, I have to say with the outsourcing of NXcg_primitive_set I feel it is damn simple now * Refactoring of NXcs base classes * First lot of refactoring EM base classes, the rest tomorrow, NXapm will not be refactored before the APT&M2023 conference in Sept2023, also because feedback from Leoben was positive enough that no immediate changes wrt to appdefs and base classes were required * Summarize the current state of the discussion about coordinate systems with the proposed NXcoordinate_system and NXcoordinate_system_set base classes and best practice recommendations how they should be used including feedback from discussions with Florian Dobner and Tamás Haraszti * Summary of the discussion how to handle conventions and method-specific conventions for EM and methods used in EM * Base classes for implemented examples for pole figures, orientation distribution function, event_data, stage_lab, and ebsd_crystal_structure_model * NXms_ipf base class and dos2unix to correct newlines * Introducing NXem_method base class as a template how to write method-specific deep base classes to describe the terms and taxononmy of method-specific branches to be hooked into appdefs like NXem * Base class inheritance proposal for a now even stronger modularized schema set for electron microscopy research, two tasks remain: i) refactor what is now a method-specific instance rather than an appdef (NXem_ebsd) (mainly to be able to fuse all examples of em-specific data converters including new ebsd examples into one em parser for pynxtools), ii) refactor NXem which is now clearly a specific appdef specializing the NXem_base deep base class, specialization work needs to define which fields and groups from all the possible ones now composable via base classes (inheritance) are required in an appdef NXem for NOMAD OASIS * NXem_ebsd refactored into a base class to use it as a method-specific group inside the NXem application definition, next step: i) refactor NXem_sim, ii) finalize refactoring of NXem appdef (for Nomad oasis) * finished draft of NXem_ebsd, NXem_correlation, and NXem appdef, cleaning the branch * Add proposal for storing mtx cfg, fixed nxtime datatypes * 2d microstructure projection * Inspection how proposed, info, N0, N, R, Z value type abbreviations, and dimensions could be added to nyamlforward * A likely too simplistic but at least working nyaml2nxdl forward mapping to explore further usage of refactored EM base classes. Info keyword has to be a child of doc or the respective text be removed from the standard and put into proposal-specific documentation, how to store what and where so that the schema docstring remain succinct and slim but all these conceptual ideas get not forgotten, typically the would be part of a tech report, i.e. in my opinion all what is under info: sections of a docstring should move to some documentation to tell the story to humans, next test these NXDLs with the NeXus documentation system * Minor fix to handle info keyword spotted while compiling the documentation * Fixes to compile with NeXus documentation test suite and sphinx * Deactivated the annoying clean yaml via make clean for dev purposes * Minor fix in em_base, this completes the appdef/base class work for now on the refactored EM, there are still some spurious info fields now, which should be removed when a decision has been made wrt to how to deal with info: keyword fields in general, next steps: i) make decision on info, ii) test refactored EM proposal with pynxtools em-parser v2, iii) implement backwards * Styling via black * Added yaml2nxdl-forward-converted NXDL files to all refactored base classes and the refactored NXem * Added NXroot header for the em appdef and its base template appdef * Continuing the refactoring of EM and APM plus related base classes for CG and MS based on suggestions from user meetings, discussions with Sandor, represents work with the MPES sprint #83 * Continuing on #83 * Continuing #83, NXcs_* * Continuing #83, ipf, pf, odf * Continuing on #83, support classes for EM * Continued on #83, coordinate system, further base classes supporting EM * Continuing #83, event_data_set and event_data description substantially condensed amongst other points * Added cross-references to base classes for rst, continuing #83 * Aligned old NXem_ebsd_conventions with NXcoordinate_system for #83 * Reviewed method-specific base classes, ebsd, eds, eels, #83 * #83, NXms_recon * #83, composed constraints on the NXem appdef * Consolidated with changes that happened in between on the fairmat branch based on 1016aa0, NXms_recon has still an issue and is therefore deactivated currently, method-specific landing pages need to be updated * Consolidated further with fairmat 15624c0 * Fixing some missing references * Fixed syntax error to compile NXms_recon, docs building also now, reviewing intro pages remains * Consistencies of dimensionality to use NX_POSINT and an enumeration * Recompiled NXDL files using new nyaml module 3d500ced7e4ca57683957c1d61a8d0cb62eccf53, removed, modified by taking the one from fairmat, and synced all files which were binarily different between this feature branch and the fairmat branch specifically commit a15798b of the fairmat branch * Deactivated em-based tests which because of a refactoring of em are not expected to work anymore * Fix improper Latex notation in math environment for polyline, face_list, nanochem * Added recompiled NXidentifier, NXserialized NXDLs which triggered pipeline errors in CatChen gh action * Some round of proof-reading * Fixed test_nxdl_utils to reflect and use refactored locations of refactored NXem * Added feedback from @phyy-nx, @PeterC-DLS, and @prjemian from discussed here https://github.com/nexusformat/definitions PR nexusformat#1271 * Black formatting * Reactivated data type check for e.g. NXem NX_NUMBER * Implementing NX_DATE_TIME suggestion of @sanbrock --------- Co-authored-by: markus.kuehbach <[email protected]> # Conflicts: # applications/NXapm.nxdl.xml # applications/NXem.nxdl.xml # base_classes/NXaberration.nxdl.xml # base_classes/NXcg_alpha_complex.nxdl.xml # base_classes/NXcg_cylinder_set.nxdl.xml # base_classes/NXcg_ellipsoid_set.nxdl.xml # base_classes/NXcg_face_list_data_structure.nxdl.xml # base_classes/NXcg_geodesic_mesh.nxdl.xml # base_classes/NXcg_grid.nxdl.xml # base_classes/NXcg_half_edge_data_structure.nxdl.xml # base_classes/NXcg_hexahedron_set.nxdl.xml # base_classes/NXcg_marching_cubes.nxdl.xml # base_classes/NXcg_parallelogram_set.nxdl.xml # base_classes/NXcg_point_set.nxdl.xml # base_classes/NXcg_polygon_set.nxdl.xml # base_classes/NXcg_polyhedron_set.nxdl.xml # base_classes/NXcg_polyline_set.nxdl.xml # base_classes/NXcg_roi_set.nxdl.xml # base_classes/NXcg_sphere_set.nxdl.xml # base_classes/NXcg_tetrahedron_set.nxdl.xml # base_classes/NXcg_triangle_set.nxdl.xml # base_classes/NXcg_triangulated_surface_mesh.nxdl.xml # base_classes/NXcg_unit_normal_set.nxdl.xml # base_classes/NXchamber.nxdl.xml # base_classes/NXcoordinate_system_set.nxdl.xml # base_classes/NXcorrector_cs.nxdl.xml # base_classes/NXcs_computer.nxdl.xml # base_classes/NXcs_filter_boolean_mask.nxdl.xml # base_classes/NXcs_prng.nxdl.xml # base_classes/NXcs_profiling.nxdl.xml # base_classes/NXcs_profiling_event.nxdl.xml # base_classes/NXdeflector.nxdl.xml # base_classes/NXebeam_column.nxdl.xml # base_classes/NXem_ebsd.nxdl.xml # base_classes/NXevent_data_em.nxdl.xml # base_classes/NXevent_data_em_set.nxdl.xml # base_classes/NXfabrication.nxdl.xml # base_classes/NXibeam_column.nxdl.xml # base_classes/NXimage_set.nxdl.xml # base_classes/NXion.nxdl.xml # base_classes/NXlens_em.nxdl.xml # base_classes/NXoptical_system_em.nxdl.xml # base_classes/NXpump.nxdl.xml # base_classes/NXscanbox_em.nxdl.xml # base_classes/NXspectrum_set.nxdl.xml # base_classes/NXstage_lab.nxdl.xml # contributed_definitions/NXaberration_model.nxdl.xml # contributed_definitions/NXaberration_model_ceos.nxdl.xml # contributed_definitions/NXaberration_model_nion.nxdl.xml # contributed_definitions/NXaperture_em.nxdl.xml # contributed_definitions/NXapm_paraprobe_results_nanochem.nxdl.xml # contributed_definitions/NXcomponent_em.nxdl.xml # contributed_definitions/NXcs_gpu_obj.nxdl.xml # contributed_definitions/NXcs_gpu_sys.nxdl.xml # contributed_definitions/NXcs_io_obj.nxdl.xml # contributed_definitions/NXcs_io_sys.nxdl.xml # contributed_definitions/NXcs_mm_sys.nxdl.xml # contributed_definitions/NXem_adf.nxdl.xml # contributed_definitions/NXem_conventions.nxdl.xml # contributed_definitions/NXem_conventions_ebsd.nxdl.xml # contributed_definitions/NXem_ebsd_crystal_structure_model.nxdl.xml # contributed_definitions/NXgraph_edge_set.nxdl.xml # contributed_definitions/NXgraph_node_set.nxdl.xml # contributed_definitions/NXgraph_root.nxdl.xml # contributed_definitions/NXimage_r_set_diff.nxdl.xml # contributed_definitions/NXimage_set_em_adf.nxdl.xml # contributed_definitions/NXimage_set_em_kikuchi.nxdl.xml # contributed_definitions/NXinteraction_vol_em.nxdl.xml # contributed_definitions/NXisocontour.nxdl.xml # contributed_definitions/NXms.nxdl.xml # contributed_definitions/NXms_feature_set.nxdl.xml # contributed_definitions/NXms_ipf.nxdl.xml # contributed_definitions/NXms_mtex_config.nxdl.xml # contributed_definitions/NXms_odf.nxdl.xml # contributed_definitions/NXms_odf_set.nxdl.xml # contributed_definitions/NXms_pf.nxdl.xml # contributed_definitions/NXms_pf_set.nxdl.xml # contributed_definitions/NXms_recon.nxdl.xml # contributed_definitions/NXms_score_results.nxdl.xml # contributed_definitions/NXorientation_set.nxdl.xml # contributed_definitions/NXrotation_set.nxdl.xml # contributed_definitions/NXspectrum_set_em_eels.nxdl.xml # contributed_definitions/NXspectrum_set_em_xray.nxdl.xml # contributed_definitions/nyaml/NXaberration.yaml # contributed_definitions/nyaml/NXaberration_model.yaml # contributed_definitions/nyaml/NXaberration_model_ceos.yaml # contributed_definitions/nyaml/NXaberration_model_nion.yaml # contributed_definitions/nyaml/NXaperture_em.yaml # contributed_definitions/nyaml/NXapm.yaml # contributed_definitions/nyaml/NXapm_paraprobe_results_nanochem.yaml # contributed_definitions/nyaml/NXcg_alpha_complex.yaml # contributed_definitions/nyaml/NXcg_cylinder_set.yaml # contributed_definitions/nyaml/NXcg_ellipsoid_set.yaml # contributed_definitions/nyaml/NXcg_face_list_data_structure.yaml # contributed_definitions/nyaml/NXcg_geodesic_mesh.yaml # contributed_definitions/nyaml/NXcg_grid.yaml # contributed_definitions/nyaml/NXcg_half_edge_data_structure.yaml # contributed_definitions/nyaml/NXcg_hexahedron_set.yaml # contributed_definitions/nyaml/NXcg_marching_cubes.yaml # contributed_definitions/nyaml/NXcg_parallelogram_set.yaml # contributed_definitions/nyaml/NXcg_point_set.yaml # contributed_definitions/nyaml/NXcg_polygon_set.yaml # contributed_definitions/nyaml/NXcg_polyhedron_set.yaml # contributed_definitions/nyaml/NXcg_polyline_set.yaml # contributed_definitions/nyaml/NXcg_roi_set.yaml # contributed_definitions/nyaml/NXcg_sphere_set.yaml # contributed_definitions/nyaml/NXcg_tetrahedron_set.yaml # contributed_definitions/nyaml/NXcg_triangle_set.yaml # contributed_definitions/nyaml/NXcg_triangulated_surface_mesh.yaml # contributed_definitions/nyaml/NXcg_unit_normal_set.yaml # contributed_definitions/nyaml/NXchamber.yaml # contributed_definitions/nyaml/NXcoordinate_system_set.yaml # contributed_definitions/nyaml/NXcorrector_cs.yaml # contributed_definitions/nyaml/NXcs_computer.yaml # contributed_definitions/nyaml/NXcs_filter_boolean_mask.yaml # contributed_definitions/nyaml/NXcs_io_obj.yaml # contributed_definitions/nyaml/NXcs_io_sys.yaml # contributed_definitions/nyaml/NXcs_mm_sys.yaml # contributed_definitions/nyaml/NXcs_prng.yaml # contributed_definitions/nyaml/NXcs_profiling.yaml # contributed_definitions/nyaml/NXcs_profiling_event.yaml # contributed_definitions/nyaml/NXdeflector.yaml # contributed_definitions/nyaml/NXebeam_column.yaml # contributed_definitions/nyaml/NXem.yaml # contributed_definitions/nyaml/NXem_ebsd.yaml # contributed_definitions/nyaml/NXevent_data_em.yaml # contributed_definitions/nyaml/NXevent_data_em_set.yaml # contributed_definitions/nyaml/NXfabrication.yaml # contributed_definitions/nyaml/NXgraph_edge_set.yaml # contributed_definitions/nyaml/NXgraph_node_set.yaml # contributed_definitions/nyaml/NXgraph_root.yaml # contributed_definitions/nyaml/NXibeam_column.yaml # contributed_definitions/nyaml/NXimage_set.yaml # contributed_definitions/nyaml/NXinteraction_vol_em.yaml # contributed_definitions/nyaml/NXion.yaml # contributed_definitions/nyaml/NXisocontour.yaml # contributed_definitions/nyaml/NXlens_em.yaml # contributed_definitions/nyaml/NXms.yaml # contributed_definitions/nyaml/NXms_feature_set.yaml # contributed_definitions/nyaml/NXms_score_results.yaml # contributed_definitions/nyaml/NXoptical_system_em.yaml # contributed_definitions/nyaml/NXpump.yaml # contributed_definitions/nyaml/NXrotation_set.yaml # contributed_definitions/nyaml/NXscanbox_em.yaml # contributed_definitions/nyaml/NXspectrum_set.yaml # contributed_definitions/nyaml/NXstage_lab.yaml # dev_tools/tests/test_nxdl_utils.py # manual/source/classes/contributed_definitions/cgms-structure.rst # manual/source/classes/contributed_definitions/em-structure.rst # manual/source/classes/contributed_definitions/icme-structure.rst
- Loading branch information
