Make working routine exporting of data/metadata from Nion to NXem in @KochLab

[mkuehbach]

Aim/Expectation: Nion microscope data should be injectable NXem-formatted in Christoph's OASIS (KPI-relevant)

• Inspect status quo of Sherjeel's work (customization of nionswift to identify where arrays are stored and how to intercept this via calls to h5py instead/additionally)
• at least some short meeting with the Nion team would be good to check that what we are developing will have some persistence, last feedback I got from Andreas Mittelberger is already 0.5year old...
• Inspect which datasets from Benedikt and others in Christoph's group are available (make them accessible/exchange these)
• Implement specific export writing from nionswift to NeXus
• May require a one/two week personal stay of Markus to work with scientists @KochLab at the microscope
• Deploy (Sherjeel is local Oasis administrator)
• Benedikt's key point: Isn't everything in nionswift already an NXprocess / workflow? Can we identify the most frequently used interaction path with nionswift to reduce our efforts and make the point for only a few use cases of nionswift?
• Would be good to finally know B1 leaders to assure investing into the support of a specific software tool like here nionswift is worthwhile

[mkuehbach]

Update 2023/05/09 @mkuehbach and @shabihsherjeel discussed how to proceed with this issue after inspecting the promising alpha material which @shabihsherjeel generated for graphically editing and setting relations between concepts via a GUI. We identified three sets of issues:

1.) Issues related to the development of the GUI, purpose: should be i) incentivize users to edit relations between metadata fields and concepts graphically, ii) generate from this a data artifact which can be parsed by readers to identify how to map data items from A to B without explicitly hardcoding this in the reader source code. The status of the GUI which is essentially a graphical modifier based node editing "widget" is "in development", support for defining for each side (input = left side) output (mapped onto, right side) versions for which specific tool and input and output (appdef) this mapping is defined is missing, also support for multi-data-element input and a mapping function to map n-quantities on one is missing, simple example ["day", "month", "year"] mapped to f"{day}/{month}/{year}" whereas also f"{year}/{month}/{day}" is possible, neither however are currently conceptualized wrt to how to describe in json let aside that support for such would be implemented, currently it is not,
Agreement next steps: @shabihsherjeel will stabilize the widget, then testing, then addition of multiple-to-one case

2.) Issues related to integrating a reader inside nionswift to export to NeXus directly, currently available is a simple demo which writes NeXus but without any specific annotations and decorators like needed for correct displaying of NXdata instances in say H5Web
Back then this implementation was directly tapping from the internal tree of python objects inside nionswift and reading from these while traversing the data tree upon exporting
Conceptually this pursuing this road inside nionswift is very similar to like "if we would have access to the ThermoFisher or JEOL source code" why not write it into the JEOL source code to export to NeXus. We both agree that this is exactly the road we as framework developers for NOMAD should not follow

3.) Alternative use the json_map reader and build on the NXiv_temp example this would essential require the following tasks:

• Study how the json_map reader works @mkuehbach

• Create an nionswift==<> + NXem<> specific json document which maps relevant fields @mkuehbach reviewed by @shabihsherjeel

• call would then be dataconverter --reader json_map --nxdl NXem --input-file mapping_table_src_version_trg_version --input-file nionswift json and npy, the downside of this approach is that "raw data from the microscope remain" as they are @mkuehbach reviewed by @shabihsherjeel

• We exchanged also about the current support for NeXus-related activities in the lab, while several scientists agree it could be useful to have processing steps documented in a differ and more complete manner than currently; there is also substantial hesitation and lacking conceptual idea how to map with this workflows and processing steps which currently everyone uses. In any case if implemented conveniently this would require customizing the nionswift source code, even more substantially than for issue set 1.) because the internal state tracking (which is known to be incomplete currently) would have to be understood first before we can intercept and harvest from it to not just dump some results from internal nionswift state data but in a way respectively and properly organized and most importantly correctly placed in the sequence the data were processed (input, output, workflows) to generate trustworthy NXprocess instances. It could be better here to have first a clearer demo how NXem data can at all be mapped using NeXus to then discuss with Nion Co. as to how this can possibly be implemented for now for state tracking we know that updating the metadata organization inside nionswift is a topic on their agenda but so far nobody clearly approaches us with supporting this nor giving away sufficient details and credible interest in pushing such issues in the priorization of sprints at Nion. Rightly so other areas may ask why only having no access to the source code is argument to not even think about pursuing this for e.g. ThermoFisher/FEI microscopes, and in fact for JEOL there is an open source scripting solution as there is for TemGYM what Dieter Weber has been working on/has worked on...

[sanbrock]

Actually, the option 2 would/could be a nice example for our technology partners, how they can implement an export function in their software solutions.

[lukaspie]

@mkuehbach I transferred this issue from data-modeling as well.