Understanding NOMAD and it's features to write better schemas

[ka-sarthak]

@aalbino2 @hampusnasstrom @budschi
Let's use this issue to collect all the possible NOMAD features or the methodologies we use to write schemas. This could also include the topics on which we get doubts when other developers extend or use our schemas as templates for their work.

[budschi]

Brainstorming

References

• NOMAD search for automated referencing, showing how to use it, difference between API and search, looking for an entry that is already exisitng in normalize function (careful usage, it can see any entries/data!) -> vs. ArchiveQuery which uses authentication <-> nomad.search. Warning: when changing schemas the query inside nomad.search must be updated. Usually the quantity path is hardcoded in the query dict and hence must be updated.

• reference strings: ../upload/raw/.../...archive.json ??? how to write that paths properly?

• accessing reference paths: how do we get that path and can put it e.g. in a ReferenceEditQuantit

new files, entries

• create_archive: two ways of parsing: through raw file (archive.json) or message file (msg) --> guidelines, functions should be part of NOMAD Metainfo https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1955#note_252942
• create new files: e.g. Jupyter Notebooks, CSV, PDF etc. Need for clear guideline, instanciate raw file and attach/reference to entry or process as own entry (relates to create_archive)
• how to treat ELN data and normalized data: normalizer should not overwrite ELN data
• update archive file (save twice...) https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/823
• merge_section: move up from standard plugins to NOMAD code

parsers

• how write file matching in parsers:

parser = Movpe2ParserEntryPoint(
    name='Movpe2Parser',
    description='Parse excel files containing growth process parameters logged manually.',
    mainfile_name_re=r'.+\.growth.movpe.ikz.xlsx',
    mainfile_mime_re=r'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet',
    mainfile_contents_dict={
        'GrowthRun': {'__has_all_keys': ['Sample Name']},
        '__has_comment': '#',
    },
)

https://github.com/FAIRmat-NFDI/nomad-measurements/blob/main/src/nomad_measurements/xrd/__init__.py#L24

• a_templates

General Metainfo understanding

• awarness about two types of contexts: server and client (and parser) context: clarifications about the contexts in NOMAD Metainfo, guidelines for us and others
• helpful functions to be used: (go through our examples and list them here!)
  • m_setdefault: instantiate sub_sections
  • m_context.process_rawfile
  • a_templates vs default
• general guidelines on pluign writing:
  • inherit
  • overwrite
  • hide quantities and subsections
  • overview true

Quantity types

• Operator, user, author --> reference to a NOMAD user
• action button

lab-features, Area A specific, BaseSections normalizers

• "batch" functionality in collection
• guidelines for plugin logic
• https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/2053

[aalbino2]

"m" methods we use:

• m_def
• m_context
• m_to_dict
• m_resolve
• m_copy

[aalbino2]

maybe also some plotting strategies would be appreciated!

This would foster from our side an extension of the docs and practical examples of that:

• plotly annotations in yaml
• plotly annotations in python (not clear to me at which extent we can even use that)
• small normalize function plotly example
• h5web plot render visualization within nomad archives (a nice treat to gift to the community)
• how to plot in one subsection and drag the plot also in other sections of the archive

[aalbino2]

@budschi I will open an MR in this repo including a folder with some python files containing snippets of code to be demonstrated and used for the workshop of 15th and 16th of january