2024-Tutorial Preparation

[budschi]

Tutorial planning

From Area A task force Meeting on Mar 4th, 2024 :

• Mai 15th 2024
• Scope:
  • BaseSections,
  • Plugins Parser App
• Targeted audience
• Format
• Responsibilities: internal A team
• brainstorming:
  • 2 sessions: user part --> Basic ELNs, app admin part --> plugin
  • concept missing, first present a clear concept
  • concept behind
  • concept of schemas
• conclusion
  • 2 parts:
    • base sections and built-in ELNs -> people understand interoperablity by data model
    • plugin
  • aim for distributing tasks next week
• Ahmed:
  • LATER abstract ready unti Apr 12th, 2024
  • Program overview and details
  • Materials
  • communicate with Carolin Rehermann
  • remind before presentation how video is produced (second of pause when answering comments)

TF Meeting 11.03.2024

Goal of this meeting:

• content of the tutorial: What is do we tell and what not?
• distribute tasks:
  • Abstract writing (goes out on Apr 12th, 2024 ) --> Sebastian
  • Introductioin part: Sebastian
    • structured and well-described data
    • plugin: aiming for automatization, visualization
    • users and roles: is done
    • overview on the whole storry
    • glossary: take from MADICES
  • part 1:
    • BaseSections: Andrea
    • Built-in ELNs: Sarthak
  • part 2:
    • Plugin --> Hampus
    • Deployment --> Hampus
  • ...
  • Tutorial materials (NOMAD example Upload, Jupyter Notebook, files, ?)
  • How long?
    • part 1a - 15min
    • part 1b - 30min
    • part 2 - 1.5h
• Next steps:
  • each responsible structures his part and brings it back for discussion
  • bring it to the TF meetings
  • aiming for finalizing it: May 6th, 2024
  • define a storyline
    • users meeting story?
    • or process story?
    • use an analogy (like cooking) knife example, for reproduction you need detailed description of everyhting including the knife

Tutorial - Detailed structure from Sebastian

• from Fz testing notebook
• Starting conditions:
  • parts
    • part 1 for all users
    • part 2 plugin writing --> appl. admins
    • part 3 plugin deployment --> appl. admins and system admins
      • use for example the XRD plugin as a demo vehicle
  • users and roles
  • preconditions:
    • structured and well-described data
    • plugin: aiming for automatization, visualization
  • make clear what people can get out of it without NOMAD
  • provide a dictionary/glossary

Part 1: BaseSections and Built-in ELNs --> all users

BaseSections:

• idea:
  • activity entity model, ensures interoperability in NOMAD by defining data structures and relationships according to the needs in (experimental) Materials Science.
  • from docs: We add a relationship between *section definitions* that allows us to create more *specialized* definitions from more *abstract* definitions. Here the *properties* of the *abstract* definition are inherited by the more *specialized definitions*
  • base sections can be used when writing custom schemas to inherit properties, and more
    importantly, functionality from already defined sections.
• Explain overall structure
• Explain one or two examples in detail
• How to use them?
  • They are supposed to be used in all schemas
  • show levels of schemas
• Transition: built-in ELNs - convenient way to learn about the BaseSections and use ELNs
• mention YAML and ref to Tutorial 8 and user metting 11-23
• data modelling
• inheriting basic concept
• mention the xistence of XRD plugin in Oasis NOMAD

Built-In ELNs

• semi-structured ELNs in NOMAD based on BaseSections:
  • some structured fields, but also large text body field for unstructured data --> similar to elabFTW etc.
  • based on BaseSections: provide the given structure and relationships
  • convenient way to familiarize yourself with NOMAD's BaseSections and start using NOMAD without writing schemas (all entries in NOMAD are based on a schema)
• learn:
  • create upload and manage it
  • instantiate template from schema
  • fill out schema
  • create references
  • get aware of tools: workflow visualizer, sample history, entry references
  • XRD file
• conclusion: further customization needed by custom schemas
• Comments:
  • start maybe with NOMAD directly

Part 2: Plugin development in Jupyter Hub --> application admin

• focus on application side
  - inheriting on basesection and/or standard plugins (inheritance from different plugins)

  • include also deployment of plugin
  • aiming for a pip installable package

• fast part 3 - deployment:

  • nomad image with plugin

• Problems:

  • testing plugin is only showing json output

Details on Plugin Schema Writing in Jupyter Hub of central NOMAD:

• I. Starting with yaml schema and using metainfoyaml2py and pip installing Plugin

• 1. write custom schema (see tutorial 8)

• 2. yaml to py conversion

install: pip install metainfoyaml2py to convert a yaml schema into a Python plugin:

run the conversion

• 3. install the plugin:

• 4. modify and improve schema if necessary, e.g. add functionality to a normalize function --> read out a file, etc.

• 5. test schema and generate archive output

--> keep editiing it until it is ready for deployment

• II. Developing Plugin directly in Jupyter Hub (like Tutorial 12)

• 1. import all necessary modules etc. --> How would you know which ones you need? Start from example plugin?

• 2. Define&edit the schema in Python (see tutorial 12)

• 3. instanciate example entries: (see tutorial 12)

• 4. generate a yaml schema if you want to see how it looks in NOMAD (without Normalizers!) (see tutorial 12)

• 5. add normalizer functions for automatization, file reading etc.

test the nomrmalizer (see tutorial 12). super(xyz).normalizer() does not work here!

• edit it schema until it is ready for deployment

• III not covered yet:

• parsers --> see tutorial 12

• plot --> can only be shown indirectly

• IV deployment: --> system admin or advanced application admin

• dev deployment: only for devs

• use docker image writing template to write Oasis image, deploy locally or at IKZ

• what is covered already in tutorials?

[budschi]

@hampusnasstrom @ka-sarthak @aalbino2 @Pepe-Marquez: please add your thoughts!

[budschi]

Conclusion from a short discussion on the technical aspects of the Tutorial with @hampusnasstrom

• proposed practice schema development
  • yaml schema --> allows you to see the schema in the GUI
    if you do this in Pyton you cannot see the gui features
  • convert to Python plugin
    • with Hampus tool
    • mention only route via example-plugin from Area D
  • implement normalize functions and parsing
  • test with nomad CLI nomad parse command
    • converge Python to yaml to test in gui before without functionality
  • after iteration and improvement deployment
    • , create docker image from template with plugin in requeriments
    • run a develop installation --> only for fulltime Oasis developers
• remarks
  • yaml schemas: explained already in tutorial 8, can be rather brief in this tutorial
  • conversion to Python: how much do we explain what is needed for a Python package? Our tool provides pip installable package without understanding it
  • plugin development
    • normalizer: show an automatization feature and explain the normalize function
    • parser: show how to develop parser --> this could be rather time consuming! no?
  • testing with NOMAD CLI
    • familiarize with CLI tool and output, generate a meaningful test.archive.yaml file
    • converge Python to Yaml to test in gui
  • plugin deployment: refer to other methods but push for deployment via docker which is the easiest
• questions
  • does nomad package work in Windows?
  • how much work is it to install docker on windows?
  • requirements:
    • technical:
      • docker installed and working for deployment
      • NOMAD account
      • browser
      • example upload with files
      • demo on Jupyter Hub
    • skills:
      • tutorial 8: yaml schema writing
      • tutorial 12: general plugin writing

[budschi]

next step, structure a good storyline for the tutorial.
What would be a good use case to make a story out of it?

[budschi]

we could reuse the Glossary from Madices

[budschi]

https://github.com/FAIRmat-NFDI/AreaA-Examples/blob/main/tutorial13/planning.md