Skip to content

Commit

Permalink
done with api review
Browse files Browse the repository at this point in the history
  • Loading branch information
jrudz committed Jan 8, 2025
1 parent 88c8a9a commit 3e2b9b4
Show file tree
Hide file tree
Showing 6 changed files with 58 additions and 177 deletions.
88 changes: 19 additions & 69 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,25 @@ Utilities for interfacing with NOMAD within workflows, e.g., via a workflow mana

This `nomad` plugin was generated with `Cookiecutter` along with `@nomad`'s [`cookiecutter-nomad-plugin`](https://github.com/FAIRmat-NFDI/cookiecutter-nomad-plugin) template.

## Usage

For direct usage and integrating the utility module into other plugins or codes, `nomad-utility-workflows` is available as a PyPI package:

```bash
pip install nomad-utility-workflows>=0.0.8
```

### Linking to your NOMAD account
Create an account on https://nomad-lab.eu/.
Store your credentials in a `.env` file in your working directory, at the root plugin directory for developers, or in some directory that is added to your `PYTHONPATH`, with the following content
```bash
NOMAD_USERNAME="MyLogin"
NOMAD_PASSWORD="MyPassWord"
```
and insert your username and password.

> [!CAUTION]
> Never push your `.env` file to a repository. This would expose your password.
## Development

Expand Down Expand Up @@ -39,18 +58,6 @@ The plugin is still under development. If you would like to contribute, install
uv pip install -e '.[dev]' --index-url https://gitlab.mpcdf.mpg.de/api/v4/projects/2187/packages/pypi/simple
```

### Linking to your NOMAD account
Create an account on https://nomad-lab.eu/.
Store your credentials in a `.env` file at the root plugin directory, with the following content
```bash
NOMAD_USERNAME="MyLogin"
NOMAD_PASSWORD="MyPassWord"
```
and insert your username and password.

> [!CAUTION]
> Never push your `.env` file to a repository. This would expose your password.
### Run the tests

You can run locally the tests:
Expand Down Expand Up @@ -125,63 +132,6 @@ python -m ipykernel install --user --name=nomad_utility_workflows
```


## Adding this plugin to NOMAD

Currently, NOMAD has two distinct flavors that are relevant depending on your role as an user:
1. [A NOMAD Oasis](#adding-this-plugin-in-your-nomad-oasis): any user with a NOMAD Oasis instance.
2. [Local NOMAD installation and the source code of NOMAD](#adding-this-plugin-in-your-local-nomad-installation-and-the-source-code-of-nomad): internal developers.

### Adding this plugin in your NOMAD Oasis

Read the [NOMAD plugin documentation](https://nomad-lab.eu/prod/v1/staging/docs/howto/oasis/plugins_install.html) for all details on how to deploy the plugin on your NOMAD instance.

### Adding this plugin in your local NOMAD installation and the source code of NOMAD

Modify the text file under `/nomad/default_plugins.txt` and add:
```sh
<other-content-in-default_plugins.txt>
nomad-utility-workflows==x.y.z
```
where `x.y.z` represents the released version of this plugin.

Then, go to your NOMAD folder, activate your NOMAD virtual environment and run:
```sh
deactivate
cd <route-to-NOMAD-folder>/nomad
source .pyenv/bin/activate
./scripts/setup_dev_env.sh
```

Alternatively and only valid for your local NOMAD installation, you can modify `nomad.yaml` to include this plugin, see [NOMAD Oasis - Install plugins](https://nomad-lab.eu/prod/v1/staging/docs/howto/oasis/plugins_install.html).


### Build the python package

The `pyproject.toml` file contains everything that is necessary to turn the project
into a pip installable python package. Run the python build tool to create a package distribution:

```sh
pip install build
python -m build --sdist
```

You can install the package with pip:

```sh
pip install dist/nomad-utility-workflows-0.1.0
```

Read more about python packages, `pyproject.toml`, and how to upload packages to PyPI
on the [PyPI documentation](https://packaging.python.org/en/latest/tutorials/packaging-projects/).


### Template update

We use cruft to update the project based on template changes. A `cruft-update.yml` is included in Github workflows to automatically check for updates and create pull requests to apply updates. Follow the [instructions](https://github.blog/changelog/2022-05-03-github-actions-prevent-github-actions-from-creating-and-approving-pull-requests/) on how to enable Github Actions to create pull requests.

To run the check for updates locally, follow the instructions on [`cruft` website](https://cruft.github.io/cruft/#updating-a-project).


## Main contributors
| Name | E-mail |
|------|------------|
Expand Down
91 changes: 20 additions & 71 deletions docs/how_to/install_this_plugin.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,28 @@
# How to install this utility module

## Usage

For direct usage and integrating the utility module into other plugins or codes, `nomad-utility-workflows` is available as a PyPI package:

```bash
pip install nomad-utility-workflows>=0.0.8
```

### Linking to your NOMAD account
Create an account on https://nomad-lab.eu/.
Store your credentials in a `.env` file in your working directory, at the root plugin directory for developers, or in some directory that is added to your `PYTHONPATH`, with the following content
```bash
NOMAD_USERNAME="MyLogin"
NOMAD_PASSWORD="MyPassWord"
```
and insert your username and password.

!!! Warning "CAUTION FOR DEVELOPERS"
Never push your `.env` file to a repository. This would expose your password.

## Development

If you want to develop locally this plugin, clone the project and in the plugin folder, create a virtual environment (you can use Python 3.9, 3.10, or 3.11):
If you want to develop this module locally, clone the project and in the plugin folder, create a virtual environment (you can use Python 3.9, 3.10, or 3.11):
```sh
git clone https://github.com/FAIRmat-NFDI/nomad-utility-workflows.git
cd nomad-utility-workflows
Expand Down Expand Up @@ -35,18 +54,6 @@ The plugin is still under development. If you would like to contribute, install
uv pip install -e '.[dev]' --index-url https://gitlab.mpcdf.mpg.de/api/v4/projects/2187/packages/pypi/simple
```

### Linking to your NOMAD account
Create an account on https://nomad-lab.eu/.
Store your credentials in a `.env` file at the root plugin directory, with the following content
```bash
NOMAD_USERNAME="MyLogin"
NOMAD_PASSWORD="MyPassWord"
```
and insert your username and password.

> [!CAUTION]
> Never push your `.env` file to a repository. This would expose your password.
### Run the tests

You can run locally the tests:
Expand Down Expand Up @@ -120,64 +127,6 @@ To run the test notebooks, create a jupyter kernel using your venv:
python -m ipykernel install --user --name=nomad_utility_workflows
```


## Adding this plugin to NOMAD

Currently, NOMAD has two distinct flavors that are relevant depending on your role as an user:
1. [A NOMAD Oasis](#adding-this-plugin-in-your-nomad-oasis): any user with a NOMAD Oasis instance.
2. [Local NOMAD installation and the source code of NOMAD](#adding-this-plugin-in-your-local-nomad-installation-and-the-source-code-of-nomad): internal developers.

### Adding this plugin in your NOMAD Oasis

Read the [NOMAD plugin documentation](https://nomad-lab.eu/prod/v1/staging/docs/howto/oasis/plugins_install.html) for all details on how to deploy the plugin on your NOMAD instance.

### Adding this plugin in your local NOMAD installation and the source code of NOMAD

Modify the text file under `/nomad/default_plugins.txt` and add:
```sh
<other-content-in-default_plugins.txt>
nomad-utility-workflows==x.y.z
```
where `x.y.z` represents the released version of this plugin.

Then, go to your NOMAD folder, activate your NOMAD virtual environment and run:
```sh
deactivate
cd <route-to-NOMAD-folder>/nomad
source .pyenv/bin/activate
./scripts/setup_dev_env.sh
```

Alternatively and only valid for your local NOMAD installation, you can modify `nomad.yaml` to include this plugin, see [NOMAD Oasis - Install plugins](https://nomad-lab.eu/prod/v1/staging/docs/howto/oasis/plugins_install.html).


### Build the python package

The `pyproject.toml` file contains everything that is necessary to turn the project
into a pip installable python package. Run the python build tool to create a package distribution:

```sh
pip install build
python -m build --sdist
```

You can install the package with pip:

```sh
pip install dist/nomad-utility-workflows-0.1.0
```

Read more about python packages, `pyproject.toml`, and how to upload packages to PyPI
on the [PyPI documentation](https://packaging.python.org/en/latest/tutorials/packaging-projects/).


### Template update

We use cruft to update the project based on template changes. A `cruft-update.yml` is included in Github workflows to automatically check for updates and create pull requests to apply updates. Follow the [instructions](https://github.blog/changelog/2022-05-03-github-actions-prevent-github-actions-from-creating-and-approving-pull-requests/) on how to enable Github Actions to create pull requests.

To run the check for updates locally, follow the instructions on [`cruft` website](https://cruft.github.io/cruft/#updating-a-project).


## Main contributors
| Name | E-mail |
|------|------------|
Expand Down
20 changes: 11 additions & 9 deletions docs/how_to/use_api_functions.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# How to use nomad-utility-workflows to perform NOMAD API Calls
# How to perform NOMAD API Calls

Imports for the following examples:

Expand Down Expand Up @@ -47,7 +47,7 @@ By default, nomad-utility-workflows uses the Test deployment of NOMAD to make AP
All API functions allow the user to specify the URL with the optional keyword argument `url`. If you want to use the central NOMAD URLs, you can simply set `url` equal to "prod", "staging", or "test", which correspond to the following deployments (see full URLs below):

- prod: the official NOMAD deployment.
- Updated most infrequently (as advertised in #software-updates on the NOMAD Discord Server)
- Updated most infrequently (as advertised in [#software-updates](https://discordapp.com/channels/1201445470485106719/1275764272122826752) on the NOMAD Discord Server&mdash;If you are not yet a member of the NOMAD server use [Invitation to Discord](https://discord.gg/Gyzx3ukUw8))
- staging: the beta version of NOMAD.
- Updated more frequently than prod in order to integrate and test new features.
- test: a test NOMAD deployment.
Expand Down Expand Up @@ -78,7 +78,7 @@ print(NOMAD_PROD_URL, NOMAD_STAGING_URL, NOMAD_TEST_URL)

## Authentication

Some API calls, e.g., making uploads or accessing your own non-published uploads, require an authentication token. To generate this token, nomad-utility-workflows expects that your NOMAD credentials are stored in a `.env` file in the plugin root directory in the format:
Some API calls, e.g., making uploads or accessing your own non-published uploads, require an authentication token. To generate this token, `nomad-utility-workflows` expects that your NOMAD credentials are stored in a `.env` file in your working directory or in a directory included in your `PYTHONPATH`:

```bash
NOMAD_USERNAME="<your_nomad_username>"
Expand All @@ -99,6 +99,8 @@ NOMAD_USERNAME
'JFRudzinski'
```

!!! Warning "CAUTION FOR DEVELOPERS"
Never push your `.env` file to a repository. This would expose your password.

Use `get_authentication_token()` with your credentials to explicitly obtain and store a token:

Expand All @@ -119,11 +121,11 @@ token
```


In practice, you do not need to obtain a token yourself when using nomad-utility-workflows. A token will automatically be obtained for API calls that require authentication. However, you may want to do the token generation yourself for custom API calls (see `Writing your own wrappers` below.)
In practice, you do not need to obtain a token yourself when using `nomad-utility-workflows`. A token will automatically be obtained for API calls that require authentication. However, you may want to do the token generation yourself for custom API calls (see [Writing your own wrappers](#writing-your-own-wrappers) below.)

### NOMAD User Metadata

nomad-utility-workflows uses the `NomadUser()` class to store the following user metadata:
`nomad-utility-workflows` uses the `NomadUser()` class to store the following user metadata:

```python
class NomadUser:
Expand Down Expand Up @@ -189,7 +191,7 @@ nomad_user

### Uploading Data

nomad-utility-workflows uses the `NomadUpload()` class to store the following upload metadata:
`nomad-utility-workflows` uses the `NomadUpload()` class to store the following upload metadata:

```python
class NomadUpload:
Expand Down Expand Up @@ -398,7 +400,7 @@ pprint(nomad_upload.process_running is False)

During the upload process, NOMAD automatically identfies representative files that indicate the presence of data that can be parsed with the plugins included within a given deployment. This means that each upload can contain multiple *entries*&mdash;the fundamental unit storage within the NOMAD database.

You can query the individual entries within a known upload with `get_entries_of_upload()`, which then returns the metadata within the `NomadEntry()` class of nomad-utility-worklfows:
You can query the individual entries within a known upload with `get_entries_of_upload()`, which then returns the metadata within the `NomadEntry()` class of `nomad-utility-workflows`:

```python
class NomadEntry:
Expand Down Expand Up @@ -862,7 +864,7 @@ for entry in dataset_entries:
'entry_id=Htbl78lHDSNAKbvPjEgEN_6sOcxF, upload_id=RdA_3ZsOTMqbtAhYLivVsw'
```

There is no "publishing" action for datasets. Instead, when the dataset is complete (i.e., you are ready to lock the contents of the dataset), you can *assign a DOI*. There is currently no API action for this within nomad-utility-workflows. You must go to the GUI of the relevant deployment, go to `PUBLISH > Datasets`, find the dataset, and then click the "assign a DOI" banner icon to the right of the dataset entry.
There is no "publishing" action for datasets. Instead, when the dataset is complete (i.e., you are ready to lock the contents of the dataset), you can *assign a DOI*. There is currently no API action for this within `nomad-utility-workflows`. You must go to the GUI of the relevant deployment, go to `PUBLISH > Datasets`, find the dataset, and then click the "assign a DOI" banner icon to the right of the dataset entry.

## Deleting Uploads and Datasets

Expand Down Expand Up @@ -950,7 +952,7 @@ except Exception:

## Useful Wrappers

nomad-utility-workflows contains a few useful wrapper functions to help users query all of their uploads and corresponding entries:
`nomad-utility-workflows` contains a few useful wrapper functions to help users query all of their uploads and corresponding entries:


```python
Expand Down
9 changes: 4 additions & 5 deletions docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,7 @@ A module with utilities for interacting with NOMAD via, e.g., a workflow manager

## Introduction

!!! note "Attention"
TODO
NOMAD's API and workflow functionalities provide users with a route to FAIR data management of complex and highthroughput processes. However, the overhead for leveraging these capabilities remains rather large. This utility module assists lowers the barrier to these tools, currently with 2 distinct sets of tools: 1. python API wrappers, and 2. automated workflow yaml generation for building custom workflows.

<div markdown="block" class="home-grid">

Expand All @@ -29,9 +28,9 @@ How-to guides provide step-by-step instructions for a wide range of tasks, with

The following examples are available for demonstration purposes:

- [Workflow - Proof of Concept](examples/workflow_proof_of_concept.md)
- [Workflow - Simulation Protocol](examples/workflow_simulation_protocol.md)
- [Workflow - Adding Custom Tasks](examples/workflow_custom_tasks.md)
- [Workflows - Proof of Concept](examples/workflow_proof_of_concept.md)
- [Workflows - Simulation Protocol](examples/workflow_simulation_protocol.md)
- [Workflows - Adding Custom Tasks](examples/workflow_custom_tasks.md)

</div>

Expand Down
8 changes: 4 additions & 4 deletions mkdocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,16 +10,16 @@ nav:
# - Tutorial: tutorial/tutorial.md
- How-to guides:
- Install: how_to/install_this_plugin.md
- API Functions: how_to/use_api_functions.md
- Perform API Calls: how_to/use_api_functions.md
# - Custom Workflows: how_to/create_custom_workflows.md
# - Install this Plugin: how_to/install_this_plugin.md
# - Use this Plugin: how_to/use_this_plugin.md
# - Contribute to this plugin: how_to/contribute_to_this_plugin.md
# - Contribute to the documentation: how_to/contribute_to_the_documentation.md
- Examples:
- Workflow - Proof of Concept: examples/workflow_proof_of_concept.md
- Workflow - Simulation Protocol: examples/workflow_simulation_protocol.md
- Workflow - Adding Custom Tasks: examples/workflow_custom_tasks.md
- Workflows - Proof of Concept: examples/workflow_proof_of_concept.md
- Workflows - Simulation Protocol: examples/workflow_simulation_protocol.md
- Workflows - Adding Custom Tasks: examples/workflow_custom_tasks.md
- Explanation:
- Workflows: explanation/workflows.md
- Reference:
Expand Down
19 changes: 0 additions & 19 deletions tests/utils/test.py

This file was deleted.

0 comments on commit 3e2b9b4

Please sign in to comment.