Skip to content

Commit

Permalink
👌 IMPROVE: Organising your data (aka. groups) (#364)
Browse files Browse the repository at this point in the history 
Make a thorough pass of the "Organising your data" module:

1. Improving the actual content - mainly minor clarifications.
2. Remove empty lines in code blocks.
3. Try to update the text according to #360.

Co-authored-by: Francisco Ramirez <[email protected]>
Co-authored-by: Marnik Bercx <[email protected]>
  • Loading branch information
@CasperWA @ramirezfranciscof @mbercx
3 people authored Jul 2, 2021
1 parent 4c8e63f commit 0c32062
Showing 1 changed file with 34 additions and 61 deletions.
95 changes: 34 additions & 61 deletions docs/sections/data/groups.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,15 +3,14 @@
# Organising your data

In this section of the tutorial we will focus on how to organise and explore the data in an AiiDA database.
As in (**TODO FIX LINK**), we will be using the previously created database entries for this tutorial.
To follow the tutorial then, you can use the profile in to which you have previously imported this data, or you may wish to create a fresh profile and import the archive into that:
For this tutorial you will be loading and using the information of an external pre-existing database.
To follow the tutorial, you can use the profile into which you have previously imported this data or import the archive now.
To do so, you can copy-paste the following code into a terminal after activating your AiiDA virtual environment:

```{code-block} console
$ verdi quicksetup --profile data
$ verdi profile setdefault data
$ verdi import https://object.cscs.ch/v1/AUTH_b1d80408b3d340db9f03d373bbde5c1e/marvel-vms/tutorials/aiida_tutorial_2020_07_perovskites_v0.9.aiida
verdi quicksetup --profile data
verdi profile setdefault data
verdi archive import https://object.cscs.ch/v1/AUTH_b1d80408b3d340db9f03d373bbde5c1e/marvel-vms/tutorials/aiida_tutorial_2020_07_perovskites_v0.9.aiida
```

## How to group nodes
Expand All @@ -23,10 +22,9 @@ A typical use case is to store all nodes that share a common property in a singl

### Listing existing groups

Lets explore the groups already present in the imported archive:
Let's explore the groups already present in the imported archive, by executing the following command:

```{code-block} console
$ verdi group list -a -A
PK Label Type string User
---- --------------- ------------- ---------------
Expand All @@ -37,28 +35,26 @@ PK Label Type string User
5 GBRV_pbesol core.upf aiida@localhost
6 GBRV_lda core.upf aiida@localhost
7 20200705-071658 core.import aiida@localhost
```

The default table shows us four pieces of information:

* **PK**: The Primary Key of the group
* **Label**: The label by which the group can be identified
* **Type string**: This tells us what "sub-class" of group this is.
* **PK**: The Primary Key of the group.
* **Label**: The label by which the group can be identified.
* **Type string**: This tells us what type of group this is.
Type strings can be used to class certain types of data, for example here we have general groups (`core`), groups containing pseudopotentials (`core.upf`), and an auto-generated group containing the nodes we imported from the archive (`core.import`).
For advanced use, you can create your own group subclass plugins, with specialised methods.
For advanced use, you can create your own group type plugins, with specialised methods by sub-classing the general `Group` class.
* **User**: The email of the user that created this group.

:::{tip}

The `-a` and `-A` flags we used above ensure that groups for all type strings and users are shown respectively.
The `-a` and `-A` flags used above ensure that groups for *all* type strings and users are shown, respectively.

:::

We can then inspect the content of a group by its label (if it is unique) or the PK:
You can then inspect the content of a group by its label (if it is unique) or the PK:

```{code-block} console
$ verdi group show tutorial_pbesol
----------------- ----------------
Group label tutorial_pbesol
Expand All @@ -71,54 +67,44 @@ PK Type Created
380 CalcJobNode 2078D:17h:46m ago
1273 CalcJobNode 2078D:18h:03m ago
...
```

Conversely, if you want to see all the groups a node belongs to, you can run:
Conversely, if you want to see all the groups a node belongs to, you can use its PK and run, (where `<PK>` should be the PK of the node):

```{code-block} console
$ verdi group list -a -A --node 380
$ verdi group list -a -A --node <PK>
PK Label Type string User
---- --------------- ------------- ---------------
1 tutorial_pbesol core aiida@localhost
7 20200705-071658 core.import aiida@localhost
```

### Creating and manipulating groups

Lets make a new group:
Let's make a new group:

```{code-block} console
$ verdi group create a_group
Success: Group created with PK = 8 and name 'a_group'
```

If we want to change the name of the group at any time:
You can change the name of a group using the `verdi group relabel` command:

```{code-block} console
$ verdi group relabel a_group my_group
Success: Label changed to my_group
```

Now we can add one or more nodes to it:
Add one or more nodes to your new group using node PKs from the `tutorial_pbesol` group we inspected earlier:

```{code-block} console
$ verdi group add-nodes -G my_group 380 1273
Do you really want to add 2 nodes to Group<my_group>? [y/N]: y
```

We can also copy the nodes from an existing group to another group:
You can also copy *all* nodes from an existing group to another group using:

```{code-block} console
$ verdi group copy tutorial_pbesol my_group
Warning: Destination group<my_group> already exists and is not empty.
Do you wish to continue anyway? [y/N]: y
Expand All @@ -135,31 +121,26 @@ PK Type Created
74 CalcJobNode 2078D:17h:51m ago
76 CalcJobNode 2078D:17h:57m ago
...
```

To remove nodes from the group run:
Removing nodes from the group is similar to adding them, try:

```{code-block} console
$ verdi group remove-nodes -G my_group 74
Do you really want to remove 1 nodes from Group<my_group>? [y/N]: y
```

and finally to remove the group entirely:

```{code-block} console
$ verdi group delete --clear my_group
Are you sure to delete Group<my_group>? [y/N]: y
Success: Group<my_group> deleted.
```

:::{important}

Any deletion operation related to groups won't affect the nodes themselves.
Any deletion operation related to groups will not affect the nodes themselves.
For example if you delete a group, the nodes that belonged to the group will remain in the database.
The same happens if you remove nodes from the group -- they will remain in the database but won't belong to the group any more.

Expand All @@ -168,65 +149,57 @@ The same happens if you remove nodes from the group -- they will remain in the d
## Organising groups in hierarchies

Earlier we mentioned that groups are like files in folders on your filesystem.
As with folders and sub-folders then, as the amount of groups we have grows, we may also wish to structure our groups in a hierarchy.
Groups in AiiDA are inherently "flat", in that groups may only contain nodes and not other groups.
However it is possible to construct *virtual* group hierarchies based on delimited group labels, using the `grouppath` utility.
Similar to folders and sub-folders, you may wish to structure your groups in a hierarchy once the number of groups in your database grows larger.
Groups in AiiDA are inherently "flat", meaning groups may only contain nodes and not other groups.
However, it is possible to construct *virtual* group hierarchies based on delimited group labels, using the `grouppath` utility.

Like folder paths grouppath requires delimitation by `/` (forward slash) characters.
Lets copy and rename the three tutorial groups:
Like folder paths on Unix systems, `grouppath` requires delimitation by forward slash (`/`) characters.
Let's copy and rename the three tutorial groups:

```{code-block} console
$ verdi group copy tutorial_lda tutorial/lda/basic
$ verdi group copy tutorial_pbe tutorial/gga/pbe
$ verdi group copy tutorial_pbesol tutorial/gga/pbesol
verdi group copy tutorial_lda tutorial/lda/basic
verdi group copy tutorial_pbe tutorial/gga/pbe
verdi group copy tutorial_pbesol tutorial/gga/pbesol
```

We can now list the groups in a new way:
You can now list the groups in a new way:

```{code-block} console
$ verdi group path ls -l
Path Sub-Groups
--------------- ------------
tutorial 3
tutorial_lda 0
tutorial_pbe 0
tutorial_pbesol 0
```

:::{note}

In the terminal, paths that contain nodes are listed in bold
In the terminal, paths that contain nodes are listed in bold.

:::

You can see that the actual groups that we create do not show, only the initial part of the "path", and how many sub-groups that path contains.
We can then step into a path:
You can see that the actual groups we just created do not appear, only the initial part of the path (`tutorial`), and how many subgroups that path contains.
You can then step into a path:

```{code-block} console
$ verdi group path ls -l tutorial
Path Sub-Groups
------------ ------------
tutorial/gga 2
tutorial/lda 1
```

This feature is also particularly useful in the verdi shell:
This feature is also particularly useful in the `verdi shell`:

```{code-block} ipython
In [1]: from aiida.tools.groups import GroupPath
In [2]: for subpath in GroupPath("tutorial/gga").walk(return_virtual=False):
...: print(subpath.get_group())
...:
"tutorial/gga/pbesol" [type core], of user aiida@localhost
"tutorial/gga/pbe" [type core], of user aiida@localhost
```

:::{seealso}
Expand Down

0 comments on commit 0c32062

Please sign in to comment.