From b6c03407840f87d164c9a8551ed16e0e0d943cb2 Mon Sep 17 00:00:00 2001 From: Blake Devcich Date: Mon, 9 Dec 2024 11:23:31 -0600 Subject: [PATCH 1/2] Update storage profiles docs for PostMount variables Signed-off-by: Blake Devcich --- docs/guides/storage-profiles/readme.md | 112 ++++++++++++++----------- external/nnf-dm | 2 +- 2 files changed, 66 insertions(+), 48 deletions(-) diff --git a/docs/guides/storage-profiles/readme.md b/docs/guides/storage-profiles/readme.md index 5aef7dc..7af9eda 100644 --- a/docs/guides/storage-profiles/readme.md +++ b/docs/guides/storage-profiles/readme.md @@ -16,13 +16,15 @@ DW directives that allocate storage on Rabbit nodes allow a `profile` parameter The administrator shall choose one profile to be the default profile that is used when a profile parameter is not specified. -# Specifying a Profile +## Specifying a Profile + To specify a profile name on a #DW directive, use the `profile` option -``` + +```shell #DW jobdw type=lustre profile=durable capacity=5GB name=example ``` -# Setting A Default Profile +## Setting A Default Profile A default profile must be defined at all times. Any #DW line that does not specify a profile will use the default profile. If a default profile is not defined, then any new workflows will be rejected. If more than one profile is marked as default then any new workflows will be rejected. @@ -36,16 +38,18 @@ nnf-system performance false 6s ``` To set the default flag on a profile + ```shell -$ kubectl patch nnfstorageprofile performance -n nnf-system --type merge -p '{"data":{"default":true}}' +kubectl patch nnfstorageprofile performance -n nnf-system --type merge -p '{"data":{"default":true}}' ``` To clear the default flag on a profile + ```shell -$ kubectl patch nnfstorageprofile durable -n nnf-system --type merge -p '{"data":{"default":false}}' +kubectl patch nnfstorageprofile durable -n nnf-system --type merge -p '{"data":{"default":false}}' ``` -# Creating The Initial Default Profile +## Creating The Initial Default Profile Create the initial default profile from scratch or by using the [NnfStorageProfile/template](https://github.com/NearNodeFlash/nnf-sos/blob/master/config/examples/nnf_v1alpha1_nnfstorageprofile.yaml) resource as a template. If `nnf-deploy` was used to install nnf-sos then the default profile described below will have been created automatically. @@ -87,13 +91,13 @@ nnf-system template false 11s The administrator should edit the `default` profile to record any cluster-specific settings. Maintain a copy of this resource YAML in a safe place so it isn't lost across upgrades. -## Keeping The Default Profile Updated +### Keeping The Default Profile Updated An upgrade of nnf-sos may include updates to the `template` profile. It may be necessary to manually copy these updates into the `default` profile. -# Profile Parameters +## Profile Parameters -## XFS +### XFS The following shows how to specify command line options for pvcreate, vgcreate, lvcreate, and mkfs for XFS storage. Optional mount options are specified one per line @@ -118,8 +122,7 @@ data: [...] ``` - -## GFS2 +### GFS2 The following shows how to specify command line options for pvcreate, lvcreate, and mkfs for GFS2. @@ -140,7 +143,7 @@ data: [...] ``` -## Lustre / ZFS +### Lustre / ZFS The following shows how to specify a zpool virtual device (vdev). In this case the default vdev is a stripe. See [zpoolconcepts(7)](https://openzfs.github.io/openzfs-docs/man/7/zpoolconcepts.7.html) for virtual device descriptions. @@ -168,7 +171,7 @@ data: [...] ``` -### ZFS dataset properties +#### ZFS dataset properties The following shows how to specify ZFS dataset properties in the `--mkfsoptions` arg for mkfs.lustre. See [zfsprops(7)](https://openzfs.github.io/openzfs-docs/man/7/zfsprops.7.html). @@ -188,9 +191,10 @@ data: [...] ``` -### Mount Options for Targets +#### Mount Options for Targets + +##### Persistent Mount Options -#### Persistent Mount Options Use the mkfs.lustre `--mountfsoptions` parameter to set persistent mount options for Lustre targets. ```yaml @@ -209,7 +213,8 @@ data: [...] ``` -#### Non-Persistent Mount Options +##### Non-Persistent Mount Options + Non-persistent mount options can be specified with the ostOptions.mountTarget parameter to the NnfStorageProfile: ```yaml @@ -232,7 +237,7 @@ data: [...] ``` -### Target Layout +#### Target Layout Users may want Lustre file systems with different performance characteristics. For example, a user job with a single compute node accessing the Lustre file system would see acceptable performance from a single OSS. An FPP workload might want as many OSSs as posible to avoid contention. @@ -269,7 +274,7 @@ data: count: 10 ``` -#### Example Layouts +##### Example Layouts `scale` with `colocateComputes=true` will likely be the most common layout type to use for `jobdw` directives. This will result in a Lustre file system whose performance scales with the number of compute nodes in the job. @@ -281,47 +286,60 @@ The `count` field may be useful when creating a persistent file system since the In general, `scale` gives a simple way for users to get a filesystem that has performance consistent with their job size. `count` is useful for times when a user wants full control of the file system layout. -# Command Line Variables +## Command Line Variables + +### pvcreate + +- `$DEVICE` - expands to the `/dev/` value for one device that has been allocated + +### vgcreate + +- `$VG_NAME` - expands to a volume group name that is controlled by Rabbit software. +- `$DEVICE_LIST` - expands to a list of space-separated `/dev/` devices. This list will contain the devices that were iterated over for the pvcreate step. -## pvcreate +### lvcreate -* `$DEVICE` - expands to the `/dev/` value for one device that has been allocated +- `$VG_NAME` - see vgcreate above. +- `$LV_NAME` - expands to a logical volume name that is controlled by Rabbit software. +- `$DEVICE_NUM` - expands to a number indicating the number of devices allocated for the volume group. +- `$DEVICE1, $DEVICE2, ..., $DEVICEn` - each expands to one of the devices from the `$DEVICE_LIST` above. -## vgcreate +### XFS mkfs -* `$VG_NAME` - expands to a volume group name that is controlled by Rabbit software. -* `$DEVICE_LIST` - expands to a list of space-separated `/dev/` devices. This list will contain the devices that were iterated over for the pvcreate step. +- `$DEVICE` - expands to the `/dev/` value for the logical volume that was created by the lvcreate step above. -## lvcreate +### GFS2 mkfs -* `$VG_NAME` - see vgcreate above. -* `$LV_NAME` - expands to a logical volume name that is controlled by Rabbit software. -* `$DEVICE_NUM` - expands to a number indicating the number of devices allocated for the volume group. -* `$DEVICE1, $DEVICE2, ..., $DEVICEn` - each expands to one of the devices from the `$DEVICE_LIST` above. +- `$DEVICE` - expands to the `/dev/` value for the logical volume that was created by the lvcreate step above. +- `$CLUSTER_NAME` - expands to a cluster name that is controlled by Rabbit Software +- `$LOCK_SPACE` - expands to a lock space key that is controlled by Rabbit Software. +- `$PROTOCOL` - expands to a locking protocol that is controlled by Rabbit Software. -## XFS mkfs +### zpool create -* `$DEVICE` - expands to the `/dev/` value for the logical volume that was created by the lvcreate step above. +- `$DEVICE_LIST` - expands to a list of space-separated `/dev/` devices. This list will contain the devices that were allocated for this storage request. +- `$POOL_NAME` - expands to a pool name that is controlled by Rabbit software. +- `$DEVICE_NUM` - expands to a number indicating the number of devices allocated for this storage request. +- `$DEVICE1, $DEVICE2, ..., $DEVICEn` - each expands to one of the devices from the `$DEVICE_LIST` above. -## GFS2 mkfs +### lustre mkfs -* `$DEVICE` - expands to the `/dev/` value for the logical volume that was created by the lvcreate step above. -* `$CLUSTER_NAME` - expands to a cluster name that is controlled by Rabbit Software -* `$LOCK_SPACE` - expands to a lock space key that is controlled by Rabbit Software. -* `$PROTOCOL` - expands to a locking protocol that is controlled by Rabbit Software. +- `$FS_NAME` - expands to the filesystem name that was passed to Rabbit software from the workflow's #DW line. +- `$MGS_NID` - expands to the NID of the MGS. If the MGS was orchestrated by nnf-sos then an appropriate internal value will be used. +- `$POOL_NAME` - see zpool create above. +- `$VOL_NAME` - expands to the volume name that will be created. This value will be `/`, and is controlled by Rabbit software. +- `$INDEX` - expands to the index value of the target and is controlled by Rabbit software. -## zpool create +### PostMount/PreUnmount and PostActivate/PreDeactivate -* `$DEVICE_LIST` - expands to a list of space-separated `/dev/` devices. This list will contain the devices that were allocated for this storage request. -* `$POOL_NAME` - expands to a pool name that is controlled by Rabbit software. -* `$DEVICE_NUM` - expands to a number indicating the number of devices allocated for this storage request. -* `$DEVICE1, $DEVICE2, ..., $DEVICEn` - each expands to one of the devices from the `$DEVICE_LIST` above. +- `$MOUNT_PATH` - expands to the mount path of the fileystem to perform certain actions on the mounted filesystem -## lustre mkfs +#### Lustre Specific -* `$FS_NAME` - expands to the filesystem name that was passed to Rabbit software from the workflow's #DW line. -* `$MGS_NID` - expands to the NID of the MGS. If the MGS was orchestrated by nnf-sos then an appropriate internal value will be used. -* `$POOL_NAME` - see zpool create above. -* `$VOL_NAME` - expands to the volume name that will be created. This value will be `/`, and is controlled by Rabbit software. -* `$INDEX` - expands to the index value of the target and is controlled by Rabbit software. +These variables are for lustre only and can be used to perform PostMount activities such are setting lustre striping. +- `$NUM_MDTS` - expands to the number of MDTs for the lustre filesystem +- `$NUM_MGTS` - expands to the number of MGTs for the lustre filesystem +- `$NUM_MGTMDTS` - expands to the number of combined MGTMDTs for the lustre filesystem +- `$NUM_OSTS` - expands to the number of OSTs for the lustre filesystem +- `$NUM_NNFNODES` - expands to the number of NNF Nodes for the lustre filesystem diff --git a/external/nnf-dm b/external/nnf-dm index 0f4cd5b..5766e13 160000 --- a/external/nnf-dm +++ b/external/nnf-dm @@ -1 +1 @@ -Subproject commit 0f4cd5bd180ebbb6bea151e1454139bbf35d5448 +Subproject commit 5766e13119f74959c006147c96ef06f668746ecd From 56b612a69cda540be6311b051bf74f7c3cd5239f Mon Sep 17 00:00:00 2001 From: Dean Roehrich Date: Mon, 9 Dec 2024 15:20:44 -0600 Subject: [PATCH 2/2] Revendor nnf-dm (#223) Signed-off-by: Dean Roehrich --- external/nnf-dm | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/external/nnf-dm b/external/nnf-dm index 5766e13..de325a4 160000 --- a/external/nnf-dm +++ b/external/nnf-dm @@ -1 +1 @@ -Subproject commit 5766e13119f74959c006147c96ef06f668746ecd +Subproject commit de325a4553c17c718e6f63018155e7e952eb46a0