Merge branch 'main' into release-v0.1.6

Signed-off-by: Anthony Floeder <[email protected]> # Conflicts: # external/nnf-dm
NearNodeFlash · Aug 22, 2024 · e2b3971 · e2b3971
2 parents 0030f36 + 4398272
commit e2b3971
Show file tree

Hide file tree

Showing 6 changed files with 288 additions and 8 deletions.
diff --git a/docs/guides/external-mgs/readme.md b/docs/guides/external-mgs/readme.md
@@ -17,6 +17,7 @@ These three methods are not mutually exclusive on the system as a whole. Individ
 
 ## Configuration with an External MGT
 
+### Storage Profile
 An existing MGT external to the NNF cluster can be used to manage the Lustre file systems on the NNF nodes. An advantage to this configuration is that the MGT can be highly available through multiple MGSs. A disadvantage is that there is only a single MGT. An MGT shared between more than a handful of Lustre file systems is not a common use case, so the Lustre code may prove less stable.
 
 The following yaml provides an example of what the `NnfStorageProfile` should contain to use an MGT on an external server.
@@ -30,12 +31,49 @@ metadata:
 data:
 [...]
   lustreStorage:
-    externalMgs: 1.2.3.4@eth0
+    externalMgs: 1.2.3.4@eth0:1.2.3.5@eth0
     combinedMgtMdt: false
     standaloneMgtPoolName: ""
 [...]
 ```
 
+### NnfLustreMGT
+
+A `NnfLustreMGT` resource tracks which fsnames have been used on the MGT to prevent fsname re-use. Any Lustre file systems that are created through the NNF software will request an fsname to use from a `NnfLustreMGT` resource. Every MGT must have a corresponding `NnfLustreMGT` resource. For MGTs that are hosted on NNF hardware, the `NnfLustreMGT` resources are created automatically. The NNF software also erases any unused fsnames from the MGT disk for any internally hosted MGTs.
+
+For a MGT hosted on an external node, an admin must create an `NnfLustreMGT` resource. This resource ensures that fsnames will be created in a sequential order without any fsname re-use. However, after an fsname is no longer in use by a file system, it will not be erased from the MGT disk. An admin may decide to periodically run the `lctl erase_lcfg [fsname]` command to remove fsnames that are no longer in use.
+
+Below is an example `NnfLustreMGT` resource. The `NnfLustreMGT` resource for external MGSs must be created in the `nnf-system` namespace.
+
+```yaml
+apiVersion: nnf.cray.hpe.com/v1alpha1
+kind: NnfLustreMGT
+metadata:
+  name: external-mgt
+  namespace: nnf-system
+spec:
+  addresses:
+  - "1.2.3.4@eth0:1.2.3.5@eth0"
+  fsNameStart: "aaaaaaaa"
+  fsNameBlackList:
+  - "mylustre"
+  fsNameStartReference:
+    name: external-mgt
+    namespace: default
+    kind: ConfigMap
+```
+
+* `addresses` - This is a list of LNet addresses that could be used for this MGT. This should match any values that are used in the `externalMgs` field in the `NnfStorageProfiles`.
+* `fsNameStart` - The first fsname to use. Subsequent fsnames will be incremented based on this starting fsname (e.g, `aaaaaaaa`, `aaaaaaab`, `aaaaaaac`). fsnames use lowercase letters `'a'`-`'z'`. `fsNameStart` should be exactly 8 characters long.
+* `fsNameBlackList` - This is a list of fsnames that should not be given to any NNF Lustre file systems. If the MGT is hosting any non-NNF Lustre file systems, their fsnames should be included in this blacklist.
+* `fsNameStartReference` - This is an optional `ObjectReference` to a `ConfigMap` that holds a starting fsname. If this field is specified, it takes precedence over the `fsNameStart` field in the spec. The `ConfigMap` will be updated to the next available fsname every time an fsname is assigned to a new Lustre file system.
+
+### ConfigMap
+
+For external MGTs, the `fsNameStartReference` should be used to point to a `ConfigMap` in the `default` namespace. The `ConfigMap` should be left empty initially. The `ConfigMap` is used to hold the value of the next available fsname, and it should not be deleted or modified while a `NnfLustreMGT` resource is referencing it. Removing the `ConfigMap` will cause the Rabbit software to lose track of which fsnames have already been used on the MGT. This is undesireable unless the external MGT is no longer being used by Rabbit software or if an admin has erased all previously used fsnames with the `lctl erase_lcfg [fsname]` command.
+
+When using the `ConfigMap`, the nnf-sos software may be undeployed and redeployed without losing track of the next fsname value. During an undeploy, the `NnfLustreMGT` resource will be removed. During a deploy, the `NnfLustreMGT` resource will read the fsname value from the `ConfigMap` if it is present. The value in the `ConfigMap` will override the fsname in the `fsNameStart` field.
+
 ## Configuration with Persistent Lustre
 
 The MGT from a persistent Lustre file system hosted on the NNF nodes can also be used as the MGT for other NNF Lustre file systems. This configuration has the advantage of not relying on any hardware outside of the cluster. However, there is no high availability, and a single MGT is still shared between all Lustre file systems created on the cluster.

diff --git a/docs/guides/index.md b/docs/guides/index.md
@@ -16,12 +16,13 @@
 * [Lustre External MGT](external-mgs/readme.md)
 * [Global Lustre](global-lustre/readme.md)
 * [Directive Breakdown](directive-breakdown/readme.md)
+* [User Interactions](user-interactions/readme.md)
 
 ## NNF User Containers
 
 * [User Containers](user-containers/readme.md)
 
 ## Node Management
 
-* [Draining A Node](node-management/drain.md)
+* [Disable or Drain a Node](node-management/drain.md)
 * [Debugging NVMe Namespaces](node-management/nvme-namespaces.md)
diff --git a/docs/guides/node-management/drain.md b/docs/guides/node-management/drain.md
@@ -1,4 +1,40 @@
-# Draining A Node
+# Disable Or Drain A Node
+
+## Disabling a node
+
+A Rabbit node can be manually disabled, indicating to the WLM that it should not schedule more jobs on the node. Jobs currently on the node will be allowed to complete at the discretion of the WLM.
+
+Disable a node by setting its Storage state to `Disabled`.
+
+```shell
+kubectl patch storage $NODE --type=json -p '[{"op":"replace", "path":"/spec/state", "value": "Disabled"}]'
+```
+
+When the Storage is queried by the WLM, it will show the disabled status.
+
+```console
+$ kubectl get storages
+NAME           STATE      STATUS     MODE   AGE
+kind-worker2   Enabled    Ready      Live   10m
+kind-worker3   Disabled   Disabled   Live   10m
+```
+
+To re-enable a node, set its Storage state to `Enabled`.
+
+```shell
+kubectl patch storage $NODE --type=json -p '[{"op":"replace", "path":"/spec/state", "value": "Enabled"}]'
+```
+
+The Storage state will show that it is enabled.
+
+```console
+kubectl get storages
+NAME           STATE     STATUS   MODE   AGE
+kind-worker2   Enabled   Ready    Live   10m
+kind-worker3   Enabled   Ready    Live   10m
+```
+
+## Draining a node
 
 The NNF software consists of a collection of DaemonSets and Deployments. The pods
 on the Rabbit nodes are usually from DaemonSets. Because of this, the `kubectl drain`
@@ -9,7 +45,11 @@ Given the limitations of DaemonSets, the NNF software will be drained by using t
 as described in
 [Taints and Tolerations](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/).
 
-## Drain NNF Pods From A Rabbit Node
+This would be used only after the WLM jobs have been removed from that Rabbit (preferably) and there is some reason to also remove the NNF software from it. This might be used before a Rabbit is powered off and pulled out of the cabinet, for example, to avoid leaving pods in "Terminating" state (harmless, but it's noise).
+
+If an admin used this taint before power-off it would mean there wouldn't be "Terminating" pods lying around for that Rabbit. After a new/same Rabbit is put back in its place, the NNF software won't jump back on it while the taint is present. The taint can be removed at any time, from immediately after the node is powered off up to some time after the new/same Rabbit is powered back on.
+
+### Drain NNF pods from a rabbit node
 
 Drain the NNF software from a node by applying the `cray.nnf.node.drain` taint.
 The CSI driver pods will remain on the node to satisfy any unmount requests from k8s
@@ -19,15 +59,33 @@ as it cleans up the NNF pods.
 kubectl taint node $NODE cray.nnf.node.drain=true:NoSchedule cray.nnf.node.drain=true:NoExecute
 ```
 
+This will cause the node's `Storage` resource to be drained:
+
+```console
+$ kubectl get storages
+NAME           STATE     STATUS    MODE   AGE
+kind-worker2   Enabled   Drained   Live   5m44s
+kind-worker3   Enabled   Ready     Live   5m45s
+```
+
+The `Storage` resource will contain the following message indicating the reason it has been drained:
+
+```console
+$ kubectl get storages rabbit1 -o json | jq -rM .status.message
+Kubernetes node is tainted with cray.nnf.node.drain
+```
+
 To restore the node to service, remove the `cray.nnf.node.drain` taint.
 
 ```shell
 kubectl taint node $NODE cray.nnf.node.drain-
 ```
 
-## The CSI Driver
+The `Storage` resource will revert to a `Ready` status.
+
+### The CSI driver
 
-While the CSI driver pods may be drained from a Rabbit node, it is advisable not to do so.
+While the CSI driver pods may be drained from a Rabbit node, it is inadvisable to do so.
 
 **Warning** K8s relies on the CSI driver to unmount any filesystems that may have
 been mounted into a pod's namespace. If it is not present when k8s is attempting