Lustre FS name is being recycled

[roehrich-hpe]

We've seen a case where the Lustre FS name is being recycled.

Code in createNnfStorage() is using the first 7 chars from the Servers resource UID to create the unique FS name:
https://github.com/NearNodeFlash/nnf-sos/blob/master/internal/controller/nnf_workflow_controller_helpers.go#L638

Unfortunately, after creating 1758 filesystems k8s created a Servers resource with a UID that shares those same first 7 chars from the earlier Servers resource UID and we collided with an old name.

The Lustre MGS does not forget a name, and can't be told to forget a name. So after we teardown an ephemeral Lustre filesystem its name will live forever in the MGS.
There may be a mount option that tells the MGS to overwrite existing records that use the name, but we'd have to be certain the previous filesystem is really gone.

[behlendorf]

The Lustre MGS does not forget a name, and can't be told to forget a name.

It turns out this can be done. lctl erase_lcfg {fsname} deletes all the config llogs for the named file system (see LCTL-LCFG(8)). It must be run on the node where the MGS is running. There are no sanity checks to prevent one from deleting llogs while someone is still using the file system.