SLURM observability cookbook

[piotrrojek]

Summary

This PR contains a cookbook for SLURM observability stack.

---

For new content

When contributing new content, read through our contribution guidelines, and mark the following action items as completed:

• I have added a new entry in registry.yaml (and, optionally, in authors.yaml) so that my content renders on the cookbook website.
• I have conducted a self-review of my content based on the contribution guidelines:
  • Relevance: This content is related to building with Crusoe Cloud and is useful to others.
  • Uniqueness: I have searched for related examples in the Crusoe Cookbook, and verified that my content offers new insights or unique information compared to existing documentation.
  • Spelling and Grammar: I have checked for spelling or grammatical mistakes.
  • Clarity: I have done a final read-through and verified that my submission is well-organized and easy to understand.
  • Correctness: The information I include is correct and all of my code executes successfully.
  • Completeness: I have explained everything fully, including all necessary references and citations.

We will rate each of these areas on a scale from 1 to 4, and will only accept contributions that score 3 or higher on all areas. Refer to our contribution guidelines for more details.

[martin-cala1]

For commands that are required to be run on all compute nodes, can we recommend using a tool like pssh or clush ( https://clustershell.readthedocs.io/en/latest/tools/clush.html ) to make it easier to get started with large cluster management? Perhaps this warrants another mini section on setting up a hostfile for parallel VM management

[piotrrojek]

After internal discussion, we suggest to use Ansible for this. In fact, the SLURM observability role is submitted as a PR (I'm going to remove draft PR mode to ready to review today after final testing).
The reason we propose that is that Ansible is already used widely in Crusoe, it's also present in GPUd cookbook. In order not to overload the main SLURM repo we can extract it to be its own playbook too.