Implement Memory Limits for User Jobs on Compute Nodes

[andriy-safe-ai]

We need to set memory limits to prevent users from over-allocating RAM on our compute nodes. Recently, we’ve experienced instances where compute nodes become unresponsive when a job consumes all available memory.

To address this, we can configure memory limits using the RealMemory and MemSpecLimit options in the /etc/slurm/slurm.conf file for our nodelist. This will ensure that users are unable to request more memory than a node can safely allocate, improving node stability and preventing system crashes.