Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allocation resources usage showing zero for rootless raw_exec driver in v1.8.0 in Linux #23425

Closed
Settler opened this issue Jun 24, 2024 · 2 comments

Comments

@Settler
Copy link

Settler commented Jun 24, 2024

Nomad version

Nomad v1.8.0
BuildDate 2024-05-28T17:38:17Z
Revision 28b82e4

Operating system and Environment details

Debian 11, raw_exec, cgroupV2

Issue

This issue possible related to #20285, but we think this problem is different.
We use nomad with raw_exec module without root (yes, we know that this is unrecommended setup). We use raw_exec because exec module has too high overhead with fs isolation. For example, if Nvidia CUDA is installed exec module copies over 10Gb of CUDA library into every task. On the other side use raw_exec with root access isn't secure enough.

So... raw_exec module can use cgroupV2 and create group for every task. We modified Nomad systemd service in this way:

[Service]
User=nomad
Group=nomad
Slice=nomad.slice
EnvironmentFile=-/etc/nomad.d/nomad.env
ExecReload=/bin/kill -HUP $MAINPID
ExecStartPre=+chown -R nomad:nomad /sys/fs/cgroup/nomad.slice
ExecStart=/usr/bin/nomad agent -config /etc/nomad.d
Delegate=yes

With that setup Nomad process under nomad user has full rights to nomad.slice, can manage resources and read stats.
In Nomad 1.7.7 everything is working perfectly. But Nomad 1.8.0 reports zero values for allocation resource usage.

Reproduction steps

Start nomad in Linux environment without root access with example systemd setup. Start any job with raw_exec driver.

Expected Result

Reported actual allocation resource usage.

Actual Result

Reported zeros for allocation resource usage.

@tgross
Copy link
Member

tgross commented Jun 24, 2024

Hi @Settler! This is a known issue, so I'm going to close this issue as a duplicate of #19828.

@tgross tgross closed this as not planned Won't fix, can't repro, duplicate, stale Jun 24, 2024
@github-project-automation github-project-automation bot moved this from Needs Triage to Done in Nomad - Community Issues Triage Jun 24, 2024
Copy link

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Dec 27, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
Development

No branches or pull requests

2 participants