waypoint runner fails to build docker image with nsenter: failed to unshare namespaces: Operation not permitted

[arbll]

Describe the bug

The Waypoint runner seem to fail to build my docker image with the following error:

Building...
nsenter: failed to unshare namespaces: Operation not permitted
exit status 20

Dockerfile

FROM node:lts-alpine
RUN npm install -g http-server
WORKDIR /app
COPY package*.json ./
RUN npm install
COPY . .
RUN npm run build
EXPOSE 8080
CMD [ "http-server", "dist", "-p", "8080" ]

I'm running the waypoint server and runner in a nomad cluster. The underlying OS is Ubuntu 20.04.

I guess it's most likely an issue with either my OS preventing https://github.com/genuinetools/img to run correctly because of some security mechanism or an issue with the way the runner use img

Waypoint Platform Versions
0.3.0

[briancain]

Hey there @arbll - Can you provide us with the runner logs for when it failed? Were you running Waypoint through the UI when you got this error, or the CLI? Thank you!

[mitchellh]

This error is caused when user namespaces aren't enabled on the underlying host OS. See the docs here: https://www.waypointproject.io/plugins/docker#dockerless-builds Namely:

Dockerless builds require user namespaces to be enabled. This is a host-level setting that is often not enabled by default. For GKE, you must not use ContainerOS. For AKS (Azure) and EKS (AWS), you must use a custom AMI that has user namespaces enabled. Please search for your distro how to enable user namespaces, it is usually a single line configuration.

I'm not sure if we can do anything more in this case, maybe we can detect this error string and link to this section.

[briancain]

Detecting the error seems good. I imagine this will be a common issue people run into, so having a good error message here seems like a good first step 👍🏻

[arbll]

Hey @mitchellh & @briancain , thanks for the quick reply.

I was suspecting the same thing but I became pretty convinced that user namespaces were correctly enabled (by default) on my ubuntu 20.04 install:

❯ sysctl kernel.unprivileged_userns_clone
kernel.unprivileged_userns_clone = 1

I have looked a bit closer at issues from the img repo and stumbled on genuinetools/img#305 which could match.

The img docs mention that to build from inside a docker container you need the following flags: --security-opt seccomp=unconfined --security-opt apparmor=unconfined
(See https://github.com/genuinetools/img#running-with-docker)

And indeed:

~
❯ sudo docker run -it r.j3ss.co/img pull ubuntu                                                           
nsenter: failed to unshare namespaces: Operation not permitted

~
❯ sudo docker run -it \                                                                                    
--security-opt seccomp=unconfined \
--security-opt apparmor=unconfined \
r.j3ss.co/img pull ubuntu
Pulling ubuntu...
Pulled: sha256:ae559937be68cd1b7ee05d8db586d630d64e22ecf1e203f087288f5f1211b46e
Size: 27.25MiB                                                                                                       

I wonder if this is what could be happening here ?

Note that I am not familiar at all with what --security-opt seccomp=unconfined --security-opt apparmor=unconfined entails so I might be spewing nonsense and they might just allow me to bypass user namespace constraints 😅

[mitchellh]

That could certainly be true, but we do launch the runner (in Docker) with that set:

waypoint/internal/serverinstall/docker.go

Lines 654 to 661 in 7376b07

	}, &container.HostConfig{
	// These security options are required for the runner so that
	// Docker daemonless image building works properly.
	SecurityOpt: []string{
	"seccomp=unconfined",
	"apparmor=unconfined",
	},
	}, &network.NetworkingConfig{

:)

[mitchellh]

Ah, you're running in Nomad! Maybe there is a way to set those on Nomad, I don't think we do.

[mitchellh]

Yep: https://www.nomadproject.io/docs/drivers/docker#security_opt

I think we need to set these.

[mitchellh]

I tried to simply add those flags and I'm getting failures starting the runner in Nomad for some reason. Ill have to look further another time.

[arbll]

Thanks for the investigation @mitchellh.
I also tried hacking something by setting those parameters as plugin config in my nomad clients but it looks like that's not possible unfortunately

I'll monitor the issue, have a good weekend

[that-guy-iain]

Hey, just want to confirm that this is a bug that will be fixed in a future version and there is nothing I can do to fix it just now, right?

[mitchellh]

Kind of yes. Waypoint 0.6 switched to using on-demand runners for everything which no longer has this issue, but we're adding Nomad support right now (PR open already and likely merging any day now). Once that is merged, that will fix this because we no longer need a privileged environment to run Docker builds.

[briancain]

I'm going to go ahead and close this! We've shipped on-demand runners with Nomad, so we should be good.