Buildx node preferences: Docker doesn't appear to be using the configured remote node

[marcelstoer]

This is in many ways strongly related to my unanswered post at https://forums.docker.com/t/docker-buildx-build-which-node-is-used-for-which-platform/135525.

I have a Buildx QEMU setup on a Linux x64 machine to which I added an Arm Mac Mini over SSH. The intention is to build linux/amd64 on this machine but build linux/arm64 on the Mac Mini.

$ docker buildx ls
NAME/NODE  DRIVER/ENDPOINT             STATUS   BUILDKIT PLATFORMS
builder    docker-container                              
  builder0 unix:///var/run/docker.sock inactive          
maven *    docker-container                              
  maven0   unix:///var/run/docker.sock running  v0.11.0  linux/amd64*, linux/amd64/v2, linux/arm64, ....
  mac-mini ssh://[email protected]    running  v0.11.5  linux/arm64*, linux/amd64, linux/amd64/v2, ...

From why understand the '*' denotes the preferred or recommended platform for the builder. I set them when I appended the node(s).

Even though I don't have hard evidence yet (-> subject of the forum post) I have strong indication that a Buildx build for platform linux/arm64 uses the QEMU infrastructure rather than the remote node over SSH. Even with demanding Docker builds the CPU on the Mac Mini shows no indication anything special is happening there.

Do I really need https://github.com/tonistiigi/binfmt from @tonistiigi in my case on the "local" machine?

[marcelstoer]

I meanwhile realized what a kinda sensible way to monitor those Buildx nodes is.

However, in order to get Docker to distribute the Buildx builds to the platform nodes as expected (linux/amd64 locally and linux/arm64 on the Mac) I needed to fix the setup.

• Delete the "default" node that was created when I initially created the runner: $ docker buildx create --name maven --node maven0 --leave.
• Recreate it but specify the desired platform explicitly: docker buildx create --name maven --node maven0 --platform linux/amd64 --append

Why that was necessary I have no idea.

When I now start a multi-platform build, I see it running on both nodes - most of the time.

However, there are situations when an Arm build is still executed on the local host through QEMU instead of on the Mini as the following screen shot shows.

Again, I have no idea why this might be.

[tonistiigi]

What version of buildx are you using?

[marcelstoer]

$ docker buildx version
github.com/docker/buildx v0.10.4 c513d34

[tonistiigi]

However, in order to get Docker to distribute the Buildx nodes to the platforms as expected

Is there a difference in buildx ls output when you use this compared to what you had before?

[marcelstoer]

Not that I could tell.

[marcelstoer]

Is there a way I could tell Buildx not to try to be clever when creating nodes in terms of detecting which platforms a node may support? In other words, when I say create --platform linux/amd64 that this becomes the only platform this node supports. Currently this appears to be interpreted as "the preferred platform" for this node rather than "the supported platform".

If I could change the configuration like that, it should stop Buildx from sending jobs to the wrong node.