[BUG] Cluster creation fails on macOS and asks for docker login repeatedly

[sumeshpremraj]

What did you do

Tried to create a cluster with 1 server and 2 agents.

• How was the cluster created?
  k3d cluster create my-cluster --servers 1 --agents 2 --port 9080:80@loadbalancer --port 9443:443@loadbalancer --api-port 6443 --k3s-arg "--disable=traefik@server:*"

It fails with a docker failed to pull the image 'docker.io/rancher/k3s:v1.25.7-k3s1': Error response from daemon: Please run 'docker login' error, even though I had previously done docker login.

If I run a docker pull 'docker.io/rancher/k3s:v1.25.7-k3s1' before the cluster creation, it works.

Screenshots or terminal output

$ docker login
Authenticating with existing credentials...
Login Succeeded

$ k3d cluster create my-cluster --servers 1 --agents 2 --port 9080:80@loadbalancer --port 9443:443@loadbalancer --api-port 6443 --k3s-arg "--disable=traefik@server:*" --verbose
DEBU[0000] DOCKER_SOCK=/var/run/docker.sock
DEBU[0000] Runtime Info:
&{Name:docker Endpoint:/var/run/docker.sock Version:24.0.5 OSType:linux OS:Docker Desktop Arch:aarch64 CgroupVersion:2 CgroupDriver:cgroupfs Filesystem:extfs InfoName:docker-desktop}
DEBU[0000] Additional CLI Configuration:
cli:
  api-port: "6443"
  env: []
  k3s-node-labels: []
  k3sargs:
  - --disable=traefik@server:*
  ports:
  - 9080:80@loadbalancer
  - 9443:443@loadbalancer
  registries:
    create: ""
  runtime-labels: []
  volumes: []
hostaliases: []
DEBU[0000] Configuration:
agents: 2
image: docker.io/rancher/k3s:v1.25.7-k3s1
network: ""
options:
  k3d:
    disableimagevolume: false
    disableloadbalancer: false
    disablerollback: false
    loadbalancer:
      configoverrides: []
    timeout: 0s
    wait: true
  kubeconfig:
    switchcurrentcontext: true
    updatedefaultkubeconfig: true
  runtime:
    agentsmemory: ""
    gpurequest: ""
    hostpidmode: false
    serversmemory: ""
registries:
  config: ""
  use: []
servers: 1
subnet: ""
token: ""
DEBU[0000] ========== Simple Config ==========
{TypeMeta:{Kind:Simple APIVersion:k3d.io/v1alpha4} ObjectMeta:{Name:} Servers:1 Agents:2 ExposeAPI:{Host: HostIP: HostPort:} Image:docker.io/rancher/k3s:v1.25.7-k3s1 Network: Subnet: ClusterToken: Volumes:[] Ports:[] Options:{K3dOptions:{Wait:true Timeout:0s DisableLoadbalancer:false DisableImageVolume:false NoRollback:false NodeHookActions:[] Loadbalancer:{ConfigOverrides:[]}} K3sOptions:{ExtraArgs:[] NodeLabels:[]} KubeconfigOptions:{UpdateDefaultKubeconfig:true SwitchCurrentContext:true} Runtime:{GPURequest: ServersMemory: AgentsMemory: HostPidMode:false Labels:[]}} Env:[] Registries:{Use:[] Create:<nil> Config:} HostAliases:[]}
==========================
DEBU[0000] ========== Merged Simple Config ==========
{TypeMeta:{Kind:Simple APIVersion:k3d.io/v1alpha4} ObjectMeta:{Name:} Servers:1 Agents:2 ExposeAPI:{Host: HostIP:0.0.0.0 HostPort:6443} Image:docker.io/rancher/k3s:v1.25.7-k3s1 Network: Subnet: ClusterToken: Volumes:[] Ports:[{Port:9080:80 NodeFilters:[loadbalancer]} {Port:9443:443 NodeFilters:[loadbalancer]}] Options:{K3dOptions:{Wait:true Timeout:0s DisableLoadbalancer:false DisableImageVolume:false NoRollback:false NodeHookActions:[] Loadbalancer:{ConfigOverrides:[]}} K3sOptions:{ExtraArgs:[{Arg:--disable=traefik NodeFilters:[server:*]}] NodeLabels:[]} KubeconfigOptions:{UpdateDefaultKubeconfig:true SwitchCurrentContext:true} Runtime:{GPURequest: ServersMemory: AgentsMemory: HostPidMode:false Labels:[]}} Env:[] Registries:{Use:[] Create:<nil> Config:} HostAliases:[]}
==========================
INFO[0000] portmapping '9080:80' targets the loadbalancer: defaulting to [servers:*:proxy agents:*:proxy]
INFO[0000] portmapping '9443:443' targets the loadbalancer: defaulting to [servers:*:proxy agents:*:proxy]
DEBU[0000] generated loadbalancer config:
ports:
  80.tcp:
  - k3d-my-cluster-server-0
  - k3d-my-cluster-agent-0
  - k3d-my-cluster-agent-1
  443.tcp:
  - k3d-my-cluster-server-0
  - k3d-my-cluster-agent-0
  - k3d-my-cluster-agent-1
  6443.tcp:
  - k3d-my-cluster-server-0
settings:
  workerConnections: 1024
DEBU[0000] ===== Merged Cluster Config =====
&{TypeMeta:{Kind: APIVersion:} Cluster:{Name:my-cluster Network:{Name:k3d-my-cluster ID: External:false IPAM:{IPPrefix:zero IPPrefix IPsUsed:[] Managed:false} Members:[]} Token: Nodes:[0x140004751e0 0x14000475380 0x14000475520 0x140004756c0] InitNode:<nil> ExternalDatastore:<nil> KubeAPI:0x14000399680 ServerLoadBalancer:0x140002f2c90 ImageVolume: Volumes:[]} ClusterCreateOpts:{DisableImageVolume:false WaitForServer:true Timeout:0s DisableLoadBalancer:false GPURequest: ServersMemory: AgentsMemory: NodeHooks:[] GlobalLabels:map[app:k3d] GlobalEnv:[] HostAliases:[] Registries:{Create:<nil> Use:[] Config:<nil>}} KubeconfigOpts:{UpdateDefaultKubeconfig:true SwitchCurrentContext:true}}
===== ===== =====
DEBU[0000] '--kubeconfig-update-default set: enabling wait-for-server
INFO[0000] Prep: Network
DEBU[0000] Found network {Name:k3d-my-cluster ID:956af13d8ad8b4381e192932424ab02eb5b3ff54a0fe3a1909c5a042868a23f3 Created:2023-08-08 06:38:48.462306428 +0000 UTC Scope:local Driver:bridge EnableIPv6:false IPAM:{Driver:default Options:map[] Config:[{Subnet:172.18.0.0/16 IPRange: Gateway:172.18.0.1 AuxAddress:map[]}]} Internal:false Attachable:false Ingress:false ConfigFrom:{Network:} ConfigOnly:false Containers:map[] Options:map[com.docker.network.bridge.enable_ip_masquerade:true] Labels:map[app:k3d] Peers:[] Services:map[]}
INFO[0000] Re-using existing network 'k3d-my-cluster' (956af13d8ad8b4381e192932424ab02eb5b3ff54a0fe3a1909c5a042868a23f3)
INFO[0000] Created image volume k3d-my-cluster-images
INFO[0000] Starting new tools node...
DEBU[0000] DOCKER_SOCK=/var/run/docker.sock
DEBU[0000] DOCKER_SOCK=/var/run/docker.sock
DEBU[0000] DOCKER_SOCK=/var/run/docker.sock
DEBU[0000] [Docker] Local DfD: using 'host.docker.internal'
DEBU[0000] [Docker] wanted to use 'host.docker.internal' as docker host, but it's not resolvable locally: lookup host.docker.internal: no such host
DEBU[0000] DOCKER_SOCK=/var/run/docker.sock
DEBU[0000] Detected CgroupV2, enabling custom entrypoint (disable by setting K3D_FIX_CGROUPV2=false)
DEBU[0000] Created container k3d-my-cluster-tools (ID: 114d7b45cef1f7231e510818ffff93b6c618b4b9c568bf9b0f43a0d0adda2ab3)
DEBU[0000] Node k3d-my-cluster-tools Start Time: 2023-08-08 08:43:30.454212 +0200 CEST m=+0.133891918
INFO[0000] Starting Node 'k3d-my-cluster-tools'
DEBU[0000] Truncated 2023-08-08 06:43:30.616273461 +0000 UTC to 2023-08-08 06:43:30 +0000 UTC
INFO[0001] Creating node 'k3d-my-cluster-server-0'
ERRO[0001] Failed Cluster Creation: failed setup of server/agent node k3d-my-cluster-server-0: failed to create node: runtime failed to create node 'k3d-my-cluster-server-0': failed to create container for node 'k3d-my-cluster-server-0': docker failed to pull image 'docker.io/rancher/k3s:v1.25.7-k3s1': docker failed to pull the image 'docker.io/rancher/k3s:v1.25.7-k3s1': Error response from daemon: Please run 'docker login'
ERRO[0001] Failed to create cluster >>> Rolling Back
INFO[0001] Deleting cluster 'my-cluster'
DEBU[0001] no netlabel present on container /k3d-my-cluster-tools
DEBU[0001] failed to get IP for container /k3d-my-cluster-tools as we couldn't find the cluster network
DEBU[0001] Cluster Details: &{Name:my-cluster Network:{Name:k3d-my-cluster ID:956af13d8ad8b4381e192932424ab02eb5b3ff54a0fe3a1909c5a042868a23f3 External:true IPAM:{IPPrefix:172.18.0.0/16 IPsUsed:[172.18.0.1] Managed:false} Members:[]} Token:UXlFIDdKpumFYrwrVMSi Nodes:[0x140004751e0 0x14000475380 0x14000475520 0x140004756c0 0x140000036c0] InitNode:<nil> ExternalDatastore:<nil> KubeAPI:0x14000399680 ServerLoadBalancer:0x140002f2c90 ImageVolume:k3d-my-cluster-images Volumes:[k3d-my-cluster-images]}
DEBU[0001] Deleting node k3d-my-cluster-serverlb ...
ERRO[0001] docker failed to remove the container 'k3d-my-cluster-serverlb': Error response from daemon: No such container: k3d-my-cluster-serverlb
DEBU[0001] Deleting node k3d-my-cluster-server-0 ...
ERRO[0001] docker failed to remove the container 'k3d-my-cluster-server-0': Error response from daemon: No such container: k3d-my-cluster-server-0
DEBU[0001] Deleting node k3d-my-cluster-agent-0 ...
ERRO[0001] docker failed to remove the container 'k3d-my-cluster-agent-0': Error response from daemon: No such container: k3d-my-cluster-agent-0
DEBU[0001] Deleting node k3d-my-cluster-agent-1 ...
ERRO[0001] docker failed to remove the container 'k3d-my-cluster-agent-1': Error response from daemon: No such container: k3d-my-cluster-agent-1
DEBU[0001] Deleting node k3d-my-cluster-tools ...
DEBU[0001] Skip deletion of cluster network 'k3d-my-cluster' because it's managed externally
INFO[0001] Deleting 1 attached volumes...
DEBU[0001] Deleting volume k3d-my-cluster-images...
FATA[0001] Cluster creation FAILED, all changes have been rolled back!

$ docker pull docker.io/rancher/k3s:v1.25.7-k3s1
v1.25.7-k3s1: Pulling from rancher/k3s
8f802876bc40: Pull complete
91fe88c41772: Pull complete
Digest: sha256:aa02a6508ec2b8aeac839443f3b69b24e7789a8d9f0d9cb24f2adc2f26d5a60a
Status: Downloaded newer image for rancher/k3s:v1.25.7-k3s1
docker.io/rancher/k3s:v1.25.7-k3s1

$ k3d cluster create my-cluster --servers 1 --agents 2 --port 9080:80@loadbalancer --port 9443:443@loadbalancer --api-port 6443 --k3s-arg "--disable=traefik@server:*"
INFO[0000] portmapping '9080:80' targets the loadbalancer: defaulting to [servers:*:proxy agents:*:proxy]
INFO[0000] portmapping '9443:443' targets the loadbalancer: defaulting to [servers:*:proxy agents:*:proxy]
INFO[0000] Prep: Network
INFO[0000] Re-using existing network 'k3d-my-cluster' (956af13d8ad8b4381e192932424ab02eb5b3ff54a0fe3a1909c5a042868a23f3)
INFO[0000] Created image volume k3d-my-cluster-images
INFO[0000] Starting new tools node...
INFO[0000] Starting Node 'k3d-my-cluster-tools'
INFO[0001] Creating node 'k3d-my-cluster-server-0'
INFO[0001] Creating node 'k3d-my-cluster-agent-0'
INFO[0001] Creating node 'k3d-my-cluster-agent-1'
INFO[0001] Creating LoadBalancer 'k3d-my-cluster-serverlb'
INFO[0001] Using the k3d-tools node to gather environment information
INFO[0001] Starting new tools node...
INFO[0001] Starting Node 'k3d-my-cluster-tools'
INFO[0002] Starting cluster 'my-cluster'
INFO[0002] Starting servers...
INFO[0002] Starting Node 'k3d-my-cluster-server-0'
INFO[0006] Starting agents...
INFO[0006] Starting Node 'k3d-my-cluster-agent-1'
INFO[0006] Starting Node 'k3d-my-cluster-agent-0'
INFO[0010] Starting helpers...
INFO[0010] Starting Node 'k3d-my-cluster-serverlb'
INFO[0016] Injecting records for hostAliases (incl. host.k3d.internal) and for 5 network members into CoreDNS configmap...
INFO[0018] Cluster 'my-cluster' created successfully!
INFO[0018] You can now use it like this:
kubectl cluster-info

Which OS & Architecture

$ k3d runtime-info
arch: aarch64
cgroupdriver: cgroupfs
cgroupversion: "2"
endpoint: /var/run/docker.sock
filesystem: extfs
infoname: docker-desktop
name: docker
os: Docker Desktop
ostype: linux
version: 24.0.5

macOS Ventura 13.5

Which version of k3d

$ k3d version
k3d version v5.4.9
k3s version v1.25.7-k3s1 (default)

Which version of docker

$ docker version; docker info
Client:
 Cloud integration: v1.0.35-desktop+001
 Version:           24.0.5
 API version:       1.43
 Go version:        go1.20.6
 Git commit:        ced0996
 Built:             Fri Jul 21 20:32:30 2023
 OS/Arch:           darwin/arm64
 Context:           desktop-linux

Server: Docker Desktop 4.22.0 (117440)
 Engine:
  Version:          24.0.5
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.6
  Git commit:       a61e2b4
  Built:            Fri Jul 21 20:35:38 2023
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.6.21
  GitCommit:        3dce8eb055cbb6872793272b4f20ed16117344f8
 runc:
  Version:          1.1.7
  GitCommit:        v1.1.7-0-g860f061
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
Client:
 Version:    24.0.5
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.11.2-desktop.1
    Path:     /Users/spremraj/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.20.2-desktop.1
    Path:     /Users/spremraj/.docker/cli-plugins/docker-compose
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.0
    Path:     /Users/spremraj/.docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.20
    Path:     /Users/spremraj/.docker/cli-plugins/docker-extension
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v0.1.0-beta.6
    Path:     /Users/spremraj/.docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /Users/spremraj/.docker/cli-plugins/docker-sbom
  scan: Docker Scan (Docker Inc.)
    Version:  v0.26.0
    Path:     /Users/spremraj/.docker/cli-plugins/docker-scan
  scout: Command line tool for Docker Scout (Docker Inc.)
    Version:  0.20.0
    Path:     /Users/spremraj/.docker/cli-plugins/docker-scout

Server:
 Containers: 5
  Running: 5
  Paused: 0
  Stopped: 0
 Images: 5
 Server Version: 24.0.5
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 3dce8eb055cbb6872793272b4f20ed16117344f8
 runc version: v1.1.7-0-g860f061
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
  cgroupns
 Kernel Version: 5.15.49-linuxkit-pr
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 5
 Total Memory: 10.7GiB
 Name: docker-desktop
 ID: c365d28b-7020-42f3-85ed-c833122e8809
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: daemon is not using the default seccomp profile

[ishanul]

Same issue in windows.

PS C:> k3d version
k3d version v5.6.0
k3s version v1.27.4-k3s1 (default)

If I do docker pull before the cluster creation, it works.

[jonathandieu]

Just encountered this issue as well on MacOS Sonoma

[iwilltry42]

I assume there's something specific to the keychains / credential stores on those hosts.
Unfortunately I cannot reproduce this on any machine or VM I have, so it's fairly difficult to debug.
The fact that the error is coming from the docker Daemon itself is a little concerning.

For everyone that's still experiencing this issue - it may be interesting to know your k3d versions and your docker infos to see if there's an issue between specific versions of k3d and docker.

[youssefcamao]

is there any solution found for this ?

[juanmolle]

any update?

[iwilltry42]

Can anyone provide any debug information or anything I asked for here?