About "pause" image

[onelucksnake]

I'm trying to use "kubeadm" [v1.26.0] to deploy a simple K8S cluster;
The scheme used is: "K8S + CRI-DOCKERD"
The command used is "kubeadm init --config kubeadm-init.yaml";
Then I couldn't start a K8S cluster normally. At first I thought it was the reason of "kubeadm-init.yaml" configuration;
but not;

When I was looking at "cri-docker.service" logs, I found these logs,

root >> K8S-MASTER:/opt/K8S_DeployData $ journalctl -xeu cri-docker
1月 03 17:28:24 K8S-MASTER cri-dockerd[584]: time="2023-01-03T17:28:24+08:00" level=info msg="Pulling the image without credentials. Image: registry.k8s.io/pause:3.6"
1月 03 17:28:28 K8S-MASTER cri-dockerd[584]: time="2023-01-03T17:28:28+08:00" level=info msg="Pulling the image without credentials. Image: registry.k8s.io/pause:3.6"
1月 03 17:29:04 K8S-MASTER cri-dockerd[584]: time="2023-01-03T17:29:04+08:00" level=info msg="Pulling the image without credentials. Image: registry.k8s.io/pause:3.6"
1月 03 17:29:04 K8S-MASTER cri-dockerd[584]: time="2023-01-03T17:29:04+08:00" level=info msg="Pulling the image without credentials. Image: registry.k8s.io/pause:3.6"
1月 03 17:29:09 K8S-MASTER cri-dockerd[584]: time="2023-01-03T17:29:09+08:00" level=info msg="Pulling the image without credentials. Image: registry.k8s.io/pause:3.6"
1月 03 17:29:10 K8S-MASTER cri-dockerd[584]: time="2023-01-03T17:29:10+08:00" level=info msg="Pulling the image without credentials. Image: registry.k8s.io/pause:3.6"
1月 03 17:29:46 K8S-MASTER cri-dockerd[584]: time="2023-01-03T17:29:46+08:00" level=info msg="Pulling the image without credentials. Image: registry.k8s.io/pause:3.6"
1月 03 17:29:48 K8S-MASTER cri-dockerd[584]: time="2023-01-03T17:29:48+08:00" level=info msg="Pulling the image without credentials. Image: registry.k8s.io/pause:3.6"
1月 03 17:29:51 K8S-MASTER cri-dockerd[584]: time="2023-01-03T17:29:51+08:00" level=info msg="Pulling the image without credentials. Image: registry.k8s.io/pause:3.6"
1月 03 17:29:56 K8S-MASTER cri-dockerd[584]: time="2023-01-03T17:29:56+08:00" level=info msg="Pulling the image without credentials. Image: registry.k8s.io/pause:3.6"
1月 03 17:30:31 K8S-MASTER cri-dockerd[584]: time="2023-01-03T17:30:31+08:00" level=info msg="Pulling the image without credentials. Image: registry.k8s.io/pause:3.6"
1月 03 17:30:33 K8S-MASTER cri-dockerd[584]: time="2023-01-03T17:30:33+08:00" level=info msg="Pulling the image without credentials. Image: registry.k8s.io/pause:3.6"
1月 03 17:30:37 K8S-MASTER cri-dockerd[584]: time="2023-01-03T17:30:37+08:00" level=info msg="Pulling the image without credentials. Image: registry.k8s.io/pause:3.6"
1月 03 17:30:42 K8S-MASTER cri-dockerd[584]: time="2023-01-03T17:30:42+08:00" level=info msg="Pulling the image without credentials. Image: registry.k8s.io/pause:3.6"
1月 03 17:31:14 K8S-MASTER cri-dockerd[584]: time="2023-01-03T17:31:14+08:00" level=info msg="Pulling the image without credentials. Image: registry.k8s.io/pause:3.6"
1月 03 17:31:16 K8S-MASTER cri-dockerd[584]: time="2023-01-03T17:31:16+08:00" level=info msg="Pulling the image without credentials. Image: registry.k8s.io/pause:3.6"

Yes, "kubeadm" asks "cri-dockerd" to download "registry.k8s.io/pause:3.6",
Since "registry.k8s.io" cannot be accessed normally in my region, this caused me to be unable to start the K8S cluster normally;
Of course, I used another method to solve the image problem;
But the problem is not about this, it's about "kubeadm config images list | sort" [v1.26.0];

$ kubeadm config images list | sort
registry.k8s.io/coredns/coredns:v1.9.3
registry.k8s.io/etcd:3.5.6-0
registry.k8s.io/kube-apiserver:v1.26.0
registry.k8s.io/kube-controller-manager:v1.26.0
registry.k8s.io/kube-proxy:v1.26.0
registry.k8s.io/kube-scheduler:v1.26.0
registry.k8s.io/pause:3.9

It clearly tells me that it is using the "pause:3.9" mirror;
I don't know if this is a bug;
This has no effect on users who can access "registry.k8s.io" normally;
But it is misleading for users who cannot access "registry.k8s.io" normally;

After running the K8S cluster normally, it also shows that "kubeadm[v1.26.0]" uses the image of "pause:3.6";

$ docker ps --format "table{{.ID}}\t{{.Image}}"
CONTAINER ID   IMAGE
8bff51455729   556768f31eb1
21fc01364de8   registry.k8s.io/pause:3.6
22bd912781d8   fce326961ae2
ddef8c2e6f12   dafd8ad70b15
418d0f321f30   5d7c5dfd3ba1
a187c60d5230   a31e1d84401e
8165804f2cfe   registry.k8s.io/pause:3.6
da3a27e00d70   registry.k8s.io/pause:3.6
2534bc8b6f5a   registry.k8s.io/pause:3.6
3d922b3529d5   registry.k8s.io/pause:3.6

[neolit123]

this sounds like a cri-dockerd problem.
kubeadm init and kubeadm config images use the same code. so if pause is version 3.9 it should be the same for the kubeadm binary.

note that cri-dockerd depending on its configuration might decide to pull it's default pause version - perhaps registry.k8s.io/pause:3.6

it is best to either configure cri-dockerd to use 3.9 or to prepull 3.6 manually.

[onelucksnake]

this sounds like a cri-dockerd problem. kubeadm init and kubeadm config images use the same code. so if pause is version 3.9 it should be the same for the kubeadm binary.

note that cri-dockerd depending on its configuration might decide to pull it's default pause version - perhaps registry.k8s.io/pause:3.6

it is best to either configure cri-dockerd to use 3.9 or to prepull 3.6 manually.

Thank you very much, it is indeed a problem with "cri-dockerd";
I found this from the help documentation of "cri-dockerd"

$ cri-dockerd --help
--pod-infra-container-image string        The image whose network/ipc namespaces containers in each pod will use (default "registry.k8s.io/pause:3.6")

[RonBarkan]

I was having the same issue when using containerd.

# kubeadm config images list --config=myconfig.yaml
localhost:5000/kube-apiserver:v1.29.6
localhost:5000/kube-controller-manager:v1.29.6
localhost:5000/kube-scheduler:v1.29.6
localhost:5000/kube-proxy:v1.29.6
localhost:5000/coredns:v1.11.1
localhost:5000/pause:3.9
localhost:5000/etcd:3.5.12-0

But, on kubeadm init, I am getting an error from kubelet saying that it was trying to pull pause:3.6.

It looks like containerd has, part of it's default config, /etc/containerd/config.toml the following:

  sandbox_image = "registry.k8s.io/pause:3.6"

Now, like this it would not matter for any machine connected to the internet, but will fail if your machine cannot access the internet. To fix, specify your registry and correct pause version:

  sandbox_image = "localhost:5000/pause:3.9"