ImageInspectError when eks try to pull the image

[itai-spiritt]

What happened:

Running in AWS EKS (version 1.21) we have started to receive the following error:

Failed to inspect image "602401143452.dkr.ecr.eu-west-1.amazonaws.com/amazon-k8s-cni-init:v1.7.5-eksbuild.1": Id or size of image "602401143452.dkr.ecr.eu-west-1.amazonaws.com/amazon-k8s-cni-init:v1.7.5-eksbuild.1" is not set

It happened suddenly, for some of our nodes (also for new nodes).

Environment:

• Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.14", GitCommit:"0f77da5bd4809927e15d1658fb4aa8f13ad890a5", GitTreeState:"clean", BuildDate:"2022-06-15T14:17:29Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"21+", GitVersion:"v1.21.14-eks-3af4770", GitCommit:"3c86b95399a496380e90327b114db108546c6345", GitTreeState:"clean", BuildDate:"2024-04-30T03:32:40Z", GoVersion:"go1.21.8", Compiler:"gc", Platform:"linux/amd64"}

[n2taylor]

Check that your node AMI was not deprecated

[orsenthil]

And v1.7.5-eksbuild.1" is very old, it is time to upgrade to latest supported version of CNI.

[itai-spiritt]

when using eks, is it sufficient to just update the cluster?

[itai-spiritt]

Check that your node AMI was not deprecated

ami-0c71ddf0753e561aa amazon-eks-node-1.18-v20201126

On 2 different nodes, the ready one is docker 20 and on the not ready the docker is 25
both have the same AMI
What causes the docker version diff? And how do I revert it back to 20?

[n2taylor]

Check that your node AMI was not deprecated

ami-0c71ddf0753e561aa amazon-eks-node-1.18-v20201126

I'm not able to find that AMI in the list of publicly searchable images in the AWS console. I would recommend getting on the latest 1.21 compatible AMI which is ami-0c9aa39ce6bc59190 amazon-eks-node-1.21-v20240615

On 2 different nodes, the ready one is docker 20 and on the not ready the docker is 25 both have the same AMI What causes the docker version diff? And how do I revert it back to 20?

Is it possible you have multiple node AMI versions running in your cluster?

[itai-spiritt]

Check that your node AMI was not deprecated

ami-0c71ddf0753e561aa amazon-eks-node-1.18-v20201126

I'm not able to find that AMI in the list of publicly searchable images in the AWS console. I would recommend getting on the latest 1.21 compatible AMI which is ami-0c9aa39ce6bc59190 amazon-eks-node-1.21-v20240615

On 2 different nodes, the ready one is docker 20 and on the not ready the docker is 25 both have the same AMI What causes the docker version diff? And how do I revert it back to 20?

Is it possible you have multiple node AMI versions running in your cluster?

no, it's the same AMI version...
I'll try update it to version 1.21, but where should I change it to be applied on the whole cluster?

[n2taylor]

I'll try update it to version 1.21, but where should I change it to be applied on the whole cluster?

You'll have to update the node group either via eksctl or modifying your launch template. See here.

[orsenthil]

This is not a VPC CNI related bug. Closing this issue.

[github-actions]

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.

[frankietoledo]

Check that your node AMI was not deprecated

+1 here, we got problems with cni and kube proxy, solved upgrading the ami.