[BUG] vm-dhcp-controller dhcpserver-agent Pod Fails to Start in Air-Gapped Environment Due to ImagePullPolicy 'Always' Setting

[zha0jf]

Describe the bug
In the prepareAgentPod function within the file vm-dhcp-controller/pkg/controller/ippool/common.go, an init container is added to the vm-dhcp-agent pod, using the image docker.io/library/busybox with the default imagePullPolicy set to Always. This configuration causes the vm-dhcp-agent's dhcpserver-agent pod to get stuck in the “Init:ImagePullBackOff” status when deploying in an air-gapped environment, even if the busybox:latest image was pre-uploaded offline. This prevents the vm-dhcp-agent from functioning in air-gapped environments.

To Reproduce
Steps to reproduce the behavior:

1. Deploy vm-dhcp-controller in an air-gapped environment.
2. Even after uploading the busybox:latest image offline, the vm-dhcp-agent dhcpserver-agent pod remains stuck in the "Init:ImagePullBackOff" status.

Expected behavior
The vm-dhcp-agent dhcpserver-agent pod should be able to pull the locally uploaded image and start successfully in an air-gapped environment.

Support bundle

Environment

• Harvester ISO version: harvester-v1.4.0-rc4-amd64.iso

Additional context
It is recommended to add the configuration ImagePullPolicy = "IfNotPresent" at line 96 in the file vm-dhcp-controller/pkg/controller/ippool/common.go to ensure that the vm-dhcp-controller can function properly in air-gapped environments.

[starbops]

Hi @zha0jf, thanks for reporting. I wonder why, even though the missing container image is imported to the node, it still complains about Init:ImagePullBackOff. Did you wait enough time to surpass the back-off time? Another simple way to test it is to simply delete the DHCP agent pod entirely.

[starbops]

imagePullPolicy defaults to Always because we specify the busybox image without a tag. As the document explains, kubelet will check if the digest exists in the cache. This involves communication between kubelet and remote registries. That might be why you constantly get an Init:ImagePullBackOff error even though the container image is manually imported.

Your fix in harvester/vm-dhcp-controller#37 can resolve the issue. However, this also reminds us that we should use any container image packed in the ISO image. I'd say that's the root cause of the problem. I'm considering adding iproute2 to the rancher/harvester-vm-dhcp-agent image and making the init container use it. We'll take care of that in following PRs. Thank you.

[zha0jf]

That will be great. Thank you.

[harvesterhci-io-github-bot]

Pre Ready-For-Testing Checklist

• If labeled: require/HEP Has the Harvester Enhancement Proposal PR submitted?
  The HEP PR is at:

• Where is the reproduce steps/test steps documented?
  The reproduce steps/test steps are at: fix(agent): use agent image for initcontainer vm-dhcp-controller#38

• Is there a workaround for the issue? If so, where is it documented?
  The workaround is at:

• Have the backend code been merged (harvester, harvester-installer, etc) (including backport-needed/*)?
  The PR is at: fix: change ippool agent pod's initcontainer's imagepullpolicy to ifn… vm-dhcp-controller#37, fix(agent): use agent image for initcontainer vm-dhcp-controller#38

  • Does the PR include the explanation for the fix or the feature?

  • Does the PR include deployment change (YAML/Chart)? If so, where are the PRs for both YAML file and Chart?
    The PR for the YAML change is at: chore: bump harvester-vm-dhcp-controller chart to 0.3.3 experimental-addons#23
    The PR for the chart change is at: chore: bump vm-dhcp-controller to 0.3.3 charts#314

• If labeled: area/ui Has the UI issue filed or ready to be merged?
  The UI issue/PR is at:~~

• If labeled: require/doc, require/knowledge-base Has the necessary document PR submitted or merged?
  The documentation/KB PR is at:

• If NOT labeled: not-require/test-plan Has the e2e test plan been merged? Have QAs agreed on the automation test case? If only test case skeleton w/o implementation, have you created an implementation issue?

  • The automation skeleton PR is at:
  • The automation test case PR is at:

• If the fix introduces the code for backward compatibility Has a separate issue been filed with the label release/obsolete-compatibility?
  The compatibility issue is filed at:

[harvesterhci-io-github-bot]

Automation e2e test issue: harvester/tests#1652

[starbops]

Hi @zha0jf, We've published harvester-vm-dhcp-controller 0.3.3. Would you like to try it to see if it solves the problem?

https://github.com/harvester/experimental-addons/blob/main/harvester-vm-dhcp-controller/harvester-vm-dhcp-controller.yaml

Thank you.

[zha0jf]

Hi @starbops，I just tested harvester-vm-dhcp-controller 0.3.3 in the harvester-v1.4.0-rc5 air-gapped environment, and the issue has been resolved. Thank you.

[starbops]

@zha0jf That's a great news! Thanks again for spotting the issue and sending a patch to us :)

Note: please don't close this issue yet. We have our pipelines and will take care of it.