Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migrate image registry to registry.k8s.io #4720

Merged
merged 1 commit into from
Mar 22, 2023
Merged

Migrate image registry to registry.k8s.io #4720

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion ci/conformance-image-config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
dockerLibraryRegistry: k8s.gcr.io/e2e-test-images
dockerLibraryRegistry: registry.k8s.io/e2e-test-images
12 changes: 6 additions & 6 deletions ci/jenkins/test.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -368,7 +368,7 @@ function deliver_antrea_windows {
echo "===== Pull necessary images on Control-Plane node ====="
harbor_images=("agnhost:2.13" "nginx:1.15-alpine")
antrea_images=("e2eteam/agnhost:2.13" "docker.io/library/nginx:1.15-alpine")
common_images=("k8s.gcr.io/e2e-test-images/agnhost:2.29")
common_images=("registry.k8s.io/e2e-test-images/agnhost:2.29")
for i in "${!harbor_images[@]}"; do
docker pull -q "${DOCKER_REGISTRY}/antrea/${harbor_images[i]}"
docker tag "${DOCKER_REGISTRY}/antrea/${harbor_images[i]}" "${antrea_images[i]}"
Expand All @@ -395,7 +395,7 @@ function deliver_antrea_windows {
# Some tests need us.gcr.io/k8s-artifacts-prod/e2e-test-images/agnhost:2.13 image but it is not for windows/amd64 10.0.17763
# Use e2eteam/agnhost:2.13 instead
harbor_images=("sigwindowstools-kube-proxy:v1.18.0" "agnhost:2.13" "agnhost:2.13" "agnhost:2.13" "agnhost:2.29" "e2eteam-jessie-dnsutils:1.0" "e2eteam-jessie-dnsutils:1.0" "e2eteam-pause:3.2" "e2eteam-pause:3.2" "e2eteam-busybox:1.29-windows-amd64-1809")
antrea_images=("sigwindowstools/kube-proxy:v1.18.0" "e2eteam/agnhost:2.13" "us.gcr.io/k8s-artifacts-prod/e2e-test-images/agnhost:2.13" "k8sprow.azurecr.io/kubernetes-e2e-test-images/agnhost:2.13" "k8s.gcr.io/e2e-test-images/agnhost:2.29" "e2eteam/jessie-dnsutils:1.0" "gcr.io/kubernetes-e2e-test-images/jessie-dnsutils:1.0" "e2eteam/pause:3.2" "k8s.gcr.io/pause:3.2" "docker.io/library/busybox:1.29")
antrea_images=("sigwindowstools/kube-proxy:v1.18.0" "e2eteam/agnhost:2.13" "us.gcr.io/k8s-artifacts-prod/e2e-test-images/agnhost:2.13" "k8sprow.azurecr.io/kubernetes-e2e-test-images/agnhost:2.13" "registry.k8s.io/e2e-test-images/agnhost:2.29" "e2eteam/jessie-dnsutils:1.0" "gcr.io/kubernetes-e2e-test-images/jessie-dnsutils:1.0" "e2eteam/pause:3.2" "registry.k8s.io/pause:3.2" "docker.io/library/busybox:1.29")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The antrea_images is used to generate a new tag for the Harbor image to avoid pull rate limit problems. As we still have a Windows Docker testbed with an old version of Kubernetes, the images used in the conformance tests on these testbeds are still in the k8s.gcr.io repository. Perhaps, for deliver_antrea_windows, we can add a new registry.k8s.io tag for e2e tests, but still keep the existing k8s.gcr.io tag for old version conformance tests. What's your opinion? @antoninbas

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry @XinShuYang, I didn't see your comment before merging this.
I guess we may need to tag the image twice (once with k8s.gcr.io and once with registry.k8s.io) to keep the tests running?

It feels like there should be a better way to handle this. The current solution seems a bit brittle, as both the Harbor registry and this script need to be updated when a new image is required by the conformance tests.

Also, one thing that comes to mind is that rate limiting doesn't apply to the K8s registries, so why don't we pull them directly?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is that why jenkins-windows-conformance failed?

Copy link
Contributor

@XinShuYang XinShuYang Mar 23, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@antoninbas Yes, our windows docker testbed is running with old version kubernetes, which means its conformance tests still use images from k8s.gcr.io repo. In CI pipeline we pull image from harbor first(which doesn't have pull rate limit problem) and add the k8s.gcr.io tag to these images, then set the image pull policy of conformance/networkpolicy tests to IfNotPresent to avoid pulling images from these unavailable repo during running tests.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One of my points is that there should be no rate limiting for k8s.gcr.io (unless I am mistaken), so why do we pull these from Harbor?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's possible that images with same version differ on harbor and dockerhub. When I ran "docker pull e2eteam-pause:3.2" on windows node, I received the error message "no matching manifest for windows/amd64 10.0.17763 in the manifest list entries". However, pulling image from "projects.registry.vmware.com/antrea/e2eteam-pause:3.2" worked fine.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps, for deliver_antrea_windows, we can add a new registry.k8s.io tag for e2e tests, but still keep the existing k8s.gcr.io tag for old version conformance tests.

@antoninbas @XinShuYang do you recommend to open a new PR for this once we have resolved the best way to tag/pull the image?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will let @XinShuYang decide what is the best course of action here.
Ideally we would not have to list all images like that (or maybe our testbed could be upgraded?).

common_images=("mcr.microsoft.com/windows/servercore/iis:latest")
# Pull necessary images in advance to avoid transient error
for i in "${!harbor_images[@]}"; do
Expand Down Expand Up @@ -479,8 +479,8 @@ function deliver_antrea_windows_containerd {
echo "===== Pull necessary images on Control-Plane node ====="
harbor_images=("agnhost:2.13" "nginx:1.15-alpine")
antrea_images=("e2eteam/agnhost:2.13" "docker.io/library/nginx:1.15-alpine")
common_images=("k8s.gcr.io/e2e-test-images/agnhost:2.29")
k8s_images=("k8s.gcr.io/e2e-test-images/agnhost:2.40" "k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.5" "k8s.gcr.io/e2e-test-images/nginx:1.14-2")
common_images=("registry.k8s.io/e2e-test-images/agnhost:2.29")
k8s_images=("registry.k8s.io/e2e-test-images/agnhost:2.40" "registry.k8s.io/e2e-test-images/jessie-dnsutils:1.5" "registry.k8s.io/e2e-test-images/nginx:1.14-2")
e2e_images=("k8sprow.azurecr.io/kubernetes-e2e-test-images/agnhost:2.40" "k8sprow.azurecr.io/kubernetes-e2e-test-images/jessie-dnsutils:1.5" "k8sprow.azurecr.io/kubernetes-e2e-test-images/nginx:1.14-2")

for i in "${!harbor_images[@]}"; do
Expand Down Expand Up @@ -529,8 +529,8 @@ function deliver_antrea_windows_containerd {
# Some tests need us.gcr.io/k8s-artifacts-prod/e2e-test-images/agnhost:2.13 image but it is not for windows/amd64 10.0.17763
# Use e2eteam/agnhost:2.13 instead
harbor_images=("sigwindowstools-kube-proxy:v1.18.0" "agnhost:2.13" "agnhost:2.13" "agnhost:2.29" "e2eteam-jessie-dnsutils:1.0" "e2eteam-pause:3.2")
antrea_images=("sigwindowstools/kube-proxy:v1.18.0" "e2eteam/agnhost:2.13" "us.gcr.io/k8s-artifacts-prod/e2e-test-images/agnhost:2.13" "k8s.gcr.io/e2e-test-images/agnhost:2.29" "e2eteam/jessie-dnsutils:1.0" "e2eteam/pause:3.2")
k8s_images=("k8s.gcr.io/e2e-test-images/agnhost:2.40" "k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.5" "k8s.gcr.io/e2e-test-images/nginx:1.14-2")
antrea_images=("sigwindowstools/kube-proxy:v1.18.0" "e2eteam/agnhost:2.13" "us.gcr.io/k8s-artifacts-prod/e2e-test-images/agnhost:2.13" "registry.k8s.io/e2e-test-images/agnhost:2.29" "e2eteam/jessie-dnsutils:1.0" "e2eteam/pause:3.2")
k8s_images=("registry.k8s.io/e2e-test-images/agnhost:2.40" "registry.k8s.io/e2e-test-images/jessie-dnsutils:1.5" "registry.k8s.io/e2e-test-images/nginx:1.14-2")
e2e_images=("k8sprow.azurecr.io/kubernetes-e2e-test-images/agnhost:2.40" "k8sprow.azurecr.io/kubernetes-e2e-test-images/jessie-dnsutils:1.5" "k8sprow.azurecr.io/kubernetes-e2e-test-images/nginx:1.14-2")
# Pull necessary images in advance to avoid transient error
for i in "${!harbor_images[@]}"; do
Expand Down
4 changes: 2 additions & 2 deletions ci/kind/test-e2e-kind.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -147,7 +147,7 @@ if $flow_visibility; then
manifest_args="$manifest_args --feature-gates FlowExporter=true --extra-helm-values-file $FLOW_VISIBILITY_HELM_VALUES"
fi

COMMON_IMAGES_LIST=("k8s.gcr.io/e2e-test-images/agnhost:2.29" \
COMMON_IMAGES_LIST=("registry.k8s.io/e2e-test-images/agnhost:2.29" \
"projects.registry.vmware.com/antrea/busybox" \
"projects.registry.vmware.com/antrea/nginx:1.21.6-alpine" \
"projects.registry.vmware.com/antrea/perftool")
Expand All @@ -163,7 +163,7 @@ else
COMMON_IMAGES_LIST+=("antrea/antrea-ubuntu:latest")
fi
if $proxy_all; then
COMMON_IMAGES_LIST+=("k8s.gcr.io/echoserver:1.10")
COMMON_IMAGES_LIST+=("registry.k8s.io/echoserver:1.10")
fi
if $flow_visibility; then
COMMON_IMAGES_LIST+=("${FLOW_VISIBILITY_IMAGE_LIST[@]}")
Expand Down
4 changes: 2 additions & 2 deletions hack/netpol-generator/test-kind.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,8 +33,8 @@ kind load docker-image antrea/antrea-ubuntu:latest
docker pull mfenwick100/cyclonus:v0.4.7
kind load docker-image mfenwick100/cyclonus:v0.4.7
# pre-load agnhost image
docker pull k8s.gcr.io/e2e-test-images/agnhost:2.29
kind load docker-image k8s.gcr.io/e2e-test-images/agnhost:2.29
docker pull registry.k8s.io/e2e-test-images/agnhost:2.29
kind load docker-image registry.k8s.io/e2e-test-images/agnhost:2.29

"$ROOT_DIR"/hack/generate-manifest.sh | kubectl apply -f -

Expand Down
5 changes: 3 additions & 2 deletions hack/netpol/pkg/utils/k8s_util.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,11 +17,12 @@ package utils
import (
"bytes"
"fmt"
"k8s.io/client-go/rest"
"os"
"path/filepath"
"strings"

"k8s.io/client-go/rest"

"github.com/pkg/errors"
log "github.com/sirupsen/logrus"
appsv1 "k8s.io/api/apps/v1"
Expand Down Expand Up @@ -219,7 +220,7 @@ func (k *Kubernetes) CreateOrUpdateDeployment(ns, deploymentName string, replica
return v1.Container{
Name: fmt.Sprintf("c%d", port),
ImagePullPolicy: v1.PullIfNotPresent,
Image: "k8s.gcr.io/e2e-test-images/agnhost:2.29",
Image: "registry.k8s.io/e2e-test-images/agnhost:2.29",
// "-k" for persistent server
Command: []string{"/agnhost", "serve-hostname", "--tcp", "--http=false", "--port", fmt.Sprintf("%d", port)},
SecurityContext: &v1.SecurityContext{},
Expand Down
2 changes: 1 addition & 1 deletion test/e2e/framework.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -119,7 +119,7 @@ const (

nameSuffixLength int = 8

agnhostImage = "k8s.gcr.io/e2e-test-images/agnhost:2.29"
agnhostImage = "registry.k8s.io/e2e-test-images/agnhost:2.29"
busyboxImage = "projects.registry.vmware.com/antrea/busybox"
mcjoinImage = "projects.registry.vmware.com/antrea/mcjoin:v2.9"
netshootImage = "projects.registry.vmware.com/antrea/netshoot:v0.1"
Expand Down