diff --git a/changelog/fragments/02-document-k8s-1-30-changes.yaml b/changelog/fragments/02-document-k8s-1-30-changes.yaml new file mode 100644 index 0000000000..f112f3a2fe --- /dev/null +++ b/changelog/fragments/02-document-k8s-1-30-changes.yaml @@ -0,0 +1,346 @@ +# entries is a list of entries to include in +# release notes and/or the migration guide +entries: + - description: > + For Go-based, Helm-based and Ansible-based operators this release moves to Kubernetes 1.30 API's and Kubebuilder + v4 Scaffolding, specifically utilizing the v4.1.1 version. The update to Kubebuiler results in some scaffolding + changes which more information can be found below: + - Discontinue usage of [kube-rbac-proxy](https://github.com/brancz/kube-rbac-proxy) in the default + scaffolding of new projects. For further information, + see: [Action Required: Ensure that you no longer use gcr.io/kubebuilder images](https://github.com/kubernetes-sigs/kubebuilder/discussions/3907) + - The `go/v2` or `go/v3` layouts have been removed, you must upgrade to `go/v4` to be compatible with this release and future updates. + To know how to upgrade,check the [migration documentation](https://book.kubebuilder.io/migration/v3vsv4). + - Re-introduces authn/authz protection for the metrics endpoint using [`WithAuthenticationAndAuthorization`](https://github.com/kubernetes-sigs/controller-runtime/blob/v0.18.4/pkg/metrics/filters/filters.go#L35) + provided by controller-runtime. which usage was [discontinued in the project](https://github.com/kubernetes-sigs/kubebuilder/discussions/3907). + Please, ensure that you no longer use the image `gcr.io/kubebuilder/kube-rbac-proxy`. Images provided under `gcr.io/kubebuilder/` will be unavailable from **March 18, 2025**. + To learn more about any of the metrics changes please look at the Kubebuilder book [metrics](https://book.kubebuilder.io/reference/metrics) page. + + For `Helm-based` and `Ansible-based` operators, a new flag called `metrics-require-rbac` was introduced into the runtime/binary, to control adding + [`WithAuthenticationAndAuthorization`](https://github.com/kubernetes-sigs/controller-runtime/blob/v0.18.4/pkg/metrics/filters/filters.go#L35) + to `Metrics.FilterProvider` of controller-runtime. This was done to ensure forwards and backwards compatibility of the binary and images with any scaffolded content. + + + + # kind is one of: + # - addition + # - change + # - deprecation + # - removal + # - bugfix + kind: "change" + + # Is this a breaking change? + breaking: false + + # NOTE: ONLY USE `pull_request_override` WHEN ADDING THIS + # FILE FOR A PREVIOUSLY MERGED PULL_REQUEST! + # + # The generator auto-detects the PR number from the commit + # message in which this file was originally added. + # + # What is the pull request number (without the "#")? + # pull_request_override: 0 + + + # Migration can be defined to automatically add a section to + # the migration guide. This is required for breaking changes. + migration: + header: Upgrade K8s versions to use 1.30 and Kubebuilder v4 + body: | + Body of the migration section. This should be formatted as markdown and can + span multiple lines. + + Using the YAML string '|' operator means that newlines in this string will + be honored and interpretted as newlines in the rendered markdown. + + // todo-adam start looking at this you can use the below as a starting point + https://github.com/operator-framework/operator-sdk/pull/6736/files#diff-8d98808d11ed4b1bd01afa7f8426135cc0eb30fd4f1003115c55b620af248f02 + + 1) [helm/v1, ansible/v1] Update the kustomize version in your Makefile + ```diff + - curl -sSLo - https://github.com/kubernetes-sigs/kustomize/releases/download/kustomize/v5.3.0/kustomize_v5.3.0_$(OS)_$(ARCH).tar.gz | \ + + curl -sSLo - https://github.com/kubernetes-sigs/kustomize/releases/download/kustomize/v5.4.2/kustomize_v5.4.2_$(OS)_$(ARCH).tar.gz | \ + ``` + + 2) [go/v4] Update your `go.mod` file to upgrade the dependencies and run `go mod tidy` to download them + ```go + go 1.22.0 + + github.com/onsi/ginkgo/v2 v2.17.1 + github.com/onsi/gomega v1.32.0 + k8s.io/api v0.30.1 + k8s.io/apimachinery v0.30.1 + k8s.io/client-go v0.30.1 + sigs.k8s.io/controller-runtime v0.18.4 + ``` + 3) [go/v4] Update your `Makefile` with the below changes: + ```diff + - ENVTEST_K8S_VERSION = 1.29.0 + + ENVTEST_K8S_VERSION = 1.30.0 + ``` + + ```diff + - KUSTOMIZE ?= $(LOCALBIN)/kustomize-$(KUSTOMIZE_VERSION) + - CONTROLLER_GEN ?= $(LOCALBIN)/controller-gen-$(CONTROLLER_TOOLS_VERSION) + - ENVTEST ?= $(LOCALBIN)/setup-envtest-$(ENVTEST_VERSION) + - GOLANGCI_LINT = $(LOCALBIN)/golangci-lint-$(GOLANGCI_LINT_VERSION) + + KUSTOMIZE ?= $(LOCALBIN)/kustomize + + CONTROLLER_GEN ?= $(LOCALBIN)/controller-gen + + ENVTEST ?= $(LOCALBIN)/setup-envtest + + GOLANGCI_LINT = $(LOCALBIN)/golangci-lint + ``` + + ```diff + - KUSTOMIZE_VERSION ?= v5.3.0 + - CONTROLLER_TOOLS_VERSION ?= v0.14.0 + - ENVTEST_VERSION ?= release-0.17 + - GOLANGCI_LINT_VERSION ?= v1.57.2 + + KUSTOMIZE_VERSION ?= v5.4.2 + + CONTROLLER_TOOLS_VERSION ?= v0.15.0 + + ENVTEST_VERSION ?= release-0.18 + + GOLANGCI_LINT_VERSION ?= v1.59.1 + ``` + + ```diff + - $(call go-install-tool,$(GOLANGCI_LINT),github.com/golangci/golangci-lint/cmd/golangci-lint,${GOLANGCI_LINT_VERSION}) + + $(call go-install-tool,$(GOLANGCI_LINT),github.com/golangci/golangci-lint/cmd/golangci-lint,$(GOLANGCI_LINT_VERSION)) + ``` + + ```diff + - @[ -f $(1) ] || { \ + + @[ -f "$(1)-$(3)" ] || { \ + echo "Downloading $${package}" ;\ + + rm -f $(1) || true ;\ + - mv "$$(echo "$(1)" | sed "s/-$(3)$$//")" $(1) ;\ + - } + + mv $(1) $(1)-$(3) ;\ + + } ;\ + + ln -sf $(1)-$(3) $(1) + ``` + + 4) [go/v4] Update your `.golangci.yml` with the below changes: + ```diff + - exportloopref + + - ginkgolinter + - prealloc + + - revive + + + +linters-settings: + + revive: + + rules: + + - name: comment-spacings + ``` + + 5) [go/v4] Update your `Dockerfile` file with the below changes: + ```diff + - FROM golang:1.21 AS builder + + FROM golang:1.22 AS builder + ``` + + 6) [go/v4] Update your `main.go` file with the below changes: + ```diff + "sigs.k8s.io/controller-runtime/pkg/log/zap" + + "sigs.k8s.io/controller-runtime/pkg/metrics/filters" + + var enableHTTP2 bool + - flag.StringVar(&metricsAddr, "metrics-bind-address", ":8080", "The address the metric endpoint binds to.") + + var tlsOpts []func(*tls.Config) + + flag.StringVar(&metricsAddr, "metrics-bind-address", "0", "The address the metrics endpoint binds to. "+ + + "Use :8443 for HTTPS or :8080 for HTTP, or leave as 0 to disable the metrics service.") + flag.StringVar(&probeAddr, "health-probe-bind-address", ":8081", "The address the probe endpoint binds to.") + flag.BoolVar(&enableLeaderElection, "leader-elect", false, + "Enable leader election for controller manager. "+ + "Enabling this will ensure there is only one active controller manager.") + - flag.BoolVar(&secureMetrics, "metrics-secure", false, + - "If set the metrics endpoint is served securely") + + flag.BoolVar(&secureMetrics, "metrics-secure", true, + + "If set, the metrics endpoint is served securely via HTTPS. Use --metrics-secure=false to use HTTP instead.") + + - tlsOpts := []func(*tls.Config){} + + + // Metrics endpoint is enabled in 'config/default/kustomization.yaml'. The Metrics options configure the server. + + // More info: + + // - https://pkg.go.dev/sigs.k8s.io/controller-runtime@v0.18.4/pkg/metrics/server + + // - https://book.kubebuilder.io/reference/metrics.html + + metricsServerOptions := metricsserver.Options{ + + BindAddress: metricsAddr, + + SecureServing: secureMetrics, + + // TODO(user): TLSOpts is used to allow configuring the TLS config used for the server. If certificates are + + // not provided, self-signed certificates will be generated by default. This option is not recommended for + + // production environments as self-signed certificates do not offer the same level of trust and security + + // as certificates issued by a trusted Certificate Authority (CA). The primary risk is potentially allowing + + // unauthorized access to sensitive metrics data. Consider replacing with CertDir, CertName, and KeyName + + // to provide certificates, ensuring the server communicates using trusted and secure certificates. + + TLSOpts: tlsOpts, + + } + + + + if secureMetrics { + + // FilterProvider is used to protect the metrics endpoint with authn/authz. + + // These configurations ensure that only authorized users and service accounts + + // can access the metrics endpoint. The RBAC are configured in 'config/rbac/kustomization.yaml'. More info: + + // https://pkg.go.dev/sigs.k8s.io/controller-runtime@v0.18.4/pkg/metrics/filters#WithAuthenticationAndAuthorization + + metricsServerOptions.FilterProvider = filters.WithAuthenticationAndAuthorization + + } + + + mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{ + - Scheme: scheme, + - Metrics: metricsserver.Options{ + - BindAddress: metricsAddr, + - SecureServing: secureMetrics, + - TLSOpts: tlsOpts, + - }, + + Scheme: scheme, + + Metrics: metricsServerOptions, + ``` + 7) [go/v4] Update your `/config/default/kustomization.yaml` file with the below changes: + ```diff + # [PROMETHEUS] To enable prometheus monitor, uncomment all sections with 'PROMETHEUS'. + #- ../prometheus + +# [METRICS] Expose the controller manager metrics service. + +- metrics_service.yaml + + +# Uncomment the patches line if you enable Metrics, and/or are using webhooks and cert-manager + patches: + -# Protect the /metrics endpoint by putting it behind auth. + -# If you want your controller-manager to expose the /metrics + -# endpoint w/o any authn/z, please comment the following line. + -- path: manager_auth_proxy_patch.yaml + +# [METRICS] The following patch will enable the metrics endpoint using HTTPS and the port :8443. + +# More info: https://book.kubebuilder.io/reference/metrics + +- path: manager_metrics_patch.yaml + + target: + + kind: Deployment + ``` + 8) [go/v4] Remove `/config/default/manager_auth_proxy_patch.yaml` and `/config/default/manager_config_patch.yaml` files. + + 9) [go/v4] Add `/config/default/manager_metrics_patch.yaml` file with the below changes: + ```diff + # This patch adds the args to allow exposing the metrics endpoint using HTTPS + - op: add + path: /spec/template/spec/containers/0/args/0 + value: --metrics-bind-address=:8443 + ``` + + 10) [go/v4] Add `/config/default/metrics_service.yaml` file with the below changes: + ```diff + apiVersion: v1 + kind: Service + metadata: + labels: + control-plane: controller-manager + app.kubernetes.io/name: + app.kubernetes.io/managed-by: kustomize + name: controller-manager-metrics-service + namespace: system + spec: + ports: + - name: https + port: 8443 + protocol: TCP + targetPort: 8443 + selector: + control-plane: controller-manager + ``` + + 11) [go/v4] Update your `/config/manager/manager.yaml` file with the below changes: + ```diff + - --leader-elect + + - --health-probe-bind-address=:8081 + ``` + + 12) [go/v4] Update your `/config/prometheus/monitor/yaml` file with the below changes: + ```diff + - path: /metrics + - port: https + + port: https # Ensure this is the name of the port that exposes HTTPS metrics + tlsConfig: + + # TODO(user): The option insecureSkipVerify: true is not recommended for production since it disables + + # certificate verification. This poses a significant security risk by making the system vulnerable to + + # man-in-the-middle attacks, where an attacker could intercept and manipulate the communication between + + # Prometheus and the monitored services. This could lead to unauthorized access to sensitive metrics data, + + # compromising the integrity and confidentiality of the information. + + # Please use the following options for secure configurations: + + # caFile: /etc/metrics-certs/ca.crt + + # certFile: /etc/metrics-certs/tls.crt + + # keyFile: /etc/metrics-certs/tls.key + insecureSkipVerify: true + ``` + + 13) [go/v4] Remove the following files from `/config/rbac` + ```diff + - auth_proxy_client_clusterrole.yaml + - auth_proxy_role.yaml + - auth_proxy_role_binding.yaml + - auth_proxy_service.yaml + ``` + + 14) [go/v4] Update your `/config/rbac/kustomization.yaml` file with the below changes: + ```diff + - leader_election_role_binding.yaml + - # Comment the following 4 lines if you want to disable + - # the auth proxy (https://github.com/brancz/kube-rbac-proxy) + - # which protects your /metrics endpoint. + - - auth_proxy_service.yaml + - - auth_proxy_role.yaml + - - auth_proxy_role_binding.yaml + - - auth_proxy_client_clusterrole.yaml + + # The following RBAC configurations are used to protect + + # the metrics endpoint with authn/authz. These configurations + + # ensure that only authorized users and service accounts + + # can access the metrics endpoint. Comment the following + + # permissions if you want to disable this protection. + + # More info: https://book.kubebuilder.io/reference/metrics.html + + - metrics_auth_role.yaml + + - metrics_auth_role_binding.yaml + + - metrics_reader_role.yaml + ``` + + 15) [go/v4] Add `/config/rbac/metrics_auth_role.yaml` file with the below changes: + ```diff + apiVersion: rbac.authorization.k8s.io/v1 + kind: ClusterRole + metadata: + name: metrics-auth-role + rules: + - apiGroups: + - authentication.k8s.io + resources: + - tokenreviews + verbs: + - create + - apiGroups: + - authorization.k8s.io + resources: + - subjectaccessreviews + verbs: + - create + ``` + + 16) [go/v4] Add `/config/rbac/metrics_auth_role_binding.yaml` file with the below changes: + ```diff + apiVersion: rbac.authorization.k8s.io/v1 + kind: ClusterRoleBinding + metadata: + name: metrics-auth-rolebinding + roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: metrics-auth-role + subjects: + - kind: ServiceAccount + name: controller-manager + namespace: system + ``` + + 17) [go/v4] Add `/config/rbac/metrics_reader_role.yaml` file with the below changes: + ```diff + apiVersion: rbac.authorization.k8s.io/v1 + kind: ClusterRole + metadata: + name: metrics-reader + rules: + - nonResourceURLs: + - "/metrics" + verbs: + - get + ```