Skip to content

Latest commit

 

History

History
808 lines (670 loc) · 89.3 KB

CHANGELOG-1.23.md

File metadata and controls

808 lines (670 loc) · 89.3 KB

v1.23.0-alpha.4

Downloads for v1.23.0-alpha.4

Source Code

filename sha512 hash
kubernetes.tar.gz aeb10a3fbb89694c52d47203cc958d3543b21426938a9664348163aacd41e20ea7670617a28d8ce6d8d51492980facd5fab062e8ad664dafd7b8dbff1c2bb54f
kubernetes-src.tar.gz b7a8999335ce15b68360478b22af4daaed10e9db50d597e077d731de194208355d1b2134f5635331d9049dc638d05f1f792d52c5890e521f0af3dc2f3e64fbb8

Client Binaries

filename sha512 hash
kubernetes-client-darwin-amd64.tar.gz 5654879ac03f4c7193a8df49cfd4b7253add031c197f50bada40942738bf5720d1c06e31a1d1a7bd1b1a540aa46897e4b34ad8a7e087bd206a7b69b9ffaf5edb
kubernetes-client-darwin-arm64.tar.gz 5dce9fee32436c971ef17595f88f3c74f5644ab3af0e3f854a79fb42f3c8d6d8f507fbb0d7b5bcba52ddf1a49ada2559c477278037cf2dadccba72f0398a1093
kubernetes-client-linux-386.tar.gz e9eb7dab22801c043da2833fde89d2cd721b9dd622df0ff42b25a6742cfab5cff8bfe3ebbda6cc584cf92db3940b95e25aff935863ed999374ee8923ca0b1215
kubernetes-client-linux-amd64.tar.gz 570eeaed029bb05235c58138a777cfd6a4b17d4d91aba346b1fc9a0e573781947599d31a8997e889165db561a18a7ab4d613c2b40a8b2dc0d0225f2411b0fd73
kubernetes-client-linux-arm.tar.gz 298923762745cc064a4489aa01d55f57076b84538aef3a6a3554b60257d9959b4eebbb8aeeecdaf14246fa4f1c17750e1b69c63d4940ae71f87010692e41675c
kubernetes-client-linux-arm64.tar.gz 498527f1cf2d16af576a6b6d27b5ddbb876e24bd85e34e2c91cf39ef467d366b2059e580fdcccb91e0b61a5f52795273b77ed94a1073b5c0bd574b8661afbe0e
kubernetes-client-linux-ppc64le.tar.gz 2632b0fb69565819ef1b6797a834e65f96629df4fd8bec01fce7370672a39afa181854d6ab44afc1c4a6b8143158cf170f5a8e61b75a48071ade2d5ab89d1b2c
kubernetes-client-linux-s390x.tar.gz b793a5a8fce9109343ada86f29cf356c6973cd80d81ca47af5c7e4fa11ffccc273f77aba52b1db42ee12abb94ee23677c21910f57c9385646e35742a1c60e17e
kubernetes-client-windows-386.tar.gz b92e34ee58e1247c1c444134dd9fa78033d0fda1f51509b43016543596cb211128f8aff730d9a3a9118dfeba139186db2a5dd45455427c7521776e63ee77218c
kubernetes-client-windows-amd64.tar.gz 0b5ea6a2de0ff6f71647f428fdbee67c7eb2b918d725cf236ce60daa02e94bd998d15ea0ebb20c4106453e220d11d31506161d5dee3cde6c616dfb5efd11c25e
kubernetes-client-windows-arm64.tar.gz a4e570be453d1df779bb85c62efb41e98209bb93b57b7655a94a737d552c90f9d3061df9088204c7787344dc6a3eb3f843c58651394c0436d2c90b55e499bfab

Server Binaries

filename sha512 hash
kubernetes-server-linux-amd64.tar.gz cf215ee7372edd7d5dbf07faee8ccf83de477c8cd431c0fac58357bb8e027349d8edf87364e7db5cec0936f991388f7b183e81e5f92cb6cdf6303efd8cd65d83
kubernetes-server-linux-arm.tar.gz d6281a6727fcdab956170dca7563fc5099ef79b06c96b2f6bc87fcd0b74f1dea0e14aca344cb41b5b6811919cf4a6d6f60cb08b7fb7034690fd0c4ead82e55ca
kubernetes-server-linux-arm64.tar.gz 4ec30cdfd8128ca405201c0c40750e10bac016e1e53a7662265328564b09e4feb831a259125bcdd64169d221145cbb166a463216e884dd76f4bb9a72a00e64e0
kubernetes-server-linux-ppc64le.tar.gz 42b31174a95d0999c78750a1d2c866918c91d11d6406df4e984913f64806708add35c27c0daf255b5d28e98eb815355d1913911f921d34e618dea4d2ebf91949
kubernetes-server-linux-s390x.tar.gz c4e2b38681c0858d560adc8a330f27e95a035cb0e426c6ff332dcd435cefe88441ea866badab5514c4055191324c48aac108d5d6934a9fd4697da179168b6632

Node Binaries

filename sha512 hash
kubernetes-node-linux-amd64.tar.gz 708b40c9c0d2cfcb6f9874aa3f1b5a27796cbe2bfe7a2345f381e0d9062df8a6769b2bf29b8b641929ef1f5952897c5739e876e8315eb51cced460b13994c247
kubernetes-node-linux-arm.tar.gz f89448106af23d6658b9c2e7b43240fb82051d2f89a302ee61fc1cc78e593535993ba12f412c3df907f415e55c38f1783ce9141075198bb9f197b6fa26328d49
kubernetes-node-linux-arm64.tar.gz bcd8d9fbb244048a3ef3f79f1d4e8f2645bbd69caf353e67ee5c5a4ffd4443da420e5984422933cd4c622c58017a942e20af076f26bfa22f5f38f73a831370ca
kubernetes-node-linux-ppc64le.tar.gz c0724d053c601d4e80ea19957bd32005aeba0cf8f5e03e8e36412aed0777e860ae680302eef632c8e7d4ef1a8e789e48dc58489ad1a7bf7fd20cb0f755e797af
kubernetes-node-linux-s390x.tar.gz 0245f592b92d79ccd102961e5b23a9f5b275829e627254fe8ce5f0a7df53ec2c4a9436942686b9d31b696635ab88131cf92e1002869369fa1cf6f080f8073b5f
kubernetes-node-windows-amd64.tar.gz 2a5c6c79ea65f47a42d25b236709a00eafb793e5d87b5f56516da16b85b06e03020679f7cabfb7dc4bc252ff57da0afa52e357915cb1d3801dc3d5c32f096edf

Changelog since v1.23.0-alpha.3

Changes by Kind

Deprecation

  • A deprecation notice has been added when using the kube-proxy Userspace proxier, which will be removed in v1.25. (#103860) (#104631, @perithompson) [SIG Network]
  • Feature-gate VolumeSubpath has been deprecated and cannot be disabled. It will be completely removed in 1.25 (#105474, @mauriciopoppe) [SIG Storage]
  • Kubeadm: remove the deprecated / NO-OP phase "update-cluster-status" in "kubeadm reset" (#105888, @neolit123) [SIG Cluster Lifecycle]
  • Removed kubectl --dry-run empty default value and boolean values. kubectl --dry-run usage must be specified with --dry-run=(server|client|none). (#105327, @julianvmodesto) [SIG CLI and Testing]

API Change

  • Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

    (#104782, @kerthcet) [SIG Scheduling and Testing]

  • Ephemeral containers have reached beta maturity and are now available by default. (#105405, @verb) [SIG API Machinery, Apps, Node and Testing]

  • Introduce OS field in the Pod Spec (#104693, @ravisantoshgudimetla) [SIG API Machinery and Apps]

  • Introduce v1beta3 api for scheduler. This version

    • increases the weight of user specifiable priorities. The weights of following priority plugins are increased

      • TaintTolerations to 3 - as leveraging node tainting to group nodes in the cluster is becoming a widely-adopted practice
      • NodeAffinity to 2
      • InterPodAffinity to 2

    • Won't have HealthzBindAddress, MetricsBindAddress fields (#104251, @ravisantoshgudimetla) [SIG Scheduling and Testing]

  • JSON log output is configurable and now supports writing info messages to stdout and error messages to stderr. Info messages can be buffered in memory. The default is to write both to stdout without buffering, as before. (#104873, @pohly) [SIG API Machinery, Architecture, CLI, Cluster Lifecycle, Instrumentation, Node and Scheduling]

  • JobTrackingWithFinalizers graduates to beta. Feature is enabled by default. (#105687, @alculquicondor) [SIG Apps and Testing]

  • Remove NodeLease feature gate that was graduated and locked to stable in 1.17 release. (#105222, @cyclinder) [SIG Apps, Node and Testing]

  • TTLAfterFinished is now GA and enabled by default (#105219, @sahilvv) [SIG API Machinery, Apps, Auth and Testing]

  • The "Generic Ephemeral Volume" feature graduates to GA. It is now enabled unconditionally. (#105609, @pohly) [SIG API Machinery, Apps, Auth, Node, Scheduling, Storage and Testing]

  • The legacy scheduler policy config is removed in v1.23, the associated flags policy-config-file, policy-configmap, policy-configmap-namespace and use-legacy-policy-config are also removed. Migrate to Component Config instead, see https://kubernetes.io/docs/reference/scheduling/config/ for details. (#105424, @kerthcet) [SIG Scheduling and Testing]

  • Track the number of Pods with a Ready condition in Job status. The feature is alpha and needs the feature gate JobReadyPods to be enabled. (#104915, @alculquicondor) [SIG API Machinery, Apps, CLI and Testing]

Feature

  • Add a new distribute-cpus-across-numa option to the static CPUManager policy. When enabled, this will trigger the CPUManager to evenly distribute CPUs across NUMA nodes in cases where more than one NUMA node is required to satisfy the allocation. (#105631, @klueska) [SIG Node]

  • Add support to generate client-side binaries for windows/arm64 platform (#104894, @pacoxu) [SIG CLI, Testing and Windows]

  • Added a new feature gate CustomResourceValidationExpressions to enable expression validation for Custom Resource. (#105107, @cici37) [SIG API Machinery]

  • Adds new [alpha] command 'kubectl events' (#99557, @bboreham) [SIG CLI]

  • Client-go, using log level 9, trace the following events of an http request: - dns lookup - tcp dialing - tls handshake - time to get a connection from the pool - time to process a request (#105156, @aojea) [SIG API Machinery]

  • Client-go: pass DeleteOptions down to the fake client Reactor (#102945, @chenchun) [SIG API Machinery, Apps and Auth]

  • Enhance scheduler volumebinding plugin to handle Lost PVC as UnschedulableAndUnresolvable during PreFilter stage (#105245, @yibozhuang) [SIG Scheduling and Storage]

  • Feature-gate StorageObjectInUseProtection has been deprecated and cannot be disabled. It will be completely removed in 1.25 (#105495, @ikeeip) [SIG Apps]

  • Kubectl will now provide shell completion choices for the --output/-o flag (#105851, @marckhouzam) [SIG CLI]

  • Kubernetes is now built with Golang 1.17.2 (#105563, @mengjiao-liu) [SIG API Machinery, Cloud Provider, Instrumentation, Release and Testing]

  • Move the getAllocatableResources endpoint in podresource-api to the beta that will make it enabled by default. (#105003, @swatisehgal) [SIG Node and Testing]

  • Node affinity, node selector and tolerations are now mutable for jobs that are suspended and have never been started (#105479, @ahg-g) [SIG Apps, Scheduling and Testing]

  • Pod template annotations and labels are now mutable for jobs that are suspended and have never been started (#105980, @ahg-g) [SIG Apps]

  • PodSecurity: add a container image and manifests for the PodSecurity validating admission webhook (#105923, @liggitt) [SIG Auth]

  • PodSecurity: in 1.23+ restricted policy levels, pods and containers which set runAsUser=0 are forbidden at admission-time; previously, they would be rejected at runtime (#105857, @liggitt) [SIG Auth]

  • Shell completion now knows to continue suggesting resource names when the command supports it. For example "kubectl get pod pod1 " will suggest more pod names. (#105711, @marckhouzam) [SIG CLI]

  • Support to enable Hyper-V in GCE Windows Nodes created with kube-up (#105999, @mauriciopoppe) [SIG Cloud Provider and Windows]

  • The CPUManager policy options are now enabled, and we introduce a graduation path for the new CPU Manager policy options. (#105012, @fromanirh) [SIG Node and Testing]

  • The etcd container image now supports Windows. (#92433, @claudiubelu) [SIG API Machinery and Windows]

  • The pods and pod controllers that are exempted from the PodSecurity admission process are now marked with the "pod-security.kubernetes.io/exempt: user/namespace/runtimeClass" annotation, based on what caused the exemption.

    The enforcement level that allowed or denied pod during PodSecurity admission is now marked by the "pod-security.kubernetes.io/enforce-policy" annotation.

    The annotation that informs about audit policy violations changed from ""pod-security.kubernetes.io/audit" to ""pod-security.kubernetes.io/audit-violation". (#105908, @stlaz) [SIG Auth]

  • When feature gate JobTrackingWithFinalizers is enabled:

    • Limit the number of pods tracked in a single job sync to avoid starvation of small jobs.
    • The metric job_pod_finished_total counts the number of finished pods tracked by the job controller (#105197, @alculquicondor) [SIG Apps, Instrumentation and Testing]

Failing Test

  • Fixes hostpath storage e2e tests within SELinux enabled env (#104551, @Elbehery) [SIG Testing]

Bug or Regression

  • (PodSecurity admission) errors validating workload resources (deployment, replicaset, etc.) no longer block admission. (#106017, @tallclair) [SIG Auth]
  • Add Pod Security admission metrics: pod_security_evaluations_total, pod_security_exemptions_total, pod_security_errors_total (#105898, @tallclair) [SIG Auth, Instrumentation and Testing]
  • Apimachinery: pretty-printed json and yaml output is now indented consistently (#105466, @liggitt) [SIG API Machinery]
  • Change kubectl diff --invalid-arg status code from 1 to 2 to match docs (#105445, @ardaguclu) [SIG CLI]
  • Client-go uses the same http client for all the generated groups and versions, allowing to share customized transports for multiple groups versions. (#105490, @aojea) [SIG API Machinery, Auth, Instrumentation and Testing]
  • Evicted and other terminated pods will no longer revert to Running phase (#105462, @ehashman) [SIG Node and Testing]
  • Fix pod name of NonIndexed jobs to not include rogue -1 substring (#105676, @alculquicondor) [SIG Apps]
  • Fix scoring for NodeResourcesBalancedAllocation plugins when nodes have containers with no requests. (#105845, @ahmad-diaa) [SIG Scheduling]
  • Fix: consolidate logs for instance not found error fix: skip not found nodes when reconciling LB backend address pools (#105188, @nilo19) [SIG Cloud Provider]
  • Fix: do not delete the lb that does not exist (#105777, @nilo19) [SIG Cloud Provider]
  • Fix: ignore not a VMSS error for VMAS nodes in EnsureBackendPoolDeleted. (#105185, @ialidzhikov) [SIG Cloud Provider]
  • Fix: leave the probe path empty for TCP probes (#105253, @nilo19) [SIG Cloud Provider]
  • Fix: remove VMSS and VMSS instances from SLB backend pool only when necessary (#105839, @nilo19) [SIG Cloud Provider]
  • Fix: skip instance not found when decoupling vmss from lb (#105666, @nilo19) [SIG Cloud Provider]
  • Fixed a bug that prevents PersistentVolume that has a Claim UID which doesn't exist in local cache but exists in ETCD from being updated to Released phase. (#105211, @xiaopingrubyist) [SIG Apps]
  • Fixed architecture within manifest for non amd64 etcd images. (#105484, @saschagrunert) [SIG API Machinery]
  • Fixes a bug that could result in the EndpointSlice controller unnecessarily updating EndpointSlices associated with a Service that had Topology Aware Hints enabled. (#105267, @llhuii) [SIG Apps and Network]
  • Fixes the should support building a client with a CSR e2e test to work with clusters configured with short certificate lifetimes (#105396, @liggitt) [SIG Auth and Testing]
  • Generic ephemeral volumes can be used also as raw block devices, but the Pod validation was refusing to create pods with that combination. (#105682, @pohly) [SIG Apps, Storage and Testing]
  • Generic ephemeral volumes were not considered properly by the the node limits scheduler filter and the kubelet hostpath check. (#100482, @pohly) [SIG Node, Scheduling, Storage and Testing]
  • Kube-apiserver: fix a memory leak when deleting multiple objects with a deletecollection. (#105606, @sxllwx) [SIG API Machinery]
  • Kubeadm: do not allow empty "--config" paths to be passed to "kubeadm kubeconfig user" (#105649, @navist2020) [SIG Cluster Lifecycle]
  • Kubelet did not report kubelet_volume_stats_* metrics for generic ephemeral voiumes. (#105569, @pohly) [SIG Node]
  • Kubelet's Node Grace Shutdown will terminate probes when shutting down. (#105215, @rphillips) [SIG Node]
  • Kubernetes object references (= name + namespace) were not logged as struct when using JSON as log output format. (#104877, @pohly) [SIG API Machinery, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation and Storage]
  • Podresources interface was changed, now it returns only isolated cpus (#97415, @AlexeyPerevalov) [SIG Node and Testing]
  • Release-note Removed error message label from kubelet_started_pods_errors_total metric (#105213, @yxxhero) [SIG Instrumentation and Node]
  • Resolves a potential issue with GC and NS controllers which may delete objects after getting a 404 response from the server during its startup. This PR ensures that requests to aggregated APIs will get 503, not 404 while the APIServiceRegistrationController hasn't finished its job. (#104748, @p0lyn0mial) [SIG API Machinery]
  • Revert building binaries with PIE mode. (#105352, @ehashman) [SIG Node, Release and Security]
  • Support more than 100 disk mounts on Windows (#105673, @andyzhangx) [SIG Storage and Windows]
  • Support using negative array index in json patch replace operations. (#105896, @zqzten) [SIG API Machinery, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation and Storage]
  • The --leader-elect* CLI args are now honored in scheduler. (#105712, @Huang-Wei) [SIG Scheduling]
  • The client-go dynamic client sets the header 'Content-Type: application/json' by default (#104327, @sxllwx) [SIG API Machinery]
  • The pods/binding subresource now honors metadata.uid and metadata.resourceVersion preconditions (#105913, @aholic) [SIG Scheduling]
  • Topology Hints now excludes control plane notes from capacity calculations. (#104744, @robscott) [SIG Apps and Network]
  • Watch requests that are delegated to aggregated apiservers no longer reserve concurrency units (seats) in the API Priority and Fairness dispatcher for their entire duration. (#105511, @benluddy) [SIG API Machinery]
  • --log-flush-frequency had no effect in several commands or was missing. Help and warning texts were not always using the right format for a command (add_dir_header instead of add-dir-header). Fixing this included cleaning up flag handling in component-base/logs: that package no longer adds flags to the global flag sets. Commands which want the klog and --log-flush-frequency flags must explicitly call logs.AddFlags; the new cli.Run does that for commands. That helper function also covers flag normalization and printing of usage and errors in a consistent way (print usage text first if parsing failed, then the error). (#105076, @pohly) [SIG API Machinery, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node, Release, Scheduling and Testing]

Other (Cleanup or Flake)

  • All klog flags except for -v and -vmodule are deprecated. Support for -vmodule is only guaranteed for the text log format. (#105042, @pohly) [SIG API Machinery, Architecture, CLI, Cluster Lifecycle and Instrumentation]
  • Kube-apiserver: requests to node, service, and pod /proxy subresources with no additional URL path now only automatically redirect GET and HEAD requests. (#95128, @Riaankl) [SIG API Machinery, Architecture and Testing]
  • Migrate pkg/scheduler/framework/plugins/interpodaffinity/filtering.go,pkg/scheduler/framework/plugins/podtopologyspread/filtering.go, pkg/scheduler/framework/plugins/volumezone/volume_zone.go to structured logging (#105931, @mengjiao-liu) [SIG Instrumentation and Scheduling]
  • Migrated cmd/kube-scheduler/app/server.go, pkg/scheduler/framework/plugins/nodelabel/node_label.go, pkg/scheduler/framework/plugins/nodevolumelimits/csi.go, pkg/scheduler/framework/plugins/nodevolumelimits/non_csi.go to structured logging (#105855, @shivanshu1333) [SIG Instrumentation and Scheduling]
  • Migrated pkg/proxy to structured logging (#104891, @shivanshu1333) [SIG Network]
  • Migrated pkg/proxy/ipvs to structured logging (#104932, @shivanshu1333) [SIG Network]
  • Support allocating whole NUMA nodes in the CPUManager when there is not a 1:1 mapping between socket and NUMA node (#102015, @klueska) [SIG Node]

Dependencies

Added

  • sigs.k8s.io/json: c049b76

Changed

Removed

Nothing has changed.

v1.23.0-alpha.3

Downloads for v1.23.0-alpha.3

Source Code

filename sha512 hash
kubernetes.tar.gz 083e6ca03c9d701768b1b5666f354223a3f7dca9fc6410ce45bbf5947152620e300b46df9b6019134e7d736ba44916537eb3bea8fa57e5f7bc3cc34898b4a5dd
kubernetes-src.tar.gz c3fc74d52e1b7e808c03b9caa30e3e73be30eb8330ce676000b93d5324bbdba93bd005d125b999ba937b79d4751af99b37986911365416f7175d223345f95914

Client Binaries

filename sha512 hash
kubernetes-client-darwin-amd64.tar.gz 31d8adc657afbd305df18bfec397a825536357e23b241a19aa538b6ddefefc59743f737db98756e04deea89cc6f260d40a80f02b4d1dc34af1d19e8d796dcd8a
kubernetes-client-darwin-arm64.tar.gz b69c4d6cde1c476bafa2ca9916ce3e5bf7286be0ff6a08193bdd1a954ba89b64b1b14193d1acec17ccc141024ee3097971448017b5c9f1327e0961b1e92b2224
kubernetes-client-linux-386.tar.gz 059f25ee48aa4b0d1621d6ba87af8fb7e765634d723d98a4e9739f50d3703e7dd3973f4d1ed886c0f3ad6eba165ed81d4e63ecde3b39e66fcbec7d3aa2dfed2e
kubernetes-client-linux-amd64.tar.gz 291dba14160803065895799adcde39bdad7a5b0372403f283d6d5e9a094fe1fc79c70e7546f93ee692b9fd297e2667cb558e4209161ecb4bf89965df5746ed4d
kubernetes-client-linux-arm.tar.gz 988e12cd7466033578acc487447df376c409e4f79726a4721af1aedbe931e927b22a93d6224891b61b55c7a0ec12e42d8cfcd40e15a9a0cbbc1dbf0e59ab0341
kubernetes-client-linux-arm64.tar.gz b3f21dac41b38e671fa7a95892468e2c27fab51abf9c77b336550e5ec213af204e16cac11dd76262fedb0087cf5ad1950af7e36599a38d50cc270cf831cd4f0b
kubernetes-client-linux-ppc64le.tar.gz beebf01e2e4ff09bb711284bb9a5c7cc519e4ac8a826dc829394fa28bd9a3149ba73088eaf6712d39a8cab96b0a1c2859e9d5955fee892b759eaddcdeaa8b93c
kubernetes-client-linux-s390x.tar.gz 87e5d3d8ba01f9fefb2300e9f06146a254d39d72eaa10cad8c444428b738b3763483ee9eb82f0a13d2ff5aba35fdcb4320598fd5a6a2a07ea3fd00b4ac682d3c
kubernetes-client-windows-386.tar.gz 71bfc5a1df9c47735476af10225830212f68c83357ff7d443e18f9b7881524db910781a95d11ff6697cb587352059b5841f7b24fda40b5302ad252bfb6da7e51
kubernetes-client-windows-amd64.tar.gz 078b0c698f9535f3eee41ecf162d57e2ace67243da36067b78b30cfbb7b27cfcf97af4c5db48cdd592953e26b42b31794002eb96317476849e89e2126c6df99d

Server Binaries

filename sha512 hash
kubernetes-server-linux-amd64.tar.gz 951b790158dadf46c32e1a1e9c12f2cc8f41e1645602ebff6b4130a08a377bc6d92549186b420332d620d67191123d98a5d717ac0f5ee9643bebe88947ead8fa
kubernetes-server-linux-arm.tar.gz 0e7a5b9f39b4f45c45bdb5a19dd3695d28f53e1039d76bc572421c707917944d28b1dbfc36e59214b5bc2b93a787900d8e6eb0b587aa801ea8a8faacdb814a4e
kubernetes-server-linux-arm64.tar.gz 921e060120b8651a0f80977360faca9f207189cee10bc61f669ceba4e540ef48c0ceff1a877ee4c7d31b01b88096bce93c577f68f93b2341c8542dfd89972b60
kubernetes-server-linux-ppc64le.tar.gz 292cde446b754a87f4ef5384fadbd30017e53ed2744d45a724be467c86ccd9837bfb490db6396642a869937f2f0d080d9655e89ca3345f8365d109a9bcdd18d9
kubernetes-server-linux-s390x.tar.gz e0ea667f828ce3b36ca4b2a05fb286da5eb321852c50caf0957694553caf2908b27bcc37a5a82277a2606cf6ff4d9e33617ad61628845d9c21f5cf68c960ca92

Node Binaries

filename sha512 hash
kubernetes-node-linux-amd64.tar.gz e13cd3f75628d354bd1544a5495600fb905741431eb4af4da3d980cc0b7565e3f9c1585d9686cc4e967e54fb854f05bbedfe0c60bb7b855fa027ac8ac45b26e0
kubernetes-node-linux-arm.tar.gz 6c91b42350528692ff558b667bffd41c5b967c7aa6101471274e4b16b0ac6f84afe01722881328fd4f6f8fe71c7852620fa000186c6f7e56e498fcc2c67ad793
kubernetes-node-linux-arm64.tar.gz 81728e1388e9cdb436d6847c868f28ab2771331e5e40cd5a7af13cb8dc80a7e4e66a215c12f8183b4884807a3962f913ef5343b889e3c4ecd0e410e8d53aaea9
kubernetes-node-linux-ppc64le.tar.gz 299649f1b25cc38f3a7543ef4d3ee6d42c85e24ac41b4eb61927bc5c5f0c533a39f9ddd4d5ad1df54c625d77aeb41f6c31b1ca7fd8983262f84fefdf1cb2cfd0
kubernetes-node-linux-s390x.tar.gz fd6cbc93f98abff9803b43215af6e75a4f7b91ca06969220a779468f34b5ec5ec69f20b529e0cd7b10ba8769bbe2507d46f84ce1d8cd0760380ab9264dd94672
kubernetes-node-windows-amd64.tar.gz a5bfaf2e3ad8d3d2127c3e3e0f131c615a03563253da6bf0e1fd793f6ef71287f341ce1bd0d35eb9a81e0721a5baf03e7c72863b5ed8eb45e8fe70573904ed54

Changelog since v1.23.0-alpha.2

Changes by Kind

Deprecation

  • Remove 'master' as a valid EgressSelection type in the EgressSelectorConfiguration API. (#102242, @pacoxu) [SIG API Machinery and Cloud Provider]
  • Remove VolumeSubpath feature gate (#105090, @saad-ali) [SIG Apps, Node and Storage]
  • The deprecated --experimental-bootstrap-kubeconfig flag has been removed. This can be set via --bootstrap-kubeconfig. (#103172, @niulechuan) [SIG Node]

API Change

  • Client-go impersonation config can specify a UID to pass impersonated uid information through in requests. (#104483, @margocrawf) [SIG API Machinery, Auth and Testing]
  • IPv6DualStack feature moved to stable. Controller Manager flags for the node IPAM controller have slightly changed:
    1. When configuring a dual-stack cluster, the user must specify both --node-cidr-mask-size-ipv4 and --node-cidr-mask-size-ipv6 to set the per-node IP mask sizes, instead of the previous --node-cidr-mask-size flag.
    2. The --node-cidr-mask-size flag is mutually exclusive with --node-cidr-mask-size-ipv4 and --node-cidr-mask-size-ipv6.
    3. Single-stack clusters do not need to change, but may choose to use the more specific flags. Users can use either the older --node-cidr-mask-size flag or one of the newer --node-cidr-mask-size-ipv4 or --node-cidr-mask-size-ipv6 flags to configure the per-node IP mask size, provided that the flag's IP family matches the cluster's IP family (--cluster-cidr). (#104691, @khenidak) [SIG API Machinery, Apps, Auth, Cloud Provider, Cluster Lifecycle, Network, Node and Testing]
  • Kubelet: turn the KubeletConfiguration v1beta1 ResolverConfig field from a string to *string. (#104624, @Haleygo) [SIG Cluster Lifecycle and Node]

Feature

  • Add mechanism to load simple sniffer class into fluentd-elasticsearch image (#92853, @cosmo0920) [SIG Cloud Provider and Instrumentation]
  • Kubeadm: do not check if the '/etc/kubernetes/manifests' folder is empty on joining worker nodes during preflight (#104942, @SataQiu) [SIG Cluster Lifecycle]
  • The kube-apiserver's Prometheus metrics have been extended with some that describe the costs of handling LIST requests. They are as follows.
    • apiserver_cache_list_total: Counter of LIST requests served from watch cache, broken down by resource_prefix and index_name
    • apiserver_cache_list_fetched_objects_total: Counter of objects read from watch cache in the course of serving a LIST request, broken down by resource_prefix and index_name
    • apiserver_cache_list_evaluated_objects_total: Counter of objects tested in the course of serving a LIST request from watch cache, broken down by resource_prefix
    • apiserver_cache_list_returned_objects_total: Counter of objects returned for a LIST request from watch cache, broken down by resource_prefix
    • apiserver_storage_list_total: Counter of LIST requests served from etcd, broken down by resource
    • apiserver_storage_list_fetched_objects_total: Counter of objects read from etcd in the course of serving a LIST request, broken down by resource
    • apiserver_storage_list_evaluated_objects_total: Counter of objects tested in the course of serving a LIST request from etcd, broken down by resource
    • apiserver_storage_list_returned_objects_total: Counter of objects returned for a LIST request from etcd, broken down by resource (#104983, @MikeSpreitzer) [SIG API Machinery and Instrumentation]
  • Turn on CSIMigrationAzureDisk by default on 1.23 (#104670, @andyzhangx) [SIG Cloud Provider]

Bug or Regression

  • Changes behaviour of kube-proxy start; does not attempt to set specific sysctl values (which does not work in recent Kernel versions anymore in non-init namespaces), when the current sysctl values are already set higher. (#103174, @Napsty) [SIG Network]
  • Fix job controller syncs: In case of conflicts, ensure that the sync happens with the most up to date information. Improves reliability of JobTrackingWithFinalizers. (#105214, @alculquicondor) [SIG Apps]
  • Fix system default topology spreading when nodes don't have zone labels. Pods correctly spread by default now. (#105046, @alculquicondor) [SIG Scheduling]
  • Headless Services with no selector which were created without dual-stack enabled will be defaulted to RequireDualStack instead of PreferDualStack. This is consistent with such Services which are created with dual-stack enabled. (#104986, @thockin) [SIG Network]
  • Kube-apiserver: events created via the events.k8s.io API group for cluster-scoped objects are now permitted in the default namespace as well for compatibility with events clients and the v1 API (#100125, @h4ghhh) [SIG API Machinery, Apps and Testing]
  • Kube-controller incorrectly enabled support for generic ephemeral inline volumes if the storage object in use protection feature was enabled. (#104913, @pohly) [SIG API Machinery]
  • Kubeadm: switch the preflight check (called 'Swap') that verifies if swap is enabled on Linux hosts to report a warning instead of an error. This is related to the graduation of the NodeSwap feature gate in the kubelet to Beta and being enabled by default in 1.23 - allows swap support on Linux hosts. In the next release of kubeadm (1.24) the preflight check will be removed, thus we recommend that you stop using it - e.g. via --ignore-preflight-errors or the kubeadm config. (#104854, @pacoxu) [SIG Cluster Lifecycle]
  • Makes the etcd client (used by the API server) retry certain types of errors. The full list of retriable (codes.Unavailable) errors can be found at https://github.com/etcd-io/etcd/blob/main/api/v3rpc/rpctypes/error.go#L72 (#105069, @p0lyn0mial) [SIG API Machinery]
  • When a static pod file is deleted and recreated while using a fixed UID, the pod was not properly restarted. (#104847, @smarterclayton) [SIG Node and Testing]
  • XFS-filesystems are now force-formatted (option -f) in order to avoid problems being formatted due to detection of magic super-blocks. This aligns with the behaviour of formatting of ext3/4 filesystems. (#104923, @davidkarlsen) [SIG Storage]

Other (Cleanup or Flake)

  • Enhanced error message for nodes not selected by scheduler due to pod's PersistentVolumeClaim(s) bound to PersistentVolume(s) that do not exist. (#105196, @yibozhuang) [SIG Scheduling and Storage]
  • Kubeadm: remove the --port flag from the manifest for the kube-scheduler since the flag has been a NO-OP since 1.23 and insecure serving was removed for the component. (#105034, @pacoxu) [SIG Cluster Lifecycle]
  • Migrate cmd/proxy/{config, healthcheck, winkernel} to structured logging (#104944, @jyz0309) [SIG Network]
  • Migrate cmd/proxy/app and pkg/proxy/meta_proxier to structured logging (#104928, @jyz0309) [SIG Apps, Cluster Lifecycle, Network, Node and Testing]
  • Migrate pkg/proxy to structured logs (#104908, @CIPHERTron) [SIG Network]
  • Migrated pkg/proxy/winuserspace to structured logging (#105035, @shivanshu1333) [SIG Network]
  • The BoundServiceAccountTokenVolume feature gate that is GA since v1.22 is unconditionally enabled, and can no longer be specified via the --feature-gates argument. (#104167, @ialidzhikov) [SIG Auth]
  • The SupportPodPidsLimit and SupportNodePidsLimit feature gates that are GA since v1.20 are unconditionally enabled, and can no longer be specified via the --feature-gates argument. (#104163, @ialidzhikov) [SIG Node]
  • Update build images to Debian 11 (Bullseye)
    • debian-base:bullseye-v1.0.0
    • debian-iptables:bullseye-v1.0.0
    • go-runner:v2.3.1-go1.17.1-bullseye.0
    • kube-cross:v1.23.0-go1.17.1-bullseye.1
    • setcap:bullseye-v1.0.0
    • cluster/images/etcd: Build 3.5.0-2 image
    • test/conformance/image: Update runner image to base-debian11 (#105158, @justaugustus) [SIG API Machinery, Architecture, Release and Testing]

Dependencies

Added

Nothing has changed.

Changed

Removed

Nothing has changed.

v1.23.0-alpha.2

Downloads for v1.23.0-alpha.2

Source Code

filename sha512 hash
kubernetes.tar.gz 121d51f42a52b28e27a4b2f914a4f80fa3fba6328e6a4a5c96dec39c5b28c05461fcc290ef35a49058e237091532b24db3cd8c61801bcb6736aee1dd7dbcffc3
kubernetes-src.tar.gz 641d47241acfadb3b13bccec57795749d2c9e3e07ffa7aa4b30df3a488643631eb8e5cd581bcfb764dff4ac5ed755f72d94e80746142123b09e1675e81421a91

Client Binaries

filename sha512 hash
kubernetes-client-darwin-amd64.tar.gz f734cb514ee56adcb2d991a6f0550df907c72f8a61cc2a13117e61b8d5826ff942a582a2e9383deb1a61d5df2243362f1327942a3b4883490eb3296647ce3737
kubernetes-client-darwin-arm64.tar.gz 24d1f851cd5782f8f39054e37beda1554dadd8a28cb3272b00d50fc095d1fc3018768c1ea72a44eda61ff0f58f71b33dd28cbdc54467d620e87c3694ecf14cc2
kubernetes-client-linux-386.tar.gz 082ad4abea58de3b629fc2ed4560a836cdbeb1adefb0c4cf47044bf33c750d8fcd8a06e2c4ce365853e83a58d52e0129d510a698dd894bd1261f8184dd1cab42
kubernetes-client-linux-amd64.tar.gz b3b0b23479c05b57ca574cf17cdcde7e716033bc4f6a80532d1175d8e533e3202bece0dcf503731d5a60319c526ce1ce4a0bc900bf87536321208a59cf890e35
kubernetes-client-linux-arm.tar.gz f5dac2976ce04310f74bba6102080554309b851fbd966ff1220d3eb23089db8eb8da519a6bd8865c94f2f24346a4d27eb40fd0a3ff06ca9c6874e1fc6f356b67
kubernetes-client-linux-arm64.tar.gz 057b372150749b13a38e04802c7cf566765e0fbb27f1b5f7bf6d3cc3f71eb3020916ea7f8579ecc7fcc10e2db1b5c8caa31a1e8a3aac80da86e4e777f515d42f
kubernetes-client-linux-ppc64le.tar.gz 9a090d22aeba011c6d039bff59dbdc23ac4a112828db3cbba588d8b0ee1cd14d16e0eacefbb000e5a3ff26bcce4730824819f86a99b7a9826f35fa9964f9f27a
kubernetes-client-linux-s390x.tar.gz 435e20055badb619289dc7c572af300bd2f86068d0b8f326e8d9abfda5347f2449e316158c412e9b946a2541208c3e8cc6e5c823946e74ac4fc2d594d410179a
kubernetes-client-windows-386.tar.gz 55f192a4d095d494bb53af1b7133124b762a677eb46247b9dba71d10ea6830b37c30d603908e7a9c63f371baff508b19406e89b231ed5ece0497627f09753f68
kubernetes-client-windows-amd64.tar.gz 944059d1f1918a793490b95be8130d06189508ba8e79e79ca8cfd2ab98bf396ac551786514b093cc6afe4b3fd15736d728cfcdce18bb32fbee41bc0a97f5c4be

Server Binaries

filename sha512 hash
kubernetes-server-linux-amd64.tar.gz a76a4b86ee151ba027f7cf4a2072451ae4c829182bb14e00ce1967421744bfc1e58f141b6eaf2ab27ece67054ae307f8e0768477ab9c3c4749eaad397d495182
kubernetes-server-linux-arm.tar.gz 95aeb4eb473ab4920d81904bc89c6126732b9c6888f9e57493ee99d692042ca44f6844ac1dade1409565f4d9fbec59445402e1f7deac6cbf5b6df16ac814b58c
kubernetes-server-linux-arm64.tar.gz 3c56e906aafc2a1ac72300352a334662bec5d59e3e523c19b9d65bc52ad9075dc2631f259513efd0f654e220fe0e7d54dfa5028d7eaad81d5d87ca251653f75d
kubernetes-server-linux-ppc64le.tar.gz b74bacafe9bb6a7cf407747b03e78ae3873e50deec4eaa08758d5e1d5287ac23af59b3ef26f888fe4cd44ccb1455beafcd1384e700230eb445720e3acae5f2e3
kubernetes-server-linux-s390x.tar.gz d3f8f8d9c233b114129f615252d42782cd366978a49506393a40af3f8b5b1250ce99e9806881675e112a69270a0411fb2f00ea19b99ad7415b9e0074beb2726d

Node Binaries

filename sha512 hash
kubernetes-node-linux-amd64.tar.gz 146e2f762c179178a57a8c7af7c26470c5d580b8ff8400615162ad1056625f87ce2b32598538d82652f88639e54afb782810529b074c36eb52cc6374414a6181
kubernetes-node-linux-arm.tar.gz 9357d1b387e1b049fb6cec06a7081afc2ce7e906484c9b061fb0449d147a6c4f9c9dc7a9219cdca5ed71df6c73784f360018d9e48d4fa2aa7eeabef60649d7a4
kubernetes-node-linux-arm64.tar.gz 8394f8f9d6ee823cb9a470ea67e15d4d0c6aca7065fe826788f50955905373fc3cdddd6db43901c07736588d8d6a3d3e2916bc8d45fd6bd06307583686137a0a
kubernetes-node-linux-ppc64le.tar.gz 7211cb426834484bff39f1ab3c9541203429039f8f5e522ca9e28c43da749e197128a3cae28db0467fc339305d2f23f85e8b4ed9ec116506c3d8076744a88d5e
kubernetes-node-linux-s390x.tar.gz a7c1a38250398171d3df5865749e9928867c4f44106ae66d44cf9f948ce4f4eed9d1f273a5d369996425b1e12482fceccde4c7652770a8c9fb3f161811323b69
kubernetes-node-windows-amd64.tar.gz 2007b3b16597cc06b486f87f35b6c637404f07c11d88b8c8e1c2c9bbea97f762bd7d4f9a31f42f78a917c595af5cb89e6885dd88f3766836dc6e4ec79cf084f2

Changelog since v1.23.0-alpha.1

Changes by Kind

Deprecation

  • Controller-manager: the following flags have no effect and would be removed in v1.24:

    • --port
    • --address The insecure port flags --port may only be set to 0 now. Also metricsBindAddress and healthzBindAddress fields from kubescheduler.config.k8s.io/v1beta1 are no-op and expected to be empty. Removed in kubescheduler.config.k8s.io/v1beta2 completely.

    In addition, please be careful that:

    • kube-scheduler MUST start with --authorization-kubeconfig and --authentication-kubeconfig correctly set to get authentication/authorization working.
    • liveness/readiness probes to kube-scheduler MUST use HTTPS now, and the default port has been changed to 10259.
    • Applications that fetch metrics from kube-scheduler should use a dedicated service account which is allowed to access nonResourceURLs /metrics. (#96345, @ingvagabund) [SIG Cloud Provider, Scheduling and Testing]

  • Removed deprecated metric scheduler_volume_scheduling_duration_seconds (#104518, @dntosas) [SIG Instrumentation, Scheduling and Storage]

API Change

  • A small regression in Service updates was fixed. The circumstances are so unlikely that probably nobody would ever hit it. (#104601, @thockin) [SIG Network]
  • Introduce v1beta2 for Priority and Fairness with no changes in API spec (#104399, @tkashem) [SIG API Machinery and Testing]
  • Kube-apiserver: Fixes handling of CRD schemas containing literal null values in enums. (#104969, @liggitt) [SIG API Machinery, Apps and Network]
  • Kubelet: turn the KubeletConfiguration v1beta1 ResolverConfig field from a string to *string. (#104624, @Haleygo) [SIG Cluster Lifecycle and Node]
  • Kubernetes is now built using go1.17 (#103692, @justaugustus) [SIG API Machinery, Apps, Architecture, Auth, Autoscaling, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node, Release, Scheduling, Storage and Testing]
  • Removed deprecated --seccomp-profile-root/seccompProfileRoot config (#103941, @saschagrunert) [SIG Node]
  • Since golang 1.17 both net.ParseIP and net.ParseCIDR rejects leading zeros in the dot-decimal notation of IPv4 addresses. Kubernetes will keep allowing leading zeros on IPv4 address to not break the compatibility. IMPORTANT: Kubernetes interprets leading zeros on IPv4 addresses as decimal, users must not rely on parser alignment to not being impacted by the associated security advisory: CVE-2021-29923 golang standard library "net" - Improper Input Validation of octal literals in golang 1.16.2 and below standard library "net" results in indeterminate SSRF & RFI vulnerabilities. Reference: https://nvd.nist.gov/vuln/detail/CVE-2021-29923 (#104368, @aojea) [SIG API Machinery, Apps, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node, Release, Scalability, Scheduling, Storage and Testing]
  • StatefulSet minReadySeconds is promoted to beta (#104045, @ravisantoshgudimetla) [SIG Apps and Testing]
  • The Service.spec.ipFamilyPolicy field is now required in order to create or update a Service as dual-stack. This is a breaking change from the beta behavior. Previously the server would try to infer the value of that field from either ipFamilies or clusterIPs, but that caused ambiguity on updates. Users who want a dual-stack Service MUST specify ipFamilyPolicy as either "PreferDualStack" or "RequireDualStack". (#96684, @thockin) [SIG API Machinery, Apps, Network and Testing]
  • Users of LogFormatRegistry in component-base must update their code to use the logr v1.0.0 API. The JSON log output now uses the format from go-logr/zapr (no v field for error messages, additional information for invalid calls) and has some fixes (correct source code location for warnings about invalid log calls). (#104103, @pohly) [SIG API Machinery, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation and Storage]
  • When creating an object with generateName, if a conflict occurs the server now returns an AlreadyExists error with a retry option. (#104699, @vincepri) [SIG API Machinery]

Feature

  • Add fish shell completion to kubectl (#92989, @WLun001) [SIG CLI]
  • Added PowerShell completion generation by running kubectl completion powershell (#103758, @zikhan) [SIG CLI]
  • Added a Processing condition for the workqueue API Changed Shutdown for the workqueue API to wait until the work queue finishes processing all in-flight items. (#101928, @alexanderConstantinescu) [SIG API Machinery and Apps]
  • Added a new flag --append-server-path to kubectl proxy that will automatically append the kube context server path to each request. (#97350, @FabianKramm) [SIG API Machinery, CLI and Testing]
  • Added support for setting controller-manager log level online (#104571, @h4ghhh) [SIG API Machinery, Apps and Cloud Provider]
  • Adding support for multiple --from-env-file flags (#104232, @lauchokyip) [SIG CLI]
  • Cloud providers can set service account names for cloud controllers. (#103178, @nckturner) [SIG API Machinery and Cloud Provider]
  • Health check of kube-controller-manager now includes each controller. (#104667, @jiahuif) [SIG API Machinery and Cloud Provider]
  • Kube-scheduler now logs node and plugin scoring even though --v<10
    • socres of the top 3 plugins in the top 3 nodes are dumped if --v=4,5
    • socres of all plugins in the top 6 nodes are dumped if --v=6,7,8,9 (#103515, @muma378) [SIG Scheduling]
  • Kubernetes is now built with Golang 1.17.1 (#104904, @cpanato) [SIG API Machinery, Cloud Provider, Instrumentation, Release and Testing]
  • The pause image list now contains Windows Server 2022 (#104438, @nick5616) [SIG Windows]
  • Updates debian-iptables to v1.6.7 to pick up CVE fixes (#104970, @PushkarJ) [SIG API Machinery, Network, Release, Security and Testing]

Documentation

  • Conformance: the test "[sig-network] EndpointSlice should have Endpoints and EndpointSlices pointing to API Server [Conformance]" only requires that there is an EndpointSlice that references the "kubernetes.default" service, it no longer requires that its named "kubernetes". (#104664, @aojea) [SIG Architecture, Network and Testing]

Bug or Regression

  • A pod that the Kubelet rejects was still considered as being accepted for a brief period of time after rejection, which might cause some pods to be rejected briefly that could fit on the node. A pod that is still terminating (but has status indicating it has failed) may also still be consuming resources and so should also be considered. (#104817, @smarterclayton) [SIG Node]
  • Changed kubectl describe to compute Age of an event using the count and lastObservedTime fields available in the event series (#104482, @harjas27) [SIG CLI]
  • Don't prematurely close reflectors in case of slow initialization in watch based manager to fix issues with inability to properly mount secrets/configmaps. (#104604, @wojtek-t) [SIG Node]
  • Fix Job tracking with finalizers for more than 500 pods, ensuring all finalizers are removed before counting the Pod. (#104666, @alculquicondor) [SIG Apps and Instrumentation]
  • Fix a regression where the Kubelet failed to exclude already completed pods from calculations about how many resources it was currently using when deciding whether to allow more pods. (#104577, @smarterclayton) [SIG Node]
  • Fix detach disk issue on deleting vmss node (#104572, @andyzhangx) [SIG Cloud Provider]
  • Fix: ensure InstanceShutdownByProviderID return false for creating Azure VMs (#104382, @feiskyer) [SIG Cloud Provider]
  • Fix: ignore the case when comparing azure tags in service annotation (#104705, @nilo19) [SIG Cloud Provider]
  • Fix: ignore the case when updating Azure tags (#104593, @nilo19) [SIG Cloud Provider]
  • Fixed bug where kubectl would emit duplicate warning messages for flag names that contain an underscore and recommend using a nonexistent flag in some cases (#103852, @brianpursley) [SIG CLI and Cluster Lifecycle]
  • Fixed client IP preservation for NodePort service with protocol SCTP in ipvs mode (#104756, @tnqn) [SIG Network]
  • Fixed occasional pod cgroup freeze when using cgroup v1 and systemd driver. (#104528, @kolyshkin) [SIG Node]
  • Fixes a regression that could cause panics in LRU caches in controller-manager, kubelet, kube-apiserver, or client-go (#104466, @stbenjam) [SIG API Machinery, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation and Storage]
  • Kube-apiserver: fixes an issue where an admission webhook can observe a v1 Pod object that does not have the defaultMode field set in the injected service account token volume (#104523, @liggitt) [SIG Auth]
  • Kube-proxy health check ports used to listen to : for each of the services. This is not needed and opens ports in addresses the cluster user may not have intended. The PR limits listening to all node address which are controlled by --nodeport-addresses flag. if no addresses are provided then we default to existing behavior by listening to : for each service (#104742, @khenidak) [SIG Network]
  • Kube-scheduler now doesn't print any usage message when unknown flag is specified (#104503, @sanposhiho) [SIG Scheduling]
  • Metrics changes: Fix exposed buckets of scheduler_volume_scheduling_duration_seconds_bucket metric (#100720, @dntosas) [SIG Apps, Instrumentation, Scheduling and Storage]
  • Scheduler resource metrics over fractional binary quantities (2.5Gi, 1.1Ki) were incorrectly reported as very small values. (#103751, @y-tag) [SIG API Machinery and Scheduling]

Other (Cleanup or Flake)

  • Generic ephemeral volumes: better pod events ("waiting for ephemeral volume controller to create the persistentvolumeclaim"" instead of "persistentvolumeclaim not found") (#104605, @pohly) [SIG Scheduling and Storage]
  • Kubeadm: remove the deprecated flags "--csr-only" and "--csr-dir" from "kubeadm certs renew". Please use "kubeadm certs generate-csr" instead. (#104796, @RA489) [SIG Cluster Lifecycle]
  • Migrate pkg/scheduler to structured logging (#99273, @yangjunmyfm192085) [SIG Scheduling]
  • Migrated pkg/proxy/userspace to structured logging (#104931, @shivanshu1333) [SIG Network]
  • More detailed logging has been added to the EndpointSlice controller for Topology Aware Hints. (#104741, @robscott) [SIG Apps and Network]
  • Support for Windows Server 2022 was added to the k8s.gcr.io/pause:3.6 image. (#104711, @claudiubelu) [SIG CLI, Cloud Provider, Cluster Lifecycle, Node, Release and Testing]
  • The maximum length of the CSINode id field has increased to 256 bytes to match the CSI spec (#104160, @pacoxu) [SIG Storage]
  • Update conformance image to use debian-base:buster-v1.9.0 (#104696, @PushkarJ) [SIG Architecture, Release, Security and Testing]
  • volume.kubernetes.io/storage-provisioner annotation will be added to dynamic provisioning required PVC. volume.beta.kubernetes.io/storage-provisioner annotation is deprecated. (#104590, @Jiawei0227) [SIG Apps and Storage]

Dependencies

Added

  • bazil.org/fuse: 371fbbd
  • github.com/go-logr/zapr: v1.1.0
  • github.com/kr/fs: v0.1.0
  • github.com/pkg/sftp: v1.10.1

Changed

Removed

v1.23.0-alpha.1

Downloads for v1.23.0-alpha.1

Source Code

filename sha512 hash
kubernetes.tar.gz f7c76f1e077b5d98019347b2c9b79eaa0c79d428542b9c15dab23886c276ca16314f200ca37af914c52264c0e1e5d0bde639d6adf37368d5e7b29d230df00d95
kubernetes-src.tar.gz f267f26eca20cd7018e68abeeed38aed5c10dbbae7c531c4e08e507196a4dd3f511eb8d41ee8b09495544337d8e1940a8ca04e94084f8dd172698a96564fb070

Client Binaries

filename sha512 hash
kubernetes-client-darwin-amd64.tar.gz deb110839c2c3cf94ca9b29df2f0b07b3fad6937d7bb6e9d2516d01345c8e324f6ab86fe1d34f1443f04c3d1fc328b53b3d756c295f4ed22f1994071fbc8c9cb
kubernetes-client-darwin-arm64.tar.gz 1473cb9fc4847b0daff6c9e3189ce55fadc22fb6190161e744e5438066a714cb467fdebfb35f6445a27f5010df94ee602fff492a2382e0f308fda111d53af1f4
kubernetes-client-linux-386.tar.gz ed5f5b0777ca51790d185764afc2c812f82ae27c35d897570fc86cabee90dc0a445d9d8c37c981bd3684ba9cd47dc0d75d0094578e79ef7b591d3c1b6564280f
kubernetes-client-linux-amd64.tar.gz 39f2a888e7a43c9e4a4018301894786f6babe23d79ab7a143e06444f69bc14aec2e158d355c5b48da4356e7bd72ec9b1268f8b12815c8b709395f36ad9a68a2f
kubernetes-client-linux-arm.tar.gz b6b8333d8adb4bc6a943bcd2c6cd1a0aeaf0b926d06aa03b759e3c723c81ccc91804debc64fedcd7d678eefdee9bdacc52b2891bd084a15fd5f7918a70e51a15
kubernetes-client-linux-arm64.tar.gz 3cb8217b9a5363cebad4989253e02c8a37259b61eafc2f08681508c11c5f68448cad43282257c3d90ad510cc9a62645b7f1adeb99fedf5e13c181495e3754ee4
kubernetes-client-linux-ppc64le.tar.gz e411700fb13b25deca6347983cdafe47199f0df00086ccd7b3e7d52a7b3bee7e96a85c2568dd52c956fd4ea8b4a6991859c57c9b73a13e06440b456c65b11687
kubernetes-client-linux-s390x.tar.gz 6c1395792a175de77436352d0893476363497b0f6a616f4415f91aed5e780d1f25b515021939a7563046237c7b651caba0d1fbf7c4c461677d1b9308b227e94c
kubernetes-client-windows-386.tar.gz f3aec7136c21d24a99145ce294a859078fcbf11bae132b8b4081555a6656c0d95ccbaca02a86dc257d557ecebc0673d0771b9cdd10593712a643e8cc0f61d681
kubernetes-client-windows-amd64.tar.gz b29697ba0a25f3d871ffbe5800dcb23ec9fd27c0122a284e17c21f1258f7dd9d341813aeb7826159c7999581a16db19fbb6eeeab48f5c89975df7595d19102c3

Server Binaries

filename sha512 hash
kubernetes-server-linux-amd64.tar.gz a5b3edca559b84cd9d22b43b23d0607951d434e185dcb313b831604d83dd306cfc017599994d3944ce77360116024eb59a302851325bb2c29c185a80db2e6eac
kubernetes-server-linux-arm.tar.gz 2334dbcff3ba22a50f252998eb63991b6c816659dbaa5f749370fc1b1f78f0af7739e50ab64c14a23c4e7dfa8917568e2a3b85bdffdb2cc691ee23ae8f5c8326
kubernetes-server-linux-arm64.tar.gz 58674443ce6e359a995dd7c4289bf730e616bcaf336837b77333a206d4e98693d9356a0a670ffbe0b274e2997a8b76a164153cf084f0ff5f91f40f00b5512684
kubernetes-server-linux-ppc64le.tar.gz f60ebdd04e2348b1ba51540cad93fa24cb133fd25db97150000bffaff8ccb41e1b6506bcde6b7d913aee7701478f975a97775430a82980105383fdb1cc13d260
kubernetes-server-linux-s390x.tar.gz ff008aa0ba1bf755f32c7251c6aceb12b6f9de00d2e2729302b51960e70e486bd82da62d21d70ad81c14e01910ab2afe0fd2509ebfdec050d36f88ee1f0330b2

Node Binaries

filename sha512 hash
kubernetes-node-linux-amd64.tar.gz 352502f10fbc4579bd9556e3f73ca7513184371ea563d12a39d655d39bb14ccf0f485f4f2b54a77d984c91ff0de2acea7225f98532a1247da5b9ecc65081bc1a
kubernetes-node-linux-arm.tar.gz af9de95e2b9e4c1f39cb9757d4dca020f7d276b6702302a2d92e7a93e9986528615ce54531e62b96f6e8a0b9863cddbb264f42b1f59374948ac3499af60d9532
kubernetes-node-linux-arm64.tar.gz 45a286cb1d469b16d046af02047cf63a8407222e4a39fe696f5652e0587e0c9ffbdbab6505ce85e2726ba10db3189a7fbe70e316bc610caedc8cbb49fed28076
kubernetes-node-linux-ppc64le.tar.gz 7a540a3ff0295998a1679b0ccd50cb1825faf1d0afd6ed08138ab3767c83a2743aa43b122c8da89ee00161f57c0af8d76012e890f9fe6d77b4ee8aff4e32e50f
kubernetes-node-linux-s390x.tar.gz 3cd7656221ac2fa161abcf237878cff26c1d97cf77d9b784736c97a56841397ff859e43947d81a83f8fe4164701da41a1dad69b551c4e1fee49b3f8196878236
kubernetes-node-windows-amd64.tar.gz 21e63913024e88a48244a598cd400fbae6ce8f8910202f1b635812fbc9281b7c6097eb10a321dd18846484a198845bba58970d83b5119a367862cf8418d4d08c

Changelog since v1.22.0

Urgent Upgrade Notes

(No, really, you MUST read this before you upgrade)

  • Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

([#104389](kubernetes#104389), [@saschagrunert](https://github.com/saschagrunert)) [SIG Node]
  • Kubeadm: remove the deprecated flag --experimental-patches for the init|join|upgrade commands. The flag --patches is no longer allowed in a mixture with the flag --config. Please use the kubeadm configuration for setting patches for a node using {Init|Join}Configuration.patches. (#104065, @pacoxu) [SIG Cluster Lifecycle]

Changes by Kind

Deprecation

  • Add apiserver_longrunning_requests metric to replace the soon to be deprecated apiserver_longrunning_gauge metric. (#103799, @jyz0309) [SIG API Machinery, Cluster Lifecycle and Instrumentation]
  • Kubeadm: remove the --port flag from the manifest for the kube-controller-manager since the flag has been a NO-OP since 1.22 and insecure serving was removed for the component. (#104157, @knight42) [SIG Cluster Lifecycle]

API Change

  • CSIDriver.Spec.StorageCapacity can now be modified. (#101789, @pohly) [SIG Storage]
  • Kube-apiserver: The rbac.authorization.k8s.io/v1alpha1 API version is removed; use the rbac.authorization.k8s.io/v1 API, available since v1.8. The scheduling.k8s.io/v1alpha1 API version is removed; use the scheduling.k8s.io/v1 API, available since v1.14. (#104248, @liggitt) [SIG API Machinery, Auth, Network and Testing]
  • Kube-controller-manager supports '--concurrent-ephemeralvolume-syncs' flag to set the number of ephemeral volume controller workers. (#102981, @SataQiu) [SIG API Machinery and Apps]

Feature

  • Adding support for multiple --from-env-file flags (#101646, @lauchokyip) [SIG CLI]

  • All folks to build kubernetes with a custom kube-cross image (#104185, @dims) [SIG Release and Testing]

  • Allow node expansion of local volumes (#102886, @gnufied) [SIG Storage and Testing]

  • Client-go event library allows customizing spam filtering function. It is now possible to override SpamKeyFunc, which is used by event filtering to detect spam in the events. (#103918, @olagacek) [SIG API Machinery and Instrumentation]

  • Constants/variables from k8s.io for STABLE metrics is now supported (#103654, @coffeepac) [SIG Auth, Instrumentation, Node and Testing]

  • Display Labels when kubectl describe ingress (#103894, @kabab) [SIG CLI]

  • Expose a NewUnstructuredExtractor from apply configurations meta/v1 package that enables extracting objects into unstructured apply configurations (#103564, @kevindelgado) [SIG API Machinery, Cluster Lifecycle, Release and Testing]

  • Introduce a feature gate DisableKubeletCloudCredentialProviders which allows disabling the in-tree kubelet credential providers.

    The DisableKubeletCloudCredentialProviders FeatureGate is currently in Alpha, which means is currently disabled by default. Once the FeatureGate moves to beta, in-tree credential providers will be disabled by default, and users will need to migrate to using external credential providers. (#102507, @ostrain) [SIG Cloud Provider]

  • Introduces a new metric: admission_webhook_request_total with the following labels: name (string) - the webhook name, type (string) - the admission type, operation (string) - the requested verb, code (int) - the HTTP status code, rejected (bool) - whether the request was rejected, namespace (string) - the namespace of the requested resource. (#103162, @rmoriar1) [SIG API Machinery and Instrumentation]

  • Kube-up.sh installs csi-proxy v1.0.1-gke.0 (#104426, @mauriciopoppe) [SIG Cloud Provider, Storage and Windows]

  • Kubeadm: add support for dry running "kubeadm join". The new flag "kubeadm join --dry-run" is similar to the existing flag for "kubeadm init/upgrade" and allows you to see what changes would be applied. (#103027, @Haleygo) [SIG Cluster Lifecycle]

  • Kubernetes is now built with Golang 1.16.7 (#104199, @cpanato) [SIG Cloud Provider, Instrumentation, Release and Testing]

  • The ServiceAccountIssuerDiscovery feature gate is removed. It reached GA in Kubernetes 1.21. (#103685, @mengjiao-liu) [SIG API Machinery and Auth]

  • Updated Cluster Autosaler to version 1.22.0. Release notes: https://github.com/kubernetes/autoscaler/releases/tag/cluster-autoscaler-1.22.0 (#104293, @x13n) [SIG Autoscaling and Cloud Provider]

  • Updates the following images to pick up CVE fixes:

    • debian to v1.9.0
    • debian-iptables to v1.6.6
    • setcap to v2.0.4 (#104142, @mengjiao-liu) [SIG API Machinery, Release and Testing]

Documentation

  • Update description of --audit-log-maxbackup to describe behavior when value = 0 (#103843, @Arkessler) [SIG API Machinery]

Bug or Regression

    1. Changes json representation for a conflicted taint to Key=Effect when a conflicted taint occurs in kubectl taint. (#104011, @manugupt1) [SIG CLI]
  • A new server run option 'shutdown-send-retry-after' has been introduced. If true the HTTP Server will continue listening until all non longrunning request(s) in flight have been drained, during this window all incoming requests will be rejected with a status code 429 and a 'Retry-After' response header. (#101257, @tkashem) [SIG API Machinery]
  • Adds Kubernetes Events to the Kubelet Graceful Shutdown feature (#101081, @rphillips) [SIG Node]
  • CA, certificate and key bundles for the generic-apiserver based servers will be reloaded immediately after the files are changed. (#104102, @tnqn) [SIG API Machinery and Testing]
  • Fix kube-apiserver metric reporting for the deprecated watch path of /api//watch/... (#104161, @wojtek-t) [SIG API Machinery and Instrumentation]
  • Fix: skip case sensitivity when checking Azure NSG rules (#104384, @feiskyer) [SIG Cloud Provider]
  • Fixed an issue which didn't append OS's environment variables with the one provided in Credential Provider Config file, which may lead to failed execution of external credential provider binary. See kubernetes#102750 (#103231, @n4j) [SIG Auth and Node]
  • Fixed architecture within manifest for non amd64 etcd images. (#104116, @saschagrunert) [SIG API Machinery]
  • Fixed bug where kubectl would emit duplicate warning messages for flag names that contain an underscore and recommend using a nonexistent flag in some cases (#103852, @brianpursley) [SIG CLI and Cluster Lifecycle]
  • Graceful node shutdown, allow the actual inhibit delay to be greater than the expected inhibit delay (#103137, @wzshiming) [SIG Node]
  • Kube-apiserver: Avoids unnecessary repeated calls to admission webhooks that reject an update or delete request. (#104182, @liggitt) [SIG API Machinery]
  • Kube-proxy: delete stale conntrack UDP entries for loadbalancer ingress IP. (#104009, @aojea) [SIG Network]
  • Kubeadm: When adding an etcd peer to an existing cluster, if an error is returned indicating the peer has already been added, this is accepted and a ListMembers call is used instead to return the existing cluster. This helps diminish the exponential backoff when the first AddMember call times out, while still retaining a similar performance when the peer had already been added from a previous call. (#104134, @ihgann) [SIG Cluster Lifecycle]
  • Pass additional flags to subpath mount to avoid flakes in certain conditions (#104253, @mauriciopoppe) [SIG Storage]
  • Update Go used to build migrate script in etcd image to v1.16.7 (#104301, @serathius) [SIG API Machinery and Release]

Other (Cleanup or Flake)

  • Deprecate apiserver_longrunning_gauge and apiserver_register_watchers in 1.23.0 (#103793, @yan-lgtm) [SIG API Machinery, Cluster Lifecycle and Instrumentation]
  • Kube-apiserver: sets an upper-bound on the lifetime of idle keep-alive connections and time to read the headers of incoming requests (#103958, @liggitt) [SIG API Machinery and Node]
  • Kubeadm: external etcd endpoints passed in the ClusterConfiguration that have Unicode characters are no longer IDNA encoded (converted to Punycode). They are now just URL encoded as per Go's implementation of RFC-3986, have duplicate "/" removed from the URL paths, and passed like that directly to the kube-apiserver --etcd-servers flag. If you have etcd endpoints that have Unicode characters, it is advisable to encode them in advance with tooling that is fully IDNA compliant. If you don't do that, the Go standard library (used in k8s and etcd) would do it for you when making requests to the endpoints. (#103801, @gkarthiks) [SIG Cluster Lifecycle]
  • Kubeadm: update references to legacy artifacts locations, the 'ci-cross' prefix has been removed from the version match as it does not exist in the new 'gs://k8s-release-dev' bucket (#103813, @SataQiu) [SIG Cluster Lifecycle]
  • Migratecmd/kube-proxy/app logs to structured logging (#98913, @yxxhero) [SIG Network]
  • Surface warning when users don't set propagationPolicy for jobs while deleting (#104080, @ravisantoshgudimetla) [SIG Apps]
  • The AllowInsecureBackendProxy feature gate is removed. It reached GA in Kubernetes 1.21. (#103796, @mengjiao-liu) [SIG API Machinery]
  • The StartupProbe feature gate that is GA since v1.20 is unconditionally enabled, and can no longer be specified via the --feature-gates argument. (#104168, @ialidzhikov) [SIG Node]
  • The apiserver exposes 4 new metrics that allow to track the status of the Service CIDRs allocations: - current number of available IPs per Service CIDR - current number of used IPs per Service CIDR - total number of allocation per Service CIDR - total number of allocation errors per ServiceCIDR (#104119, @aojea) [SIG Apps, Instrumentation and Network]
  • The flag --deployment-controller-sync-period has no effect now, deprecate it and will be removed in v1.24. (#103538, @Pingan2017) [SIG Apps]
  • Troubleshooting: informers log handlers that take more than 100 milliseconds to process an object if the DeltaFIFO queue starts to grow beyond 10 elements. (#103917, @aojea) [SIG API Machinery]
  • Update cri-tools dependency to v1.22.0 (#104430, @saschagrunert) [SIG Cloud Provider and Node]
  • gcr.io/kubernetes-e2e-test-images will no longer be used in E2E / CI testing, k8s.gcr.io/e2e-test-images will be used instead. (#103724, @claudiubelu) [SIG API Machinery and Testing]

Dependencies

Added

Changed

Removed