Kubernetes native sidecar not supported

[andreasgeisslerdt]

What happened?

Since K8S 1.28 the native sidecars feature (https://kubernetes.io/blog/2023/08/25/native-sidecar-containers/) and in Istio this feature can be enabled to start the istio-proxy as init container with the option
restartPolicy: Always

When I enable this feature and try to create a cassandra cluster using the k8ssandra-operator, the Statefulset is failing with the failure:

Warning  FailedCreate      94s (x17 over 7m3s)  statefulset-controller  create Pod cassandra-dc1-default-sts-0 in StatefulSet cassandra-dc1-default-sts failed error: Pod "cassandra-dc1-default-sts-0" is invalid: [spec.initContainers[1].lifecycle: Forbidden: may not be set for init containers without restartPolicy=Always, spec.initContainers[1].readinessProbe: Forbidden: may not be set for init containers without restartPolicy=Always, spec.initContainers[1].startupProbe: Forbidden: may not be set for init containers without restartPolicy=Always]

The same problem is reported in other cases, e.g.:
istio/istio#49221
Azure/azure-workload-identity#1312
istio/istio#49054

The might be related to the fact, that the cass-operator (until version 1.19.0) uses an old k8s.io version.
Maybe it will be resolved in the version 1.19.1 which will be included in the next k8ssandra-operator version.

What did you expect to happen?

init-container sidecars are added by istio

How can we reproduce it (as minimally and precisely as possible)?

• Enable "native-sidecars" feature in Kubernetes
• Enable "ENABLE_NATIVE_SIDECARS" in istiod
• create a cassandra cluster

cass-operator version

v1.19.0

Kubernetes version

v1.28.6

Method of installation

Helm

Anything else we need to know?

Using K8ssandra-Operator v1.14.0

[burmanm]

The Kubernetes dependencies in cass-operator are updated as part of 1.20 (k8ssandra/cass-operator#619), 1.19.x branch remains at 1.26.

However, since this project does not ship with mutating webhook, the issues in related tickets does not seem to apply here. I don't know how Istio works with native sidecars, but I'm assuming it has a mutating webhook targeting Pods when they're deployed.

The incompatibility could be part of k8ssandra-operator's mutating webhook. That is, the order in which k8ssandra-operator and Istio modify the Pods is incorrect. I'll transfer this ticket to k8ssandra-operator where the Kubernetes dependency update should happen at some point.

[andreasgeisslerdt]

Hi, @burmanm
Thanks for the clarification.
You are right, the mutating webhook of the k8ssandra-operator is causing the problem.
Like described for the otel-operator in istio/istio#49054 (comment)

• Istio injects the istio-proxy container to the cassandra pod.
  It adds "restartPolicy=Always" into the init container
• The K8ssandra webhook injector is also triggered. it takes the JSON, marshals to a v1.Pod, mutates things, then marshals back.
  Because it is on old k8s libraries, this round trip is lossy and loses the "restartPolicy=Always"
• Kubernetes marks the Pod as broken, as the restartPolicy is missing

So the solution will be, as you suggested to update the k8s version in k8ssandra-operator.
Thanks for the clarification
Andreas

[andreasgeisslerdt]

Hi again,
I restarted the tests with the k8ssandra-operator version v1.16.0 (which contains cass-operator 1.20), but the result is the same:

istio                  	istio-system      	1       	2024-05-17 12:14:31.140405187 +0000 UTC	deployed	base-1.22.0                  	1.22.0     
istio-cni              	istio-system      	1       	2024-05-17 12:15:00.529044755 +0000 UTC	deployed	cni-1.22.0                   	1.22.0     
istio-ingress          	istio-ingress     	1       	2024-05-17 12:15:08.246708008 +0000 UTC	deployed	gateway-1.22.0               	1.22.0     
istiod                 	istio-system      	1       	2024-05-17 12:14:35.936156823 +0000 UTC	deployed	istiod-1.22.0                	1.22.0     
k8ssandra-operator     	k8ssandra-operator	1       	2024-05-17 12:20:10.460073671 +0000 UTC	deployed	k8ssandra-operator-1.16.0    	1.16.0     

cassandra STS:

Events:
  Type     Reason            Age                  From                    Message
  ----     ------            ----                 ----                    -------
  Normal   SuccessfulCreate  7m3s                 statefulset-controller  create Claim server-data-cassandra-dc1-default-sts-0 Pod cassandra-dc1-default-sts-0 in StatefulSet cassandra-dc1-default-sts success
  Warning  FailedCreate      94s (x17 over 7m3s)  statefulset-controller  create Pod cassandra-dc1-default-sts-0 in StatefulSet cassandra-dc1-default-sts failed error: Pod "cassandra-dc1-default-sts-0" is invalid: [spec.initContainers[1].lifecycle: Forbidden: may not be set for init containers without restartPolicy=Always, spec.initContainers[1].readinessProbe: Forbidden: may not be set for init containers without restartPolicy=Always, spec.initContainers[1].startupProbe: Forbidden: may not be set for init containers without restartPolicy=Always]

[andreasgeisslerdt]

Oh I see, the k8ssandra-operator was not updated to a new k8s version yet, so it cannot work, I guess:
https://github.com/k8ssandra/k8ssandra-operator/blob/d7f059e51ab80a3a8692ad1db7d12522a79e0d5b/go.mod#L33C1-L36C1

	k8s.io/api v0.26.4
	k8s.io/apimachinery v0.26.4
	k8s.io/client-go v0.26.4

[andreasgeisslerdt]

I tested the version v1.17.0 successfully.
Kubernetes "Native Sidecar" is now supported.
Will close the ticket.