The ingress and egress gateways stuck on pending

[jtr109]

Hi @morlay:

First of all, thanks for sharing the ARM Istio Images.

I met a problem after followed the installation guide in README. I'm not sure if my operation is wrong.

Description

I am using an istioctl with version 1.10.2. I initialized the operator by executing the command below:

istioctl operator init --hub=docker.io/querycapistio --tag=1.9.2

The operator is running as expected:

❯ k -n istio-operator get pod                                
NAME                              READY   STATUS    RESTARTS   AGE
istio-operator-59588b8ff6-95s8m   1/1     Running   0          9m40s

I found the namespace istio-system is created after initialization and I installed istio:

kubectl apply -f - <<EOF
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  namespace: istio-system
  name: example-istiocontrolplane
spec:
  hub: docker.io/querycapistio
  profile: demo
EOF

The operator realized the installation request, and the two pods istio-ingressgateway and istio-ingressgateway were created. However, they were stuck at pending because affinity did not match. I found there is no value arm64 added in spec.affinity.

Other information:

• system: macOS
• arch: arm64
• cluster is built by minikube

Another try

I also tried to set spec.components.*.k8s.affinity and create a new manifest below:

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  namespace: istio-system
  name: example-istiocontrolplane
spec:
  hub: docker.io/querycapistio
  profile: demo
  components:
    pilot:
      k8s: # each components have to set this
        affinity: &affinity
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
                - matchExpressions:
                    - key: kubernetes.io/arch
                      operator: In
                      values:
                        - arm64
                        - amd64
    egressGateways:
      - k8s:
          affinity: *affinity
    ingressGateways:
      - k8s:
          affinity: *affinity

After applied that, the operator logged an error:

2021-09-17T02:59:08.754190Z	info	installer	Reconciling IstioOperator
2021-09-17T02:59:08.755945Z	info	installer	Updating IstioOperator
2021-09-17T02:59:08.756054Z	info	installer	Loading values from compiled in VFS at path profiles/demo.yaml
2021-09-17T02:59:08.756076Z	info	installer	Loading values from compiled in VFS at path profiles/default.yaml
2021-09-17T02:59:08.879809Z	error	installer	failed to merge base profile with user IstioOperator CR example-istiocontrolplane, json merge error (map: map[k8s:map[affinity:map[nodeAffinity:map[requiredDuringSchedulingIgnoredDuringExecution:map[nodeSelectorTerms:[map[matchExpressions:[map[key:kubernetes.io/arch operator:In values:[arm64 amd64]]]]]]]]]] does not contain declared merge key: name) for base object: 
{"apiVersion":"install.istio.io/v1alpha1","kind":"IstioOperator","metadata":{"namespace":"istio-system"},"spec":{"components":{"base":{"enabled":true},"cni":{"enabled":false},"egressGateways":[{"enabled":true,"k8s":{"resources":{"requests":{"cpu":"10m","memory":"40Mi"}}},"name":"istio-egressgateway"}],"ingressGateways":[{"enabled":true,"k8s":{"resources":{"requests":{"cpu":"10m","memory":"40Mi"}},"service":{"ports":[{"name":"status-port","port":15021,"targetPort":15021},{"name":"http2","port":80,"targetPort":8080},{"name":"https","port":443,"targetPort":8443},{"name":"tcp","port":31400,"targetPort":31400},{"name":"tls","port":15443,"targetPort":15443}]}},"name":"istio-ingressgateway"}],"istiodRemote":{"enabled":false},"pilot":{"enabled":true,"k8s":{"env":[{"name":"PILOT_TRACE_SAMPLING","value":"100"}],"resources":{"requests":{"cpu":"10m","memory":"100Mi"}}}}},"hub":"docker.io/istio","meshConfig":{"accessLogFile":"/dev/stdout","defaultConfig":{"proxyMetadata":{}},"enablePrometheusMerge":true},"tag":"1.9.2","values":{"base":{"enableCRDTemplates":false,"validationURL":""},"gateways":{"istio-egressgateway":{"autoscaleEnabled":false,"env":{},"name":"istio-egressgateway","secretVolumes":[{"mountPath":"/etc/istio/egressgateway-certs","name":"egressgateway-certs","secretName":"istio-egressgateway-certs"},{"mountPath":"/etc/istio/egressgateway-ca-certs","name":"egressgateway-ca-certs","secretName":"istio-egressgateway-ca-certs"}],"type":"ClusterIP","zvpn":{}},"istio-ingressgateway":{"autoscaleEnabled":false,"env":{},"name":"istio-ingressgateway","secretVolumes":[{"mountPath":"/etc/istio/ingressgateway-certs","name":"ingressgateway-certs","secretName":"istio-ingressgateway-certs"},{"mountPath":"/etc/istio/ingressgateway-ca-certs","name":"ingressgateway-ca-certs","secretName":"istio-ingressgateway-ca-certs"}],"type":"LoadBalancer","zvpn":{}}},"global":{"arch":{"amd64":2,"ppc64le":2,"s390x":2},"configValidation":true,"defaultNodeSelector":{},"defaultPodDisruptionBudget":{"enabled":true},"defaultResources":{"requests":{"cpu":"10m"}},"imagePullPolicy":"","imagePullSecrets":[],"istioNamespace":"istio-system","istiod":{"enableAnalysis":false},"jwtPolicy":"third-party-jwt","logAsJson":false,"logging":{"level":"default:info"},"meshNetworks":{},"mountMtlsCerts":false,"multiCluster":{"clusterName":"","enabled":false},"network":"","omitSidecarInjectorConfigMap":false,"oneNamespace":false,"operatorManageWebhooks":false,"pilotCertProvider":"istiod","priorityClassName":"","proxy":{"autoInject":"enabled","clusterDomain":"cluster.local","componentLogLevel":"misc:error","enableCoreDump":false,"excludeIPRanges":"","excludeInboundPorts":"","excludeOutboundPorts":"","image":"proxyv2","includeIPRanges":"*","logLevel":"warning","privileged":false,"readinessFailureThreshold":30,"readinessInitialDelaySeconds":1,"readinessPeriodSeconds":2,"resources":{"limits":{"cpu":"2000m","memory":"1024Mi"},"requests":{"cpu":"10m","memory":"40Mi"}},"statusPort":15020,"tracer":"zipkin"},"proxy_init":{"image":"proxyv2","resources":{"limits":{"cpu":"2000m","memory":"1024Mi"},"requests":{"cpu":"10m","memory":"10Mi"}}},"sds":{"token":{"aud":"istio-ca"}},"sts":{"servicePort":0},"tracer":{"datadog":{},"lightstep":{},"stackdriver":{},"zipkin":{}},"useMCP":false},"istiodRemote":{"injectionURL":""},"pilot":{"autoscaleEnabled":false,"autoscaleMax":5,"autoscaleMin":1,"configMap":true,"cpu":{"targetAverageUtilization":80},"deploymentLabels":null,"enableProtocolSniffingForInbound":true,"enableProtocolSniffingForOutbound":true,"env":{},"image":"pilot","keepaliveMaxServerConnectionAge":"30m","nodeSelector":{},"replicaCount":1,"traceSampling":1},"telemetry":{"enabled":true,"v2":{"enabled":true,"metadataExchange":{"wasmEnabled":false},"prometheus":{"enabled":true,"wasmEnabled":false},"stackdriver":{"configOverride":{},"enabled":false,"logging":false,"monitoring":false,"topology":false}}}}}}
 override object: 
{"apiVersion":"install.istio.io/v1alpha1","kind":"IstioOperator","metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"install.istio.io/v1alpha1\",\"kind\":\"IstioOperator\",\"metadata\":{\"annotations\":{},\"name\":\"example-istiocontrolplane\",\"namespace\":\"istio-system\"},\"spec\":{\"components\":{\"egressGateways\":[{\"k8s\":{\"affinity\":{\"nodeAffinity\":{\"requiredDuringSchedulingIgnoredDuringExecution\":{\"nodeSelectorTerms\":[{\"matchExpressions\":[{\"key\":\"kubernetes.io/arch\",\"operator\":\"In\",\"values\":[\"arm64\",\"amd64\"]}]}]}}}}}],\"ingressGateways\":[{\"k8s\":{\"affinity\":{\"nodeAffinity\":{\"requiredDuringSchedulingIgnoredDuringExecution\":{\"nodeSelectorTerms\":[{\"matchExpressions\":[{\"key\":\"kubernetes.io/arch\",\"operator\":\"In\",\"values\":[\"arm64\",\"amd64\"]}]}]}}}}}],\"pilot\":{\"k8s\":{\"affinity\":{\"nodeAffinity\":{\"requiredDuringSchedulingIgnoredDuringExecution\":{\"nodeSelectorTerms\":[{\"matchExpressions\":[{\"key\":\"kubernetes.io/arch\",\"operator\":\"In\",\"values\":[\"arm64\",\"amd64\"]}]}]}}}}}},\"hub\":\"docker.io/querycapistio\",\"profile\":\"demo\"}}\n"},"creationTimestamp":"2021-09-17T02:58:47Z","finalizers":["istio-finalizer.install.istio.io"],"generation":"1","managedFields":[{"apiVersion":"install.istio.io/v1alpha1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:kubectl.kubernetes.io/last-applied-configuration":{}}},"f:spec":{".":{},"f:components":{".":{},"f:egressGateways":{},"f:ingressGateways":{},"f:pilot":{".":{},"f:k8s":{".":{},"f:affinity":{".":{},"f:nodeAffinity":{".":{},"f:requiredDuringSchedulingIgnoredDuringExecution":{".":{},"f:nodeSelectorTerms":{}}}}}}},"f:hub":{},"f:profile":{}}},"manager":"kubectl-client-side-apply","operation":"Update","time":"2021-09-17T02:58:47Z"},{"apiVersion":"install.istio.io/v1alpha1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:finalizers":{".":{},"v:\"istio-finalizer.install.istio.io\"":{}}}},"manager":"operator","operation":"Update","time":"2021-09-17T02:58:47Z"}],"name":"example-istiocontrolplane","namespace":"istio-system","resourceVersion":"572","uid":"46203465-810c-46fe-bcc8-81cdcb6e6dda"},"spec":{"components":{"egressGateways":[{"k8s":{"affinity":{"nodeAffinity":{"requiredDuringSchedulingIgnoredDuringExecution":{"nodeSelectorTerms":[{"matchExpressions":[{"key":"kubernetes.io/arch","operator":"In","values":["arm64","amd64"]}]}]}}}}}],"ingressGateways":[{"k8s":{"affinity":{"nodeAffinity":{"requiredDuringSchedulingIgnoredDuringExecution":{"nodeSelectorTerms":[{"matchExpressions":[{"key":"kubernetes.io/arch","operator":"In","values":["arm64","amd64"]}]}]}}}}}],"pilot":{"k8s":{"affinity":{"nodeAffinity":{"requiredDuringSchedulingIgnoredDuringExecution":{"nodeSelectorTerms":[{"matchExpressions":[{"key":"kubernetes.io/arch","operator":"In","values":["arm64","amd64"]}]}]}}}}}},"hub":"docker.io/querycapistio","profile":"demo"}}	moreInfo=The values in the selected spec.profile could not be merged with the user IstioOperator resource. impact=The operator controller cannot create and act upon the user defined IstioOperator resource. The Istio control plane will not be installed or updated. action=Check that the IstioOperator resource has the correct syntax. If you are sure your configuration is correct, see https://istio.io/latest/about/bugs for possible solutions. likelyCause=The likely cause is an incorrect or badly formatted configuration.Another possible cause could be an issue with the Istio code. 

[morlay]

This error in log is clear

does not contain declared merge key: name

You should declare name in each component, follow the demo profile
https://github.com/istio/istio/blob/master/manifests/profiles/demo.yaml#L8
https://github.com/istio/istio/blob/master/manifests/profiles/demo.yaml#L17

Like:

spec:
  components:
    pilot:
      k8s: # each components have to set this
        affinity: &affinity
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
                - matchExpressions:
                    - key: kubernetes.io/arch
                      operator: In
                      values:
                        - arm64
                        - amd64
    egressGateways:
      - name: "istio-egressgateway"
        k8s:
          affinity: *affinity
    ingressGateways:
      - name: "istio-ingressgateway"
        k8s:
          affinity: *affinity

However, consider use 1.10+, the spec.components.*.k8s.affinity is not need any more.