Using ReadinessGate for the canary ReplicaSet

[johnniee]

Summary

as part of the migration of my Deployment manifest into Rollout, I have trouble using the ReadinessGate. My Infra runs on AWS EKS using ALB. I'm using the following example https://github.com/argoproj/argo-rollouts/blob/master/docs/getting-started/alb/rollout.yaml
with the addition of this ReadinessGate block:

readinessGates:
      - conditionType: target-health.alb.ingress.k8s.aws/rollouts-demo-ingress_rollouts-demo-stable_80
      - conditionType: target-health.alb.ingress.k8s.aws/rollouts-demo-ingress_rollouts-demo-canary_80

But obviously, pods can't pass the readinessGate since the pods aren't being registered under the new Target Group.

  - lastProbeTime: null
    lastTransitionTime: "2020-10-19T07:14:04Z"
    message: corresponding condition of pod readiness gate "target-health.alb.ingress.k8s.aws/rollouts-demo-ingress_rollouts-demo-stable_80"
      does not exist.
    reason: ReadinessGatesNotReady
    status: "False"
    type: Ready

Is there any workaround for this issue so I will be able to keep the ReadinessGate?

What do you want to know about this project?

Motivation

Stability. You can find the motivations for using ReadinessGate here: https://kubernetes-sigs.github.io/aws-alb-ingress-controller/guide/ingress/pod-conditions/

Why do you need to know this, any examples or use cases you could include?
The AWS ALB ingress controller can set such a condition on your pods. This is needed under certain circumstances to achieve full zero downtime rolling deployments. Consider the following example: * low number of replicas in a deployment (e.g. one to three) * start a rolling update of the deployment * rollout of new pods takes less time than it takes the ALB ingress controller to register the new pods and for their health state turn »Healthy« in the target group * at some point during this rolling update, the target group might only have registered targets that are in »Initial« or »Draining« state; this results in service outage
(taken from https://kubernetes-sigs.github.io/aws-alb-ingress-controller/guide/ingress/pod-conditions/ )

[jessesuen]

But obviously, pods can't pass the readinessGate since the pods aren't being registered under the new Target Group.

This is isn't obvious to me. Why wouldn't pods be registered under the new target group?

[johnniee]

@jessesuen Thanks for your reply.
The new pods are not being registered behind the stable target group. See the message:

    message: corresponding condition of pod readiness gate "target-health.alb.ingress.k8s.aws/rollouts-demo-ingress_rollouts-demo-stable_80"
      does not exist.

(rollouts-demo-ingress_rollouts-demo-stable_80)
This makes sense because if the canary pods will be registered behind the stable TargetGroup, they will receive unwanted traffic.

[jessesuen]

@johnniee

I think to solve this, you may need a third service for this, that targets all of the Rollout pods. The readiness gate should reference a single root service, which selects on app: rollouts-demo

readinessGates:
      - conditionType: target-health.alb.ingress.k8s.aws/rollouts-demo-ingress_rollouts-demo-root_80

[johnniee]

@jessesuen Thanks for the idea. The readiness gate should mark my registered pod as healthy/unhealthy during all the time pod is up. Creating a demo target group/service won't really help since it could just give me the indication that something isn't working well, but It won't help me during the pod lifetime.

I think that this can be solved by register the pods to the canary service, connect the target group to the ALB (with 0% traffic), and then check the pods.

[jessesuen]

I think that this can be solved by register the pods to the canary service, connect the target group to the ALB (with 0% traffic), and then check the pods.

I didn't quite follow your last comment, did you find a workaround/solution? Or are you suggesting some type of new behavior that needs to be done in rollout controller to support this?