Redis sentinel does not work, chart throws error on install #191

sarg3nt · 2024-03-22T17:01:55Z

The helm chart throws an error during install with sessionStorage.rdis.clientType: sentinel
I've tried all kinds of configs and cannot get it to run properly.
I haven't been able to determine if there is a value I'm not setting that it doesn't like or if it's just broken.

Error:
Deployment in version "v1" cannot be handled as a Deployment: json: cannot unmarshal array into Go struct field EnvVar.spec.template.spec.containers.env.value of type strin

Config:

config:
  clientID: <redacted>
  clientSecret: <redacted>
  configFile: |-
    pass_basic_auth = false
    pass_access_token = true
    set_authorization_header = true
    pass_authorization_header = true
    reverse_proxy = true
    whitelist_domains = [".bob.com"]
    email_domains = [
      "*",
    ]
    allowed_groups = [
      "redacted"
    ]
    azure_tenant = "redacted"
    oidc_issuer_url = "https://redacted"
    provider = "azure"
    upstreams = ["file:///dev/null"]
    http_address = "0.0.0.0:4180"
    cookie_domains = ["k8s.bob.com"]
  cookieSecret: redacted
extraArgs:
  redis-insecure-skip-tls-verify: "true"
extraEnv:
- name: HTTP_PROXY
   value: <redacted along with rest of env vars>
image:
  pullPolicy: Always
ingress:
  annotations:
    ingress.kubernetes.io/ssl-redirect: "true"
    kubernetes.io/ingress.class: nginx
    kubernetes.io/tls-acme: "true"
  enabled: true
  hosts:
  - oauth2.bob.com
  path: /
  tls:
  - hosts:
    - oauth2.bob.com
    secretName: oauth2-proxy-tls
redis:
  enabled: true
  global:
    imageRegistry: <redacted>
    redis:
      password: <redacted>
  sentinel:
    enabled: true
  tls:
    authClients: false
    autoGenerated: true
    enabled: true
replicaCount: 3
resources:
  limits:
    cpu: 500m
    memory: 300Mi
  requests:
    cpu: 10m
    memory: 30Mi
sessionStorage:
  redis:
    clientType: sentinel
    password: <redacted>
    sentinel:
      # The following are fake, I've also tried sending it a TF array.
      connectionUrls: '["redis://127.0.0.1:8000","redis://127.0.0.1:8000","redis://127.0.0.1:8000"]'
      masterName: master
      password: <redacted>
  type: redis

The text was updated successfully, but these errors were encountered:

pierluigilenoci · 2024-04-03T11:02:49Z

There is an indentation problem in your file.
This is the correct way to pass the extra env:

extraEnv:
  - name: <ENV_NAME>
    value: <ENV_VALUE>

For your reference, here is the code that generates the manifest:
https://github.com/oauth2-proxy/manifests/blob/main/helm/oauth2-proxy/templates/deployment.yaml#L211-L213

If even solving this you have issues, we can try to understand the further problem.

sarg3nt · 2024-04-04T17:40:44Z

@pierluigilenoci
That must have been an error I introduced while cleaning up the code for this post.
The current prod version of that file is using a secret for the env vars now anyway.
Still getting the same error.
Let's zoom in on the changes, I'm going to paste just the relavent lines of config.

This works

sessionStorage:
  # Can be one of the supported session storage cookie|redis
  type: redis
  redis:
    # Redis password value. Applicable for all Redis configurations. Taken from redis subchart secret if not set. `sessionStorage.redis.existingSecret` takes precedence
    password: "${redis_password}"
    # Can be one of standalone|cluster|sentinel
    # Sentinel and cluster do not work, chart won't install, could not figure it out.  Sarge.
    clientType: "standalone"
    # NOTE: Sentinel does not work, have open case https://github.com/oauth2-proxy/manifests/issues/191
    # sentinel:
    #   # Name of the Kubernetes secret containing the redis sentinel password value (see also `sessionStorage.redis.sentinel.passwordKey`). Default: `sessionStorage.redis.existingSecret`
    #   #existingSecret: "none"
    #   # Redis sentinel password. Used only for sentinel connection; any redis node passwords need to use `sessionStorage.redis.password`
    #   password: "${redis_password}"
    #   # Key of the Kubernetes secret data containing the redis sentinel password value
    #   passwordKey: "redis-sentinel-password"
    #   # Redis sentinel master name
    #   masterName: "master"
    #   # List of Redis sentinel connection URLs (e.g. `["redis://127.0.0.1:8000", "redis://127.0.0.1:8000"]`)
    #   connectionUrls: '["redis://127.0.0.1:8000","redis://127.0.0.1:8000","redis://127.0.0.1:8000"]'

# Enables and configure the automatic deployment of the redis subchart
redis:
  global:
    imageRegistry: sel-docker.artifactory.metro.ad.selinc.com
    redis:
      password: ${redis_password}
  # provision an instance of the redis sub-chart
  enabled: true
  tls:
    enabled: true
    authClients: false
    autoGenerated: true
  # Redis specific helm chart settings, please see:
  # https://github.com/bitnami/charts/tree/master/bitnami/redis#parameters
  # redisPort: 6379
  # sentinel:
  #   enabled: true

This does not

sessionStorage:
  # Can be one of the supported session storage cookie|redis
  type: redis
  redis:
    # Redis password value. Applicable for all Redis configurations. Taken from redis subchart secret if not set. `sessionStorage.redis.existingSecret` takes precedence
    password: "${redis_password}"
    # Can be one of standalone|cluster|sentinel
    # Sentinel and cluster do not work, chart won't install, could not figure it out.  Sarge.
    clientType: "sentinel"
    # NOTE: Sentinel does not work, have open case https://github.com/oauth2-proxy/manifests/issues/191
    sentinel:
      # Name of the Kubernetes secret containing the redis sentinel password value (see also `sessionStorage.redis.sentinel.passwordKey`). Default: `sessionStorage.redis.existingSecret`
      #existingSecret: "none"
      # Redis sentinel password. Used only for sentinel connection; any redis node passwords need to use `sessionStorage.redis.password`
      password: "${redis_password}"
      # Key of the Kubernetes secret data containing the redis sentinel password value
      #passwordKey: "redis-sentinel-password"
      # Redis sentinel master name
      masterName: "master"
      # List of Redis sentinel connection URLs (e.g. `["redis://127.0.0.1:8000", "redis://127.0.0.1:8000"]`)
      connectionUrls: '["redis://127.0.0.1:8000","redis://127.0.0.1:8000","redis://127.0.0.1:8000"]'

# Enables and configure the automatic deployment of the redis subchart
redis:
  global:
    imageRegistry: sel-docker.artifactory.metro.ad.selinc.com
    redis:
      password: ${redis_password}
  # provision an instance of the redis sub-chart
  enabled: true
  tls:
    enabled: true
    authClients: false
    autoGenerated: true
  # Redis specific helm chart settings, please see:
  # https://github.com/bitnami/charts/tree/master/bitnami/redis#parameters
  # redisPort: 6379
  sentinel:
    enabled: true

And to dig a little deeper, if you start with working example above and change sessionStorage.redis.clientType from standalone to sentinel the error occurs.

Here's the error again:
Deployment in version "v1" cannot be handled as a Deployment: json: cannot unmarshal array into Go struct field EnvVar.spec.template.spec.containers.env.value of type string

pierluigilenoci · 2024-04-08T09:35:50Z

To enable sentinel, a couple of parameters must be configured correctly:

masterName
connectionUrls

Ref: https://github.com/oauth2-proxy/manifests/blob/main/helm/oauth2-proxy/templates/deployment.yaml#L194-L197

The problem could be the quoting.

Try to validate your values.yaml with the helm template command.

Further detail: https://github.com/msepp/oauth2_proxy/blob/v4.2.2/pkg/apis/options/sessions.go#L25

Pionerd · 2024-04-09T19:36:17Z

As briefly mentioned in a comment, this also applies when clientType is set to cluster.

I do not see any logic in the helm chart that transforms the list of connectionUrls to something environment variable friendly.

pierluigilenoci · 2024-04-10T10:42:55Z

@Pionerd, there is no logic.
The list is passed directly into the environment.
But it must be compatible with the environment and the helm chart.

pierluigilenoci · 2024-04-30T14:27:22Z

@sarg3nt Were you able to solve it?

sarg3nt · 2024-05-03T23:49:56Z

@pierluigilenoci Sorry for the delay, was sidetracked on other projects.
I've gotten further. I"m wondering if Helm changed since your values.yaml file had

(e.g. ["redis://127.0.0.1:8000", "redis://127.0.0.1:8000"])

inserted into it. Turns out if Helm sees the square brackets, it removes the quotes, regardless of quote type and turns it into an actual array. It then fails because an array is not a string.
helm template worked regardless, it just generated an invalid value for that env var.
I switched it to a comma separated string and got passed that bit.

The redis cluster seems to be bootstrapping just fine but the outh2-proxy pods stay in an error state with the following:
Error from server (NotFound): pods "oauth2-proxy-redis-master-0" not found
The redis pods are oauth2-proxy-redis-node-0 etc.
Doesn't matter what I set masterName: to whether it be masterName: "node" or masterName: "oauth2-proxy-redis-node-0", what the oath2 pods are looking for never changes, it's always oauth2-proxy-redis-master-0.

So either masterName is not the thing I need to change or I'm missing something else.

I set connectionUrls to connectionUrls: "redis://oauth2-proxy-redis-node-0:26379,redis://oauth2-proxy-redis-node-1:26379,redis://oauth2-proxy-redis-node-2:26379"
As that seems to match what is being deployed by the redis chart.

Note: From what I can tell from here there is no Redis master deployed when the cluster is in sentinel mode. So not sure what the deal is with it looking for a master and with the masterName parameter.

Here's the new latter half of the config

# Configure the session storage type, between cookie and redis
sessionStorage:
  # Can be one of the supported session storage cookie|redis
  type: redis
  redis:
    # Redis password value. Applicable for all Redis configurations. Taken from redis subchart secret if not set. `sessionStorage.redis.existingSecret` takes precedence
    password: "${redis_password}"
    # Can be one of standalone|cluster|sentinel
    # Sentinel and cluster do not work, chart won't install, could not figure it out.  Sarge.
    clientType: "sentinel"
    # NOTE: Sentinel does not work, have open case https://github.com/oauth2-proxy/manifests/issues/191
    sentinel:
      # Name of the Kubernetes secret containing the redis sentinel password value (see also `sessionStorage.redis.sentinel.passwordKey`). Default: `sessionStorage.redis.existingSecret`
      #existingSecret: "none"
      # Redis sentinel password. Used only for sentinel connection; any redis node passwords need to use `sessionStorage.redis.password`
      password: "${redis_password}"
      # Key of the Kubernetes secret data containing the redis sentinel password value
      #   passwordKey: "redis-sentinel-password"
      # Redis sentinel master name
      masterName: "oauth2-proxy-redis-node-0"
      # Error from server (NotFound): pods "oauth2-proxy-redis-master-0" not found

      # List of Redis sentinel connection URLs (e.g. `["redis://127.0.0.1:8000", "redis://127.0.0.1:8000"]`)
      connectionUrls: "redis://oauth2-proxy-redis-node-0:26379,redis://oauth2-proxy-redis-node-1:26379,redis://oauth2-proxy-redis-node-2:26379"
      #oauth2-proxy-redis-node-0:26379

# Enables and configure the automatic deployment of the redis subchart
redis:
  global:
    imageRegistry: sel-docker.artifactory.metro.ad.selinc.com
    redis:
      password: ${redis_password}
  # provision an instance of the redis sub-chart
  enabled: true
  tls:
    enabled: true
    authClients: false
    autoGenerated: true
  # Redis specific helm chart settings, please see:
  # https://github.com/bitnami/charts/tree/master/bitnami/redis#parameters
  # redisPort: 6379
  sentinel:
    enabled: true

And some screenshots:

andreasgeisslerdt · 2024-05-07T13:24:16Z

Hi,
I am also trying the Sentinel option and was able to solve the configuration issue as you described above.
But now the next problem occurs wnich is the wait-for-redis.sh.
As $OAUTH2_PROXY_REDIS_USE_CLUSTER and $OAUTH2_PROXY_REDIS_USE_SENTINEL are not defined for the container the script cannot handle neither Cluster nor Sentinel:

# Main
if [ "$OAUTH2_PROXY_REDIS_USE_CLUSTER" = "true" ]; then
    echo "Checking Redis in cluster mode..."
    echo "$OAUTH2_PROXY_REDIS_CLUSTER_CONNECTION_URLS" | tr ',' '\n' | while read -r addr; do
        parse_and_check $addr || exit 1
    done
elif [ "$OAUTH2_PROXY_REDIS_USE_SENTINEL" = "true" ]; then
    echo "Checking Redis in sentinel mode..."
    echo "$OAUTH2_PROXY_REDIS_SENTINEL_CONNECTION_URLS" | tr ',' '\n' | while read -r addr; do
        parse_and_check $addr || exit 1
    done
elif [ -n "$OAUTH2_PROXY_REDIS_CONNECTION_URL" ]; then
    echo "Checking standalone Redis..."
    parse_and_check "$OAUTH2_PROXY_REDIS_CONNECTION_URL" || exit 1
else
    echo "Redis configuration not specified."
    exit 1
fi

Solution would be:

# Main
if [ -n "$OAUTH2_PROXY_REDIS_CLUSTER_CONNECTION_URLS" ]; then
    echo "Checking Redis in cluster mode..."
    echo "$OAUTH2_PROXY_REDIS_CLUSTER_CONNECTION_URLS" | tr ',' '\n' | while read -r addr; do
        parse_and_check $addr || exit 1
    done
elif [ -n "$OAUTH2_PROXY_REDIS_SENTINEL_CONNECTION_URLS" ]; then
    echo "Checking Redis in sentinel mode..."
    echo "$OAUTH2_PROXY_REDIS_SENTINEL_CONNECTION_URLS" | tr ',' '\n' | while read -r addr; do
        parse_and_check $addr || exit 1
    done
elif [ -n "$OAUTH2_PROXY_REDIS_CONNECTION_URL" ]; then
    echo "Checking standalone Redis..."
    parse_and_check "$OAUTH2_PROXY_REDIS_CONNECTION_URL" || exit 1
else
    echo "Redis configuration not specified."
    exit 1
fi

Pionerd · 2024-05-07T14:50:11Z

@andreasgeisslerdt thank you for the suggestion, please see #205

pierluigilenoci · 2024-05-07T16:14:46Z

@andreasgeisslerdt @Pionerd @sarg3nt Thank you for the collective effort to resolve this issue.

sarg3nt · 2024-05-08T23:00:02Z

This has fixed my problem. Below is my config that works in case future users need some help.
@pierluigilenoci @Pionerd and @andreasgeisslerdt I don't know what connectionUrls is actually supposed to be. I could not find a way of feeding it three direct connections to the redis servers as the only things that would resolve within our k8s cluster was the service the Redis chart stood up and the ephemeral IPs which you cannot predict and should not try to talk to. I used the svc and that seems to be working just fine. Feels like it is wanting all three so it can do that init check to make sure all three are up before starting the oauth2-proxy container but I'm not sure.
If any of you have a better idea of what should be used there I would love to here them.

Once we figure out if there is / is not a better solution to connectionUrls I suggest the values.yaml file be updated to reflect a better starting state and examples for sentinel mode.

NOTE: ${redis_password} is a Terraform variable and will need to be replaced. This is the sessionStorage and redis sections of the values file only.

# Configure the session storage type, between cookie and redis
sessionStorage:
  # Can be one of the supported session storage cookie|redis
  type: redis
  redis:
    # Redis password value. Applicable for all Redis configurations. Taken from redis subchart secret if not set. `sessionStorage.redis.existingSecret` takes precedence
    password: "${redis_password}"
    # Can be one of standalone|cluster|sentinel
    clientType: "sentinel"
    sentinel:
      # Name of the Kubernetes secret containing the redis sentinel password value (see also `sessionStorage.redis.sentinel.passwordKey`). Default: `sessionStorage.redis.existingSecret`
      existingSecret: ""
      # Redis sentinel password. Used only for sentinel connection; any redis node passwords need to use `sessionStorage.redis.password`
      password: "${redis_password}"
      # Key of the Kubernetes secret data containing the redis sentinel password value
      passwordKey: "redis-sentinel-password" # This is not used as we are passing in a  password from Terraform.
      # Redis sentinel master name, default in the chart is "mymaster"
      masterName: "mymaster"
      # List of Redis sentinel connection URLs (e.g. `["redis://127.0.0.1:8000", "redis://127.0.0.1:8000"]`)
      # Only thing I could get to work was to point at the service.
      connectionUrls: "redis://oauth2-proxy-redis:26379"

# Enables and configure the automatic deployment of the redis subchart
redis:
  global:
    redis:
      password: ${redis_password}
  # provision an instance of the redis sub-chart
  enabled: true
  tls:
    enabled: true
    authClients: false
    autoGenerated: true
  # Redis specific helm chart settings, please see:
  # https://github.com/bitnami/charts/tree/master/bitnami/redis#parameters
  # redisPort: 6379
  sentinel:
    enabled: true
    # This is the default according to the chart docs https://github.com/bitnami/charts/tree/main/bitnami/redis#parameters
    # Setting it here to be extra safe in case the default changes at some point.
    masterSet: "mymaster"

Pionerd mentioned this issue Apr 9, 2024

Improve wait-for-redis logic #197

Merged

Pionerd mentioned this issue May 7, 2024

Fix wait for redis #205

Merged

pierluigilenoci closed this as completed in #205 May 7, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Redis sentinel does not work, chart throws error on install #191

Redis sentinel does not work, chart throws error on install #191

sarg3nt commented Mar 22, 2024 •

edited

Loading

pierluigilenoci commented Apr 3, 2024 •

edited

Loading

sarg3nt commented Apr 4, 2024

pierluigilenoci commented Apr 8, 2024 •

edited

Loading

Pionerd commented Apr 9, 2024

pierluigilenoci commented Apr 10, 2024

pierluigilenoci commented Apr 30, 2024

sarg3nt commented May 3, 2024

andreasgeisslerdt commented May 7, 2024

Pionerd commented May 7, 2024

pierluigilenoci commented May 7, 2024

sarg3nt commented May 8, 2024

Redis sentinel does not work, chart throws error on install #191

Redis sentinel does not work, chart throws error on install #191

Comments

sarg3nt commented Mar 22, 2024 • edited Loading

pierluigilenoci commented Apr 3, 2024 • edited Loading

sarg3nt commented Apr 4, 2024

pierluigilenoci commented Apr 8, 2024 • edited Loading

Pionerd commented Apr 9, 2024

pierluigilenoci commented Apr 10, 2024

pierluigilenoci commented Apr 30, 2024

sarg3nt commented May 3, 2024

andreasgeisslerdt commented May 7, 2024

Pionerd commented May 7, 2024

pierluigilenoci commented May 7, 2024

sarg3nt commented May 8, 2024

sarg3nt commented Mar 22, 2024 •

edited

Loading

pierluigilenoci commented Apr 3, 2024 •

edited

Loading

pierluigilenoci commented Apr 8, 2024 •

edited

Loading