sotw: auth

[guicassolato]

State-of-the-world reconciler – Auth workflow

• spec.targetRef.sectionName
• spec.(defaults|overrides).strategy
• Defaults & Overrides merge strategies (RFC-0009)
• Effective AuthPolicy
• Authorino AuthConfigs
• Istio cluster (EnvoyFilter)
• istio extension (WasmPlugin)
• envoy cluster (EnvoyPatchPolicy)
• envoy extension (EnvoyExtensionPolicy)
• Accepted status condition
• Enforced status condition – mising status of the AuthConfigs only
• Rename spec.rules.response.success.dynamicMetadata → spec.rules.response.success.filters

TODOs

• Incorporate status of the AuthConfigs into the AuthPolicy Enforced status condition
• Fix integration tests
• Dry-run desired AuthConfig to ensure defaults are applied –– probably in a separate PR, so a more holistic approach can be employed, covering all objects that are updated (not only AuthConfigs)

Closes #820
Closes #825
Closes #822

Verification steps

Setup the environment:

make local-setup

Enable Envoy Gateway alongside with Istio:

make envoy-gateway-install

# Restart the Kuadrant Operator so it can acknowledge the presence of Envoy Gateway
kubectl rollout restart deployment/kuadrant-operator-controller-manager -n kuadrant-system

kubectl apply -f -<<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: envoygateway
spec:
  controllerName: gateway.envoyproxy.io/gatewayclass-controller
EOF

kubectl apply -n gateway-system -f -<<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: kuadrant-envoygateway
spec:
  gatewayClassName: envoygateway
  listeners:
  - name: https
    hostname: "*.eg.apps.io"
    protocol: HTTPS
    port: 443
    tls:
      mode: Terminate
      certificateRefs:
      - name: kuadrant-envoygateway-cert
        kind: Secret
    allowedRoutes:
      namespaces:
        from: All
EOF

Configure TLS on the Envoy Gateway-provided gateway:

kubectl apply -n gateway-system -f -<<EOF
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: kuadrant-ca
spec:
  selfSigned: {}
---
apiVersion: kuadrant.io/v1alpha1
kind: TLSPolicy
metadata:
  name: kuadrant-envoygateway-tls
  namespace: gateway-system
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: Gateway
    name: kuadrant-envoygateway
  issuerRef:
    group: cert-manager.io
    kind: ClusterIssuer
    name: kuadrant-ca
EOF

Deploy an application:

kubectl apply -f examples/toystore/toystore.yaml

kubectl apply -f - <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: toystore
spec:
  parentRefs:
  - name: kuadrant-ingressgateway
    namespace: gateway-system
  - name: kuadrant-envoygateway
    namespace: gateway-system
  rules:
  - backendRefs:
    - name: toystore
      port: 80
    matches:
    - method: GET
  - backendRefs:
    - name: toystore
      port: 80
    matches:
    - method: POST
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: other
spec:
  hostnames:
  - other.example.com
  parentRefs:
  - name: kuadrant-ingressgateway
    namespace: gateway-system
  rules:
  - backendRefs:
    - name: toystore
      port: 80
EOF

(From now on and at anytime) Send requests to the application:

export ISTIO_GATEWAY_IP=$(kubectl get gateway/kuadrant-ingressgateway -n gateway-system -o jsonpath='{.status.addresses[0].value}')

curl --resolve toystore.example.com:80:$ISTIO_GATEWAY_IP http://toystore.example.com --write-out '%{http_code}\n' --silent --output /dev/null
curl --resolve other.example.com:80:$ISTIO_GATEWAY_IP http://other.example.com --write-out '%{http_code}\n' --silent --output /dev/null

export ENVOY_GATEWAY_IP=$(kubectl get gateway/kuadrant-envoygateway -n gateway-system -o jsonpath='{.status.addresses[0].value}')

curl --resolve toystore.eg.apps.io:443:$ENVOY_GATEWAY_IP https://toystore.eg.apps.io --write-out '%{http_code}\n' --silent --output /dev/null --insecure

Deploy Kuadrant:

kubectl -n kuadrant-system apply -f - <<EOF
apiVersion: kuadrant.io/v1beta1
kind: Kuadrant
metadata:
  name: kuadrant
spec: {}
EOF

Create a gateway atomic default RateLimitPolicy:

kubectl apply -n gateway-system -f - <<EOF
apiVersion: kuadrant.io/v1beta3
kind: RateLimitPolicy
metadata:
  name: gw-rlp
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: Gateway
    name: kuadrant-ingressgateway
  defaults:
    limits:
      "global":
        rates:
        - limit: 5
          duration: 10
          unit: second
        when:
        - selector: source.address
          operator: neq
          value: 127.0.0.1
EOF

Create a route RateLimitPolicy:

kubectl apply -f - <<EOF
apiVersion: kuadrant.io/v1beta3
kind: RateLimitPolicy
metadata:
  name: route-rlp
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: toystore
  limits:
    "specific":
      rates:
      - limit: 3
        duration: 5
        unit: second
      - limit: 20
        duration: 1
        unit: minute
EOF

Modify the gateway RateLimitPolicy to atomic override strategy:

kubectl apply -n gateway-system -f - <<EOF
apiVersion: kuadrant.io/v1beta3
kind: RateLimitPolicy
metadata:
  name: gw-rlp
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: Gateway
    name: kuadrant-ingressgateway
  overrides:
    limits:
      "global":
        rates:
        - limit: 5
          duration: 10
          unit: second
        when:
        - selector: source.address
          operator: neq
          value: 127.0.0.1
EOF

Modify the gateway RateLimitPolicy to merge override strategy:

kubectl apply -n gateway-system -f - <<EOF
apiVersion: kuadrant.io/v1beta3
kind: RateLimitPolicy
metadata:
  name: gw-rlp
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: Gateway
    name: kuadrant-ingressgateway
  overrides:
    limits:
      "global":
        rates:
        - limit: 5
          duration: 10
          unit: second
        when:
        - selector: source.address
          operator: neq
          value: 127.0.0.1
    strategy: merge
EOF

Create a route AuthPolicy:

kubectl apply -f - <<EOF
apiVersion: kuadrant.io/v1beta3
kind: AuthPolicy
metadata:
  name: route-auth
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: toystore
  rules:
    authentication:
      "api-keys-authn":
        apiKey:
          selector:
            matchLabels:
              app: toystore
EOF

Create a gateway merge override AuthPolicy:

kubectl apply -n gateway-system -f - <<EOF
apiVersion: kuadrant.io/v1beta3
kind: AuthPolicy
metadata:
  name: route-auth
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: Gateway
    name: kuadrant-envoygateway
  overrides:
    rules:
      authorization:
        "forbidden-ip":
          patternMatching:
            patterns:
            - selector: source.address.@extract{"sep":":"}
              operator: neq
              value: "177.30.200.1"
    strategy: merge
EOF

Try other use cases not covered in the verification steps.

In general, this PR should enable:

• AuthPolicies targeting a Gateway
• AuthPolicies targeting a specific Listener of a Gateway (with spec.targetRef.sectionName)
• AuthPolicies targeting a HTTPRoute
• AuthPolicies targeting a specific HTTPRouteRule of a HTTPRoute (with spec.targetRef.sectionName)
• Multiple AuthPolicies targeting a same resource - different sections of the resource
• Multiple AuthPolicies targeting a same resource - same section of the resource
• Multiple AuthPolicies targeting a same resource - entire resource and section of the resource
• Multiple AuthPolicies targeting a same resource - entire resource
• Defaults and Overrides in AuthPolicies targeting Gateways
• Defaults and Overrides in AuthPolicies targeting HTTPRoutes
• Defaults and Overrides' policy merge strategy in AuthPolicies (spec.defaults.strategy: merge, spec.overrides.strategy: merge)
• Multiple Gateways parenting a HTTPRoute with AuthPolicies attached at any level
• Gateways controlled by Istio only
• Gateways controlled by Envoy Gateway only
• Mixed Gateways controlled by Istio and Envoy Gateway
• Status of Gateways and HTTPRoutes reflected in the status Enforced condition of the policies
• Status of Authorino CR reflected in the status Enforced condition of the policies
• Presence of Istio WasmPlugin CRs reflected in the status Enforced condition of the policies
• Presence of Istio EnvoyFilter CRs reflected in the status Enforced condition of the policies
• Presence of Envoy Gateway EnvoyExtensionPolicy CRs reflected in the status Enforced condition of the policies
• Presence of Envoy Gateway EnvoyPatchPolicy CRs reflected in the status Enforced condition of the policies

[codecov]

Codecov Report

Attention: Patch coverage is 86.25162% with 212 lines in your changes missing coverage. Please review.

Project coverage is 76.55%. Comparing base (63f1d28) to head (6138e34).
Report is 40 commits behind head on main.

Files with missing lines	Patch %	Lines
api/v1beta3/authpolicy_types.go	85.28%	38 Missing and 1 partial ⚠️
...ntrollers/envoy_gateway_auth_cluster_reconciler.go	80.13%	25 Missing and 4 partials ⚠️
controllers/istio_auth_cluster_reconciler.go	80.82%	24 Missing and 4 partials ⚠️
pkg/istio/utils.go	51.78%	18 Missing and 9 partials ⚠️
controllers/authconfigs_reconciler.go	86.59%	19 Missing and 7 partials ⚠️
controllers/auth_policy_status_updater.go	91.32%	12 Missing and 5 partials ⚠️
api/v1/merge_strategies.go	60.71%	11 Missing ⚠️
controllers/auth_workflow_helpers.go	91.66%	6 Missing and 3 partials ⚠️
pkg/envoygateway/utils.go	78.04%	6 Missing and 3 partials ⚠️
controllers/effective_auth_policies_reconciler.go	88.23%	5 Missing and 1 partial ⚠️
... and 6 more

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #952      +/-   ##
==========================================
- Coverage   81.49%   76.55%   -4.94%     
==========================================
  Files         102      113      +11     
  Lines        7177     9056    +1879     
==========================================
+ Hits         5849     6933    +1084     
- Misses        898     1813     +915     
+ Partials      430      310     -120     

Flag	Coverage Δ
bare-k8s-integration	10.76% <12.12%> (+1.87%)	⬆️
controllers-integration	63.78% <86.25%> (-1.55%)	⬇️
envoygateway-integration	37.69% <30.02%> (-12.61%)	⬇️
gatewayapi-integration	13.71% <12.45%> (-0.70%)	⬇️
istio-integration	39.46% <30.28%> (-14.06%)	⬇️
unit	25.19% <0.19%> (-3.15%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
api/v1beta1 (u)	92.18% <100.00%> (+1.27%)	⬆️
api/v1beta2 (u)	∅ <ø> (∅)
pkg/common (u)	87.67% <ø> (-0.47%)	⬇️
pkg/istio (u)	47.03% <ø> (-24.49%)	⬇️
pkg/log (u)	93.18% <ø> (-1.56%)	⬇️
pkg/reconcilers (u)	∅ <ø> (∅)
pkg/rlptools (u)	∅ <ø> (∅)
controllers (i)	83.48% <85.01%> (+0.42%)	⬆️

Files with missing lines	Coverage Δ
api/v1beta1/topology.go	100.00% <100.00%> (ø)
api/v1beta3/ratelimitpolicy_types.go	89.62% <100.00%> (+15.93%)	⬆️
controllers/auth_policies_validator.go	100.00% <100.00%> (ø)
controllers/data_plane_policies_workflow.go	100.00% <100.00%> (ø)
...rollers/effective_ratelimit_policies_reconciler.go	88.23% <100.00%> (ø)
...lers/envoy_gateway_ratelimit_cluster_reconciler.go	80.00% <100.00%> (ø)
controllers/ratelimit_policies_validator.go	100.00% <100.00%> (ø)
controllers/ratelimit_policy_status_updater.go	88.81% <100.00%> (ø)
controllers/state_of_the_world.go	92.16% <100.00%> (-5.07%)	⬇️
controllers/test_common.go	100.00% <ø> (ø)
... and 22 more

... and 48 files with indirect coverage changes

[maleck13]

@guicassolato who is the right person to review this? I am happy to take a look to help it move forward, but someone who has worked on auth or sotw might be a better candidate?
@mikenairn @KevFan ?

[Boomatang]

Here are some of the things I noticed while reviewing the code.

[adam-cattermole]

I was giving this a go following the authenticated rate limiting guide and am hitting an error trying to define an AuthPolicy with spec.rules.response.success.filters - is there an issue in the way I've defined this:

kubectl apply -f - <<EOF
apiVersion: kuadrant.io/v1beta3
kind: AuthPolicy
metadata:
  name: toystore
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: toystore
  rules:
    authentication:
      "api-key-users":
        apiKey:
          selector:
            matchLabels:
              app: toystore
          allNamespaces: true
        credentials:
          authorizationHeader:
            prefix: APIKEY
    response:
      success:
        filters:
          "identity":
            json:
              properties:
                "userid":
                  selector: auth.identity.metadata.annotations.secret\.kuadrant\.io/user-id
EOF

The operator crash loops panicking but is fine once I remove the response block.

The panic:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x151d4a0]

goroutine 5078 [running]:
github.com/kuadrant/kuadrant-operator/api/v1beta3.(*MergeableDenyWithSpec).GetSource(0x17d3980?)
	/workspace/api/v1beta3/authpolicy_types.go:554
github.com/kuadrant/kuadrant-operator/api/v1.NewMergeableRule({0x1e2d110, 0x0}, {0x4000884210, 0x27})
	/workspace/api/v1/merge_strategies.go:37 +0x34
github.com/kuadrant/kuadrant-operator/api/v1beta3.(*AuthPolicy).Rules(0x4000143380)
	/workspace/api/v1beta3/authpolicy_types.go:171 +0x9c4
github.com/kuadrant/kuadrant-operator/api/v1.copyMergeablePolicy({0xfffe6b897270, 0x400086f1e0})
	/workspace/api/v1/merge_strategies.go:224 +0x98
github.com/kuadrant/kuadrant-operator/api/v1.AtomicDefaultsMergeStrategy({0x1e470c0, 0x400086f1e0}, {0x1e470c0?, 0x400086f1e0})
	/workspace/api/v1/merge_strategies.go:73 +0x100
github.com/kuadrant/kuadrant-operator/api/v1beta3.(*AuthPolicy).Merge(0x1e45f40?, {0x1e470c0?, 0x400086f1e0?})
	/workspace/api/v1beta3/authpolicy_types.go:117 +0x164
github.com/kuadrant/kuadrant-operator/api/v1.EffectivePolicyForPath[...].func1({0x1e470c0?, 0x400086f1e0?}, 0x4000ee8101?)
	/workspace/api/v1/merge_strategies.go:187 +0x3c
github.com/samber/lo.ReduceRight[...]({0x40004e4af0?, 0x5, 0x1884c60?}, 0x40003ad968?, {0x1e470c0, 0x400086f1e0?})
	/go/pkg/mod/github.com/samber/[email protected]/slice.go:82 +0x5c
github.com/kuadrant/kuadrant-operator/api/v1.EffectivePolicyForPath[...]({0x40004e4a50, 0x5, 0x5}, 0x4000ee81e0)
	/workspace/api/v1/merge_strategies.go:186 +0xd4
github.com/kuadrant/kuadrant-operator/controllers.(*EffectiveAuthPolicyReconciler).calculateEffectivePolicies(0x400077f6e0?, {0x1e3cf70?, 0x400017aaf0?}, 0x400011dac0, {0x1e407a0, 0x40009ec500}, 0x400011dae0)
	/workspace/controllers/effective_auth_policies_reconciler.go:74 +0x3d0
github.com/kuadrant/kuadrant-operator/controllers.(*EffectiveAuthPolicyReconciler).Reconcile(0x4000028310, {0x1e3cf70, 0x400017aaf0}, {0x1e5be08?, 0x400105bc20?, 0x1e5be08?}, 0x400011dac0, {0x0?, 0x0?}, 0x400011dae0)
	/workspace/controllers/effective_auth_policies_reconciler.go:49 +0x148
github.com/kuadrant/policy-machinery/controller.Subscription.Reconcile({0x40001a0420?, {0x2f4d800?, 0x40008796f8?, 0x150e288?}}, {0x1e3cf70, 0x400017aaf0}, {0x400097b950?, 0x400097b950?, 0x2?}, 0x400011dac0, ...)
	/go/pkg/mod/github.com/kuadrant/[email protected]/controller/subscriber.go:34 +0xc4
github.com/kuadrant/policy-machinery/controller.(*Workflow).Run.func1()
	/go/pkg/mod/github.com/kuadrant/[email protected]/controller/workflow.go:42 +0x48
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78 +0x58
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 5077
	/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x98

[guicassolato]

I was giving this a go following the authenticated rate limiting guide and am hitting an error trying to define an AuthPolicy with spec.rules.response.success.filters - is there an issue in the way I've defined this:

[…]

The operator crash loops panicking but is fine once I remove the response block.

Thanks @adam-cattermole! Silly one this bug. Fixed.

[KevFan]

@guicassolato Is there anything from the guides that should not be working currently from this change? 🤔

I know you've mentioned before that API Key Secret's should be created in the kuadrant-system namespace where the AuthConfig's are now created but following https://github.com/Kuadrant/kuadrant-operator/blob/main/doc/user-guides/auth-for-app-devs-and-platform-engineers.md#-protect-the-toy-store-application-persona-app-developer
at:

curl -H 'Host: api.toystore.com' -H 'Authorization: APIKEY iamaregularuser' http://$GATEWAY_URL/cars -i
# HTTP/1.1 200 OK

gives a 403:

# HTTP/1.1 403 Forbidden                     
HTTP/1.1 403 Forbidden
x-ext-auth-reason: Unauthorized
date: Thu, 31 Oct 2024 12:29:44 GMT
server: istio-envoy
content-length: 0

[thomasmaas]

Lets merge move on.

[guicassolato]

@guicassolato Is there anything from the guides that should not be working currently from this change? 🤔

I know you've mentioned before that API Key Secret's should be created in the kuadrant-system namespace where the AuthConfig's are now created but following https://github.com/Kuadrant/kuadrant-operator/blob/main/doc/user-guides/auth-for-app-devs-and-platform-engineers.md#-protect-the-toy-store-application-persona-app-developer at:

curl -H 'Host: api.toystore.com' -H 'Authorization: APIKEY iamaregularuser' http://$GATEWAY_URL/cars -i
# HTTP/1.1 200 OK

gives a 403:

# HTTP/1.1 403 Forbidden                     
HTTP/1.1 403 Forbidden
x-ext-auth-reason: Unauthorized
date: Thu, 31 Oct 2024 12:29:44 GMT
server: istio-envoy
content-length: 0

@KevFan, I've added another commit updating the doc.

TL;DR – What you experienced when trying the user guide is not an issue of this PR. Rather, it's a combination of 2 things:

1. The removal of routeSelector (UPDATE: bump authPolicy to v1beta3 in guides #949) without an equivalent using sectionName (only possible after this PR).
2. (TBC) What I believe could be a bug in Authorino that linked the API key Secrets created in the default namespace to AuthConfigs in the kuadrant-system one, despite apiKey.allNamespaces set to false, due to the empty apiKey.selector.

[KevFan]

Looks good to me! 👍

[Boomatang]

Happy to see this merge