EnvoyFilter LocalRateLimit with actions and descriptors does not work as expected

[serhii-ciq]

Hi there!

I am trying to create the local rate limit and seems this does not work as expected.
My idea is to have several usage plans and a common limit.
Usage plans are based on the header X-USAGE-PLAN: basic or X-USAGE-PLAN: pro and X-USAGE-PLAN: enterprise

This is the current Envoy configuration

Envoy config

                  {
                      "name": "envoy.filters.network.http_connection_manager",
                      "typedConfig": {
                          "@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager",
                          "statPrefix": "inbound_0.0.0.0_8080;",
                          "routeConfig": {
                              "name": "inbound|8080||",
                              "virtualHosts": [
                                  {
                                      "name": "inbound|http|8000",
                                      "domains": [
                                          "*"
                                      ],
                                      "routes": [
                                          {
                                              "name": "default",
                                              "match": {
                                                  "prefix": "/"
                                              },
                                              "route": {
                                                  "cluster": "inbound|8080||",
                                                  "timeout": "0s",
                                                  "maxStreamDuration": {
                                                      "maxStreamDuration": "0s",
                                                      "grpcTimeoutHeaderMax": "0s"
                                                  }
                                              },
                                              "decorator": {
                                                  "operation": "httpbin.httpbin.svc.cluster.local:8000/*"
                                              }
                                          }
                                      ],
                                      "rateLimits": [
                                          {
                                              "actions": [
                                                  {
                                                      "requestHeaders": {
                                                          "headerName": "X-USAGE-PLAN",
                                                          "descriptorKey": "header-rate-limit"
                                                      }
                                                  }
                                              ]
                                          }
                                      ]
                                  }
                              ],
                              "validateClusters": false
                          },
                          "httpFilters": [
                              {
                                  "name": "envoy.filters.http.local_ratelimit",
                                  "typedConfig": {
                                      "@type": "type.googleapis.com/udpa.type.v1.TypedStruct",
                                      "typeUrl": "type.googleapis.com/envoy.extensions.filters.http.local_ratelimit.v3.LocalRateLimit",
                                      "value": {
                                          "descriptors": [
                                              {
                                                  "entries": [
                                                      {
                                                          "key": "header-rate-limit",
                                                          "value": "basic"
                                                      }
                                                  ],
                                                  "token_bucket": {
                                                      "fill_interval": "30s",
                                                      "max_tokens": 5,
                                                      "tokens_per_fill": 5
                                                  }
                                              },
                                              {
                                                  "entries": [
                                                      {
                                                          "key": "header-rate-limit",
                                                          "value": "pro"
                                                      }
                                                  ],
                                                  "token_bucket": {
                                                      "fill_interval": "30s",
                                                      "max_tokens": 10,
                                                      "tokens_per_fill": 5
                                                  }
                                              },
                                              {
                                                  "entries": [
                                                      {
                                                          "key": "header-rate-limit",
                                                          "value": "enterprise"
                                                      }
                                                  ],
                                                  "token_bucket": {
                                                      "fill_interval": "30s",
                                                      "max_tokens": 50000,
                                                      "tokens_per_fill": 50000
                                                  }
                                              }
                                          ],
                                          "filter_enabled": {
                                              "default_value": {
                                                  "numerator": 100
                                              },
                                              "runtime_key": "local_rate_limit_enabled"
                                          },
                                          "filter_enforced": {
                                              "default_value": {
                                                  "numerator": 100
                                              },
                                              "runtime_key": "local_rate_limit_enforced"
                                          },
                                          "response_headers_to_add": [
                                              {
                                                  "header": {
                                                      "key": "x-rate-limited",
                                                      "value": "TOO_MANY_REQUESTS"
                                                  }
                                              }
                                          ],
                                          "stat_prefix": "http_local_rate_limiter",
                                          "status": {
                                              "code": "TooManyRequests"
                                          },
                                          "token_bucket": {
                                              "fill_interval": "30s",
                                              "max_tokens": 1,
                                              "tokens_per_fill": 1
                                          }
                                      }
                                  }
                              },

The issue is:

Currently, only a common bucket does work. Whether I send requests with special headers or not they are throttled by a common bucket.
Seems the buckets for certain headers do not work

How it looks

Console output

(venv) ❯❯  istio-1.23.3  10:26  http https://httpbin.company.com/status/200 "X-USAGE-PLAN: enterprise"
HTTP/1.1 200 OK
Connection: keep-alive
Content-Length: 0
Content-Type: text/plain; charset=utf-8
Date: Thu, 21 Nov 2024 08:26:11 GMT
access-control-allow-credentials: true
access-control-allow-origin: *
server: istio-envoy
x-envoy-upstream-service-time: 1

(venv) ❯❯  istio-1.23.3  10:26  http https://httpbin.company.com/status/200 "X-USAGE-PLAN: enterprise"
HTTP/1.1 429 Too Many Requests
Connection: keep-alive
Content-Length: 18
Content-Type: text/plain
Date: Thu, 21 Nov 2024 08:26:14 GMT
server: istio-envoy
x-envoy-upstream-service-time: 0
x-rate-limited: TOO_MANY_REQUESTS

local_rate_limited

I configured the metrics for http. and the metric

envoy_http_downstream_rq_http1_total{http_conn_manager_prefix="inbound_0.0.0.0_8080"} 27

increased each time I sent a request

Envoy version

envoy  version: cbd889517ed13455bf2d88facc5685d958eb54a6/1.31.3-dev/Clean/RELEASE/BoringSSL

[serhii-ciq]

Seems like I found out the root cause of this issue:

The global bucket is used FIRST

If a request for a certain bucket exhausts the GLOBAL bucket then this request will be throttled regardless of the bucket configuration that request belongs to!

[soulxu]

cc @wbpcode