bug: openstack: XFF headers not available on CCEE clusters

[pbochynski]

Description
I followed a tutorial: https://github.com/kyma-project/api-gateway/blob/main/docs/user/tutorials/01-50-expose-and-secure-a-workload/v2alpha1/01-55-ip-based-access-with-xff.md
My cluster runs on OpenStack (CCEE).
In that case external Ip and XFF headers are not populated by istio-ingress-gateway.

Expected result

I can access client IP from my workload

Suggestions
I found that openstack has possibility to configure load balancer through service annotations:
https://github.com/kubernetes/cloud-provider-openstack/blob/master/docs/openstack-cloud-controller-manager/expose-applications-using-loadbalancer-type-service.md#use-proxy-protocol-to-preserve-client-ip

[pbochynski]

I managed to get client IP address by applying some annotation in the load balancer service. This is my test deployment and service:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: httpbin
  name: httpbin
spec:
  replicas: 1
  selector:
    matchLabels:
      app: httpbin
  template:
    metadata:
      labels:
        app: httpbin
    spec:
      containers:
      - image: kennethreitz/httpbin
        imagePullPolicy: Always
        name: httpbin
---
apiVersion: v1
kind: Service
metadata:
  annotations:
    loadbalancer.openstack.org/proxy-protocol: v2
    loadbalancer.openstack.org/x-forwarded-for: "true"
  name: test-lb-1
spec:
  externalTrafficPolicy: Local
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: httpbin
  type: LoadBalancer

Apply it, wait for external IP address and then you can check if you get your IP from curl {externalIP}/ip

I applied the same annotations on the existing istio-ingressgateway service, but it didn't work for me (maybe it has to be created with the proper annotations).

[Ressetkk]

The issue is not about the External IP not being available on a LB service, but about XFF header not being in a request.
/retitle bug: openstack: XFF headers not available on CCEE clusters

First of all - setting both annotations loadbalancer.openstack.org/proxy-protocol and loadbalancer.openstack.org/x-forwarded-for is not supported. When you apply both annotations and see the events on istio-ingressgateway service you can observe an error:

  Warning  SyncLoadBalancerFailed  26s (x7 over 5m41s)  service-controller  Error syncing load balancer: failed to ensure load balancer: annotation loadbalancer.openstack.org/proxy-protocol and loadbalancer.openstack.org/x-forwarded-for cannot be used together

That's most likely why setting it up didn't work for you.

Second of all, istio module deploys EnvoyFilter that allows using proxy protocol, but it seems in OpenStack case, it only supports proxy protocol in version 1. Trying to use proxy-protocol in version 2 using annotation loadbalancer.openstack.org/proxy-protocol=v2 will make istio-ingressgateway connectivity broken with error that proxy protocol header exceeds allowed v1 length.

There is also seems to be another error when you annotate the existing load balancer which seems to be out-of-scope of the istio module. We do not change any configuration internally and this error comes from the external Load Balancer service used in CCEE:

  Warning  SyncLoadBalancerFailed  3m45s (x2 over 5m21s)   service-controller  Error syncing load balancer: failed to ensure load balancer: failed to create listener for loadbalancer 2f787089-8a96-44fb-8228-8ca329da9e25: Expected HTTP response code [201 202] when accessing [POST https://{...}/v2.0/lbaas/listeners], but got 409 instead: {"faultcode": "Client", "faultstring": "Another Listener on this Load Balancer is already using protocol TCP and protocol_port 15021", "debuginfo": null}

I'm not really sure about the configuration of the load-balancer itself, but in the openstack-lb docu there is a mention that this annotation is working only when using Octavia. That means if the cluster is using anything other than Octavia, the loadbalancer.openstack.org/x-forwarded-for will be useless. But since, it's generating the event on the resource, I assume CCEE uses Octavia as LB backend.

If 'true', X-Forwarded-For is inserted into the HTTP headers which contains the original client IP address so that the backend HTTP service is able to get the real source IP of the request. Please note that the cloud provider will force the creation of an Octavia listener of type HTTP if this option is set. Only applies when using Octavia.

k annotate service -n istio-system istio-ingressgateway loadbalancer.openstack.org/x-forwarded-for=true

References:
https://github.com/kubernetes/cloud-provider-openstack/blob/master/docs/openstack-cloud-controller-manager/expose-applications-using-loadbalancer-type-service.md

[Ressetkk]

I can confirm, when you apply loadbalancer.openstack.org/proxy-protocol: v1 annotation on a resource, XFF header is passed to the request. You need just more time for LoadBalancer to reconfigure the resource.