Crowdsec bans the wrong IP when OpenAppSec is behind a reverse proxy

[alnviana]

I have the following structure:
Internet -> Reverse Proxy 1 -> OpenAppSec -> App1, App2, etc

The first proxy is configured to fill and pass the X-Forwarding-For, so it's value is "Real IP, Proxy 1 IP".
I configured OpenAppSec Source Identity to use this header instead of using Source IP.
I did a SQL Injection test, a event was triggered:

{
  "eventData": {
    "proxyIP": "<REAL IP>",
    "sourceIP": "<PROXY 1 IP>",
    "httpSourceId": "<REAL IP>",
  }
}

Crowdsec reads the logs, parses the sourceIP and bans Proxy 1 IP.

Adding the Proxy 1's IP as a hop on Source Identity doesn't help, resulting in:

{
  "eventData": {
    "sourceIP": "<PROXY 1 IP>",
    "httpSourceId": "<PROXY 1 IP>",
  }
}

With OpenAppSec directly exposed (Without Proxy 1) the Real IP appears as sourceIP, so it is banned correctly.

So I'm thinking of the following possible reasons for it not working:

• The parser was planned to work just when OpenAppSec is directly exposed to the internet.
  • If this is the case, can someone tell me the correct way to configure it? I know I can create a parser manually, but since it exists and is a prerequisite for collection, I think I need to do something else to avoid banning duplicates with the wrong IP.
• OpenAppSec is filling in the fields incorrectly.
  • I tried to find the documentation for these fields, what they should contain and how they should be used, but I couldn't find it.
  • It's strange that the Real IP is placed in the proxyIP field, after all, this is the IP of the source and not the Proxy, but that depends on your point of view. So, I think the name of the proxyIP field might not be the best, or at least it should have documentation explaining it. (I know, it's not up to you)
• The parser is using the wrong field.
  • It could be changed to use the proxyIP when it exists. Or maybe it shouldn't use sourceIP but httpSourceId... But as I said above, I couldn't find any information about these fields.

If anyone can help me, I'd be grateful. :)

[LaurenceJJones]

Hey 👋🏻

Thank you for a detailed issue.

I can confirm when we was initially testing and publishing the collection we did expose it directly to the internet with no testing of being behind an upstream proxy.

I reach out to our contact at checkpoint to clarify a few points, as from our own interpretation of the field names I would suspect the same as you, proxyIP to be the upstream IP address and sourceIP to be set to the actual user IP.

However, before making changes to the parser I will wait for confirmation this is the intended behavior and if so will make the changes to blindly "trust" the proxyIP if it is set (I say blindly because CrowdSec doesnt have the context to know if the IP is trusted as in the upstream so we have to just trust it).

Edit: or switch to httpSourceId depending on the outcome of the clarification.

[alnviana]

Thanks for the quick reply. :)
I agree with you that it is good to plan this change well, as it could cause unwanted effects on those who are already using it. This care is the basics when it comes to security and resilience, which makes me like the project even more.

If you need any more information or help testing something, feel free to ask me.

[chkp-sergeysh]

Hello @alnviana,

Thank you for providing such a detailed report. To gain a clearer understanding of the issue related to the X-Forwarded-For source identifier, we need to examine the request headers at each stage from the client to the open-appsec endpoint. This information will help us identify any discrepancies that may be occurring along the path.

Could you please provide the following details?

1. Complete Request Headers:

   • Capture the full request headers at each stage as the request travels from:
     • The client
     • Proxy 1
     • The open-appsec endpoint

2. X-Forwarded-For Header Details:

   • Document the contents of the X-Forwarded-For header, noting any changes observed at each hop.

3. Network Configuration:

   • If possible, include a brief description of any network or proxy configurations that might impact header forwarding. Additionally, please share the open-appsec Asset policy settings where the X-Forwarded-For header is configured as the source identifier.

These details will enable us to replicate the environment and more accurately pinpoint any configuration or handling issues. Please let us know if you need assistance capturing this data.

Thank you for your cooperation.

Best regards,
Sergey
open-appsec Development Team

[alnviana]

Of course, I can. Just one observation:
Nginx doesn't seem to have a native way of logging all the headers of a request. I only know how to do this through a customized log_format, but I'd have to enter each of the headers manually, and there might be some missing that I don't know about. Because of this I decided to do it a little differently, but I can test it another way if you tell me it's not right.

Moving on, as the intention is to know how the headers are received and passed on, I decided to use an OpenResty container configured to record all the headers, replacing each hop at each stage. I understand that the ideal would be to use the scenario as closely as possible, but in this way it is still possible to know the “input/output” of each stage.

To make it easier to understand the structure at each stage, I've created the diagram below to illustrate:

OpenResty configuration

log_format log_req_resp '$remote_addr - $remote_user [$time_local] '
  '"$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" '
  '$request_time req_header:"$req_header" resp_header:"$resp_header" '
  'req_body:"$request_body" resp_body:"$resp_body"';

server {
    listen       80;
    server_name  localhost;

    location / {
      set $req_header "";
      set $resp_header "";
      header_filter_by_lua_block{
        local h = ngx.req.get_headers();
        for k, v in pairs(h) do
          ngx.var.req_header = ngx.var.req_header .. k.."="..v.." ";
        end
        local rh = ngx.resp.get_headers();
        for k, v in pairs(rh) do
          ngx.var.resp_header = ngx.var.resp_header .. k.."="..v.." ";
        end
      }

      lua_need_request_body on;
      set $resp_body "";
      body_filter_by_lua_block {
        local resp_body = string.sub(ngx.arg[1], 1, 1000)
        ngx.ctx.buffered = (ngx.ctx.buffered or "") .. resp_body
        if ngx.arg[2] then
          ngx.var.resp_body = ngx.ctx.buffered
        end
      }

      access_log  /dev/stdout log_req_resp;

      add_header Content-Type application/json;
      echo_read_request_body;
      echo '{ "result": "OK" }';
    }
}

These are the settings used in Bunkerweb, which refer to the headers

# /usr/share/bunkerweb/core/reverseproxy/confs/server-http/reverse-proxy.conf
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-Protocol $scheme;
proxy_set_header X-Forwarded-Ssl $proxy_x_forwarded_ssl;
proxy_set_header X-Forwarded-Host $http_host;
proxy_set_header X-Forwarded-Port $server_port;
proxy_set_header X-Forwarded-Prefix "{{ url }}";
proxy_set_header X-Original-URI $request_uri;

# /data/configs/http/custom-maps.conf
map $http_x_forwarded_proto $proxy_x_forwarded_proto {
    default $http_x_forwarded_proto;
    '' $scheme;
}
map $proxy_x_forwarded_proto $proxy_x_forwarded_ssl {
  default off;
  https on;
}

These are the settings used in Open-Appsec, which refer to the headers

# /etc/nginx/conf.d/default.conf
map $http_x_forwarded_proto $proxy_x_forwarded_proto {
    default $http_x_forwarded_proto;
    '' $scheme;
}
map $http_x_forwarded_host $proxy_x_forwarded_host {
    default $http_x_forwarded_host;
    '' $http_host;
}
map $http_x_forwarded_port $proxy_x_forwarded_port {
    default $http_x_forwarded_port;
    '' $server_port;
}
map $server_port $host_port {
    default :$server_port;
    80 '';
    443 '';
}
map $http_upgrade $proxy_connection {
    default upgrade;
    '' $proxy_connection_noupgrade;
}
map $upstream_keepalive $proxy_connection_noupgrade {
    # Preserve nginx's default behavior (send "Connection: close").
    default close;
    # Use an empty string to cancel nginx's default behavior.
    true '';
}
map "" $upstream_keepalive {
    # The value here should not matter because it should always be overridden in
    # a location block (see the "location" template) for all requests where the
    # value actually matters.
    default false;
}
# Apply fix for very long server names
server_names_hash_bucket_size 128;
# Set appropriate X-Forwarded-Ssl header based on $proxy_x_forwarded_proto
map $proxy_x_forwarded_proto $proxy_x_forwarded_ssl {
    default off;
    https on;
}
gzip_types text/plain text/css application/javascript application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;
log_format vhost escape=default '$host $remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" "$upstream_addr"';
access_log off;

error_log /dev/stderr;
resolver 127.0.0.11;
# HTTP 1.1 support
proxy_http_version 1.1;
proxy_set_header Host $http_host;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $proxy_connection;
proxy_set_header X-Real-IP $http_x_real_ip;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $proxy_x_forwarded_host;
proxy_set_header X-Forwarded-Proto $proxy_x_forwarded_proto;
#proxy_set_header X-Forwarded-Protocol $proxy_x_forwarded_proto;
proxy_set_header X-Forwarded-Ssl $proxy_x_forwarded_ssl;
proxy_set_header X-Forwarded-Port $proxy_x_forwarded_port;
proxy_set_header X-Forwarded-Prefix $http_x_original_uri;
proxy_set_header X-Original-URI $http_x_original_uri;
# Mitigate httpoxy attack (see README for details)
proxy_set_header Proxy "";

upstream backend {
    server prd-nginx-proxy-proxy-1;
    keepalive 2;
}

server {
    server_name _;
    access_log /var/log/nginx/access.log vhost;
    http2 on;
    listen 80 default_server;

    location / {
        proxy_pass http://backend/;
        set $upstream_keepalive true;
    }

    location /proxy-health {
        access_log off;
        add_header 'Content-Type' 'application/json';
        return 200 '{"status":"UP"}';
    }
}

Docker Networks

4: prd-bunkerweb: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    inet 10.255.15.1/24 brd 10.255.15.255 scope global prd-bunkerweb
15: prd-open-appsec: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    inet 10.255.1.1/24 brd 10.255.1.255 scope global prd-open-appsec
7: prd-nginx-proxy: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    inet 10.255.10.1/24 brd 10.255.10.255 scope global prd-nginx-proxy

Headers

Request made to: http://subdomain.domain.com/test/

Client Firefox - Request Headers:

GET /test/ HTTP/1.1
Host: subdomain.domain.com
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:131.0) Gecko/20100101 Firefox/131.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/png,image/svg+xml,*/*;q=0.8
Accept-Language: pt-BR,en-US;q=0.7,en;q=0.3
Accept-Encoding: gzip, deflate
DNT: 1
Sec-GPC: 1
Connection: keep-alive
Upgrade-Insecure-Requests: 1
Priority: u=0, i
Pragma: no-cache
Cache-Control: no-cache

Stage 1 - Request received by OpenResty as if it were Proxy 1:

192.168.15.10 - - [28/Oct/2024:23:35:30 -0300] "GET /test/ HTTP/1.1" 200 30 "-"
"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:131.0) Gecko/20100101 Firefox/131.0" 0.000 

req_header:"accept-language=pt-BR,en-US;q=0.7,en;q=0.3
pragma=no-cache
accept-encoding=gzip, deflate
cache-control=no-cache
connection=keep-alive
dnt=1
host=subdomain.domain.com
sec-gpc=1
user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:131.0) Gecko/20100101 Firefox/131.0
upgrade-insecure-requests=1
accept=text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/png,image/svg+xml,*/*;q=0.8 priority=u=0, i "

resp_header:"content-type=application/octet-stream connection=keep-alive "
req_body:"-"
resp_body:"{ \x22result\x22: \x22OK\x22 }\x0A"

Stage 2 - Request received by OpenResty as if it were Open-Appsec:

10.255.1.2 - - [28/Oct/2024:23:32:24 -0300] "GET /test/ HTTP/1.1" 200 30 "-"
"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:131.0) Gecko/20100101 Firefox/131.0" 0.000

req_header:"x-forwarded-protocol=http
x-forwarded-ssl=off
x-forwarded-host=subdomain.domain.com
x-forwarded-port=80
x-forwarded-prefix=/
x-original-uri=/test/
connection=close
user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:131.0) Gecko/20100101 Firefox/131.0
accept=text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/png,image/svg+xml,*/*;q=0.8
accept-language=pt-BR,en-US;q=0.7,en;q=0.3
accept-encoding=gzip, deflate
dnt=1
sec-gpc=1
upgrade-insecure-requests=1
priority=u=0, i
pragma=no-cache
host=subdomain.domain.com
x-forwarded-for=192.168.15.10
cache-control=no-cache
x-real-ip=192.168.15.10
x-forwarded-proto=http "

resp_header:"content-type=application/octet-stream connection=close "
req_body:"-"
resp_body:"{ \x22result\x22: \x22OK\x22 }\x0A"

Stage 3 - Request received by OpenResty as if it were the Server:

10.255.10.2 - - [28/Oct/2024:23:29:53 -0300] "GET /test/ HTTP/1.1" 200 30 "-"
"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:131.0) Gecko/20100101 Firefox/131.0" 0.000

req_header:"host=subdomain.domain.com
x-real-ip=192.168.15.10
x-forwarded-for=192.168.15.10, 10.255.1.2
x-forwarded-host=subdomain.domain.com
x-forwarded-proto=http
x-forwarded-ssl=off
x-forwarded-port=80
x-forwarded-prefix=/test/
x-original-uri=/test/
x-forwarded-protocol=http
user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:131.0) Gecko/20100101 Firefox/131.0
accept=text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/png,image/svg+xml,*/*;q=0.8
accept-language=pt-BR,en-US;q=0.7,en;q=0.3
accept-encoding=gzip, deflate
dnt=1
sec-gpc=1
upgrade-insecure-requests=1
priority=u=0, i
pragma=no-cache
cache-control=no-cache "

resp_header:"content-type=application/octet-stream connection=keep-alive "
req_body:"-"
resp_body:"{ \x22result\x22: \x22OK\x22 }\x0A"

X-Forwarded-For Source Identity

I was going to post the policy here, but I didn't realize it was so big. So I hope the screenshot of the SaaS option is enough.

I hope this information is what you need. If there's anything missing, let me know. :)

[chkp-sergeysh]

Hello @alnviana,

Thank you for your quick replies and the detailed information about the issue you're experiencing. We’re having some difficulty reproducing the problem on our end, so a brief remote session would be very helpful to diagnose it directly.

Could you let me know a few times that work best for you in the coming days? I’ll do my best to accommodate.

For privacy, please feel free to email me any relevant details or setup configurations at [email protected]. This will help us prepare for the session and ensure we make the most of our time together.

Looking forward to resolving this with you!

Best regards,
Sergey
open-appsec Developer Team

[chkp-sergeysh]

Hello @alnviana,

Thank you again for your willingness to arrange a remote session. I wanted to let you know that I was able to successfully reproduce the issue on my end, so there’s no need for the session after all.

We’ll be working on a fix and will update you here once it’s resolved.

Best regards,
Sergey
open-appsec Developer Team

[rr404]

this should fix the issue : #1181

[he2ss]

Hi @alnviana,

Can you update your hub index and upgrade the openappsec collection and check if it's working now please.

[alnviana]

Crowdsec updated to version 1.6.4, Hub updated.
SQL Injection attempt blocked by Crowdsec as expected. Thanks for your attention! :D