forwarding client_ip with proxy and logs config

[doums]

Hi, not a bug but rather I have a few questions. First off, this is my caddy config:

{
	email [email protected]
	dynamic_dns […]
	crowdsec […]
	order crowdsec first

	servers {
		metrics
	}

	# default logger
	log default {
		output stdout
		format console
	}

	# write http access logs to 2 different outputs
	# see https://caddyserver.com/docs/caddyfile/directives/log#examples

	# HTTP logger - console
	log access-console {
		include http.log.access.http-logger
		output stdout
		format console
	}

	# HTTP logger - file JSON
	log access-json {
		include http.log.access.http-logger
		output file /log/access.log
		format json
	}

	# l4 logger
	log l4 {
		include layer4.l4
		output stdout
		format console
	}

	layer4 {
                 log l4 # tried this

		0.0.0.0:8054 {
                         log l4 # tried this

			@foo tls sni sub.mydomain.com
			route @foo {
				tls
				proxy service:1234
			}
		}
	}
}

*.mydomain.com {
	tls  […]

	log http-logger

	[…] some reverse proxy blocks
}

1. How to configure access logs correctly for layer4? Especially to logs the requests going through the tls matcher (:8054)? Right now I don't get any logs but in the other side of the proxy (service:1234), I can see the requests received as expected.
2. Is there a way to forward the initial client_ip to the proxied service in layer4?

[francislavoie]

There's no such thing as access logs for l4 right now.

Is there a way to forward the initial client_ip to the proxied service in layer4?

You can use PROXY procotol to do that.

[mholt]

Well, this module does log connections (it shows you how many bytes are read/written), but they're not like HTTP access logs where you can see the decoded request. We can improve logging for sure though.

[doums]

The issue is I don't even see those connections log you mentioned. Are they emitted at a lower level of info?
Also, I'm not sure where to put the log directive inside the layer4 app?

[mholt]

Oh, they might be at DEBUG level come to think of it. You don't need to enable them (because they're at a more verbose level.)

We can rework this I suppose, I just whipped up the logging quickly.

[doums]

Ok cool. Yea the 'logic' would be to have some trace logged (at INFO level so) whenever some bytes hit the tls handler. So that we're able to see the activities at least.

[doums]

You can use PROXY procotol to do that.

@francislavoie thx for the suggestion. But I'm still failing to get this working.
Also note that the target service is a blocky instance. And the purpose is to tap into its dns endpoint, raw no tls, since tls is terminated at l4 proxy level.

I tried to change the handler to this but without success

                         route @foo {
				tls
				proxy {
					proxy_protocol v1 # also tried v2
					upstream blocky:53
				}
			}

[francislavoie]

Your upstream would need to be ready to handle PROXY protocol as well. Basically it prepends some bytes to the start of the connection which includes the original client IP. Looks like it doesn't support it yet 0xERR0R/blocky#320

[alexander-potemkin]

Oh, they might be at DEBUG level come to think of it. You don't need to enable them (because they're at a more verbose level.)

We can rework this I suppose, I just whipped up the logging quickly.

Thank you! I'm trying to get at least some logs as well - to troubleshoot my config. Could you please, advice how to enable logging / debugging?

My current Caddyfile looks like that:

{
    layer4 {
        0.0.0.0:443 {
            @s5 socks5
            route @s5 {
                proxy IP:443
            }
            @secure tls
            route @secure {
                proxy some_site:443
            }
        }
    }
}

I've tried all kinds of log directive, and searched for an answer in the source code, but it didn't work...

[mholt]

Put the word debug in your global options: https://caddyserver.com/docs/caddyfile/options#debug