Caddy panics when binding systemd socket to http/s server #6833

chovanecadam · 2025-02-09T22:21:03Z

Description

SystemD creates file descriptors fd/3 (80/tcp), fd/4 (443/tcp), and fd/5 (443/udp) and passes them to Caddy. The following Caddyfile is functional and works as expected, that is Caddy listens on port 80 for HTTP/1.1, on port 443/tcp for HTTP/1.1 and HTTP/2.0, and on port 443/udp for HTTP3.0.

{
        default_sni example.com
        debug
        log stderr

        admin fd/6
}

http://example.com {
        bind fd/3 {
                protocols h1
        }
        respond "hello world"
}

https://example.com {
        bind fd/4 {
                protocols h1 h2
        }

        bind fdgram/5 {
                protocols h3
        }
        respond "hello world"
}

However, this forces me to repeat the bind directive for every server individually, which is not ideal. I expected the following Caddyfile to work, that is I expected that the bind directives in catch-all http:// block will be applied to all other blocks, thus not needing to repeat the directive over and over again. However, this results in Caddy exiting with a panic.

{
        default_sni example.com
        debug
        log stderr

        admin fd/6
}

http:// {
        bind fd/3 {
                protocols h1
        }
}

https:// {
        bind fd/4 {
                protocols h1 h2
        }

        bind fdgram/5 {
                protocols h3
        }
}

http://example.com {
        respond "hello world"
}

https://example.com {
        respond "hello world"
}

caddy.socket:

[Socket]
ListenStream=[::]:80
ListenStream=[::]:443
ListenDatagram=[::]:443

# %t: Runtime directory root
ListenStream=%t/caddy.sock
SocketMode=0600

[Install]
WantedBy=sockets.target

caddy.container:

[Unit]
Description=Caddy proxy

BindsTo=caddy.socket
After=caddy.socket

[Service]
Restart=always
TimeoutStartSec=900

[Install]
WantedBy=multi-user.target default.target

[Container]
ContainerName=caddy
Image=docker.io/caddy:2.9

Volume=../assets/caddy/Caddyfile:/etc/caddy/Caddyfile:U,ro
Volume=caddy_data.volume:/data
Volume=caddy_config.volume:/config:U,ro
Network=caddy.network
Notify=true

log

Feb 09 21:56:28 my-server caddy[171142]: {"level":"info","ts":1739138188.474818,"msg":"using config from file","file":"/etc/caddy/Caddyfile"}
Feb 09 21:56:28 my-server caddy[171142]: {"level":"info","ts":1739138188.4776082,"msg":"adapted config to JSON","adapter":"caddyfile"}
Feb 09 21:56:28 my-server caddy[171142]: {"level":"info","ts":1739138188.4787242,"logger":"admin","msg":"admin endpoint started","address":"fd/6","enforce_origin":false,"origins":["","//127.0.0.1","//::1"]}
Feb 09 21:56:28 my-server caddy[171142]: {"level":"info","ts":1739138188.47906,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
Feb 09 21:56:28 my-server caddy[171142]: {"level":"info","ts":1739138188.479078,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
Feb 09 21:56:28 my-server caddy[171142]: {"level":"info","ts":1739138188.479086,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv2"}
Feb 09 21:56:28 my-server caddy[171142]: {"level":"warn","ts":1739138188.479102,"logger":"http.auto_https","msg":"server is listening only on the HTTP port, so no automatic HTTPS will be applied to this server","server_name":"srv3","http_port":80}
Feb 09 21:56:28 my-server caddy[171142]: {"level":"debug","ts":1739138188.4791498,"logger":"http.auto_https","msg":"adjusted config","tls":{"automation":{"policies":[{}]}},"http":{"servers":{"remaining_auto_https_redirects":{"listen":[":80"],"routes":[{},{}]},"srv0":{"listen":[":443"],"routes":[{"handle":[{"handler":"subroute","routes":[{"handle":[{"body":"hello world","handler":"static_response"}]}]}],"terminal":true}],"tls_connection_policies":[{}],"automatic_https":{}},"srv1":{"listen":["fd/3"],"routes":[{},{}],"automatic_https":{},"listen_protocols":[["h1"]]},"srv2":{"listen":["fd/4","fdgram/5"],"routes":[{},{}],"tls_connection_policies":[{"default_sni":"www.example.com"}],"automatic_https":{},"listen_protocols":[["h1","h2"],["h3"]]},"srv3":{"automatic_https":{"disable":true}}}}}
Feb 09 21:56:28 my-server caddy[171142]: {"level":"info","ts":1739138188.4797654,"logger":"http.log","msg":"server running","name":"srv3","protocols":["h1","h2","h3"]}
Feb 09 21:56:28 my-server caddy[171142]: {"level":"info","ts":1739138188.4800358,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc000537080"}
Feb 09 21:56:28 my-server caddy[171142]: {"level":"debug","ts":1739138188.480103,"logger":"http","msg":"starting server loop","address":"[::]:80","tls":false,"http3":false}
Feb 09 21:56:28 my-server caddy[171142]: {"level":"warn","ts":1739138188.4801283,"logger":"http","msg":"HTTP/2 skipped because it requires TLS","network":"tcp","addr":":80"}
Feb 09 21:56:28 my-server caddy[171142]: {"level":"warn","ts":1739138188.4801338,"logger":"http","msg":"HTTP/3 skipped because it requires TLS","network":"tcp","addr":":80"}
Feb 09 21:56:28 my-server caddy[171142]: {"level":"info","ts":1739138188.4801414,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
Feb 09 21:56:28 my-server caddy[171142]: {"level":"debug","ts":1739138188.4801967,"logger":"http","msg":"starting server loop","address":"[::]:443","tls":true,"http3":false}
Feb 09 21:56:28 my-server caddy[171142]: {"level":"info","ts":1739138188.4802055,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
Feb 09 21:56:28 my-server caddy[171142]: {"level":"info","ts":1739138188.4806733,"msg":"failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 7168 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details."}
Feb 09 21:56:28 my-server caddy[171142]: {"level":"info","ts":1739138188.4808042,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
Feb 09 21:56:28 my-server caddy[171142]: {"level":"debug","ts":1739138188.480846,"logger":"http","msg":"starting server loop","address":"[::]:80","tls":false,"http3":false}
Feb 09 21:56:28 my-server caddy[171142]: {"level":"info","ts":1739138188.4808524,"logger":"http.log","msg":"server running","name":"srv1","protocols":["h1","h2","h3"]}
Feb 09 21:56:28 my-server caddy[171142]: {"level":"debug","ts":1739138188.4808836,"logger":"http","msg":"starting server loop","address":"[::]:443","tls":true,"http3":false}
Feb 09 21:56:28 my-server caddy[171142]: {"level":"info","ts":1739138188.48089,"logger":"http","msg":"enabling HTTP/3 listener","addr":"5"}
Feb 09 21:56:28 my-server systemd[1]: Started Caddy proxy.
Feb 09 21:56:28 my-server caddy[171142]: panic: connection already exists
Feb 09 21:56:28 my-server caddy[171142]: goroutine 1 [running]:                                                                                                             21:56:28 [24/1818]
Feb 09 21:56:28 my-server caddy[171142]: github.com/quic-go/quic-go.(*connMultiplexer).AddConn(0xc0002c9000, {0x76efd0459038?, 0xc000301430?})
Feb 09 21:56:28 my-server caddy[171142]:         github.com/quic-go/[email protected]/multiplexer.go:59 +0x165
Feb 09 21:56:28 my-server caddy[171142]: github.com/quic-go/quic-go.(*Transport).init.func1()
Feb 09 21:56:28 my-server caddy[171142]:         github.com/quic-go/[email protected]/transport.go:266 +0x3f0
Feb 09 21:56:28 my-server caddy[171142]: sync.(*Once).doSlow(0x10?, 0x10?)
Feb 09 21:56:28 my-server caddy[171142]:         sync/once.go:76 +0xb4
Feb 09 21:56:28 my-server caddy[171142]: sync.(*Once).Do(...)
Feb 09 21:56:28 my-server caddy[171142]:         sync/once.go:67
Feb 09 21:56:28 my-server caddy[171142]: github.com/quic-go/quic-go.(*Transport).init(0xc000527b00, 0x90?)
Feb 09 21:56:28 my-server caddy[171142]:         github.com/quic-go/[email protected]/transport.go:225 +0x45
Feb 09 21:56:28 my-server caddy[171142]: github.com/quic-go/quic-go.(*Transport).createServer(0xc000527b00, 0xc0005b9dc0, 0xc00059ec60, 0x1)
Feb 09 21:56:28 my-server caddy[171142]:         github.com/quic-go/[email protected]/transport.go:175 +0x11d
Feb 09 21:56:28 my-server caddy[171142]: github.com/quic-go/quic-go.(*Transport).ListenEarly(0xc0000addd0?, 0xc0005b9880?, 0x0?)
Feb 09 21:56:28 my-server caddy[171142]:         github.com/quic-go/[email protected]/transport.go:153 +0x18
Feb 09 21:56:28 my-server caddy[171142]: github.com/caddyserver/caddy/v2.NetworkAddress.ListenQUIC.func1()
Feb 09 21:56:28 my-server caddy[171142]:         github.com/caddyserver/caddy/[email protected]/listeners.go:465 +0x3d2
Feb 09 21:56:28 my-server caddy[171142]: github.com/caddyserver/caddy/v2.(*UsagePool).LoadOrNew(0x2a14400, {0x1613640, 0xc000771410}, 0xc000858b78)
Feb 09 21:56:28 my-server caddy[171142]:         github.com/caddyserver/caddy/[email protected]/usagepool.go:93 +0x185
Feb 09 21:56:28 my-server caddy[171142]: github.com/caddyserver/caddy/v2.NetworkAddress.ListenQUIC({{0x18f8525, 0x6}, {0xc00032db2f, 0x1}, 0x0, 0x0}, {0x1da9ec8, 0xc000380460}, 0x0, {0x0, ...}, ...)
Feb 09 21:56:28 my-server caddy[171142]:         github.com/caddyserver/caddy/[email protected]/listeners.go:436 +0x21d
Feb 09 21:56:28 my-server caddy[171142]: github.com/caddyserver/caddy/v2/modules/caddyhttp.(*Server).serveHTTP3(0xc000594388, {{0x18f8525, 0x6}, {0xc00032db2f, 0x1}, 0x0, 0x0}, 0xc0005b9880)
Feb 09 21:56:28 my-server caddy[171142]:         github.com/caddyserver/caddy/[email protected]/modules/caddyhttp/server.go:611 +0x15d
Feb 09 21:56:28 my-server caddy[171142]: github.com/caddyserver/caddy/v2/modules/caddyhttp.(*App).Start(0xc000591860)
Feb 09 21:56:28 my-server caddy[171142]:         github.com/caddyserver/caddy/[email protected]/modules/caddyhttp/app.go:619 +0x2005
Feb 09 21:56:28 my-server caddy[171142]: github.com/caddyserver/caddy/v2.run.func1({{0x1daa0c0, 0xc00053d1d0}, 0xc00001c2d0, 0xc000451b00, {0x0, 0x0, 0x0}, {0x0, 0x0, 0x0}, ...})
Feb 09 21:56:28 my-server caddy[171142]:         github.com/caddyserver/caddy/[email protected]/caddy.go:422 +0x111
Feb 09 21:56:28 my-server caddy[171142]: github.com/caddyserver/caddy/v2.run(0xc?, 0x1)
Feb 09 21:56:28 my-server caddy[171142]:         github.com/caddyserver/caddy/[email protected]/caddy.go:438 +0x158
Feb 09 21:56:28 my-server caddy[171142]: github.com/caddyserver/caddy/v2.unsyncedDecodeAndRun({0xc000702800, 0x200, 0x200}, 0x1)
Feb 09 21:56:28 my-server caddy[171142]:         github.com/caddyserver/caddy/[email protected]/caddy.go:343 +0x145
Feb 09 21:56:28 my-server caddy[171142]: github.com/caddyserver/caddy/v2.changeConfig({0x18f47c0, 0x4}, {0x18fb5ee, 0x7}, {0xc000702600, 0x200, 0x200}, {0x0, 0x0}, 0x1)
Feb 09 21:56:28 my-server caddy[171142]:         github.com/caddyserver/caddy/[email protected]/caddy.go:234 +0x6b6
Feb 09 21:56:28 my-server caddy[171142]: github.com/caddyserver/caddy/v2.Load({0xc000702600, 0x200, 0x200}, 0x1)
Feb 09 21:56:28 my-server caddy[171142]:         github.com/caddyserver/caddy/[email protected]/caddy.go:133 +0x22d
Feb 09 21:56:28 my-server caddy[171142]: github.com/caddyserver/caddy/v2/cmd.cmdRun({0x0?})
Feb 09 21:56:28 my-server caddy[171142]:         github.com/caddyserver/caddy/[email protected]/cmd/commandfuncs.go:235 +0x831
Feb 09 21:56:28 my-server caddy[171142]: github.com/caddyserver/caddy/v2/cmd.init.1.func2.WrapCommandFuncForCobra.1(0xc0006c6908, {0x18f481c?, 0x4?, 0x18f47f0?})
Feb 09 21:56:28 my-server caddy[171142]:         github.com/caddyserver/caddy/[email protected]/cmd/cobra.go:141 +0x2f
Feb 09 21:56:28 my-server caddy[171142]: github.com/spf13/cobra.(*Command).execute(0xc0006c6908, {0xc000206e00, 0x4, 0x4})
Feb 09 21:56:28 my-server caddy[171142]:         github.com/spf13/[email protected]/command.go:985 +0xaaa
Feb 09 21:56:28 my-server caddy[171142]: github.com/spf13/cobra.(*Command).ExecuteC(0xc0006c6308)
Feb 09 21:56:28 my-server caddy[171142]:         github.com/spf13/[email protected]/command.go:1117 +0x3ff
Feb 09 21:56:28 my-server caddy[171142]: github.com/spf13/cobra.(*Command).Execute(...)
Feb 09 21:56:28 my-server caddy[171142]:         github.com/spf13/[email protected]/command.go:1041
Feb 09 21:56:28 my-server caddy[171142]: github.com/caddyserver/caddy/v2/cmd.Main()
Feb 09 21:56:28 my-server caddy[171142]:         github.com/caddyserver/caddy/[email protected]/cmd/main.go:75 +0x1dd
Feb 09 21:56:28 my-server caddy[171142]: main.main()
Feb 09 21:56:28 my-server caddy[171142]:         caddy/main.go:39 +0xf

The text was updated successfully, but these errors were encountered:

mholt · 2025-02-10T15:40:40Z

Hmm! I am not very well versed with the systemd sockets, but @MayCXC may have a clue on this one.

MayCXC · 2025-02-14T10:14:08Z

@chovanecadam if you need a quick workaround, see if default_bind works for your configuration: #6573 (comment)

after some digging, it looks like quic-go/quic-go#3727 is relevant, and the update in quic-go/[email protected] is already required by caddyserver/caddy/v2@master. are you able to test your configuration with the latest caddy compiled from source, and see if it works as expected?

chovanecadam · 2025-02-14T15:32:22Z

I compiled from source with CGO_ENABLED=0 go build -tags nobadger -trimpath -ldflags="-w -s" -v and copied the static binary to my docker container.

Here's my Caddyfile. The rest of the configuration (all systemd sockets, etc.) are the same.

{
        default_sni example.com
        debug
        log stderr

        admin fd/6
        auto_https disable_redirects
}

http:// {
        bind fd/3 {
                protocols h1
        }
}

https:// {
        bind fd/4 {
                protocols h1 h2
        }

        bind fdgram/5 {
                protocols h3
        }
}

http://example.com {
        respond "hello world"
}

https://example.com {
        respond "hello world"
}

Caddy responds with status code 200 and Content-Length 0 to any request to http or https. The expected behavior is that caddy responds with "hello world".

If I replace respond directives with file_server and root, caddy always responds with 200 and content-length 0. The expected behavior is that Caddy servers the files in root.

If I remove auto_https disable_redirects, then curl http://example.com -Lv redirects to port 0:

HTTP/1.1 308 Permanent Redirect
Connection: close
Location: https://example.com:0/
Server: Caddy
Date: Fri, 14 Feb 2025 14:56:54 GMT
Content-Length: 0

It does not make a difference if I specify only one example.com server without a protocol, or specify http://example.com and https://example.com separately.

chovanecadam · 2025-02-14T15:38:18Z

@MayCXC A proper workaround is to specify bind directives explicitly for every server. The default_bind directive does not work for Caddyfiles that serve both HTTP and HTTPS. But I guess that's expected, because Caddy sees HTTPS server, configures all sockets to HTTPS, then sees the HTTP server, tries to configure the sockets to HTTP, and fails.

Writing this just to clear things up that default_bind is not a workaround in this case.

{
        default_sni example.com
        debug
        log stderr

        admin fd/6
        auto_https disable_redirects

        default_bind fd/3 {
                protocols h1
        }

        default_bind fd/4 {
                protocols h1 h2
        }

        default_bind fdgram/5 {
                protocols h3
        }
}

https://example.com {
        respond "hello from https"
}

http://example.com {
        respond "hello from http"
}

# vim: noexpandtab

Error: adapting config using caddyfile: server listening on [fd/3 fd/4 fdgram/5] is configured for HTTPS and cannot natively multiplex HTTP and HTTPS: http://example.com

MartinSchmidt · 2025-02-15T12:09:49Z

I know this might not be exactly what you are looking for but I got my setup working like this:

{
    acme_dns cloudflare {env.CLOUDFLARE_API_TOKEN}
    default_bind fd/4 {
        protocols h1 h2
    }
}

http:// {
    bind fd/3 {
        protocols h1
    }
}

example.com {
    log
    reverse_proxy whoami:80
}

Seems to for me, I was also looking for how to get around repeating the bind section, I hope it helps :)

chovanecadam · 2025-02-16T14:27:57Z

@MartinSchmidt that does not work for me. Accessing http://example.com results in redirection to https://example.com:0 which fails because of the wrong port. However, setting auto_https disable_redirects with explicit redir https://{host}{uri} works very well. Thanks for the hint!

Here's my working config:

{
        default_sni example.com
        auto_https disable_redirects

        default_bind fd/4 {
                protocols h1 h2
        }

        default_bind fdgram/5 {
                protocols h3
        }
}

http:// {
        bind fd/3 {
                protocols h1
        }

        redir https://{host}{uri}
}

example.com {
        respond "hello from example.com"
}

MartinSchmidt · 2025-02-16T18:13:00Z

@chovanecadam weird with the redirect to port 0, works fine here :/
I am using the pre-build caddy:2.9.1 image.

$ curl -vv http://example.com
* Host example.com:80 was resolved.
* IPv6: (none)
* IPv4: 192.168.12.95
*   Trying 192.168.12.95:80...
* Connected to example.com (192.168.12.95) port 80
> GET / HTTP/1.1
> Host: example.com
> User-Agent: curl/8.9.1
> Accept: */*
> 
* Request completely sent off
< HTTP/1.1 308 Permanent Redirect
< Connection: close
< Location: https://example.com/
< Server: Caddy
< Date: Sun, 16 Feb 2025 18:08:27 GMT
< Content-Length: 0
< 
* shutting down connection #0

But I am glad that the hint helped :)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Caddy panics when binding systemd socket to http/s server #6833

Caddy panics when binding systemd socket to http/s server #6833

chovanecadam commented Feb 9, 2025

mholt commented Feb 10, 2025

MayCXC commented Feb 14, 2025 •

edited

Loading

chovanecadam commented Feb 14, 2025 •

edited

Loading

chovanecadam commented Feb 14, 2025 •

edited

Loading

MartinSchmidt commented Feb 15, 2025

chovanecadam commented Feb 16, 2025 •

edited

Loading

MartinSchmidt commented Feb 16, 2025

Caddy panics when binding systemd socket to http/s server #6833

Caddy panics when binding systemd socket to http/s server #6833

Comments

chovanecadam commented Feb 9, 2025

mholt commented Feb 10, 2025

MayCXC commented Feb 14, 2025 • edited Loading

chovanecadam commented Feb 14, 2025 • edited Loading

chovanecadam commented Feb 14, 2025 • edited Loading

MartinSchmidt commented Feb 15, 2025

chovanecadam commented Feb 16, 2025 • edited Loading

MartinSchmidt commented Feb 16, 2025

MayCXC commented Feb 14, 2025 •

edited

Loading

chovanecadam commented Feb 14, 2025 •

edited

Loading

chovanecadam commented Feb 14, 2025 •

edited

Loading

chovanecadam commented Feb 16, 2025 •

edited

Loading