Caddy config not reloaded

[radokristof]

Most of the time, the config is not reloaded automatically, I need to restart the caddy-docker-proxy container to reload the config.
For example, I had an already added domain, I added basicauth to that domain via docker-compose labels. The change was not visible, until I restarted the proxy container.

Where should I look, what is causing this issue?

[francislavoie]

You need to run docker compose up -d to have Docker recreate the containers that had labels changed. Labels are set at container creation time, they aren't changed if you just save your compose.yml.

If that's not the problem, then please share more about your setup and the steps you take. Can't help without more information, we're forced to make assumptions.

[radokristof]

Sorry, yes I was a vague about my setup. I actually do recreate the containers after changing the labels. I just can't see any logs/sign that the proxy even notices the change in the labels.
Caddy version: v2.7.6
My Dockerfile:

ARG CADDY_VERSION=latest
FROM caddy:builder AS builder

RUN xcaddy build \
    --with github.com/lucaslorentz/caddy-docker-proxy/v2 \
    --with github.com/caddy-dns/cloudflare \
    --with github.com/greenpau/caddy-security \
    --with github.com/WeidiDeng/caddy-cloudflare-ip

FROM caddy:${CADDY_VERSION}

COPY --from=builder /usr/bin/caddy /usr/bin/caddy

CMD ["caddy", "docker-proxy"]

An example docker-compose.yml:

services:
  frontail:
    container_name: frontail
    build: .
    command: /log/logger/logfile --ui-highlight --ui-highlight-preset /preset/basic.json --theme basic -l 5000 -n 500 --disable-usage-stats
    volumes:
      - ./preset:/preset
    restart: always
    networks:
      - default
      - caddy
    labels:
      caddy: test.example.com
      caddy.reverse_proxy: "{{ upstreams 9001 }}"
      caddy.basicauth.user: ${AUTH_KEY}
    env_file:
      - ./.env
    environment:
      TZ: Europe/Budapest

networks:
  caddy:
    name: caddy
    external: true

[radokristof]

Here is my redacted caddy-config, which is generated upon container restart:

{
	order authenticate before respond
	order authorize before reverse_proxy
	acme_dns cloudflare REDACTED_API_KEY
	email [email protected]
	log access {
		format json
		include http.log.access http.handlers.reverse_proxy
		level DEBUG
		output file /var/log/caddy/access.log {
			roll_keep 5
			roll_keep_for 2160h
			roll_size 1gb
		}
	}
	log default {
		exclude http.log.access
		format json
		level INFO
		output file /var/log/caddy/runtime.log {
			roll_keep 3
			roll_keep_for 720h
			roll_size 500mb
		}
	}
	security {
		authentication portal example_sso {
			crypto default token lifetime 3600
			enable identity provider generic
			cookie domain example.hu
			ui {
				links {
					Profile /auth/profile icon "las la-user-cog"
					"Who Am I" /auth/whoami icon "las la-user"
				}
				meta author "Example"
				meta description "Authentication Portal"
				meta title "Web Authentication Portal"
				static_asset assets/images/example.png images/png /assets/img/example.png
				static_asset assets/images/example_logo.png images/png /assets/img/example_logo.png
				template login /assets/templates/example/login.template
				template portal /assets/templates/example/portal.template
				template whoami /assets/templates/example/whoami.template
			}
			transform user {
				action add role authp/admin
				match realm generic
			}
		}
		authorization policy logger_policy {
			set auth url https://auth.example.com/auth
			allow roles authp/admin
		}
		oauth identity provider generic {
			realm generic
			driver generic
			client_id caddy_REDACTED
			client_secret REDACTED_35cf5ab1e1c823d44d86ade590562fd4
			scopes openid email profile
			base_auth_url https://sso.example.com
			metadata_url https://sso.example.com/.well-known/openid-configuration
		}
		authentication portal example_sso_cloud {
			crypto default token lifetime 3600
			enable identity provider generic
			cookie domain example.cloud
			ui {
				links {
					Profile /auth/profile icon "las la-user-cog"
					"Who Am I" /auth/whoami icon "las la-user"
				}
				meta author "example"
				meta description "Authentication Portal"
				meta title "Web Authentication Portal"
				static_asset assets/images/example.png images/png /assets/img/example.png
				static_asset assets/images/example_logo.png images/png /assets/img/example_logo.png
				template login /assets/templates/example/login.template
				template portal /assets/templates/example/portal.template
				template whoami /assets/templates/example/whoami.template
			}
			transform user {
				action add role authp/admin
				match realm generic
			}
		}
	}
	servers {
		trusted_proxies cloudflare {
			interval 12h
			timeout 15s
		}
	}
}
:2019 {
	metrics
}
auth.example.cloud {
	route /auth* {
		authenticate with example_sso_cloud
	}
	route /* {
		redir https://auth.example.cloud/auth/ 302
	}
}
auth.example.hu {
	route /auth* {
		authenticate with example_sso
	}
	route /* {
		redir https://auth.example.hu/auth/ 302
	}
}
automate.example.cloud {
	reverse_proxy 192.168.5.10:8080
}
log.example.cloud {
	authorize with logger_policy
	reverse_proxy 192.168.5.2:9001
}

dev.phpadmin.example.cloud {
	reverse_proxy 192.168.5.8:80
}
docs.example.cloud {
	reverse_proxy 192.168.5.13:3000
}
drone.example.hu {
	reverse_proxy 192.168.5.9:80
}
git.example.cloud {
	reverse_proxy 192.168.5.22:80
}

graphs.example.cloud {
	reverse_proxy 192.168.5.21:3000
}
influx.example.cloud {
	reverse_proxy 192.168.5.18:8086
}
log.example.cloud {
	authorize with logger_policy
	reverse_proxy 192.168.5.5:9001
}
example.app, www.example.app {
	redir * https://example.hu/
}
example.cloud, erp.example.cloud {
	reverse_proxy 192.168.5.12:80 {
		header_down Strict-Transport-Security max-age=15552000;
	}
}

[radokristof]

@francislavoie Do you have any idea why it is not working like this? I have tried rebuilding, removing parts of config, etc... still no luck.

[francislavoie]

Like I said, you need to recreate the containers to have label changes take effect. You can't just edit your docker-compose.yml and expect it to have an effect.

[radokristof]

And as I said before, I recreate the containers all the time.
Usually my workflow which produces this error:

1. Container updated, pulled from repo: docker compose pull
2. Recreate container to update to its new version: docker compose up -d
3. Container's IP address changes in the caddy network
4. caddy-docker-proxy still has the old IP address, config is not updated.

[7-zete-7]

@francislavoie, I encountered a similar problem. In my case the configuration changes, but Caddy continues to access old addresses.
In my case, this only happens if Caddy is behind another proxy.

docker compose version

Docker Compose version v2.30.3-desktop.1

docker version

Client:
 Version:           27.3.1
 API version:       1.47
 Go version:        go1.22.7
 Git commit:        ce12230
 Built:             Fri Sep 20 11:39:44 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Desktop  ()
 Engine:
  Version:          27.3.1
  API version:      1.47 (minimum version 1.24)
  Go version:       go1.22.7
  Git commit:       41ca978
  Built:            Fri Sep 20 11:41:11 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.7.21
  GitCommit:        472731909fa34bd7bc9c087e4c27943f9835f111
 runc:
  Version:          1.1.13
  GitCommit:        v1.1.13-0-g58aa920
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

compose.yaml

services:
  traefik:
    image: traefik:v3.1
    networks:
      - proxy
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
    command:
      - --entrypoints.web.address=:80

      - --providers.docker
      - --providers.docker.exposedbydefault=false

    ports:
      - "80:80/tcp"

  caddy:
    image: lucaslorentz/caddy-docker-proxy:ci-alpine
    networks:
      - proxy
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
    labels:
      # region Caddy
      # global configs
      caddy_0.order: respond after reverse_proxy
      caddy_0.debug:

      # `:80` servers configs
      caddy_1.servers: :80
      caddy_1.servers.protocols: h1 h2c
      caddy_1.servers.trusted_proxies: static private_ranges

      # `localhost:80` address configs
      caddy_2: localhost:80
      caddy_2.import: app

      # dynamic `localhost:80` address configs
      caddy_3: (app)
      caddy_3.respond: "503"
      # endregion Caddy

      # region Traefik
      traefik.enable: "true"

      traefik.http.services.caddy.loadbalancer.server.port: "80"
      traefik.http.services.caddy.loadbalancer.server.scheme: h2c

      traefik.http.routers.caddy.entrypoints: web
      traefik.http.routers.caddy.rule: Host(`localhost`)
      traefik.http.routers.caddy.service: caddy@docker
      # endregion Traefik

  app:
    image: traefik/whoami:latest
    scale: 2
    networks:
      - proxy
    labels:
      caddy_ingress_network: "${COMPOSE_PROJECT_NAME}_proxy"

      caddy: (app)
      caddy.reverse_proxy: "{{upstreams 80}}"

networks:
  proxy:

After launching the project

docker compose up --wait

, everything works normally.

At this moment the containers are running:

• ...-app-1
• ...-app-2
• ...-caddy-1
• ...-traefik-1

Next, we increase the number of containers of the app service to 4.

docker compose up --wait --scale app=4

Now we have the following containers:

• ...-app-1
• ...-app-2
• ...-app-3
• ...-app-4
• ...-caddy-1
• ...-traefik-1

Contents of the /config/caddy/Caddyfile.autosave file at this point

{
	debug
	order respond after reverse_proxy
	servers :80 {
		protocols h1 h2c
		trusted_proxies static private_ranges
	}
}
(app) {
	respond 503
	reverse_proxy 172.18.0.6:80 172.18.0.7:80 172.18.0.2:80 172.18.0.5:80
}
localhost:80 {
	import app
}

But requests come only to containers ...-app-1 and ...app-2.

Now let's stop the ...-app-1 and ...app-2 containers.

docker compose stop ...-app-1
docker compose stop ...-app-2

Now Caddy starts responding with a 502 HTTP code.

Contents of the /config/caddy/Caddyfile.autosave file at this point

{
	debug
	order respond after reverse_proxy
	servers :80 {
		protocols h1 h2c
		trusted_proxies static private_ranges
	}
}
(app) {
	respond 503
	reverse_proxy 172.18.0.6:80 172.18.0.7:80
}
localhost:80 {
	import app
}

Caddy logs at the moment of error

caddy-1  | {"level":"debug","ts":1733335948.490328,"logger":"...","msg":"Skipping default Caddyfile because no path is set"}
caddy-1  | {"level":"debug","ts":1733335948.4903579,"logger":"...","msg":"Skipping swarm config caddyfiles because swarm is not available"}
caddy-1  | {"level":"debug","ts":1733335948.5433204,"logger":"...","msg":"Skipping swarm services because swarm is not available"}
caddy-1  | {"level":"debug","ts":1733335958.583889,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"172.18.0.5:80","total_upstreams":2}
caddy-1  | {"level":"debug","ts":1733335961.586447,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"172.18.0.5:80","duration":3.002495039,"request":{"remote_ip":"172.18.0.3","remote_port":"50786","client_ip":"172.18.0.1","proto":"HTTP/2.0","method":"GET","host":"localhost","uri":"/","headers":{"X-Forwarded-Server":["7cf08c882dba"],"X-Forwarded-Host":["localhost"],"X-Forwarded-For":["172.18.0.1, 172.18.0.3"],"X-Forwarded-Port":["80"],"X-Real-Ip":["172.18.0.1"],"X-Forwarded-Proto":["http"]}},"error":"dial tcp 172.18.0.5:80: i/o timeout"}
caddy-1  | {"level":"error","ts":1733335961.5865045,"logger":"http.log.error","msg":"dial tcp 172.18.0.5:80: i/o timeout","request":{"remote_ip":"172.18.0.3","remote_port":"50786","client_ip":"172.18.0.1","proto":"HTTP/2.0","method":"GET","host":"localhost","uri":"/","headers":{"X-Forwarded-Host":["localhost"],"X-Forwarded-For":["172.18.0.1"],"X-Real-Ip":["172.18.0.1"],"X-Forwarded-Server":["7cf08c882dba"],"X-Forwarded-Proto":["http"],"X-Forwarded-Port":["80"]}},"duration":3.002690556,"status":502,"err_id":"icquvupft","err_trace":"reverseproxy.statusError (reverseproxy.go:1269)"}

Caddy logs for the entire time the above actions were repeated

caddy-1  | {"level":"info","ts":1733336630.892053,"logger":"...","msg":"Running caddy proxy server"}
caddy-1  | {"level":"info","ts":1733336630.8928719,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
caddy-1  | {"level":"info","ts":1733336630.893155,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
caddy-1  | {"level":"info","ts":1733336630.8931603,"logger":"...","msg":"Running caddy proxy controller"}
caddy-1  | {"level":"info","ts":1733336630.8973975,"logger":"...","msg":"Start","CaddyfilePath":"","EnvFile":"","LabelPrefix":"caddy","PollingInterval":30,"ProxyServiceTasks":true,"ProcessCaddyfile":true,"ScanStoppedContainers":false,"IngressNetworks":"[]","DockerSockets":[""],"DockerCertsPath":[""],"DockerAPIsVersion":[""]}
caddy-1  | {"level":"info","ts":1733336630.8976839,"logger":"...","msg":"Caddy ContainerID","ID":"5db95da3f64d3304d76d42c886f28f98fa1110b918e38088a963596502471585"}
caddy-1  | {"level":"info","ts":1733336630.9009056,"logger":"...","msg":"Connecting to docker events","DockerSocket":""}
caddy-1  | {"level":"info","ts":1733336630.9106565,"logger":"...","msg":"IngressNetworksMap","ingres":"map[361b3af628964b758e64140d4a9777167084546246db3764b235bb468149ede0:true caddy-proxy_proxy:true]"}
caddy-1  | {"level":"info","ts":1733336630.9269004,"logger":"...","msg":"Swarm is available","new":false}
caddy-1  | {"level":"info","ts":1733336630.9545295,"logger":"...","msg":"New Caddyfile","caddyfile":"{\n\tdebug\n\torder respond after reverse_proxy\n\tservers :80 {\n\t\tprotocols h1 h2c\n\t\ttrusted_proxies static private_ranges\n\t}\n}\n(app) {\n\trespond 503\n}\nlocalhost:80 {\n\timport app\n}\n"}
caddy-1  | {"level":"info","ts":1733336630.9566166,"logger":"...","msg":"New Config JSON","json":"{\"logging\":{\"logs\":{\"default\":{\"level\":\"DEBUG\"}}},\"apps\":{\"http\":{\"servers\":{\"srv0\":{\"listen\":[\":80\"],\"routes\":[{\"match\":[{\"host\":[\"localhost\"]}],\"handle\":[{\"handler\":\"subroute\",\"routes\":[{\"handle\":[{\"handler\":\"static_response\",\"status_code\":503}]}]}],\"terminal\":true}],\"trusted_proxies\":{\"ranges\":[\"192.168.0.0/16\",\"172.16.0.0/12\",\"10.0.0.0/8\",\"127.0.0.1/8\",\"fd00::/8\",\"::1\"],\"source\":\"static\"},\"protocols\":[\"h1\",\"h2c\"]}}}}}"}
caddy-1  | {"level":"info","ts":1733336630.9566493,"logger":"...","msg":"Sending configuration to","server":"localhost"}
caddy-1  | {"level":"info","ts":1733336630.9664147,"logger":"admin.api","msg":"received request","method":"POST","host":"localhost:2019","uri":"/load","remote_ip":"127.0.0.1","remote_port":"43770","headers":{"Accept-Encoding":["gzip"],"Content-Length":["465"],"Content-Type":["application/json"],"User-Agent":["Go-http-client/1.1"]}}
caddy-1  | {"level":"info","ts":1733336630.9669645,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
caddy-1  | {"level":"warn","ts":1733336630.9671369,"logger":"http.auto_https","msg":"server is listening only on the HTTP port, so no automatic HTTPS will be applied to this server","server_name":"srv0","http_port":80}
caddy-1  | {"level":"debug","ts":1733336630.9671662,"logger":"http.auto_https","msg":"adjusted config","tls":{"automation":{"policies":[{}]}},"http":{"servers":{"srv0":{"listen":[":80"],"routes":[{"handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"static_response","status_code":503}]}]}],"terminal":true}],"automatic_https":{"disable":true},"trusted_proxies":{"ranges":["192.168.0.0/16","172.16.0.0/12","10.0.0.0/8","127.0.0.1/8","fd00::/8","::1"],"source":"static"},"protocols":["h1","h2c"]}}}}
caddy-1  | {"level":"debug","ts":1733336630.967426,"logger":"http","msg":"starting server loop","address":"[::]:80","tls":false,"http3":false}
caddy-1  | {"level":"info","ts":1733336630.9674501,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2c"]}
caddy-1  | {"level":"info","ts":1733336630.9675312,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
caddy-1  | {"level":"info","ts":1733336630.9675488,"logger":"admin.api","msg":"load complete"}
caddy-1  | {"level":"info","ts":1733336630.9677083,"logger":"admin","msg":"stopped previous server","address":"localhost:2019"}
caddy-1  | {"level":"info","ts":1733336630.9677482,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0006c3180"}
caddy-1  | {"level":"info","ts":1733336630.9703016,"logger":"...","msg":"Successfully configured","server":"localhost"}
caddy-1  | {"level":"info","ts":1733336630.978914,"logger":"tls","msg":"cleaning storage unit","storage":"FileStorage:/data/caddy"}
caddy-1  | {"level":"info","ts":1733336630.9793568,"logger":"tls","msg":"finished cleaning storage units"}
caddy-1  | {"level":"debug","ts":1733336631.0715556,"logger":"...","msg":"Skipping default Caddyfile because no path is set"}
caddy-1  | {"level":"debug","ts":1733336631.0715854,"logger":"...","msg":"Skipping swarm config caddyfiles because swarm is not available"}
caddy-1  | {"level":"debug","ts":1733336631.1523685,"logger":"...","msg":"Skipping swarm services because swarm is not available"}
caddy-1  | {"level":"info","ts":1733336631.153941,"logger":"...","msg":"New Caddyfile","caddyfile":"{\n\tdebug\n\torder respond after reverse_proxy\n\tservers :80 {\n\t\tprotocols h1 h2c\n\t\ttrusted_proxies static private_ranges\n\t}\n}\n(app) {\n\trespond 503\n\treverse_proxy 172.18.0.3:80\n}\nlocalhost:80 {\n\timport app\n}\n"}
caddy-1  | {"level":"info","ts":1733336631.1557615,"logger":"...","msg":"New Config JSON","json":"{\"logging\":{\"logs\":{\"default\":{\"level\":\"DEBUG\"}}},\"apps\":{\"http\":{\"servers\":{\"srv0\":{\"listen\":[\":80\"],\"routes\":[{\"match\":[{\"host\":[\"localhost\"]}],\"handle\":[{\"handler\":\"subroute\",\"routes\":[{\"handle\":[{\"handler\":\"reverse_proxy\",\"upstreams\":[{\"dial\":\"172.18.0.3:80\"}]},{\"handler\":\"static_response\",\"status_code\":503}]}]}],\"terminal\":true}],\"trusted_proxies\":{\"ranges\":[\"192.168.0.0/16\",\"172.16.0.0/12\",\"10.0.0.0/8\",\"127.0.0.1/8\",\"fd00::/8\",\"::1\"],\"source\":\"static\"},\"protocols\":[\"h1\",\"h2c\"]}}}}}"}
caddy-1  | {"level":"info","ts":1733336631.1558566,"logger":"...","msg":"Sending configuration to","server":"localhost"}
caddy-1  | {"level":"info","ts":1733336631.1612098,"logger":"admin.api","msg":"received request","method":"POST","host":"localhost:2019","uri":"/load","remote_ip":"127.0.0.1","remote_port":"43784","headers":{"Accept-Encoding":["gzip"],"Content-Length":["532"],"Content-Type":["application/json"],"User-Agent":["Go-http-client/1.1"]}}
caddy-1  | {"level":"info","ts":1733336631.1626773,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
caddy-1  | {"level":"warn","ts":1733336631.1629126,"logger":"http.auto_https","msg":"server is listening only on the HTTP port, so no automatic HTTPS will be applied to this server","server_name":"srv0","http_port":80}
caddy-1  | {"level":"debug","ts":1733336631.163006,"logger":"http.auto_https","msg":"adjusted config","tls":{"automation":{"policies":[{}]}},"http":{"servers":{"srv0":{"listen":[":80"],"routes":[{"handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"reverse_proxy","upstreams":[{"dial":"172.18.0.3:80"}]},{"handler":"static_response","status_code":503}]}]}],"terminal":true}],"automatic_https":{"disable":true},"trusted_proxies":{"ranges":["192.168.0.0/16","172.16.0.0/12","10.0.0.0/8","127.0.0.1/8","fd00::/8","::1"],"source":"static"},"protocols":["h1","h2c"]}}}}
caddy-1  | {"level":"debug","ts":1733336631.1635418,"logger":"http","msg":"starting server loop","address":"[::]:80","tls":false,"http3":false}
caddy-1  | {"level":"info","ts":1733336631.1636004,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2c"]}
caddy-1  | {"level":"info","ts":1733336631.1636324,"logger":"http","msg":"servers shutting down with eternal grace period"}
caddy-1  | {"level":"info","ts":1733336631.1640606,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
caddy-1  | {"level":"info","ts":1733336631.1640759,"logger":"admin.api","msg":"load complete"}
caddy-1  | {"level":"info","ts":1733336631.1686296,"logger":"...","msg":"Successfully configured","server":"localhost"}
caddy-1  | {"level":"info","ts":1733336631.1713405,"logger":"admin","msg":"stopped previous server","address":"localhost:2019"}
caddy-1  | {"level":"debug","ts":1733336631.4833872,"logger":"...","msg":"Skipping default Caddyfile because no path is set"}
caddy-1  | {"level":"debug","ts":1733336631.4834192,"logger":"...","msg":"Skipping swarm config caddyfiles because swarm is not available"}
caddy-1  | {"level":"debug","ts":1733336631.535616,"logger":"...","msg":"Skipping swarm services because swarm is not available"}
caddy-1  | {"level":"info","ts":1733336631.5361838,"logger":"...","msg":"New Caddyfile","caddyfile":"{\n\tdebug\n\torder respond after reverse_proxy\n\tservers :80 {\n\t\tprotocols h1 h2c\n\t\ttrusted_proxies static private_ranges\n\t}\n}\n(app) {\n\trespond 503\n\treverse_proxy 172.18.0.3:80 172.18.0.5:80\n}\nlocalhost:80 {\n\timport app\n}\n"}
caddy-1  | {"level":"info","ts":1733336631.5365298,"logger":"...","msg":"New Config JSON","json":"{\"logging\":{\"logs\":{\"default\":{\"level\":\"DEBUG\"}}},\"apps\":{\"http\":{\"servers\":{\"srv0\":{\"listen\":[\":80\"],\"routes\":[{\"match\":[{\"host\":[\"localhost\"]}],\"handle\":[{\"handler\":\"subroute\",\"routes\":[{\"handle\":[{\"handler\":\"reverse_proxy\",\"upstreams\":[{\"dial\":\"172.18.0.3:80\"},{\"dial\":\"172.18.0.5:80\"}]},{\"handler\":\"static_response\",\"status_code\":503}]}]}],\"terminal\":true}],\"trusted_proxies\":{\"ranges\":[\"192.168.0.0/16\",\"172.16.0.0/12\",\"10.0.0.0/8\",\"127.0.0.1/8\",\"fd00::/8\",\"::1\"],\"source\":\"static\"},\"protocols\":[\"h1\",\"h2c\"]}}}}}"}
caddy-1  | {"level":"info","ts":1733336631.5365846,"logger":"...","msg":"Sending configuration to","server":"localhost"}
caddy-1  | {"level":"info","ts":1733336631.5372775,"logger":"admin.api","msg":"received request","method":"POST","host":"localhost:2019","uri":"/load","remote_ip":"127.0.0.1","remote_port":"43794","headers":{"Accept-Encoding":["gzip"],"Content-Length":["557"],"Content-Type":["application/json"],"User-Agent":["Go-http-client/1.1"]}}
caddy-1  | {"level":"info","ts":1733336631.5377276,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
caddy-1  | {"level":"warn","ts":1733336631.5378516,"logger":"http.auto_https","msg":"server is listening only on the HTTP port, so no automatic HTTPS will be applied to this server","server_name":"srv0","http_port":80}
caddy-1  | {"level":"debug","ts":1733336631.5378785,"logger":"http.auto_https","msg":"adjusted config","tls":{"automation":{"policies":[{}]}},"http":{"servers":{"srv0":{"listen":[":80"],"routes":[{"handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"reverse_proxy","upstreams":[{"dial":"172.18.0.3:80"},{"dial":"172.18.0.5:80"}]},{"handler":"static_response","status_code":503}]}]}],"terminal":true}],"automatic_https":{"disable":true},"trusted_proxies":{"ranges":["192.168.0.0/16","172.16.0.0/12","10.0.0.0/8","127.0.0.1/8","fd00::/8","::1"],"source":"static"},"protocols":["h1","h2c"]}}}}
caddy-1  | {"level":"debug","ts":1733336631.538043,"logger":"http","msg":"starting server loop","address":"[::]:80","tls":false,"http3":false}
caddy-1  | {"level":"info","ts":1733336631.5380654,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2c"]}
caddy-1  | {"level":"info","ts":1733336631.5380723,"logger":"http","msg":"servers shutting down with eternal grace period"}
caddy-1  | {"level":"info","ts":1733336631.5382814,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
caddy-1  | {"level":"info","ts":1733336631.5383086,"logger":"admin.api","msg":"load complete"}
caddy-1  | {"level":"info","ts":1733336631.5384116,"logger":"...","msg":"Successfully configured","server":"localhost"}
caddy-1  | {"level":"info","ts":1733336631.539603,"logger":"admin","msg":"stopped previous server","address":"localhost:2019"}
caddy-1  | {"level":"debug","ts":1733336633.5820024,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"172.18.0.3:80","total_upstreams":2}
caddy-1  | {"level":"debug","ts":1733336633.5830715,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"172.18.0.3:80","duration":0.001009587,"request":{"remote_ip":"172.18.0.4","remote_port":"49344","client_ip":"172.18.0.1","proto":"HTTP/2.0","method":"GET","host":"localhost","uri":"/","headers":{"X-Forwarded-For":["172.18.0.1, 172.18.0.4"],"X-Forwarded-Proto":["http"],"X-Real-Ip":["172.18.0.1"],"X-Forwarded-Host":["localhost"],"X-Forwarded-Port":["80"],"X-Forwarded-Server":["10eb5b693ceb"]}},"headers":{"Date":["Wed, 04 Dec 2024 18:23:53 GMT"],"Content-Length":["1003"],"Content-Type":["text/plain; charset=utf-8"]},"status":200}
caddy-1  | {"level":"debug","ts":1733336634.5985658,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"172.18.0.5:80","total_upstreams":2}
caddy-1  | {"level":"debug","ts":1733336634.599954,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"172.18.0.5:80","duration":0.001328178,"request":{"remote_ip":"172.18.0.4","remote_port":"49344","client_ip":"172.18.0.1","proto":"HTTP/2.0","method":"GET","host":"localhost","uri":"/","headers":{"X-Forwarded-Proto":["http"],"X-Forwarded-Host":["localhost"],"X-Forwarded-Port":["80"],"X-Real-Ip":["172.18.0.1"],"X-Forwarded-Server":["10eb5b693ceb"],"X-Forwarded-For":["172.18.0.1, 172.18.0.4"]}},"headers":{"Date":["Wed, 04 Dec 2024 18:23:54 GMT"],"Content-Length":["1003"],"Content-Type":["text/plain; charset=utf-8"]},"status":200}
caddy-1  | {"level":"debug","ts":1733336640.9364047,"logger":"...","msg":"Skipping default Caddyfile because no path is set"}
caddy-1  | {"level":"debug","ts":1733336640.936437,"logger":"...","msg":"Skipping swarm config caddyfiles because swarm is not available"}
caddy-1  | {"level":"debug","ts":1733336640.9885507,"logger":"...","msg":"Skipping swarm services because swarm is not available"}
caddy-1  | {"level":"debug","ts":1733336641.3052762,"logger":"...","msg":"Skipping default Caddyfile because no path is set"}
caddy-1  | {"level":"debug","ts":1733336641.3053012,"logger":"...","msg":"Skipping swarm config caddyfiles because swarm is not available"}
caddy-1  | {"level":"debug","ts":1733336641.3627765,"logger":"...","msg":"Skipping swarm services because swarm is not available"}
caddy-1  | {"level":"info","ts":1733336641.363941,"logger":"...","msg":"New Caddyfile","caddyfile":"{\n\tdebug\n\torder respond after reverse_proxy\n\tservers :80 {\n\t\tprotocols h1 h2c\n\t\ttrusted_proxies static private_ranges\n\t}\n}\n(app) {\n\trespond 503\n\treverse_proxy 172.18.0.6:80 172.18.0.3:80 172.18.0.5:80\n}\nlocalhost:80 {\n\timport app\n}\n"}
caddy-1  | {"level":"info","ts":1733336641.3645043,"logger":"...","msg":"New Config JSON","json":"{\"logging\":{\"logs\":{\"default\":{\"level\":\"DEBUG\"}}},\"apps\":{\"http\":{\"servers\":{\"srv0\":{\"listen\":[\":80\"],\"routes\":[{\"match\":[{\"host\":[\"localhost\"]}],\"handle\":[{\"handler\":\"subroute\",\"routes\":[{\"handle\":[{\"handler\":\"reverse_proxy\",\"upstreams\":[{\"dial\":\"172.18.0.6:80\"},{\"dial\":\"172.18.0.3:80\"},{\"dial\":\"172.18.0.5:80\"}]},{\"handler\":\"static_response\",\"status_code\":503}]}]}],\"terminal\":true}],\"trusted_proxies\":{\"ranges\":[\"192.168.0.0/16\",\"172.16.0.0/12\",\"10.0.0.0/8\",\"127.0.0.1/8\",\"fd00::/8\",\"::1\"],\"source\":\"static\"},\"protocols\":[\"h1\",\"h2c\"]}}}}}"}
caddy-1  | {"level":"info","ts":1733336641.3645325,"logger":"...","msg":"Sending configuration to","server":"localhost"}
caddy-1  | {"level":"info","ts":1733336641.3652577,"logger":"admin.api","msg":"received request","method":"POST","host":"localhost:2019","uri":"/load","remote_ip":"127.0.0.1","remote_port":"48104","headers":{"Accept-Encoding":["gzip"],"Content-Length":["582"],"Content-Type":["application/json"],"User-Agent":["Go-http-client/1.1"]}}
caddy-1  | {"level":"info","ts":1733336641.3657737,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
caddy-1  | {"level":"warn","ts":1733336641.365866,"logger":"http.auto_https","msg":"server is listening only on the HTTP port, so no automatic HTTPS will be applied to this server","server_name":"srv0","http_port":80}
caddy-1  | {"level":"debug","ts":1733336641.365878,"logger":"http.auto_https","msg":"adjusted config","tls":{"automation":{"policies":[{}]}},"http":{"servers":{"srv0":{"listen":[":80"],"routes":[{"handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"reverse_proxy","upstreams":[{"dial":"172.18.0.6:80"},{"dial":"172.18.0.3:80"},{"dial":"172.18.0.5:80"}]},{"handler":"static_response","status_code":503}]}]}],"terminal":true}],"automatic_https":{"disable":true},"trusted_proxies":{"ranges":["192.168.0.0/16","172.16.0.0/12","10.0.0.0/8","127.0.0.1/8","fd00::/8","::1"],"source":"static"},"protocols":["h1","h2c"]}}}}
caddy-1  | {"level":"debug","ts":1733336641.3660278,"logger":"http","msg":"starting server loop","address":"[::]:80","tls":false,"http3":false}
caddy-1  | {"level":"info","ts":1733336641.3660321,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2c"]}
caddy-1  | {"level":"info","ts":1733336641.3660376,"logger":"http","msg":"servers shutting down with eternal grace period"}
caddy-1  | {"level":"info","ts":1733336641.366212,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
caddy-1  | {"level":"info","ts":1733336641.366238,"logger":"admin.api","msg":"load complete"}
caddy-1  | {"level":"info","ts":1733336641.3663638,"logger":"...","msg":"Successfully configured","server":"localhost"}
caddy-1  | {"level":"info","ts":1733336641.3674102,"logger":"admin","msg":"stopped previous server","address":"localhost:2019"}
caddy-1  | {"level":"debug","ts":1733336641.6386697,"logger":"...","msg":"Skipping default Caddyfile because no path is set"}
caddy-1  | {"level":"debug","ts":1733336641.638713,"logger":"...","msg":"Skipping swarm config caddyfiles because swarm is not available"}
caddy-1  | {"level":"debug","ts":1733336641.6947596,"logger":"...","msg":"Skipping swarm services because swarm is not available"}
caddy-1  | {"level":"info","ts":1733336641.6952639,"logger":"...","msg":"New Caddyfile","caddyfile":"{\n\tdebug\n\torder respond after reverse_proxy\n\tservers :80 {\n\t\tprotocols h1 h2c\n\t\ttrusted_proxies static private_ranges\n\t}\n}\n(app) {\n\trespond 503\n\treverse_proxy 172.18.0.6:80 172.18.0.7:80 172.18.0.3:80 172.18.0.5:80\n}\nlocalhost:80 {\n\timport app\n}\n"}
caddy-1  | {"level":"info","ts":1733336641.6956055,"logger":"...","msg":"New Config JSON","json":"{\"logging\":{\"logs\":{\"default\":{\"level\":\"DEBUG\"}}},\"apps\":{\"http\":{\"servers\":{\"srv0\":{\"listen\":[\":80\"],\"routes\":[{\"match\":[{\"host\":[\"localhost\"]}],\"handle\":[{\"handler\":\"subroute\",\"routes\":[{\"handle\":[{\"handler\":\"reverse_proxy\",\"upstreams\":[{\"dial\":\"172.18.0.6:80\"},{\"dial\":\"172.18.0.7:80\"},{\"dial\":\"172.18.0.3:80\"},{\"dial\":\"172.18.0.5:80\"}]},{\"handler\":\"static_response\",\"status_code\":503}]}]}],\"terminal\":true}],\"trusted_proxies\":{\"ranges\":[\"192.168.0.0/16\",\"172.16.0.0/12\",\"10.0.0.0/8\",\"127.0.0.1/8\",\"fd00::/8\",\"::1\"],\"source\":\"static\"},\"protocols\":[\"h1\",\"h2c\"]}}}}}"}
caddy-1  | {"level":"info","ts":1733336641.69566,"logger":"...","msg":"Sending configuration to","server":"localhost"}
caddy-1  | {"level":"info","ts":1733336641.6963708,"logger":"admin.api","msg":"received request","method":"POST","host":"localhost:2019","uri":"/load","remote_ip":"127.0.0.1","remote_port":"48112","headers":{"Accept-Encoding":["gzip"],"Content-Length":["607"],"Content-Type":["application/json"],"User-Agent":["Go-http-client/1.1"]}}
caddy-1  | {"level":"info","ts":1733336641.6969,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
caddy-1  | {"level":"warn","ts":1733336641.6969864,"logger":"http.auto_https","msg":"server is listening only on the HTTP port, so no automatic HTTPS will be applied to this server","server_name":"srv0","http_port":80}
caddy-1  | {"level":"debug","ts":1733336641.6970072,"logger":"http.auto_https","msg":"adjusted config","tls":{"automation":{"policies":[{}]}},"http":{"servers":{"srv0":{"listen":[":80"],"routes":[{"handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"reverse_proxy","upstreams":[{"dial":"172.18.0.6:80"},{"dial":"172.18.0.7:80"},{"dial":"172.18.0.3:80"},{"dial":"172.18.0.5:80"}]},{"handler":"static_response","status_code":503}]}]}],"terminal":true}],"automatic_https":{"disable":true},"trusted_proxies":{"ranges":["192.168.0.0/16","172.16.0.0/12","10.0.0.0/8","127.0.0.1/8","fd00::/8","::1"],"source":"static"},"protocols":["h1","h2c"]}}}}
caddy-1  | {"level":"debug","ts":1733336641.6971881,"logger":"http","msg":"starting server loop","address":"[::]:80","tls":false,"http3":false}
caddy-1  | {"level":"info","ts":1733336641.6972058,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2c"]}
caddy-1  | {"level":"info","ts":1733336641.697212,"logger":"http","msg":"servers shutting down with eternal grace period"}
caddy-1  | {"level":"info","ts":1733336641.6973526,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
caddy-1  | {"level":"info","ts":1733336641.6973782,"logger":"admin.api","msg":"load complete"}
caddy-1  | {"level":"info","ts":1733336641.697441,"logger":"...","msg":"Successfully configured","server":"localhost"}
caddy-1  | {"level":"info","ts":1733336641.698533,"logger":"admin","msg":"stopped previous server","address":"localhost:2019"}
caddy-1  | {"level":"debug","ts":1733336643.9553964,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"172.18.0.3:80","total_upstreams":2}
caddy-1  | {"level":"debug","ts":1733336643.9562013,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"172.18.0.3:80","duration":0.000718651,"request":{"remote_ip":"172.18.0.4","remote_port":"49344","client_ip":"172.18.0.1","proto":"HTTP/2.0","method":"GET","host":"localhost","uri":"/","headers":{"X-Real-Ip":["172.18.0.1"],"X-Forwarded-Proto":["http"],"X-Forwarded-Port":["80"],"X-Forwarded-Server":["10eb5b693ceb"],"X-Forwarded-Host":["localhost"],"X-Forwarded-For":["172.18.0.1, 172.18.0.4"]}},"headers":{"Date":["Wed, 04 Dec 2024 18:24:03 GMT"],"Content-Length":["1003"],"Content-Type":["text/plain; charset=utf-8"]},"status":200}
caddy-1  | {"level":"debug","ts":1733336644.5550253,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"172.18.0.5:80","total_upstreams":2}
caddy-1  | {"level":"debug","ts":1733336644.5555682,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"172.18.0.5:80","duration":0.000482951,"request":{"remote_ip":"172.18.0.4","remote_port":"49344","client_ip":"172.18.0.1","proto":"HTTP/2.0","method":"GET","host":"localhost","uri":"/","headers":{"X-Forwarded-Server":["10eb5b693ceb"],"X-Forwarded-For":["172.18.0.1, 172.18.0.4"],"X-Real-Ip":["172.18.0.1"],"X-Forwarded-Host":["localhost"],"X-Forwarded-Port":["80"],"X-Forwarded-Proto":["http"]}},"headers":{"Content-Length":["1003"],"Content-Type":["text/plain; charset=utf-8"],"Date":["Wed, 04 Dec 2024 18:24:04 GMT"]},"status":200}
caddy-1  | {"level":"debug","ts":1733336645.2101312,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"172.18.0.3:80","total_upstreams":2}
caddy-1  | {"level":"debug","ts":1733336645.2107072,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"172.18.0.3:80","duration":0.000506808,"request":{"remote_ip":"172.18.0.4","remote_port":"49344","client_ip":"172.18.0.1","proto":"HTTP/2.0","method":"GET","host":"localhost","uri":"/","headers":{"X-Forwarded-Port":["80"],"X-Forwarded-Proto":["http"],"X-Forwarded-For":["172.18.0.1, 172.18.0.4"],"X-Forwarded-Host":["localhost"],"X-Real-Ip":["172.18.0.1"],"X-Forwarded-Server":["10eb5b693ceb"]},"headers":{"Date":["Wed, 04 Dec 2024 18:24:05 GMT"],"Content-Length":["1003"],"Content-Type":["text/plain; charset=utf-8"]},"status":200}
caddy-1  | {"level":"debug","ts":1733336645.830422,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"172.18.0.3:80","total_upstreams":2}
caddy-1  | {"level":"debug","ts":1733336645.8309786,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"172.18.0.3:80","duration":0.000496155,"request":{"remote_ip":"172.18.0.4","remote_port":"49344","client_ip":"172.18.0.1","proto":"HTTP/2.0","method":"GET","host":"localhost","uri":"/","headers":{"X-Forwarded-Proto":["http"],"X-Forwarded-Host":["localhost"],"X-Real-Ip":["172.18.0.1"],"X-Forwarded-For":["172.18.0.1, 172.18.0.4"],"X-Forwarded-Port":["80"],"X-Forwarded-Server":["10eb5b693ceb"]}},"headers":{"Date":["Wed, 04 Dec 2024 18:24:05 GMT"],"Content-Length":["1003"],"Content-Type":["text/plain; charset=utf-8"]},"status":200}
caddy-1  | {"level":"debug","ts":1733336654.0969012,"logger":"...","msg":"Skipping default Caddyfile because no path is set"}
caddy-1  | {"level":"debug","ts":1733336654.0969603,"logger":"...","msg":"Skipping swarm config caddyfiles because swarm is not available"}
caddy-1  | {"level":"debug","ts":1733336654.1506946,"logger":"...","msg":"Skipping swarm services because swarm is not available"}
caddy-1  | {"level":"info","ts":1733336654.1511412,"logger":"...","msg":"New Caddyfile","caddyfile":"{\n\tdebug\n\torder respond after reverse_proxy\n\tservers :80 {\n\t\tprotocols h1 h2c\n\t\ttrusted_proxies static private_ranges\n\t}\n}\n(app) {\n\trespond 503\n\treverse_proxy 172.18.0.6:80 172.18.0.7:80 172.18.0.5:80\n}\nlocalhost:80 {\n\timport app\n}\n"}
caddy-1  | {"level":"info","ts":1733336654.1514466,"logger":"...","msg":"New Config JSON","json":"{\"logging\":{\"logs\":{\"default\":{\"level\":\"DEBUG\"}}},\"apps\":{\"http\":{\"servers\":{\"srv0\":{\"listen\":[\":80\"],\"routes\":[{\"match\":[{\"host\":[\"localhost\"]}],\"handle\":[{\"handler\":\"subroute\",\"routes\":[{\"handle\":[{\"handler\":\"reverse_proxy\",\"upstreams\":[{\"dial\":\"172.18.0.6:80\"},{\"dial\":\"172.18.0.7:80\"},{\"dial\":\"172.18.0.5:80\"}]},{\"handler\":\"static_response\",\"status_code\":503}]}]}],\"terminal\":true}],\"trusted_proxies\":{\"ranges\":[\"192.168.0.0/16\",\"172.16.0.0/12\",\"10.0.0.0/8\",\"127.0.0.1/8\",\"fd00::/8\",\"::1\"],\"source\":\"static\"},\"protocols\":[\"h1\",\"h2c\"]}}}}}"}
caddy-1  | {"level":"info","ts":1733336654.1514645,"logger":"...","msg":"Sending configuration to","server":"localhost"}
caddy-1  | {"level":"info","ts":1733336654.152151,"logger":"admin.api","msg":"received request","method":"POST","host":"localhost:2019","uri":"/load","remote_ip":"127.0.0.1","remote_port":"44658","headers":{"Accept-Encoding":["gzip"],"Content-Length":["582"],"Content-Type":["application/json"],"User-Agent":["Go-http-client/1.1"]}}
caddy-1  | {"level":"info","ts":1733336654.1526415,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
caddy-1  | {"level":"warn","ts":1733336654.1527417,"logger":"http.auto_https","msg":"server is listening only on the HTTP port, so no automatic HTTPS will be applied to this server","server_name":"srv0","http_port":80}
caddy-1  | {"level":"debug","ts":1733336654.1527526,"logger":"http.auto_https","msg":"adjusted config","tls":{"automation":{"policies":[{}]}},"http":{"servers":{"srv0":{"listen":[":80"],"routes":[{"handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"reverse_proxy","upstreams":[{"dial":"172.18.0.6:80"},{"dial":"172.18.0.7:80"},{"dial":"172.18.0.5:80"}]},{"handler":"static_response","status_code":503}]}]}],"terminal":true}],"automatic_https":{"disable":true},"trusted_proxies":{"ranges":["192.168.0.0/16","172.16.0.0/12","10.0.0.0/8","127.0.0.1/8","fd00::/8","::1"],"source":"static"},"protocols":["h1","h2c"]}}}}
caddy-1  | {"level":"debug","ts":1733336654.1528835,"logger":"http","msg":"starting server loop","address":"[::]:80","tls":false,"http3":false}
caddy-1  | {"level":"info","ts":1733336654.1528866,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2c"]}
caddy-1  | {"level":"info","ts":1733336654.1528916,"logger":"http","msg":"servers shutting down with eternal grace period"}
caddy-1  | {"level":"info","ts":1733336654.1530569,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
caddy-1  | {"level":"info","ts":1733336654.1530628,"logger":"admin.api","msg":"load complete"}
caddy-1  | {"level":"info","ts":1733336654.1531422,"logger":"...","msg":"Successfully configured","server":"localhost"}
caddy-1  | {"level":"info","ts":1733336654.1542761,"logger":"admin","msg":"stopped previous server","address":"localhost:2019"}
caddy-1  | {"level":"debug","ts":1733336656.7490783,"logger":"...","msg":"Skipping default Caddyfile because no path is set"}
caddy-1  | {"level":"debug","ts":1733336656.7491026,"logger":"...","msg":"Skipping swarm config caddyfiles because swarm is not available"}
caddy-1  | {"level":"debug","ts":1733336656.790554,"logger":"...","msg":"Skipping swarm services because swarm is not available"}
caddy-1  | {"level":"info","ts":1733336656.7910807,"logger":"...","msg":"New Caddyfile","caddyfile":"{\n\tdebug\n\torder respond after reverse_proxy\n\tservers :80 {\n\t\tprotocols h1 h2c\n\t\ttrusted_proxies static private_ranges\n\t}\n}\n(app) {\n\trespond 503\n\treverse_proxy 172.18.0.6:80 172.18.0.7:80\n}\nlocalhost:80 {\n\timport app\n}\n"}
caddy-1  | {"level":"info","ts":1733336656.791442,"logger":"...","msg":"New Config JSON","json":"{\"logging\":{\"logs\":{\"default\":{\"level\":\"DEBUG\"}}},\"apps\":{\"http\":{\"servers\":{\"srv0\":{\"listen\":[\":80\"],\"routes\":[{\"match\":[{\"host\":[\"localhost\"]}],\"handle\":[{\"handler\":\"subroute\",\"routes\":[{\"handle\":[{\"handler\":\"reverse_proxy\",\"upstreams\":[{\"dial\":\"172.18.0.6:80\"},{\"dial\":\"172.18.0.7:80\"}]},{\"handler\":\"static_response\",\"status_code\":503}]}]}],\"terminal\":true}],\"trusted_proxies\":{\"ranges\":[\"192.168.0.0/16\",\"172.16.0.0/12\",\"10.0.0.0/8\",\"127.0.0.1/8\",\"fd00::/8\",\"::1\"],\"source\":\"static\"},\"protocols\":[\"h1\",\"h2c\"]}}}}}"}
caddy-1  | {"level":"info","ts":1733336656.7914805,"logger":"...","msg":"Sending configuration to","server":"localhost"}
caddy-1  | {"level":"info","ts":1733336656.7920027,"logger":"admin.api","msg":"received request","method":"POST","host":"localhost:2019","uri":"/load","remote_ip":"127.0.0.1","remote_port":"44672","headers":{"Accept-Encoding":["gzip"],"Content-Length":["557"],"Content-Type":["application/json"],"User-Agent":["Go-http-client/1.1"]}}
caddy-1  | {"level":"info","ts":1733336656.792609,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
caddy-1  | {"level":"warn","ts":1733336656.7927058,"logger":"http.auto_https","msg":"server is listening only on the HTTP port, so no automatic HTTPS will be applied to this server","server_name":"srv0","http_port":80}
caddy-1  | {"level":"debug","ts":1733336656.7927165,"logger":"http.auto_https","msg":"adjusted config","tls":{"automation":{"policies":[{}]}},"http":{"servers":{"srv0":{"listen":[":80"],"routes":[{"handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"reverse_proxy","upstreams":[{"dial":"172.18.0.6:80"},{"dial":"172.18.0.7:80"}]},{"handler":"static_response","status_code":503}]}]}],"terminal":true}],"automatic_https":{"disable":true},"trusted_proxies":{"ranges":["192.168.0.0/16","172.16.0.0/12","10.0.0.0/8","127.0.0.1/8","fd00::/8","::1"],"source":"static"},"protocols":["h1","h2c"]}}}}
caddy-1  | {"level":"debug","ts":1733336656.7928324,"logger":"http","msg":"starting server loop","address":"[::]:80","tls":false,"http3":false}
caddy-1  | {"level":"info","ts":1733336656.7928593,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2c"]}
caddy-1  | {"level":"info","ts":1733336656.7928643,"logger":"http","msg":"servers shutting down with eternal grace period"}
caddy-1  | {"level":"info","ts":1733336656.793035,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
caddy-1  | {"level":"info","ts":1733336656.7930682,"logger":"admin.api","msg":"load complete"}
caddy-1  | {"level":"info","ts":1733336656.793158,"logger":"...","msg":"Successfully configured","server":"localhost"}
caddy-1  | {"level":"info","ts":1733336656.7941995,"logger":"admin","msg":"stopped previous server","address":"localhost:2019"}
caddy-1  | {"level":"debug","ts":1733336658.2171938,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"172.18.0.3:80","total_upstreams":2}
caddy-1  | {"level":"debug","ts":1733336661.2198925,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"172.18.0.3:80","duration":3.00260833,"request":{"remote_ip":"172.18.0.4","remote_port":"49344","client_ip":"172.18.0.1","proto":"HTTP/2.0","method":"GET","host":"localhost","uri":"/","headers":{"X-Real-Ip":["172.18.0.1"],"X-Forwarded-Proto":["http"],"X-Forwarded-Port":["80"],"X-Forwarded-Server":["10eb5b693ceb"],"X-Forwarded-For":["172.18.0.1, 172.18.0.4"],"X-Forwarded-Host":["localhost"]}},"error":"dial tcp 172.18.0.3:80: i/o timeout"}
caddy-1  | {"level":"error","ts":1733336661.2199938,"logger":"http.log.error","msg":"dial tcp 172.18.0.3:80: i/o timeout","request":{"remote_ip":"172.18.0.4","remote_port":"49344","client_ip":"172.18.0.1","proto":"HTTP/2.0","method":"GET","host":"localhost","uri":"/","headers":{"X-Forwarded-Server":["10eb5b693ceb"],"X-Forwarded-Host":["localhost"],"X-Forwarded-For":["172.18.0.1"],"X-Real-Ip":["172.18.0.1"],"X-Forwarded-Port":["80"],"X-Forwarded-Proto":["http"]}},"duration":3.002829856,"status":502,"err_id":"axru540dq","err_trace":"reverseproxy.statusError (reverseproxy.go:1269)"}

[francislavoie]

Stopping containers is not the same as removing them.

[7-zete-7]

Stopped containers can be removed - the problem will not be solved.

In any case: accessing stopped containers is also not a good idea, as is not accessing new ones that are running.

[7-zete-7]

It looks like the error is not in this project.

compose.yaml

services:
  traefik:
    image: traefik:v3.1
    networks:
      - traefik
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
    command:
      - --entrypoints.web.address=:80

      - --providers.docker
      - --providers.docker.network=${COMPOSE_PROJECT_NAME}_traefik
      - --providers.docker.exposedbydefault=false

    ports:
      - "80:80/tcp"

  caddy-server:
    image: caddy:2.8.4-alpine
    networks:
      - traefik
      - caddy
      - controller
    volumes:
      - caddy-data:/data
    environment:
      CADDY_ADMIN: :2019
    labels:
      # region Caddy
      caddy_controlled_server:

      # global configs
      caddy_0.order: respond after reverse_proxy
      caddy_0.debug:

      # `:80` servers configs
      caddy_1.servers: :80
      caddy_1.servers.protocols: h1 h2c
      caddy_1.servers.trusted_proxies: static private_ranges

      # `localhost:80` address configs
      caddy_2: localhost:80
      caddy_2.import: app

      # dynamic `localhost:80` address configs
      caddy_3: (app)
      caddy_3.respond: "503"
      # endregion Caddy

      # region Traefik
      traefik.enable: "true"

      traefik.http.services.caddy.loadbalancer.server.port: "80"
      traefik.http.services.caddy.loadbalancer.server.scheme: h2c

      traefik.http.routers.caddy.entrypoints: web
      traefik.http.routers.caddy.rule: Host(`localhost`)
      traefik.http.routers.caddy.service: caddy@docker
      # endregion Traefik

  caddy-controller:
    image: lucaslorentz/caddy-docker-proxy:ci-alpine
    networks:
      - controller
    environment:
      CADDY_DOCKER_MODE: controller
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro

  app:
    image: traefik/whoami:latest
    scale: 2
    networks:
      - caddy
    labels:
      caddy_ingress_network: "${COMPOSE_PROJECT_NAME}_caddy"

      caddy: (app)
      caddy.reverse_proxy: "{{upstreams 80}}"

volumes:
  caddy-data:

networks:
  traefik:
  caddy:
  controller:

I made a separate service controller and configured it to manage the standard Caddy — the same error is observed.
Checked on Caddy versions 2.8.4 and 2.9.0-beta.3.

The service controller generates a JSON configuration and sends it via the Caddy API method POST /load.
Caddy continues to access old addresses.

When calling the Caddy API method GET /reverse_proxy/upstreams I get a list of new addresses.
Caddy still accesses only old addresses.

The problem goes away either after restarting Caddy or restarting Traefik.

Looks like the problem is keeping the connection to Caddy open.

[radokristof]

@francislavoie any updates on this? Regarding my case, as stated above, it is not about stopped / not recreated containers.
I always do a docker compose up -d after if I check the containers IP address in caddy network it will not be the same as visible in the caddy config reported by caddy-docker-proxy. Restarting the caddy-docker-proxy container solves the issue.

[luckylinux]

@francislavoie, @7-zete-7, @francislavoie: Not sure how relevant, but I had (well ... still have) a similar Issue on podman with traefik Reverse Proxy.

I need to restart the Reverse Proxy Container (traefik in my Case), whenever another Container is (re)started, independently if this is due to a Configuration Change or Crash etc.

You might have a look at how I handle traefik with podman although some Modifications will be of course required: Script from my Repository

I was looking into a Way to handle caddy in a Similar way to traefik, that's why I found this Plugin. But I'm still not 100% sure whether to go this Road. Currently I either use traefik + N x Containers or I use caddy + app and simply repeat that N times (so there are N Caddy instances running, each with a different Host GUA IPv6-bound IP address & Port 80/443).

How close is this plugin Implementation to full-featured like traefik ? I'd like to have a "basic" Caddyfile where I specify e.g. the TLS Certificate to use and Log Configuration, yet have the possibility to use Labels like with traefik to add new Hosts/Containers, without a need to expose Host:8081 for Container 1, Host:8082 for Container 2, Host:8083 for Container 3, etc.