Got exception when needed to restart

[karserasl]

restarter | 2023-09-24T12:30:02.509038316Z Traceback (most recent call last): restarter | 2023-09-24T12:30:02.509065623Z File "/usr/src/app/./restarter.py", line 32, in <module> restarter | 2023-09-24T12:30:02.509143644Z service = event["Actor"]["Attributes"]["com.docker.compose.service"] restarter | 2023-09-24T12:30:02.509219177Z ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ restarter | 2023-09-24T12:30:02.509233587Z KeyError: 'com.docker.compose.service'

[karserasl]

Im using socket-proxy for docker host

restarter: <<: *common-keys-core # See EXTENSION FIELDS at the top image: ghcr.io/cascandaliato/docker-restarter container_name: restarter networks: - socket_proxy - t2_proxy environment: <<: *default-tz-puid-pgid DOCKER_HOST: tcp://socket-proxy:2375

[cascandaliato]

I pushed a fix (0c6057a). You probably started or stopped a container defined outside of a docker compose file.

Is Tecnativa/docker-socket-proxy the proxy you are talking about? If so, this project needs the following permissions: EVENTS (enabled by default?), CONTAINERS, POST, ALLOW_RESTARTS.

[karserasl]

Thanks alot
Gonna try

[karserasl]

I think using ALLOW_RESTARTS=1 and the latest version fixed everything

Great work

[karserasl]

Talked too fast.
Im getting this exception now.

restarter | 2023-09-24T15:01:52.654079529Z Restarting the following container(s) in 10 seconds because container qbittorrent (re)started: cross-seed, cross-seed-search
restarter | 2023-09-24T15:33:28.054311911Z Restarting the following container(s) in 10 seconds because container gluetun (re)started: autobrr, joal, portcheck, qbittorrent, thelounge
restarter | 2023-09-24T15:33:29.823397734Z Traceback (most recent call last):
restarter | 2023-09-24T15:33:29.823410523Z File "/usr/local/lib/python3.11/site-packages/docker/api/client.py", line 268, in _raise_for_status
restarter | 2023-09-24T15:33:29.833985297Z response.raise_for_status()
restarter | 2023-09-24T15:33:29.834000573Z File "/usr/local/lib/python3.11/site-packages/requests/models.py", line 1021, in raise_for_status
restarter | 2023-09-24T15:33:29.839697374Z raise HTTPError(http_error_msg, response=self)
restarter | 2023-09-24T15:33:29.840563152Z requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http://socket-proxy:2375/v1.43/containers/4f6a65c0047b48d46bc1a962696dea9226fc7f5a7938bb81c815755e0ec0d6c8/restart?t=10
restarter | 2023-09-24T15:33:29.840567853Z
restarter | 2023-09-24T15:33:29.840569867Z The above exception was the direct cause of the following exception:
restarter | 2023-09-24T15:33:29.840571908Z
restarter | 2023-09-24T15:33:29.840573848Z Traceback (most recent call last):
restarter | 2023-09-24T15:33:29.840575881Z File "/usr/src/app/./restarter.py", line 40, in
restarter | 2023-09-24T15:33:29.840639762Z docker_client.containers.get(depender).restart()
restarter | 2023-09-24T15:33:29.840687543Z File "/usr/local/lib/python3.11/site-packages/docker/models/containers.py", line 395, in restart
restarter | 2023-09-24T15:33:29.851685047Z return self.client.api.restart(self.id, **kwargs)
restarter | 2023-09-24T15:33:29.851714893Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
restarter | 2023-09-24T15:33:29.851739577Z File "/usr/local/lib/python3.11/site-packages/docker/utils/decorators.py", line 19, in wrapped
restarter | 2023-09-24T15:33:29.856188061Z return f(self, resource_id, *args, **kwargs)
restarter | 2023-09-24T15:33:29.856212190Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
restarter | 2023-09-24T15:33:29.856216523Z File "/usr/local/lib/python3.11/site-packages/docker/api/container.py", line 1089, in restart
restarter | 2023-09-24T15:33:29.858447220Z self._raise_for_status(res)
restarter | 2023-09-24T15:33:29.858459696Z File "/usr/local/lib/python3.11/site-packages/docker/api/client.py", line 270, in _raise_for_status
restarter | 2023-09-24T15:33:29.858578592Z raise create_api_error_from_http_exception(e) from e
restarter | 2023-09-24T15:33:29.858594269Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
restarter | 2023-09-24T15:33:29.858608526Z File "/usr/local/lib/python3.11/site-packages/docker/errors.py", line 39, in create_api_error_from_http_exception
restarter | 2023-09-24T15:33:29.868615961Z raise cls(e, response=response, explanation=explanation) from e
restarter | 2023-09-24T15:33:29.868635785Z docker.errors.APIError: 500 Server Error for http://socket-proxy:2375/v1.43/containers/4f6a65c0047b48d46bc1a962696dea9226fc7f5a7938bb81c815755e0ec0d6c8/restart?t=10: Internal Server Error ("Cannot restart container 4f6a65c0047b48d46bc1a962696dea9226fc7f5a7938bb81c815755e0ec0d6c8: No such container: c6205526a4ce82f0d4ff8b090d1138d701ecf9910dd571e02a5ab221004b066d")

[cascandaliato]

I've added more log traces. The code doesn't expect (yet) any container to disappear so it won't work if there are services that remove containers (watchtower?) but I suspect it's something else because the proxy API is logging two different container ids in the same error message:

docker.errors.APIError: 500 Server Error for http://socket-proxy:2375/v1.43/containers/4f6a65c0047b48d46bc1a962696dea9226fc7f5a7938bb81c815755e0ec0d6c8/restart?t=10: Internal Server Error ("Cannot restart container 4f6a65c0047b48d46bc1a962696dea9226fc7f5a7938bb81c815755e0ec0d6c8: No such container: c6205526a4ce82f0d4ff8b090d1138d701ecf9910dd571e02a5ab221004b066d")

Do you mind trying again and sharing the container logs? Thank you for your patience.

[vdrover]

Just adding a note that watchtower support would be wonderful.

[karserasl]

Yeah, it was the watchtower that updated Gluetun.
So, that's the main problem.

Gluetun pushed a new release and Watchtower updates it, which caused the whole stack depending on it to lose connection.

[cascandaliato]

Ok, that makes sense. I'll make the script handle this case and only log an error if a container is not available anymore when we try to restart it, for example because watchtower has updated it, the stack has been rebuilt or the user just removed it with docker rm.

[cascandaliato]

After commit 7295fca I don't expect the container to crash if it's trying to restart a container while watchtower is removing and updating it. Please give it a try when possible.

[karserasl]

im still seeing this

restarter  | 2023-09-28T15:08:03.586274819Z Container gluetun (id cef21da8dee3, service gluetun) restarted
restarter  | 2023-09-28T15:08:03.606773656Z The following containers depend on service gluetun and will be restarted in 30 seconds:
restarter  | 2023-09-28T15:08:03.608356355Z   qbittorrent (id d2543234000c, service qbittorrent)

restarter  | 2023-09-28T15:08:38.071354033Z Restarting container qbittorrent (id d2543234000c, service qbittorrent)
restarter  | 2023-09-28T15:08:49.117722348Z Could not restart container qbittorrent (id d2543234000c, service qbittorrent) because of error: 500 Server Error for http://socket-proxy:2375/v1.43/containers/d2543234000cc6be03349c23270e65c22135499cdfa6cbd8a1703ac4e2f95406/restart?t=10: Internal Server Error ("Cannot restart container d2543234000cc6be03349c23270e65c22135499cdfa6cbd8a1703ac4e2f95406: No such container: a7a0622a5331c1e838b668c6e7c5ad808c0ca8f78213f95eec29e3d07331b6e6")

was after gluetun update from watchtower

[cascandaliato]

Thank you for taking the time to test the latest version.

Do you know if watchtower also updated the qbittorrent container?

Here are some things that might help me:

• Capture the output of these two commands

  docker inspect d2543234000c --format "{{.Name}}"
  docker inspect a7a0622a5331 --format "{{.Name}}"
  

• Share watchtower logs
• Run restarter without the Docker socket proxy

If you prefer, you can reach me on Twitter or Reddit (same username as here).

[cascandaliato]

Nevermind, I have replicated the issue so there is no need to collect logs, etc. It's the same issue qdm12/deunhealth#11 that Quentin has been working on. I'll read his notes and see if I can do my own implementation.