Synology: WatchTower deletes or does not restore iobroker container after update

[mtoensing]

Describe the bug
I use many containers like minecraft, iobroker, mysql and so on. I keep them up to date with watchtower. One of this containers disappears after a crash in watchtower every single time. It is https://github.com/buanet/docker-iobroker from @buanet It happend 5 times. I completely reinstalled docker on my nas on a different volume and removed the images. Nothing helps. Since my home automation system relies on this container and i can not exclude it from watchtower on synology I am not able to prevent this. Any idea? Here is the log of watchtower:

2020-08-18 05:04:24 | stderr | /home/circleci/repo/main.go:13 +0x20
2020-08-18 05:04:24 | stderr | main.main()
2020-08-18 05:04:24 | stderr | /home/circleci/repo/cmd/root.go:55 +0x2d
2020-08-18 05:04:24 | stderr | github.com/containrrr/watchtower/cmd.Execute()
2020-08-18 05:04:24 | stderr | /go/pkg/mod/github.com/spf13/[email protected]/command.go:800
2020-08-18 05:04:24 | stderr | github.com/spf13/cobra.(*Command).Execute(...)
2020-08-18 05:04:24 | stderr | /go/pkg/mod/github.com/spf13/[email protected]/command.go:852 +0x2ea
2020-08-18 05:04:24 | stderr | github.com/spf13/cobra.(*Command).ExecuteC(0x1137100, 0x405b6f, 0xc000102058, 0x0)
2020-08-18 05:04:24 | stderr | /go/pkg/mod/github.com/spf13/[email protected]/command.go:766 +0x29d
2020-08-18 05:04:24 | stderr | github.com/spf13/cobra.(*Command).execute(0x1137100, 0xc00011e050, 0x3, 0x3, 0x1137100, 0xc00011e050)
2020-08-18 05:04:24 | stderr | /home/circleci/repo/cmd/root.go:135 +0x143
2020-08-18 05:04:24 | stderr | github.com/containrrr/watchtower/cmd.Run(0x1137100, 0xc0003c6060, 0x0, 0x3)
2020-08-18 05:04:24 | stderr | /home/circleci/repo/cmd/root.go:205 +0xbe
2020-08-18 05:04:24 | stderr | github.com/containrrr/watchtower/cmd.runUpdatesWithNotifications(0xc0003757b0)
2020-08-18 05:04:24 | stderr | /home/circleci/repo/internal/actions/update.go:53 +0x37a
2020-08-18 05:04:24 | stderr | github.com/containrrr/watchtower/internal/actions.Update(0xca5880, 0xc000380be0, 0xc0003757b0, 0x1, 0x2540be400, 0x0, 0xc00025fcd8, 0x74751d)
2020-08-18 05:04:24 | stderr | /home/circleci/repo/internal/actions/update.go:96 +0x241
2020-08-18 05:04:24 | stderr | github.com/containrrr/watchtower/internal/actions.restartContainersInSortedOrder(0xc000082480, 0xf, 0x10, 0xca5880, 0xc000380be0, 0xc0003757b0, 0x1, 0x2540be400, 0x0)
2020-08-18 05:04:24 | stderr | /home/circleci/repo/internal/actions/update.go:121 +0xdb
2020-08-18 05:04:24 | stderr | github.com/containrrr/watchtower/internal/actions.restartStaleContainer(0xac0100, 0xc0001b60c0, 0xc0005ae420, 0xca5880, 0xc000380be0, 0xc0003757b0, 0x1, 0x2540be400, 0x0)
2020-08-18 05:04:24 | stderr | /home/circleci/repo/pkg/container/client.go:168 +0x86
2020-08-18 05:04:24 | stderr | github.com/containrrr/watchtower/pkg/container.dockerClient.StartContainer(0xcb4da0, 0xc000378700, 0x10001, 0x900100, 0xc0001b60c0, 0xc0005ae420, 0xc0001e3800, 0x40, 0x0, 0x0)
2020-08-18 05:04:24 | stderr | /home/circleci/repo/pkg/container/container.go:203 +0x41c
2020-08-18 05:04:24 | stderr | github.com/containrrr/watchtower/pkg/container.Container.runtimeConfig(0xc000270100, 0xc0001b60c0, 0xc0005ae420, 0xc000442ca2)
2020-08-18 05:04:24 | stderr | goroutine 1 [running]:
2020-08-18 05:04:24 | stderr |  
2020-08-18 05:04:24 | stderr | panic: assignment to entry in nil map
2020-08-18 05:04:23 | stderr | time="2020-08-18T05:04:23Z" level=info msg="Creating /overviewer-test-genpoi"
2020-08-18 05:04:22 | stderr | time="2020-08-18T05:04:22Z" level=info msg="Creating /overviewer-test"
2020-08-18 05:04:20 | stderr | time="2020-08-18T05:04:20Z" level=info msg="Creating /overviewer-genpoi"
2020-08-18 05:04:19 | stderr | time="2020-08-18T05:04:19Z" level=info msg="Creating /overviewer"
2020-08-18 05:04:16 | stderr | time="2020-08-18T05:04:16Z" level=info msg="Creating /mcbedrock"
2020-08-18 05:04:13 | stderr | time="2020-08-18T05:04:13Z" level=info msg="Creating /mcserver"

To Reproduce
Steps to reproduce the behavior:

1. Install iobroker docker container
2. Install watchtower docker container
3. Wait until watchtower updates the containers.
4. Only iobroker container is missing after that.

Expected behavior
Watchtower should not delete images without restoring them afterwards

Environment

• Platform
• Architecture
• Docker version

Logs from running watchtower with the --debug option

Additional context
Add any other context about the problem here.

[github-actions]

Hi there! 👋🏼 As you're new to this repo, we'd like to suggest that you read our code of conduct as well as our contribution guidelines. Thanks a bunch for opening your first issue! 🙏

[zoispag]

Since my home automation system relies on this container and i can not exclude it from watchtower on synology

You should be able to selectively include or exclude containers from watchtower automation. See container-selection. Unless there is something special about Synology that I am not aware off. @simskij might have another suggestion

[mtoensing]

@zoispag I use the Synology GUI and Portainer. I don't see an option to add a "label".

[zoispag]

what about this @mtoensing ?

[mtoensing]

Ah, only on create? I'll try that

[zoispag]

Labels can be set only on create afaik

[piksel]

This seems to be caused by the containerInfo.Config having ExposedPorts set to nil. This scenario could be handled and skipped in the same manner as #612 (since I am guessing that this container DOES have exposed ports, so restarting without them would be just as bad, right?).

Still unclear why this is happening, but I wonder if the modified dockerd is the cause...

[Bustout]

I'm having a similar issue with watchtower deleting my Minecraft and Booksonic Dockers from my Synology. It seems to only delete them if there is an update to install. Here is my log:

Task: Watchtower
Start time: Sun, 08 Nov 2020 00:00:02 GMT Stop time: Sun, 08 Nov 2020 00:01:13 GMT Current status: 2 (Interrupted) Standard output/error:

2020-11-08T08:00:09+00 [I] Running a one time update.
2020-11-08T08:00:11+00 [W] Failed to retrieve container image info: Error: No such image: sha256:333738857dc9ab1d7feb2fcff5af0c1893472022d33e884a6377250b3f94f6c7
2020-11-08T08:00:36+00 [I] Found new marctv/minecraft-bedrock-server:latest image (sha256:bcf39e4cdd9f455e26bbfae2517aa7c8db0f09163286925499b5c9d5f3378d30)
2020-11-08T08:00:37+00 [I] Found new marctv/minecraft-bedrock-server:latest image (sha256:bcf39e4cdd9f455e26bbfae2517aa7c8db0f09163286925499b5c9d5f3378d30)
2020-11-08T08:00:42+00 [I] Found new containrrr/watchtower:latest image sha256:aed26b939d83a57d3c57685f9a675b53b83910bc49a591b5ad9d005a2d9baa98)
2020-11-08T08:00:42+00 [I] Stopping /Minecraft_Creative_2 (cb53da810946fbe254394fc2f7b70b99091652fbe5185aa353562f19a3aa061e) with SIGTERM
2020-11-08T08:00:57+00 [I] Stopping /Minecraft_Creative_1 (a6bec0766e86d9b3d023d8c81d265a4f2521ec2d3318c7b74cb91f3093ccc3fe) with SIGTERM

panic: runtime error: invalid memory address or nil pointer dereference 
[signal SIGSEGV: segmentation violation code=0x1 addr=0xa0 pc=0x922a74]

goroutine 1 [running]:
github.com/containrrr/watchtower/pkg/container.Container.runtimeConfig(0x100,
0xc000416570, 0xc00040d600, 0x0)
	/home/circleci/repo/pkg/container/container.go:198 +0x74 github.com/containrrr/watchtower/pkg/container.dockerClient.StartContainer(0xc82d00,
0xc00036e600, 0x10001, 0x6f00000100, 0xc000416570, 0xc00040d600, 0xc000143d70, 0x20, 0x0, 0x0)
	/home/circleci/repo/pkg/container/client.go:167 +0x86 github.com/containrrr/watchtower/internal/actions.restartStaleContainer(0xc01e190100,
0xc000416570, 0xc00040d600, 0xc7c8a0, 0xc0003b80c0, 0xc00036b4a0, 0x1, 0x2540be400, 0x0)
	/home/circleci/repo/internal/actions/update.go:149 +0xdd github.com/containrrr/watchtower/internal/actions.restartContainersInSortedOrder(0xc00035e780,
0x6, 0x8, 0xc7c8a0, 0xc0003b80c0, 0xc00036b4a0, 0x1, 0x2540be400, 0x0)
	/home/circleci/repo/internal/actions/update.go:119 +0x166 github.com/containrrr/watchtower/internal/actions.Update(0xc7c8a0,
0xc0003b80c0, 0xc00036b4a0, 0x1, 0x2540be400, 0x0, 0xc00025fce0,
0x7576a5)
	/home/circleci/repo/internal/actions/update.go:61 +0x5eb
github.com/containrrr/watchtower/cmd.runUpdatesWithNotifications(0xc00036b4a0)
	/home/circleci/repo/cmd/root.go:229 +0xbe github.com/containrrr/watchtower/cmd.Run(0x10235c0, 0xc000353ef0, 0x0, 0x3)
	/home/circleci/repo/cmd/root.go:146 +0x15a github.com/spf13/cobra.(*Command).execute(0x10235c0, 0xc000108050, 0x3, 0x3, 0x10235c0, 0xc000108050)
	/go/pkg/mod/github.com/spf13/[email protected]/command.go:766 +0x2c2 github.com/spf13/cobra.(*Command).ExecuteC(0x10235c0, 0x406305, 0xc000062058, 0x0)
	/go/pkg/mod/github.com/spf13/[email protected]/command.go:852 +0x2fe
github.com/spf13/cobra.(*Command).Execute(...)
	/go/pkg/mod/github.com/spf13/[email protected]/command.go:800
github.com/containrrr/watchtower/cmd.Execute()
	/home/circleci/repo/cmd/root.go:57 +0x31
main.main()
	/home/circleci/repo/main.go:13 +0x25

[piksel]

@Bustout could you create a new issue, since it's another error. I suspect it might have to do with there being two identical containers, but I am not sure why it happens. The warning says that it failed to retrieve the image, that should cause it to skip the update since it cannot be performed:

watchtower/internal/actions/update.go

Line 31 in e735cc8

if stale && !params.NoRestart && !params.MonitorOnly && !targetContainer.HasImageInfo() {

[Bustout]

#677

[simskij]

#923 explains our current stance on Synology. With that said, however, we do have a fix in the latest release that we believe will solve this for most users (see #1183 for details)