HostManager failed to remove container

[mhasself]

Odd problem today (finally resolved at 2024-01-18 14:30:06 UTC) when ACU agent was misbehaving, and hostmanager "down" could not kill it (twice, though first attempt was brief). HM logs show:

2024-01-18T14:23:51+0000 start called for update
2024-01-18T14:23:51+0000 update:140 Status is now "starting".
2024-01-18T14:23:51+0000 update:140 Status is now "running".
2024-01-18T14:23:51+0000 update:140 Update requested.
2024-01-18T14:23:51+0000 update:140 Status is now "done".
2024-01-18T14:23:52+0000 manager:0 Requesting termination of ACUAgent:acu
2024-01-18T14:23:57+0000 manager:0 Agent instance ACUAgent:acu refused to die.
2024-01-18T14:23:58+0000 manager:0 Detected unexpected session for ACUAgent:acu (probably docker); it will ...
2024-01-18T14:24:03+0000 manager:0 Requesting termination of ACUAgent:acu
2024-01-18T14:24:08+0000 manager:0 Agent instance ACUAgent:acu refused to die.
2024-01-18T14:24:09+0000 manager:0 Detected unexpected session for ACUAgent:acu (probably docker); it will ...
2024-01-18T14:24:13+0000 manager:0 Requesting termination of ACUAgent:acu
2024-01-18T14:24:19+0000 manager:0 Agent instance ACUAgent:acu refused to die.
...

But running docker-compose rm --stop --force ocs-acu, as ocs user, did bring down the container, without any issues. (That command should be exactly what the HM runs.)

I have no useful analysis yet...

[BrianJKoopman]

This happened again today on satp2 with a Lakeshore372 agent. A normal docker stop <container> run manually stopped the container. After that I was able to start with the HM, then subsequent "down" commands from HM worked fine.

EDIT: I should also note that the 372 agent was in a semi-crashed state, having had a broken pipe error in the main acq process.

[BrianJKoopman]

This problem continues to occur. It'd be great to figure out a way to reproduce this state. Recent occurrences I've seen have coincided with the agent's loss of connection to its device. For instance, when the cryomech agent had a broken pipe to the compressor, or the 372 agent to the 372.