Dock Action Server Succeeds But Sometimes Does Not Send Completion To Client

[slowrunner]

How are you connecting to your Create 3?

Wi-Fi (ROS 2)

Computer(s) Model(s) and Operating System(s)

Ubuntu 22.04 in Docker over Raspberry Pi 5 PiOS Bookworm

Which version of ROS 2 is installed on your computer?

Humble

Which firmware version is installed on your robot?

H.2.3

Which RMW is your robot running?

FastRTPS

Does your robot have an assigned namespace? If so, please share.

No response

Is the robot connected to a network? If so, what is the network type?

Yes: 2.4GHz Home WiFi

Are there multiple Create 3 robots connected to your network?

No

Is multicast enabled?

Yes

What is the Adapter Board's USB/BLE Toggle currently switched to?

Bluetooth (default)

Describe your question.

Dock Action Server Succeeds But Fails To Send Completion To Client

My ROS 2 Humble wali_node.py is successfully controlling my Create3:

• Undocks at battery_state.percentage > 0.995
• Rotates to face dock "ready2dock" at battery < 17%
• Sends Dock action server goal at battery < 15%

But recently the Dock action server has been successfully docking the Create3, but failing to send the completion to the client. (Can't say if it is every time or just the last two times, but rotate_angle has never missed a result message):

pi@WaLiPi5:~/wali_pi5/c3ws $ tail -50 ../logs/life.log
2024-01-06 08:03|[logMaintenance.py.main]** Create3 would not undock, application restarted **
2024-01-06 08:05|wali_node.py| ** WaLI node started - Dock below 15% **
2024-01-06 08:07|wali_node.py| ** WaLI Undocking: success at battery 100% **
2024-01-06 08:12|[logMaintenance.py.main]** Manual Docking: success (assumed) **
2024-01-06 08:13|wali_node.py| ** WaLI Undocking: success at battery 100% **
2024-01-06 11:14|wali_node.py| ** WaLI Ready To Dock at battery 17% **
2024-01-06 11:22|wali_node.py| ** WaLI Docking: success at battery 14% **              <<--- HERE IT WORKED AS DESIGNED
2024-01-06 11:41|[logMaintenance.py.main]** Issued reset_pose **
2024-01-06 11:43|[logMaintenance.py.main]** Manual Undocking: success (assumed) **
2024-01-06 11:44|[logMaintenance.py.main]** Dead-Reckoning To Kitchen From Undocked **
2024-01-06 11:49|[logMaintenance.py.main]** Arrived at Kitchen **
2024-01-06 12:00|[logMaintenance.py.main]** Dead-Reckoning To Undocked position From Kitchen **
2024-01-06 12:02|[logMaintenance.py.main]** Arrived at Undocked Position **
2024-01-06 12:04|wali_node.py| ** WaLI Ready To Dock at battery 17% **
2024-01-06 12:08|[logMaintenance.py.main]** Manual Docking: success (assumed) **
2024-01-06 12:08|[logMaintenance.py.main]** Issued reset_pose **
2024-01-06 12:11|[logMaintenance.py.main]** Manual Undocking: success (assumed) **
2024-01-06 12:12|wali_node.py| ** WaLI Ready To Dock at battery 17% **
2024-01-06 12:15|[logMaintenance.py.main]** Manual Docking: success (assumed) **
2024-01-06 12:16|[logMaintenance.py.main]** Issued reset_pose **
2024-01-06 14:15|wali_node.py| ** WaLI Undocking: success at battery 100% **
2024-01-06 17:22|wali_node.py| ** WaLI Ready To Dock at battery 17% **
2024-01-06 17:44|wali_node.py| ** WaLI node started - Dock below 15% **              <<-- HERE missing, restarted client 
2024-01-06 19:30|wali_node.py| ** WaLI Undocking: success at battery 100% **
2024-01-06 22:38|wali_node.py| ** WaLI Ready To Dock at battery 17% **               <<-- HERE rotate_angle action server sent result successfully
2024-01-06 22:43|[logMaintenance.py.main]** Manual Docking: success (assumed) **
2024-01-06 22:43|[logMaintenance.py.main]** Issued reset_pose **
2024-01-06 23:00|[logMaintenance.py.main]** Manual Undocking: success (assumed) **
2024-01-06 23:00|[logMaintenance.py.main]** Dead-Reckoning To Kitchen From Undocked (and back) **
2024-01-06 23:02|[logMaintenance.py.main]** Arrived at Kitchen **
2024-01-06 23:03|[logMaintenance.py.main]** Dead-Reckoning Back To Undocked position From Kitchen **
2024-01-06 23:05|[logMaintenance.py.main]** Arrived Back at Undocked Position **
2024-01-06 23:28|wali_node.py| ** WaLI Ready To Dock at battery 17% **
2024-01-06 23:50|lifelogger.dEmain execution: 16.15                                             <<---- HERE missing docking success
2024-01-06 23:50|[logMaintenance.py.main]** WaLiPi5 is going to nap for 7.6 hours **
2024-01-07 07:28|------------ boot ------------
2024-01-07 07:28|wali_node.py| ** WaLI node started - Dock below 15% **
2024-01-07 07:29|wali_node.py| ** WaLI Undocking: success at battery 100% **
2024-01-07 10:25|lifelogger.dEmain execution: 2.95
2024-01-07 10:25|[logMaintenance.py.main]** rebooting to try lower volume ip_feedback **
2024-01-07 10:27|------------ boot ------------
2024-01-07 10:27|wali_node.py| ** WaLI node started - Dock below 15% **
2024-01-07 10:32|wali_node.py| ** WaLI Ready To Dock at battery 17% **
2024-01-07 11:15|lifelogger.dEmain execution: 0.80                                              <<--- HERE missing success status
2024-01-07 11:17|[logMaintenance.py.main]** WaLi Docking: success (no completion status at client) **    

Adding log from Create3:
ActionServerSuccessDidNotSendCompletion.txt

I posted a design question for this issue on robotics.stackexchange.com

[shamlian]

It is possible you are running into a bug we have fixed with H.2.4, which is about to be released; please give it a try and let us know. It is also possible that there's something else going on.

[slowrunner]

So far H.2.4 returned the first Dock goal result in three weeks. I'll keep an eye on it.

(Times are EST not UTC)

2024-01-09 11:37|wali_node.py| ** WaLI Ready To Dock at battery 17% **
2024-01-09 11:44|wali_node.py| ** WaLI Noticed Docking: success at battery 15% **  <--- GOAL RESULT WAS NOT RETURNED
2024-01-09 13:24|[logMaintenance.py.main]** Stopping Wali_Node to try H.2.4 Firmware **  <-- H.2.4 NEW F/W
2024-01-09 13:26|[logMaintenance.py.main]** WaLiPi5 is going to nap for 0.5 hours **
2024-01-09 13:27|lifelogger.dEmain execution: 51.00
2024-01-09 13:57|------------ boot ------------
2024-01-09 13:57|wali_node.py| ** WaLI node started - Undock:99% Rotate:18% Dock:15% **
2024-01-09 13:58|wali_node.py| ** WaLI Undocking: success at battery 100% **
2024-01-09 17:04|wali_node.py| ** WaLI Ready To Dock at battery 17% **
2024-01-09 17:11|wali_node.py| ** WaLI dock goal result - Docking: success at battery 14% **  <---- GOAL RESULT WAS RETURNED

[shamlian]

Thanks and especially for sharing your work and asking all these questions, it's really helpful to have you in the community and fun to see what you are building!

[slowrunner]

Wali has faithfully responded properly with dock goal result since updating to H.2.4 - 6 dock goal requests in 24 hours of testing. Closing this issue:

2024-01-09 17:11|wali_node.py| ** WaLI dock goal result - Docking: success at battery 14% **
2024-01-09 19:18|wali_node.py| ** WaLI dock goal result - Docking: success at battery 14% **
2024-01-10 00:38|wali_node.py| ** WaLI dock goal result - Docking: success at battery 14% **
2024-01-10 05:56|wali_node.py| ** WaLI dock goal result - Docking: success at battery 14% **
2024-01-10 12:31|wali_node.py| ** WaLI dock goal result - Docking: success at battery 14% **
2024-01-10 17:48|wali_node.py| ** WaLI dock goal result - Docking: success at battery 14% **