Dart sshnpd hanging

[cpswan]

Describe the bug

Customer reports that an SSH session became unresponsive and they needed to reboot their device to regain access.

They've observed that every 5m or so the connection from sshnpd to its atServer fails and needs to be re-established:

Nov 18 09:15:27 router sshnpd[2350]: SEVERE|2024-11-18 09:15:27.684479|AtLookup|Exception in sending to server, Exception: Waited for 10000 millis. No response after 2024-11-18 09:15:17.672609
Nov 18 09:15:27 router sshnpd[2350]: SHOUT|2024-11-18 09:15:27.688202| sshnpd |connection lost
Nov 18 09:15:32 router sshnpd[2350]: INFO|2024-11-18 09:15:32.672239|AtLookup|Creating new connection
Nov 18 09:15:33 router sshnpd[2350]: INFO|2024-11-18 09:15:33.170159|AtLookup|New connection created OK
Nov 18 09:15:33 router sshnpd[2350]: SHOUT|2024-11-18 09:15:33.353140| sshnpd |connection available
Nov 18 09:20:42 router sshnpd[2350]: INFO|2024-11-18 09:20:42.681929|OutboundConnectionImpl|close(): calling socket.destroy() on connection to InternetAddress('35.224.23.101', IPv4):7454
Nov 18 09:20:42 router sshnpd[2350]: SEVERE|2024-11-18 09:20:42.691957|AtLookup|Exception in sending to server, Exception: Waited for 10000 millis. No response after 2024-11-18 09:20:32.673361
Nov 18 09:20:42 router sshnpd[2350]: SHOUT|2024-11-18 09:20:42.692465| sshnpd |connection lost
Nov 18 09:20:47 router sshnpd[2350]: INFO|2024-11-18 09:20:47.672539|AtLookup|Creating new connection
Nov 18 09:20:48 router sshnpd[2350]: INFO|2024-11-18 09:20:48.201517|AtLookup|New connection created OK
Nov 18 09:20:48 router sshnpd[2350]: SHOUT|2024-11-18 09:20:48.383361| sshnpd |connection available

Steps to reproduce

We need to create a test environment to reproduce.

Expected behavior

sshnpd works reliably

Additional context

Customer has been asked for more details of the device (e.g. RAM)

[cpswan]

Device is a Robustel EV8100 Elevator Voice Gateway

CPU: ARM Cortex-A7, 792MHz
RAM: 512 MB DDR3
Flash: 8 GB eMMC

[gkc]

@gkc and @cpswan are working with the customer to troubleshoot. No resolution as yet.

Story so far

• Tests to date
  • Upgraded to latest version 5.7.0 - problems persist
  • Ping test - no observed issues
  • iperf3 test for 16 minutes to elsewhere on the internet from a laptop connected to router - no observed issues
  • iperf3 test for 16 minutes from router itself - no observed issues
• Next steps
  • Retest with 5.7.0 and observe whether the srv process which was spawned is still running or not
    • This will tell us whether or not the o/s is killing the srv for some reason
  • Retest with sshnpd running on a computer connected to the router rather than on the router itself
    • This will tell us whether or not the problem is specific to running sshnpd on the router
  • Retest with the C versions of sshnpd and srv running on the router from pre-release c0.2.4
    • This will tell us whether or not the problem is specific to the Dart implementation