Random EdgeX services facing flaky startup issue "Failed to start HTTP server"

[sagargaikwad-eaton]

We are facing flaky EdgeX services startup issue with error Fatal startup error: 5: Failed to start HTTP server this error comes from MHD_start_daemon API which is standard libmicrohttpd library function
I have added libmicrohttpd logging mechanism which is recently merged by @iain-anderson once that issue got reproduced the error logs as msg="microhttpd error: Failed to bind to port 59798: Invalid argument
59798 is valid port number and service is running on 59789 port fine. but sometimes any random services are failing with error bind failure Invalid arguments.
so not able to get why bind failure is happening is this a system issue ?
Also I have observed some logs on my board which is related to systemd which is as below
systemd-networkd[317]: eth0: Lost carrier Jan 02 05:28:15 BL-Edge-Gateway-42-82-F4 kernel: stm32-dwmac 5800a000.ethernet eth0: Link is Down

[iain-anderson]

The message "Invalid argument" is coming back from the attempted bind() call in libmicrohttpd. On some O/S this might be a more descriptive error, like "Cannot assign requested address" or "Address already in use".
If you have LogLevel set to Info or above, there should be a log message indicating what interface and port is being used. At Debug level we additionally log the resolution of the interface name (unless the service is configured for all interfaces).
Assuming there aren't other services running that might use the required port, it could be that systemd is deconfiguring the interface when the link goes down?

[sagargaikwad-eaton]

@iain-anderson We have added retry mechanism which will reattempts to start http server but its also getting failed can be seen in below logs ..
NOTE : network-manager here is our internal service and this issue can randomly occur in any other service

30:09 BL-Edge-Gateway-12-34-02 network-manager[2606]: level=INFO ts=2023-02-23T10:30:09Z msg="Starting HTTP server on interface 127.0.0.1, port 59798"
Feb 23 05:30:09 BL-Edge-Gateway-12-34-02 network-manager[2606]: level=ERROR ts=2023-02-23T10:30:09Z msg="microhttpd error: Failed to bind to port 59798: Invalid argument
Feb 23 05:30:09 BL-Edge-Gateway-12-34-02 network-manager[2606]: "
Feb 23 05:30:14 BL-Edge-Gateway-12-34-02 network-manager[2606]: level=INFO ts=2023-02-23T10:30:14Z msg="Reattempting to start HTTP server on interface 127.0.0.1, port 59798"
Feb 23 05:30:14 BL-Edge-Gateway-12-34-02 network-manager[2606]: level=ERROR ts=2023-02-23T10:30:14Z msg="microhttpd error: Failed to bind to port 59798: Invalid argument
Feb 23 05:30:14 BL-Edge-Gateway-12-34-02 network-manager[2606]: "
Feb 23 05:30:19 BL-Edge-Gateway-12-34-02 network-manager[2599]: Port is not yet open
Feb 23 05:30:19 BL-Edge-Gateway-12-34-02 network-manager[2606]: level=INFO ts=2023-02-23T10:30:19Z msg="Reattempting to start HTTP server on interface 127.0.0.1, port 59798"
Feb 23 05:30:19 BL-Edge-Gateway-12-34-02 network-manager[2606]: level=ERROR ts=2023-02-23T10:30:19Z msg="microhttpd error: Failed to bind to port 59798: Invalid argument
Feb 23 05:30:19 BL-Edge-Gateway-12-34-02 network-manager[2606]: "
Feb 23 05:30:19 BL-Edge-Gateway-12-34-02 network-manager[2606]: level=ERROR ts=2023-02-23T10:30:19Z msg="Unable to start HTTP server"
Feb 23 05:30:19 BL-Edge-Gateway-12-34-02 network-manager[2606]: Fatal Startup Error: 5: Failed to start HTTP server
Feb 23 05:30:19 BL-Edge-Gateway-12-34-02 network-manager[2606]: level=INFO ts=2023-02-23T10:30:19Z msg="isSubscribe not set so no need of unsubscribing from data events"
Feb 23 05:30:22 BL-Edge-Gateway-12-34-02 network-manager[2599]: /usr/bin/network-manager exited already, failing
Feb 23 05:30:22 BL-Edge-Gateway-12-34-02 systemd[1]: network-manager.service: Killing process 2606 (n/a) with signal SIGKILL.
Feb 23 05:30:22 BL-Edge-Gateway-12-34-02 systemd[1]: network-manager.service: Failed with result 'protocol'.
Feb 23 05:30:22 BL-Edge-Gateway-12-34-02 systemd[1]: Failed to start Eaton EdgeX Network Manager Service

Here we are using localhost 127.0.0.1 and port 59798 while Port is not yet open this message coming from out internal script but its not related to bind failure .
so what could be cause of bind failure ? if it's system issue then what could be the possible reasons or component which are responsible for this