Destroyed AsyncClient object is still used and causes a crash #17
Comments
|
I got the problem, see following trace with additional detail: void AsyncClient::_notifyWrittenBuffers(std::deque<notify_writebuf> & notifyqueue, int write_errno)
{
while (notifyqueue.size() > 0) {
if (notifyqueue.front().length > 0 && _sent_cb) {
_sent_cb(_sent_cb_arg, this, notifyqueue.front().length, notifyqueue.front().delay);
}
notifyqueue.pop_front();
}
if (write_errno != 0) _error(write_errno);
}
What is your suggestion how to solve it? |
|
Do you have a particular connection sequence that reliably triggers this crash? For my own projects, I use branch yuboxfixes-0xFEEDC0DE64-cleanup from the fork at https://github.com/yubox-node-org/ESPAsyncWebServer . It is not clear which particular fork and branch you are using to reproduce this crash. Could you please try switching to yuboxfixes-0xFEEDC0DE64-cleanup and attempt to reproduce? |
|
Yes, I can reproduce it with 90% probability.
I will try it with your variant of the webserver and come back. |
|
I can not reproduce it with your variant of the webserver. I think I will jump on yours for my project. :-) What I figured out so far is in my problem use case, the WebResponse is closed too fast (state = RESPONSE_END). size_t AsyncAbstractResponse::_ack(AsyncWebServerRequest *request, size_t len, uint32_t time)
...
if(!_sendContentLength || _ackedLength >= _writtenLength){
_state = RESPONSE_END;
if(!_chunked && !_sendContentLength)
request->client()->close(true);
}
...Too fast means, the response is written back with two write() operations, because of less space. What I am not sure whether its a bug, but it solves the problem as well, is removing the What's your opinion? I am not so deep familiar here. |
|
I'm using https://github.com/yubox-node-org/ESPAsyncWebServer as well and the exact same crash happening to me too. |
|
I can reproduce even if i comment out |
Could you please build a minimal sketch that will reproduce the issue and post it here. I have been using the libraries for months on my own projects without issue.
This explanation is suspect anyway. The real question is why would the code enter the |
|
I will definitely make one because iam struggling with this async server for months now. I even got a said perfect fork of both of the libs from github somewhere else and I still got crashes. Unfortunately Iam not able to make my own fork since I don't understand that so deeply. My tests will include several front end files and it's gonna crash |
|
Here is the test repo: https://github.com/zekageri/AsyncServerTest/tree/main I connected 3 clients. Two from a desktop and one from a mobile. I'm testing large payloads as well as frequent messages for now. |
I am currently unable to reproduce the crash in my setup with your code.
Is there something I should be doing differently? |
|
@zekageri I noticed you appear to be using ESP32-S2, so I tried replicating the test there. However, my ESP32-S2 keeps getting a different IP every time I reboot. |
|
I forced a static IP with WiFi.config(), then repeated the test on the ESP32-S2 with the same methodology. Again no crash after repeated reboots. |
|
Iam using wrom in my tests and not s2. In my real project iam using wrover with psram. So far I can not reproduce the same crash with the wrom. Will try with psram. Iam really confused as in my real project every clue is pointing to the Async lib. Will test further. Thank you for your tests. I really appreciate your help |
|
I can't belive this. Meanwhile in my big project it just happened again in front of my eyes. assert failed: xQueueSemaphoreTake queue.c:1545 (( pxQueue ))
Backtrace: 0x40083d69:0x3ffd9820 0x4008eeb1:0x3ffd9840 0x400948b5:0x3ffd9860 0x4008fec1:0x3ffd9990 0x401a7b0b:0x3ffd99d0 0x401a7c6c:0x3ffd99f0 0x401a7db3:0x3ffd9a20 0x401a80c5:0x3ffd9a50 0x401a766b:0x3ffd9ab0
#0 0x40083d69:0x3ffd9820 in panic_abort at /Users/ficeto/Desktop/ESP32/ESP32S2/esp-idf-public/components/esp_system/panic.c:402
#1 0x4008eeb1:0x3ffd9840 in esp_system_abort at /Users/ficeto/Desktop/ESP32/ESP32S2/esp-idf-public/components/esp_system/esp_system.c:128
#2 0x400948b5:0x3ffd9860 in __assert_func at /Users/ficeto/Desktop/ESP32/ESP32S2/esp-idf-public/components/newlib/assert.c:85
#3 0x4008fec1:0x3ffd9990 in xQueueSemaphoreTake at /Users/ficeto/Desktop/ESP32/ESP32S2/esp-idf-public/components/freertos/queue.c:1549 (discriminator 1)
#4 0x401a7b0b:0x3ffd99d0 in AsyncClient::_clearWriteQueue() at lib/AsyncTCPSock/src/AsyncTCP.cpp:1077
#5 0x401a7c6c:0x3ffd99f0 in AsyncClient::_error(signed char) at lib/AsyncTCPSock/src/AsyncTCP.cpp:998
#6 0x401a7db3:0x3ffd9a20 in AsyncClient::_notifyWrittenBuffers(std::deque<AsyncClient::notify_writebuf, std::allocator<AsyncClient::notify_writebuf> >&, int) at lib/AsyncTCPSock/src/AsyncTCP.cpp:846 (discriminator 1)
#7 0x401a80c5:0x3ffd9a50 in AsyncClient::_sockIsWriteable() at lib/AsyncTCPSock/src/AsyncTCP.cpp:733
#8 0x401a766b:0x3ffd9ab0 in _asynctcpsock_task(void*) at lib/AsyncTCPSock/src/AsyncTCP.cpp:141 (discriminator 1)
ELF file SHA256: 83667e7d6bcd3ad4
Rebooting... |
|
Three client refresh at the same time. |
|
I have created a task which periodically sends a message to all clients. void setup() {
Serial.begin(115200);
if (!LittleFS.begin()) {
Serial.println("Failed to mount file system!");
ESP.restart();
}
setupWifi();
setupEndpoints();
socketHandler.init(&server, "/main");
server.serveStatic("/", LittleFS, "/");
server.begin();
xTaskCreateUniversal(testTask, "testTask", 2000, NULL, 2, NULL, -1);
}
void loop() {
socketHandler.run();
}
void testTask(void* parameter){
int core = xPortGetCoreID();
for(;;){
socketHandler.sendAll("Ping from core %d",core);
vTaskDelay(1000);
}
}And i was able to trigger the error. I just refreshed 3 clients web page. 2 out of 3 was not able to load every asset and eventually the esp died. Error without backtrace for now |
|
For some reason if the clients disconnected, and try to reconnect this happens 18 client connected.
There are 3 client(s).
19 client connected.
There are 4 client(s).
20 client connected.
There are 5 client(s).
21 client connected.
There are 6 client(s).
22 client connected.
There are 7 client(s).
23 client connected.
There are 8 client(s).
[1469677][E][AsyncTCP.cpp:797] _flushWriteQueue(): socket 50 lwip_write() failed errno=104
assert failed: xQueueSemaphoreTake queue.c:1545 (( pxQueue ))
Backtrace: 0x400836b9:0x3ffd87e0 0x4008b7a9:0x3ffd8800 0x40090db9:0x3ffd8820 0x4008c7b9:0x3ffd8950 0x4014cb03:0x3ffd8990 0x4014cc4a:0x3ffd89b0 0x4014cd72:0x3ffd89e0 0x4014d052:0x3ffd8a10 0x4014c4cf:0x3ffd8a70
#0 0x400836b9:0x3ffd87e0 in panic_abort at /Users/ficeto/Desktop/ESP32/ESP32S2/esp-idf-public/components/esp_system/panic.c:402
#1 0x4008b7a9:0x3ffd8800 in esp_system_abort at /Users/ficeto/Desktop/ESP32/ESP32S2/esp-idf-public/components/esp_system/esp_system.c:128
#2 0x40090db9:0x3ffd8820 in __assert_func at /Users/ficeto/Desktop/ESP32/ESP32S2/esp-idf-public/components/newlib/assert.c:85
#3 0x4008c7b9:0x3ffd8950 in xQueueSemaphoreTake at /Users/ficeto/Desktop/ESP32/ESP32S2/esp-idf-public/components/freertos/queue.c:1549 (discriminator 1)
#4 0x4014cb03:0x3ffd8990 in AsyncClient::_clearWriteQueue() at lib/AsyncTCPSock-master/src/AsyncTCP.cpp:1076
#5 0x4014cc4a:0x3ffd89b0 in AsyncClient::_error(signed char) at lib/AsyncTCPSock-master/src/AsyncTCP.cpp:997
#6 0x4014cd72:0x3ffd89e0 in AsyncClient::_notifyWrittenBuffers(std::deque<AsyncClient::notify_writebuf, std::allocator<AsyncClient::notify_writebuf> >&, int) at lib/AsyncTCPSock-master/src/AsyncTCP.cpp:845 (discriminator 1)
#7 0x4014d052:0x3ffd8a10 in AsyncClient::_sockIsWriteable() at lib/AsyncTCPSock-master/src/AsyncTCP.cpp:732
#8 0x4014c4cf:0x3ffd8a70 in _asynctcpsock_task(void*) at lib/AsyncTCPSock-master/src/AsyncTCP.cpp:141
ELF file SHA256: a9e3b59c17e742c9
Rebooting...
ets Jun 8 2016 00:22:57
rst:0xc (SW_CPU_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:1
load:0x3fff0030,len:1344
load:0x40078000,len:13836
load:0x40080400,len:3608
entry 0x400805f0
Wifi connected
IP: 192.168.10.114
1 client connected.
There are 0 client(s).
2 client connected.
There are 1 client(s).
3 client connected.
There are 2 client(s).
4 client connected.
There are 3 client(s).
5 client connected.
There are 4 client(s).
6 client connected.
There are 5 client(s).
7 client connected.
There are 6 client(s).
8 client connected.
There are 7 client(s).
9 client connected.
There are 8 client(s).
Client disconnected
There are 8 client(s).
10 client connected.
There are 8 client(s).
Client disconnected
There are 8 client(s).
11 client connected.
There are 8 client(s).
[ 4137][E][AsyncTCP.cpp:797] _flushWriteQueue(): socket 51 lwip_write() failed errno=104
[ 4137][E][AsyncTCP.cpp:797] _flushWriteQueue(): socket 51 lwip_write() failed errno=128
[ 4142][E][AsyncTCP.cpp:797] _flushWriteQueue(): socket 51 lwip_write() failed errno=128
[ 4150][E][AsyncTCP.cpp:797] _flushWriteQueue(): socket 51 lwip_write() failed errno=128
[ 4157][E][AsyncTCP.cpp:797] _flushWriteQueue(): socket 51 lwip_write() failed errno=128
[ 4165][E][AsyncTCP.cpp:797] _flushWriteQueue(): socket 51 lwip_write() failed errno=128
[ 4173][E][AsyncTCP.cpp:797] _flushWriteQueue(): socket 51 lwip_write() failed errno=128
[ 4181][E][AsyncTCP.cpp:797] _flushWriteQueue(): socket 51 lwip_write() failed errno=128
[ 4189][E][AsyncTCP.cpp:797] _flushWriteQueue(): socket 51 lwip_write() failed errno=128
[ 4197][E][AsyncTCP.cpp:797] _flushWriteQueue(): socket 51 lwip_write() failed errno=128
[ 4205][E][AsyncTCP.cpp:797] _flushWriteQueue(): socket 51 lwip_write() failed errno=128
[ 4213][E][AsyncTCP.cpp:797] _flushWriteQueue(): socket 51 lwip_write() failed errno=128
[ 4221][E][AsyncTCP.cpp:797] _flushWriteQueue(): socket 51 lwip_write() failed errno=128
[ 4228][E][AsyncTCP.cpp:797] _flushWriteQueue(): socket 51 lwip_write() failed errno=128
[ 4236][E][AsyncTCP.cpp:797] _flushWriteQueue(): socket 51 lwip_write() failed errno=128
[ 4245][E][AsyncTCP.cpp:797] _flushWriteQueue(): socket 51 lwip_write() failed errno=128
[ 4252][E][AsyncTCP.cpp:797] _flushWriteQueue(): socket 51 lwip_write() failed errno=128
[ 4260][E][AsyncTCP.cpp:797] _flushWriteQueue(): socket 51 lwip_write() failed errno=128
[ 4268][E][AsyncTCP.cpp:797] _flushWriteQueue(): socket 51 lwip_write() failed errno=128
[ 4276][E][AsyncTCP.cpp:797] _flushWriteQueue(): socket 51 lwip_write() failed errno=128
[ 4284][E][AsyncTCP.cpp:797] _flushWriteQueue(): socket 51 lwip_write() failed errno=128
[ 4292][E][AsyncTCP.cpp:797] _flushWriteQueue(): socket 51 lwip_write() failed errno=128
[ 4300][E][AsyncTCP.cpp:797] _flushWriteQueue(): socket 51 lwip_write() failed errno=128
[ 4307][E][AsyncTCP.cpp:797] _flushWriteQueue(): socket 51 lwip_write() failed errno=128
[ 4316][E][AsyncTCP.cpp:797] _flushWriteQueue(): socket 51 lwip_write() failed errno=128
[ 4324][E][AsyncTCP.cpp:797] _flushWriteQueue(): socket 51 lwip_write() failed errno=128
[ 4332][E][AsyncTCP.cpp:797] _flushWriteQueue(): socket 51 lwip_write() failed errno=128
[ 4339][E][AsyncTCP.cpp:797] _flushWriteQueue(): socket 51 lwip_write() failed errno=128
[ 4347][E][AsyncTCP.cpp:797] _flushWriteQueue(): socket 51 lwip_write() failed errno=128
[ 4355][E][AsyncTCP.cpp:797] _flushWriteQueue(): socket 51 lwip_write() failed errno=128
[ 4363][E][AsyncTCP.cpp:797] _flushWriteQueue(): socket 51 lwip_write() failed errno=128
[ 4371][E][AsyncTCP.cpp:797] _flushWriteQueue(): socket 51 lwip_write() failed errno=128
[ 4379][E][AsyncTCP.cpp:797] _flushWriteQueue(): socket 51 lwip_write() failed errno=128
[ 4386][E][AsyncTCP.cpp:797] _flushWriteQueue(): socket 51 lwip_write() failed errno=128
[ 4394][E][AsyncTCP.cpp:797] _flushWriteQueue(): socket 51 lwip_write() failed errno=128
[ 4402][E][AsyncTCP.cpp:797] _flushWriteQueue(): socket 51 lwip_write() failed errno=128
[ 4410][E][AsyncTCP.cpp:797] _flushWriteQueue(): socket 51 lwip_write() failed errno=128
Client disconnected
assert failed: xQueueSemaphoreTake queue.c:1545 (( pxQueue ))
Backtrace: 0x400836b9:0x3ffd8610 0x4008b7a9:0x3ffd8630 0x40090db9:0x3ffd8650 0x4008c7b9:0x3ffd8780 0x4014cb03:0x3ffd87c0 0x4014cc4a:0x3ffd87e0 0x4014cd72:0x3ffd8810 0x4014d052:0x3ffd8840 0x4014c4cf:0x3ffd88a0
#0 0x400836b9:0x3ffd8610 in panic_abort at /Users/ficeto/Desktop/ESP32/ESP32S2/esp-idf-public/components/esp_system/panic.c:402
#1 0x4008b7a9:0x3ffd8630 in esp_system_abort at /Users/ficeto/Desktop/ESP32/ESP32S2/esp-idf-public/components/esp_system/esp_system.c:128
#2 0x40090db9:0x3ffd8650 in __assert_func at /Users/ficeto/Desktop/ESP32/ESP32S2/esp-idf-public/components/newlib/assert.c:85
#3 0x4008c7b9:0x3ffd8780 in xQueueSemaphoreTake at /Users/ficeto/Desktop/ESP32/ESP32S2/esp-idf-public/components/freertos/queue.c:1549 (discriminator 1)
#4 0x4014cb03:0x3ffd87c0 in AsyncClient::_clearWriteQueue() at lib/AsyncTCPSock-master/src/AsyncTCP.cpp:1076
#5 0x4014cc4a:0x3ffd87e0 in AsyncClient::_error(signed char) at lib/AsyncTCPSock-master/src/AsyncTCP.cpp:997
#6 0x4014cd72:0x3ffd8810 in AsyncClient::_notifyWrittenBuffers(std::deque<AsyncClient::notify_writebuf, std::allocator<AsyncClient::notify_writebuf> >&, int) at lib/AsyncTCPSock-master/src/AsyncTCP.cpp:845 (discriminator 1)
#7 0x4014d052:0x3ffd8840 in AsyncClient::_sockIsWriteable() at lib/AsyncTCPSock-master/src/AsyncTCP.cpp:732
#8 0x4014c4cf:0x3ffd88a0 in _asynctcpsock_task(void*) at lib/AsyncTCPSock-master/src/AsyncTCP.cpp:141
ELF file SHA256: a9e3b59c17e742c9
Rebooting...You must leave it for a while and let the clients disconnect. After like 20 min you click into the pages one by one, they will try to reconnect and this happens. |
|
It seems to me that the lib does not prevent additional clients to join if the count is bigger then the max? |
The shown log is in combination with ESPAsyncWebServer, which destroys the AsyncClient always after getting a disconnect indication via callback: https://github.com/me-no-dev/ESPAsyncWebServer/blob/f71e3d427b5be9791a8a2c93cf8079792c3a9a26/src/WebRequest.cpp#L74
The log shows that after _close() the destructor is called. Anyway later the _error() is called, which causes the crash.
The text was updated successfully, but these errors were encountered: