Truncated UART packets when BREAK_ERROR occours

[hitecSmartHome]

Board

esp32 wrover

Device Description

psram, eth, flash etc..

Hardware Configuration

psram, eth, flash etc..

Version

latest master (checkout manually)

IDE Name

PlatformIO

Operating System

Windows 10

Flash frequency

80

PSRAM enabled

yes

Upload speed

115200

Description

I often get truncated packets on uart when accessing either the flash with LittleFS wrapper or when making an http call from client to esp32. The way I test this is that I get an interrupt from Serial1.onReceive() with 0x00 data in it when I save something to flash or when making an http request. http server uses the IDF server implementation as well as the arduino Serial wrapper.

I expect the onReceive function to call my cb only when the set timeout is triggered. It is UART_SYMBOL_TIMEOUT 1 in this case.
The problem is triggered every time when I write to flash. I can access menuconfig so I made sure that the UART functions and variables are in IRAM and interrupts too.

Sketch

void Modbus::init() {
    Serial1.setRxBufferSize(MAX_RX_BUFFER_SIZE);
    Serial1.setTxBufferSize(MAX_MBUS_DATA_LENGTH);
    Serial1.begin(MBUS_BAUD, SERIAL_8N1, MBUS_RX, MBUS_TX);
    Serial1.setPins(-1, -1, -1, MBUS_RTS);
    Serial1.setMode(UART_MODE_RS485_HALF_DUPLEX);
    Serial1.setRxTimeout(MBUS_RX_TIMEOUT);
    
    Serial1.onReceive(
        std::bind(&Modbus::handlePacket, this),
        PACKET_TRIGGER_ONLY_ON_TIMEOUT
    );
    Serial1.onReceiveError(
        std::bind(&Modbus::handleReceiveError, this, std::placeholders::_1)
    );
}

Debug Message

Debug messages coming from my API

E (39603) Modbus: CRC error in response packet: 0x00
E (39603) Modbus: Invalid packet. Can't process it.
Raw Packet: 00
E (39608) HardwareHandler: Packet error code: 1

Other Steps to Reproduce

Give repeated modbus packets to your esp, wait for them and access the flash meanwhile.

I have checked existing issues, online documentation and the Troubleshooting Guide

• I confirm I have checked existing issues, online documentation and Troubleshooting guide.

[SuGlider]

onReceive() will be called only when:
1- The UART FIFO is full.
2- When RX timeout occurs.

From the comments in https://github.com/espressif/arduino-esp32/blob/master/cores/esp32/HardwareSerial.h

  // onReceive will setup a callback that will be called whenever an UART interruption occurs (UART_INTR_RXFIFO_FULL or UART_INTR_RXFIFO_TOUT)
  // UART_INTR_RXFIFO_FULL interrupt triggers at UART_FULL_THRESH_DEFAULT bytes received (defined as 120 bytes by default in IDF)
  // UART_INTR_RXFIFO_TOUT interrupt triggers at UART_TOUT_THRESH_DEFAULT symbols passed without any reception (defined as 10 symbols by default in IDF)
  // onlyOnTimeout parameter will define how onReceive will behave:
  // Default: true -- The callback will only be called when RX Timeout happens.
  //                  Whole stream of bytes will be ready for being read on the callback function at once.
  //                  This option may lead to Rx Overflow depending on the Rx Buffer Size and number of bytes received in the streaming
  //         false -- The callback will be called when FIFO reaches 120 bytes and also on RX Timeout.
  //                  The stream of incommig bytes will be "split" into blocks of 120 bytes on each callback.
  //                  This option avoid any sort of Rx Overflow, but leaves the UART packet reassembling work to the Application.
  void onReceive(OnReceiveCb function, bool onlyOnTimeout = false);

What is the value of PACKET_TRIGGER_ONLY_ON_TIMEOUT?

[SuGlider]

Could it be a memory allocation issue?
Please provide a minimum sketch that can be used to confirm the issue and to debug it.

[hitecSmartHome]

PACKET_TRIGGER_ONLY_ON_TIMEOUT is true in this case. The fifo can not be full since I got messages only when asked and no message is bigger than the fifo

[hitecSmartHome]

These are standard modbus messages

[SuGlider]

OK. I can't relate UART interrupts to flash accessing / httpd task.
From your description, it sounds like this is an Arduino as IDF component project with specific IDF sdkconfig settings that may be very particular to this project.

Could you please provide a minimum sketch that can be built using Arduino Core standard sdkconfig that demonstrates such issue?

[SuGlider]

What is the value of MAX_RX_BUFFER_SIZE, MAX_MBUS_DATA_LENGTH and MBUS_BAUD?

[SuGlider]

OK. I can't relate UART interrupts to flash accessing / httpd task. From your description, it sounds like this is an Arduino as IDF component project with specific IDF sdkconfig settings that may be very particular to this project.

Could you please provide a minimum sketch that can be built using Arduino Core standard sdkconfig that demonstrates such issue?

Could it be an issue with flash access / httpd tasks?
Arduino setup()/loop() will run under the lowest FreeRTOS priority.
By other hand onReceive() related task will run under priority ARDUINO_SERIAL_EVENT_TASK_PRIORITY.
This can be set as a global #define for the whole project.

#ifndef ARDUINO_SERIAL_EVENT_TASK_PRIORITY
#define ARDUINO_SERIAL_EVENT_TASK_PRIORITY (configMAX_PRIORITIES - 1)
#endif

You can also change the ESP32-WROVER CPU that is running the UART event Loop with ARDUINO_SERIAL_EVENT_TASK_RUNNING_CORE

#ifndef ARDUINO_SERIAL_EVENT_TASK_RUNNING_CORE
#define ARDUINO_SERIAL_EVENT_TASK_RUNNING_CORE -1
#endif

Or change the Task Stack size with ARDUINO_SERIAL_EVENT_TASK_STACK_SIZE

#ifndef ARDUINO_SERIAL_EVENT_TASK_STACK_SIZE
#define ARDUINO_SERIAL_EVENT_TASK_STACK_SIZE 2048
#endif

[SuGlider]

About stack size... onReceive() callback will be called from a separated UART Event Task.
If the callback function overflows the Task stack, it can cause a similar issue.

[hitecSmartHome]

#define MBUS_BAUD 115200
#define PACKET_TRIGGER_ONLY_ON_TIMEOUT true
#define MBUS_RX_TIMEOUT 1
#define MAX_MBUS_DATA_LENGTH 256
#define MAX_RX_BUFFER_SIZE 80

• I don't have any arduino setup and loop. I have disabled them in the menuconfig.
• It is not so specific build, the latest arduino core and the latest idf.

This is from my sdkconfig.

CONFIG_ARDUINO_SERIAL_EVENT_RUN_NO_AFFINITY=y
CONFIG_ARDUINO_SERIAL_EVENT_TASK_RUNNING_CORE=-1
CONFIG_ARDUINO_SERIAL_EVENT_TASK_STACK_SIZE=4048
CONFIG_ARDUINO_SERIAL_EVENT_TASK_PRIORITY=24

I can increase the event stack size for testing and also the rx buffer size.

Could you please provide a minimum sketch that can be built using Arduino Core standard sdkconfig that demonstrates such issue?

I will create one for sure because I want to get to the bottom of this.

[hitecSmartHome]

Here is my whole sdkconfig: https://pastebin.com/w4wPRxDK

[SuGlider]

#define MAX_RX_BUFFER_SIZE 80
Serial1.setRxBufferSize(MAX_RX_BUFFER_SIZE);

It can't be lower than 129.
Please enable DEBUG level at least to INFO in order to check error and warning messages.

From HardwareSerial Source Code:

// minimum total RX Buffer size is the UART FIFO space (128 bytes for most SoC) + 1. IDF imposition.
size_t HardwareSerial::setRxBufferSize(size_t new_size) {

  if (_uart) {
    log_e("RX Buffer can't be resized when Serial is already running. Set it before calling begin().");
    return 0;
  }

  if (new_size <= SOC_UART_FIFO_LEN) {
    log_w("RX Buffer set to minimum value: %d.", SOC_UART_FIFO_LEN + 1);  // ESP32, S2, S3 and C3 means higher than 128
    new_size = SOC_UART_FIFO_LEN + 1;
  }

  _rxBufferSize = new_size;
  return _rxBufferSize;
}

[hitecSmartHome]

Well, I see that it changes the value back.

[hitecSmartHome]

Firstly

• Increased MAX_RX_BUFFER_SIZE to 256
• Increased CONFIG_ARDUINO_SERIAL_EVENT_TASK_STACK_SIZE to 8000
• Increased log level verbosity to DEBUG

I couldn't see shit so decreased debug back to info.
I could reproduce it without flash calls.

I (117321) System: Getting task info
E (117329) Modbus: CRC error in response packet: 0x00
E (117329) Modbus: Invalid packet. Can't process it.
Raw Packet: 00
E (117334) HardwareHandler: Packet error code: 1


E (117339) Modbus: CRC error in response packet: 0x14
E (117345) Modbus: Invalid packet. Can't process it.
Raw Packet: 14 00 00 ed 85
E (117353) HardwareHandler: Packet error code: 1

The packet also has missing bytes and it is splitted.
The task priority is default 19 and it is amongst the biggest priority tasks the system has.

uart_event_task is priority 24
wifi is 23
esp_timer is 22
sys_evt is 20
and arduino_events is 19

I can increase the arduino_events task further hovewer. ( I want to try these things before I go write a reproduction example because it will probably involve http requests. )

[hitecSmartHome]

For some strange reason, in the menuconfig under Arduino configuration the Serial Event task priority shows up as 24 but when I do a check with uxTaskGetSystemState(taskStatusArray, uxArraySize, &ulTotalRunTime); it reports the arduino_events priority as 19. Strange...

[hitecSmartHome]

Does Arduino Serial Event task the same as arduino_events ?

[SuGlider]

Does Arduino Serial Event task the same as arduino_events ?

No. It is a separated task and settings:

arduino-esp32/cores/esp32/HardwareSerial.cpp

Lines 14 to 24 in 0d5d50e

	#ifndef ARDUINO_SERIAL_EVENT_TASK_STACK_SIZE
	#define ARDUINO_SERIAL_EVENT_TASK_STACK_SIZE 2048
	#endif
	#ifndef ARDUINO_SERIAL_EVENT_TASK_PRIORITY
	#define ARDUINO_SERIAL_EVENT_TASK_PRIORITY (configMAX_PRIORITIES - 1)
	#endif
	#ifndef ARDUINO_SERIAL_EVENT_TASK_RUNNING_CORE
	#define ARDUINO_SERIAL_EVENT_TASK_RUNNING_CORE -1
	#endif

arduino-esp32/cores/esp32/HardwareSerial.cpp

Lines 123 to 132 in 0d5d50e

	void HardwareSerial::_createEventTask(void *args) {
	// Creating UART event Task
	xTaskCreateUniversal(
	_uartEventTask, "uart_event_task", ARDUINO_SERIAL_EVENT_TASK_STACK_SIZE, this, ARDUINO_SERIAL_EVENT_TASK_PRIORITY, &_eventTask,
	ARDUINO_SERIAL_EVENT_TASK_RUNNING_CORE
	);
	if (_eventTask == NULL) {
	log_e(" -- UART%d Event Task not Created!", _uart_nr);
	}
	}

[hitecSmartHome]

So the ARDUINO_SERIAL_EVENT_TASK name is uart_event_task.
In this case, the uart_event_task priority is 24 for me...

What does arduino_events task do? Like OTA and things like that?

[SuGlider]

What does arduino_events task do? Like OTA and things like that?

It is a general Event Loop that catches most of internal Arduino Core events.
Most Arduino Event Callback are based on it.

[SuGlider]

@hitecSmartHome - I have marked this issue as awaiting response because we need a minimum sketch that demonstrates the issue in order to move forward.

[hitecSmartHome]

Yeah, sorry Iam thinking and testing what could cause this. At first I tought that it is general flash access but it is triggered on other http requests too. I also put a lot of critical things into iram like flash functions and rtos functions as well as uart interrupts. I can definitely trigger it with http requests but iam not sure about what operation does it exactly. Sometimes it is easily triggered with some request other times the same request does nothing even with multiple tries.

[SuGlider]

We can keep this issue open for longer.
Take your time. Let us know about your progress.
Thanks.

[hitecSmartHome]

Thank you very much

[hitecSmartHome]

There is definietly something to do with the rx buffer size.
If I read a lot of bytes from the slaves, let's say 32x2 bytes + header and crc so like max 80 bytes it seems to overflow something.
I can see the packet second half as a separate pacage and it blows my data validator since the CRC is wrong for both half.

It doesn't matter how much I increase the RX_BUFFER_SIZE.
I have tried to call uart_flush_input(UART_NUM_1); on every packet so it should empty the fifo or reset the ring buffer so it can start over after every packet but it did not help.

[hitecSmartHome]

Can this be? I added 1ms wait after my callback called but before I read the serial data and it seems to work reliably. I can read huge amounts of data and I also uncommented uart_flush_input

Serial1.onReceive(
    std::bind(&Modbus::handlePacket, this),
    PACKET_TRIGGER_ONLY_ON_TIMEOUT
);

void Modbus::handlePacket() {
    delayMicroseconds(1000); // Added 1ms wait
    int available = Serial1.available();
    uint8_t rawPacket[available] = {0};
    int readBytes = Serial1.readBytes(rawPacket, available);
    //uart_flush_input(UART_NUM_1);
    if (!isPacketValid(rawPacket, readBytes)) {
        
        return;
    }
}

Further testing required...

[SuGlider]

The data packet is sent in a single flow?
What if the slave sends a "first half" of data, delays a bit and then sends the remaining "second half".
It would generate an "RX Timeout" after the first half, hence it would call onReceive for this "first half".

What if the RX Timeout is set to the maximum allowed by the protocol?
It could force the "onReceive callback" to be called only after the system is sure that no more "extra" data could arrive.

[SuGlider]

It may not be "so simple as that".
Looking into that, I just guess that the issue may be related to a BREAK detection and UART adding 0x00 to the FIFO.
BREAK also forces onReceive() to be called.

It will require a lot of data analysis and checking the logs.

[SuGlider]

@hitecSmartHome -

Also sometimes there is a 0x00 at the first byte of the data which we do not see with an oscilloscope. That 0x00 sometimes there even with this 1ms delay after an onRecive callback.

00 0e 03 01 00 00 1c 45 should be 0e 03 01 00 00 1c 45 ( address, func code, reg num, data first, data second, crc, crc ) registers are 16 bit and sending two 8 bit data. But that 0x00 is mistery still.

This is consistent with a BREAK received in the beginning of the packet transmission.
As side effect, it adds a 0x00 into FIFO when a BREAK is detected. This is the HW that adds it.

Unfortunatelly, disabling BREAK INT doesn't fix this extra 0x00.

[SuGlider]

Fixing it would require some sort of combination of detecting the BREAK and ignoring any "extra 0x00"...
Using the BREAK receive example:
1- when BREAK is received at the end of the packet:
(a) BREAK Error is detected and onReceiveError callback can see all the data + extra 0x00 at the end
(b) onReceive callback gets an extra 0x00 is added at the end
2- when BREAK is received at the beginning of the packet:
(a) BREAK Error is detected and onReceiveError callback no data
(b) onReceive callback gets an extra 0x00 is added at the beginning of the packet.

The output below considers a FIFO Full as 120 bytes.

================================
Test Case #1 BREAK at END
================================

Testing onReceive for receiving 26 bytes at 115200 baud, using RX FIFO Full = 120.
onReceive is called on both FIFO Full and RX Timeout events.
BREAK event will be sent at the END of the 26 bytes
[   678][V][HardwareSerial.cpp:276] _uartEventTask(): UART1 RX break.

-- onReceiveError [ERR#1:UART_BREAK_ERROR] 
-- onReceiveError:: There are 27 bytes available.
onReceive Callback:: There are 27 bytes available: {0x41='A' 0x42='B' 0x43='C' 0x44='D' 0x45='E' 0x46='F' 0x47='G' 0x48='H' 0x49='I' 0x4a='J' 0x4b='K' 0x4c='L' 0x4d='M' 0x4e='N' 0x4f='O' 0x50='P' 0x51='Q' 0x52='R' 0x53='S' 0x54='T' 0x55='U' 0x56='V' 0x57='W' 0x58='X' 0x59='Y' 0x5a='Z' 0x0=''}

Sent String: ABCDEFGHIJKLMNOPQRSTUVWXYZ

It has sent 26 bytes from Serial1 TX to Serial1 RX
onReceive() has read a total of 27 bytes


================================
Test Case #1 BREAK at BEGINNING
================================

Testing onReceive for receiving 26 bytes at 115200 baud, using RX FIFO Full = 120.
onReceive is called on both FIFO Full and RX Timeout events.
BREAK event will be sent at the BEGINNING of the 26 bytes
[   756][V][HardwareSerial.cpp:276] _uartEventTask(): UART1 RX break.

-- onReceiveError [ERR#1:UART_BREAK_ERROR] 
-- onReceiveError:: There are 0 bytes available.

Sent String: ABCDEFGHIJKLMNOPQRSTUVWXYZ
onReceive Callback:: There are 27 bytes available: {0x0='' 0x41='A' 0x42='B' 0x43='C' 0x44='D' 0x45='E' 0x46='F' 0x47='G' 0x48='H' 0x49='I' 0x4a='J' 0x4b='K' 0x4c='L' 0x4d='M' 0x4e='N' 0x4f='O' 0x50='P' 0x51='Q' 0x52='R' 0x53='S' 0x54='T' 0x55='U' 0x56='V' 0x57='W' 0x58='X' 0x59='Y' 0x5a='Z'}

It has sent 26 bytes from Serial1 TX to Serial1 RX
onReceive() has read a total of 27 bytes
========================
Finished!

[hitecSmartHome]

That is.... funny.
So I need to listen to a break error and expect a 0x00 somewhere in the packet...

Break is not an error in this case. It should be there after every message since there is a packet timeout.

But why don't i get this extra byte every single time? And why is the uart acting and placing random bytes into the buffer in my behalf? Even on cases where it is an error, ( like streams ) it should not alter the buffer...

[hitecSmartHome]

So In my testing

void Modbus::handleReceiveError(hardwareSerial_error_t error) {
    if (error == UART_NO_ERROR) { return; }
    ESP_LOGE(MBUS_DEBUG_TAG, "Packet error code: %d", error);
    ESP_LOGE(MBUS_DEBUG_TAG, "Packet error description: %s", uartErrorStrings[error]);
    if( error == UART_BREAK_ERROR ){ return; }
    translateUartError(error);
    callErrorCb(lastPacketError);
}

I get a BREAK_ERROR after every message.

E (376014) Modbus: Packet error code: 1
E (376018) Modbus: Packet error description: UART_BREAK_ERROR
E (376025) Modbus: Packet error code: 1
E (376029) Modbus: Packet error description: UART_BREAK_ERROR
E (376036) Modbus: Packet error code: 1
E (376040) Modbus: Packet error description: UART_BREAK_ERROR
E (376046) Modbus: Packet error code: 1
E (376051) Modbus: Packet error description: UART_BREAK_ERROR
E (376057) Modbus: Packet error code: 1
E (376061) Modbus: Packet error description: UART_BREAK_ERROR
E (376068) Modbus: Packet error code: 1
E (376072) Modbus: Packet error description: UART_BREAK_ERROR

This is consistent and comes after every packet and there are no 0x00 byte in my actual packets.
It does not split in half or anything.

[hitecSmartHome]

Previously I had this error handler like this

void Modbus::handleReceiveError(hardwareSerial_error_t error) {
    if (error == UART_NO_ERROR || error == UART_BREAK_ERROR) { return; }
    ESP_LOGE(MBUS_DEBUG_TAG, "Packet error code: %d", error);
    ESP_LOGE(MBUS_DEBUG_TAG, "Packet error description: %s", uartErrorStrings[error]);
    translateUartError(error);
    callErrorCb(lastPacketError);
}

So I ignored the BREAK_ERROR since there is a break after every message as it should be.

[hitecSmartHome]

What is the purpose of the onReceive callback if the framework forces me to do my own packet framing because it is unreliable and useless? UART_BREAK should not be an error in the case of UART_MODE_RS485_HALF_DUPLEX because it will fire no matter what and it should not alter my receive buffer. Now I have to implement a workaround into my packet handler to check for single 0 bytes and check the first and last byte of my packet and continue to ignore BREAK

[hitecSmartHome]

From this

void Modbus::handlePacket() {
    int available = Serial1.available();
    uint8_t rawPacket[available] = {0};
    int readBytes = Serial1.readBytes(rawPacket, available);
    if (!isPacketValid(rawPacket, readBytes)) {
        ESP_LOGE(MBUS_DEBUG_TAG, "Invalid packet. Can't process it.");
        utils.printRawPacket("MODBUS INVALID PACKET", rawPacket, readBytes);
        callErrorCb(rawPacket[0]);
        return;
    }
    parseScanPacket(rawPacket, readBytes);
    if (packetCallback && !isScanning) {
        packetCallback(rawPacket, readBytes);
    }
}

I can workaround the 0x00 issue like this

void Modbus::handlePacket() {
    int available = Serial1.available();
    if (available <= 0) { return; }

    uint8_t rawPacket[available] = {0};
    int readBytes = Serial1.readBytes(rawPacket, available);

    // Check and remove 0x00 at the start
    int startIdx = 0;
    if (rawPacket[0] == 0x00) {
        ESP_LOGW(MBUS_DEBUG_TAG, "Detected 0x00 at start - removing.");
        startIdx = 1; // Skip the first byte
    }

    // Check and remove 0x00 at the end
    int endIdx = readBytes;
    if (rawPacket[readBytes - 1] == 0x00) {
        ESP_LOGW(MBUS_DEBUG_TAG, "Detected 0x00 at end - removing.");
        endIdx = readBytes - 1; // Ignore the last byte
    }

    // Calculate the new length after trimming
    int trimmedLength = endIdx - startIdx;
    if (trimmedLength <= 0) {
        ESP_LOGW(MBUS_DEBUG_TAG, "Packet length is zero after trimming - ignoring packet.");
        return;
    }

    // Create a trimmed packet array if necessary
    uint8_t trimmedPacket[trimmedLength];
    memcpy(trimmedPacket, rawPacket + startIdx, trimmedLength);

    if (!isPacketValid(trimmedPacket, trimmedLength)) {
        ESP_LOGE(MBUS_DEBUG_TAG, "Invalid packet. Can't process it.");
        utils.printRawPacket("MODBUS INVALID PACKET", trimmedPacket, trimmedLength);
        callErrorCb(trimmedPacket[0]);
        return;
    }
    parseScanPacket(trimmedPacket, trimmedLength);
    if (packetCallback && !isScanning) {
        packetCallback(trimmedPacket, trimmedLength);
    }
}

This way I can detect single 0x00 or at the start or end but I still got truncated packets like this

E (11162) Modbus: CRC error in response packet: 0x03
CRC error raw packet: : 0x03 0x03 0x20 0x00 0x05 0x00 0x00 0x00 0x04 0x00 0x00 0x00 0x04 0x00 0x00 0x00 0x04 0x00 0x00 0x00 0x03 0x00 0x00 0x00 0x02 0x00 0x00 0x00 0x03 0x00 0x00 0x00 0x02 0x00 0x00 0xfe
E (11175) Modbus: Invalid packet. Can't process it.
MODBUS INVALID PACKET: 0x03 0x03 0x20 0x00 0x05 0x00 0x00 0x00 0x04 0x00 0x00 0x00 0x04 0x00 0x00 0x00 0x04 0x00 0x00 0x00 0x03 0x00 0x00 0x00 0x02 0x00 0x00 0x00 0x03 0x00 0x00 0x00 0x02 0x00 0x00 0xfe
E (11198) HardwareHandler: Packet error code: 1


E (11216) Modbus: CRC error in response packet: 0x89
CRC error raw packet: : 0x89
E (11219) Modbus: Invalid packet. Can't process it.
MODBUS INVALID PACKET: 0x89
E (11227) HardwareHandler: Packet error code: 1

The driver cuts the last byte of the frame and makes an other frame with it with an other callback...

[hitecSmartHome]

This frame does not even overflow the buffer. This is 37 bytes and cuts the last byte.

0x03 0x03 0x20 0x00 0x05 0x00 0x00 0x00 0x04 0x00 0x00 0x00 0x04 0x00 0x00 0x00 0x04 0x00 0x00 0x00 0x03 0x00 0x00 0x00 0x02 0x00 0x00 0x00 0x03 0x00 0x00 0x00 0x02 0x00 0x00 0xfe 0x89

[hitecSmartHome]

I must switch to interrupts handling and process the incoming bytes myself...

[SuGlider]

As you say, onReceive() may not fit the application needs.
It works based on how IDF and the HW works, as described in #10578 (comment)

It may be usefull withSerial1.setRxFIFOFull(1); in order to use an async callback for each single incomming byte.
Regarding the BREAK signal, this is how ESP32 HW works. If the BREAK is too long, it will add 0x00 to the UART RX FIFO.

Don't bother if BREAK is called error. This is just a denomination.

[hitecSmartHome]

Thank you for your response and your help.

As you say, onReceive() may not fit the application needs.

What application would need a truncated/clipped frame provider callback?

Regarding the BREAK signal, this is how ESP32 HW works. If the BREAK is too long, it will add 0x00 to the UART RX FIFO.

Why?

[SuGlider]

Anyway, regarding the initial topic of this issue, there is no relationship between HTTP / Flash Accessing Tasks and the truncated UART packets.

[SuGlider]

What application would need a truncated/clipped frame provider callback?

onReceive() is just a way to emulate an UART ISR (async callback). It can be configured to be used in different ways using Timeout or FIFO Full events.

Other possible application: an UART console.

Regarding the BREAK signal, this is how ESP32 HW works. If the BREAK is too long, it will add 0x00 to the UART RX FIFO.
Why?

This is a question for the Espressif Digital Design team... I have no answer.

[SuGlider]

@hitecSmartHome - thinking about the BREAK = received 0x00:

What if used Parity is ODD instead of EVEN?
0x00 ==> Start Bit + DATA + Parity Bit + STOPBIT.
BREAK is when ParityBit doesn't match the necessary STOPBIT which shall be HIGH when the Line is hold LOW.

But if Parity is ODD, Parity Bit should be 1 instead of 0.
Therefore you may get a BREAK associated to a PARITY error instead of BREAK error...
This may work in the sense of not adding 0x00 to FIFO.

Just guessing...

[SuGlider]

@hitecSmartHome - thinking about the BREAK = received 0x00:

What if used Parity is ODD instead of EVEN? 0x00 ==> Start Bit + DATA + Parity Bit + STOPBIT. BREAK is when ParityBit doesn't match the necessary STOPBIT which shall be HIGH when the Line is hold LOW.

But if Parity is ODD, Parity Bit should be 1 instead of 0. Therefore you may get a BREAK associated to a PARITY error instead of BREAK error... This may work in the sense of not adding 0x00 to FIFO.

Just guessing...

It doesn't help... I've just tested it using the OnReceiveError_BREAK_Demo.ino example code, but it still adds 0x00 to FIFO.
I have tried EVEN/ODD, 7 bits ... no solution.

Another possible way would be using some SoftwareSerial Library to deal with UART.
It is CPU intensive, but it may give the application a better control over bytes and timing.

Just a suggestion.

[SuGlider]

As you say, onReceive() may not fit the application needs. It works based on how IDF and the HW works, as described in #10578 (comment)

It may be usefull withSerial1.setRxFIFOFull(1); in order to use an async callback for each single incomming byte. Regarding the BREAK signal, this is how ESP32 HW works. If the BREAK is too long, it will add 0x00 to the UART RX FIFO.

Don't bother if BREAK is called error. This is just a denomination.

Well... as final testing: Serial1.setRxFIFOFull(1); will execute the callback on every single byte, but if the callback takes too long, it could be called when FIFO has more bytes waiting for processing. It means that a single byte will force a callback, but it doens't mean that there is just one byte waiting in FIFO, therefore, Serial1.available() may return higher than 1.

[hitecSmartHome]

Anyway, regarding the initial topic of this issue, there is no relationship between HTTP / Flash Accessing Tasks and the truncated UART packets.

Yes, that is true.

[hitecSmartHome]

Technically the single 0x00 byte "packet" callback happens because of the uart driver puts it when BREAK happens.
But the last byte clipping is an other issue it seems. I can workaround the 0x00 byte as I shown but I don't know about the last byte clip. Needs more testing

[hitecSmartHome]

just to see if I understand correctly:

• UART_BREAK happens if the flow of data stops for x time
• If UART_BREAK happens, the driver puts a 0x00 byte into the buffer, indicating a break
• This break signaling byte will show up at the first byte of the next packet if the data flow started right after a break but the timeout does not fired
• If the data flow does not start right away, I get an onReceive callback with only the 0x00 break byte since the timeout is fired.

[hitecSmartHome]

Well, interestingly the application wrote a file automatically meanwhile and I got this on the serial

I (10830070) DataBase: Writing json: /saves/users.json
W (10830075) Modbus: Detected single byte frame: 0x00. Ignoring it...
W (10830092) Modbus: Detected single byte frame: 0x00. Ignoring it...
W (10830103) Modbus: Detected single byte frame: 0x00. Ignoring it...
W (10830111) Modbus: Detected single byte frame: 0x00. Ignoring it...
W (10830125) Modbus: Detected single byte frame: 0x00. Ignoring it...
W (10830130) Modbus: Detected single byte frame: 0x00. Ignoring it...
W (10830132) Modbus: Detected single byte frame: 0x00. Ignoring it...
W (10830138) Modbus: Detected single byte frame: 0x00. Ignoring it...
W (10830146) Modbus: Detected single byte frame: 0x00. Ignoring it...
W (10830152) Modbus: Detected single byte frame: 0x00. Ignoring it...
W (10830171) Modbus: Detected single byte frame: 0x00. Ignoring it...
W (10830173) Modbus: Detected single byte frame: 0x00. Ignoring it...
W (10830179) Modbus: Detected single byte frame: 0x00. Ignoring it...
W (10830199) Modbus: Detected single byte frame: 0x00. Ignoring it...
W (10830201) Modbus: Detected single byte frame: 0x00. Ignoring it...
W (10830209) Modbus: Detected single byte frame: 0x00. Ignoring it...
W (10830217) Modbus: Detected single byte frame: 0x00. Ignoring it...
W (10830222) Modbus: Detected single byte frame: 0x00. Ignoring it...
W (10830228) Modbus: Detected single byte frame: 0x00. Ignoring it...
W (10830236) Modbus: Detected single byte frame: 0x00. Ignoring it...
W (10830242) Modbus: Detected single byte frame: 0x00. Ignoring it...
W (10830247) Modbus: Detected single byte frame: 0x00. Ignoring it...
W (10830253) Modbus: Detected single byte frame: 0x00. Ignoring it...
W (10830260) Modbus: Detected single byte frame: 0x00. Ignoring it...
W (10830267) Modbus: Detected single byte frame: 0x00. Ignoring it...
W (10830280) Modbus: Detected single byte frame: 0x00. Ignoring it...
W (10830294) Modbus: Detected single byte frame: 0x00. Ignoring it...
W (10830299) Modbus: Detected single byte frame: 0x00. Ignoring it...
W (10830301) Modbus: Detected single byte frame: 0x00. Ignoring it...
W (10830307) Modbus: Detected single byte frame: 0x00. Ignoring it...
W (10830315) Modbus: Detected single byte frame: 0x00. Ignoring it...
W (10830329) Modbus: Detected single byte frame: 0x00. Ignoring it...
W (10830341) Modbus: Detected single byte frame: 0x00. Ignoring it...
W (10830347) Modbus: Detected single byte frame: 0x00. Ignoring it...
W (10830349) Modbus: Detected single byte frame: 0x00. Ignoring it...
W (10830353) Modbus: Detected single byte frame: 0x00. Ignoring it...
W (10830360) Modbus: Detected single byte frame: 0x00. Ignoring it...
W (10830369) Modbus: Detected single byte frame: 0x00. Ignoring it...
W (10830374) Modbus: Detected single byte frame: 0x00. Ignoring it...
W (10830387) Modbus: Detected single byte frame: 0x00. Ignoring it...
W (10830389) Modbus: Detected single byte frame: 0x00. Ignoring it...
W (10830395) Modbus: Detected single byte frame: 0x00. Ignoring it...
W (10830402) Modbus: Detected single byte frame: 0x00. Ignoring it...
W (10830409) Modbus: Detected single byte frame: 0x00. Ignoring it...
W (10830417) Modbus: Detected single byte frame: 0x00. Ignoring it...
W (10830424) Modbus: Detected single byte frame: 0x00. Ignoring it...
W (10830436) Modbus: Detected single byte frame: 0x00. Ignoring it...
W (10830441) Modbus: Detected single byte frame: 0x00. Ignoring it...
W (10830445) Modbus: Detected single byte frame: 0x00. Ignoring it...
W (10830452) Modbus: Detected single byte frame: 0x00. Ignoring it...
W (10830467) Modbus: Detected single byte frame: 0x00. Ignoring it...
W (10830473) Modbus: Detected single byte frame: 0x00. Ignoring it...
W (10830477) Modbus: Detected single byte frame: 0x00. Ignoring it...
W (10830481) Modbus: Detected single byte frame: 0x00. Ignoring it...
W (10830487) Modbus: Detected single byte frame: 0x00. Ignoring it...
W (10830494) Modbus: Detected single byte frame: 0x00. Ignoring it...
W (10830504) Modbus: Detected single byte frame: 0x00. Ignoring it...
W (10830508) Modbus: Detected single byte frame: 0x00. Ignoring it...
W (10830516) Modbus: Detected single byte frame: 0x00. Ignoring it...
W (10830525) Modbus: Detected single byte frame: 0x00. Ignoring it...
W (10830575) Modbus: Detected single byte frame: 0x00. Ignoring it...
W (10830586) Modbus: Detected single byte frame: 0x00. Ignoring it...
W (10830602) Modbus: Detected single byte frame: 0x00. Ignoring it...
W (10830605) Modbus: Detected single byte frame: 0x00. Ignoring it...
W (10830611) Modbus: Detected single byte frame: 0x00. Ignoring it...
W (10830614) Modbus: Detected single byte frame: 0x00. Ignoring it...
W (10830621) Modbus: Detected single byte frame: 0x00. Ignoring it...
W (10830628) Modbus: Detected single byte frame: 0x00. Ignoring it...
W (10830635) Modbus: Detected single byte frame: 0x00. Ignoring it...
W (10830653) Modbus: Detected single byte frame: 0x00. Ignoring it...
W (10830656) Modbus: Detected single byte frame: 0x00. Ignoring it...
W (10830661) Modbus: Detected single byte frame: 0x00. Ignoring it...
W (10830665) Modbus: Detected single byte frame: 0x00. Ignoring it...
W (10830674) Modbus: Detected single byte frame: 0x00. Ignoring it...
W (10830693) Modbus: Detected single byte frame: 0x00. Ignoring it...
W (10830695) Modbus: Detected single byte frame: 0x00. Ignoring it...
W (10830706) Modbus: Detected single byte frame: 0x00. Ignoring it...
W (10830712) Modbus: Detected single byte frame: 0x00. Ignoring it...
W (10830716) Modbus: Detected single byte frame: 0x00. Ignoring it...
W (10830733) Modbus: Detected single byte frame: 0x00. Ignoring it...
W (10830741) Modbus: Detected single byte frame: 0x00. Ignoring it...
W (10830743) Modbus: Detected single byte frame: 0x00. Ignoring it...
W (10830749) Modbus: Detected single byte frame: 0x00. Ignoring it...
I (10830774) UserHandler: Saved 1 users to file: /saves/users.json

[hitecSmartHome]

As you can see, it started writing I (10830070) DataBase: Writing json: /saves/users.json -> bunch of UART BREAK BYTES coming in -> stopped writing to flash I (10830774) UserHandler: Saved 1 users to file: /saves/users.json -> no further errors

[hitecSmartHome]

god help us

[SuGlider]

This frame does not even overflow the buffer. This is 37 bytes and cuts the last byte.

0x03 0x03 0x20 0x00 0x05 0x00 0x00 0x00 0x04 0x00 0x00 0x00 0x04 0x00 0x00 0x00 0x04 0x00 0x00 0x00 0x03 0x00 0x00 0x00 0x02 0x00 0x00 0x00 0x03 0x00 0x00 0x00 0x02 0x00 0x00 0xfe 0x89

I have tested sending this sequence using the OnReceiveError_BREAK_Demo.ino example code.
I see no cut when FIFO Full is 120 bytes.

It adds the 0x00 because of the BREAK (at the end or beginning of the package), but that's all.

Could it be a problem with the Line? Bad impedance or parasitic capacitance?

[SuGlider]

just to see if I understand correctly:

• UART_BREAK happens if the flow of data stops for x time
• If UART_BREAK happens, the driver puts a 0x00 byte into the buffer, indicating a break
• This break signaling byte will show up at the first byte of the next packet if the data flow started right after a break but the timeout does not fired
• If the data flow does not start right away, I get an onReceive callback with only the 0x00 break byte since the timeout is fired.

Yes, correct!

[SuGlider]

This has been an excellent issue report.
I'll add that as a reference discussion later on.

[hitecSmartHome]

Could it be a problem with the Line? Bad impedance or parasitic capacitance?

Yes, it can be line problem or the slave itself. Iam testing it meanwhile and so far it occoures on the same slave over and over again. Requires further testing.

[hitecSmartHome]

But it seems flash write also triggers uart break. Hehe

[hitecSmartHome]

Will test this as well