I2C Fail 3.2.0

[xblack0]

Board

ESP32 - WROOM

Device Description

Custom ESE32 Wroom .

Hardware Configuration

GPIO 33,32, Wire library.

Version

v3.2.0

IDE Name

ARD

Operating System

Windows 11

Flash frequency

40

PSRAM enabled

yes

Upload speed

921600

Description

Compare to 3.1.3

3.2.0 Reported with 2 issues :
1-Executing i2c call time increase.
2-Deadlock [I2C hardware NACK detected] .

Sketch

Board Package from 3.1.3 To 3.2.0

Debug Message

I2C hardware NACK detected.

Dead Lock

Other Steps to Reproduce

No response

I have checked existing issues, online documentation and the Troubleshooting Guide

• I confirm I have checked existing issues, online documentation and Troubleshooting guide.

[enjoyneering]

I also see strange behavior on the I2c bus. Serial debug is disabled, but I get the message - i2c.master: i2c transaction timeout detected. Full log:

E (1124) i2c.master: I2C transaction timeout detected
E (1125) i2c.master: probe device timeout. Please check if xfer_timeout_ms and pull-ups are correctly set up
E (1138) i2c.master: I2C hardware NACK detected
E (1139) i2c.master: I2C transaction unexpected nack detected
E (1144) i2c.master: s_i2c_synchronous_transaction(924): I2C transaction failed
E (1151) i2c.master: i2c_master_multi_buffer_transmit(1186): I2C transaction failed
E (1161) i2c.master: I2C hardware timeout detected
E (1163) i2c.master: s_i2c_synchronous_transaction(924): I2C transaction failed
E (1170) i2c.master: i2c_master_multi_buffer_transmit(1186): I2C transaction failed
E (1180) i2c.master: I2C hardware timeout detected
E (1182) i2c.master: s_i2c_synchronous_transaction(924): I2C transaction failed
E (1189) i2c.master: i2c_master_multi_buffer_transmit(1186): I2C transaction failed
E (1203) i2c.master: I2C hardware timeout detected
E (1203) i2c.master: s_i2c_synchronous_transaction(924): I2C transaction failed
E (1208) i2c.master: i2c_master_multi_buffer_transmit(1186): I2C transaction failed
E (1217) i2c.master: I2C hardware timeout detected
E (1219) i2c.master: s_i2c_synchronous_transaction(924): I2C transaction failed
E (1226) i2c.master: i2c_master_multi_buffer_transmit(1186): I2C transaction failed
E (1236) i2c.master: I2C hardware timeout detected
E (1238) i2c.master: s_i2c_synchronous_transaction(924): I2C transaction failed
E (1245) i2c.master: i2c_master_multi_buffer_transmit(1186): I2C transaction failed

[me-no-dev]

1. The underlying ESP-IDF I2C driver has been changed
2. The information provided here is not enough to test/confirm any issue

Pleas provide full information and minimal compilable code example

[MarksWizard]

I am also experiencing this issue. It occurred after Updating Arduino IDE to 2.3.5 from 2.3.4. The ESP IDF also updated at this time. I had never seen this error previously.

@me-no-dev espressif/esp-idf#15734

[MarksWizard]

@lucasssvaz I noticed some changes attributed to you in the most recent update, any chances that is what’s causing the issue?

test(wokwi): Add I2C Master test and enable GPIO and PSRAM tests by @lucasssvaz in #10848

test(i2c): Add test to scan bus by @lucasssvaz in #11022

test(i2c): Do not use delta as Wokwi timing can be inconsistent by @lucasssvaz in #11080

[lucasssvaz]

@MarksWizard These PRs are related only to our I2C test in the CI. It does not affect how I2C works in any way.

[MarksWizard]

After a bunch of testing, and even more hair pulling... I think it ended up being a wiring issue for me. I finally got code to run without the error, but it would still be nice if that message didn’t spam the serial like it does.

espressif/esp-idf#15734 (comment)

[HanksSaloon]

I see the same issue, but I can confirm that it does not seem to be a wiring issue.
Board: DoIT ESP32 DEVKIT V1
Arduino IDE 2.3.6
When the sketch is compiled with ESP32 library 3.2.0 it fails, (error messages as above) but after a downgrade to ESP32 ver 3.1.3 it works. No changes made to hardware, nor the sketch, nor other libraries.

[me-no-dev]

Please list devices and libraries that you use with them, so that we can try to replicate any of the confirmed non-wiring issues.

[MarksWizard]

@me-no-dev

If everything is wired correctly, the sketch runs fine. If you disconnect a single wire from the OLED and reboot it will trigger the issue. It’s not specifically related to the OLED though, the issue seems to occur when you initialize an I2C device that isn’t connected. Unfortunately the solution isn’t simply “check your wiring”, because this completely eliminates the ability to test code “headless” without a sensor or screen attached. Often times I test code on just an unwired board just to make sure everything is working correctly before I upload to the actual device. Previous to the ESP update, typical initialization code would just tell you the sensor wasn’t present and would maybe report something like -127 or whatever, which is perfectly fine when I’m testing something that doesn’t actually need the sensor to be present. Now it spams the serial with the error messages and the official ESP IDE I2C_oled example I tested caused the device into a boot/crash loop.

I can get it to trigger using the code below, it should trigger before printing “Wire Started” if you don’t have an I2C device connected and should run fine if you do.

#include <Arduino.h> //Library for using Arduinio IDE specific commands
#include <Adafruit_GFX.h> //https://github.com/adafruit/Adafruit-GFX-Library
#include <Adafruit_SSD1306.h> //https://github.com/adafruit/Adafruit_SSD1306
#define SCREEN_WIDTH 128 // OLED display width, in pixels
#define SCREEN_HEIGHT 64 // OLED display height, in pixels

Adafruit_SSD1306 display(SCREEN_WIDTH, SCREEN_HEIGHT, &Wire, -1);

int timer = 0;
#define SDA 12 //variables for OLED display
#define SCL 13

void setup() {
delay(50);

Serial.begin(115200); //Begin serial connection

Serial.println("Initializing display”);
delay(1000);

Wire.begin(SDA, SCL);
Serial.println("Wire started”);
delay(1000);
if(!display.begin(SSD1306_SWITCHCAPVCC, 0x3C)) { // Address 0x3D for 128x64
Serial.println(F("SSD1306 allocation failed”));
}

display.clearDisplay();
Serial.println("Display Ready”);
}

void loop() {

while (timer < 10){
display.clearDisplay();
display.setTextColor(WHITE);
display.setTextSize(3);
display.setCursor(0,0);
display.print(timer);
display.display();
timer = timer + 1;
delay(1000);
}
}

[me-no-dev]

@MarksWizard so this is rather a logging issue. Your terminal gets flooded when there is no device. This I would say is oK. You should check if the device is there and not issue any more I2C transactions. That would stop the logs. I understand this is change in behavior, but that change comes from ESP-IDF's driver and not from Arduino, so we can not control it (other than blank stop it and not allow any logs, which is also not a good idea).

I am looking for a case where everything was working before and is no longer working at all with the new driver and proper wiring.

[HanksSaloon]

@me-no-dev
To add to @MarksWizard comments: I can submit an export from the logic analyzer (PulseView) showing the I2C behaviour using ESP32 ver 3.1.3 and using ESP32 ver 3.2.0 respectively, if that would help. (?)
My setup is a bit more complex, actually using both kernels separately and both I2C channels, so there might be an issue with the ESP-IDF driver timings. However, with 3.1.3 it works, with 3.2.0 it doesn't.

[me-no-dev]

@HanksSaloon sure! Please also give some basic info of the setup. What devices you have connected, how do you manage communication with them and pointers to what should we look at in the traces

[MarksWizard]

@me-no-dev first thanks for all of your help with this, and everything you do on the other projects.. I would not be able to play with these things without all of the hard work you have contributed.

I understand that it’s not an issue with Arduino, since the ESP IDF example code causes the issue. With the new update they changed some I2C stuff and added these error messages I guess. However it’s bad code behavior as there is no rate limit, flag or anything to limit the error messages. In the ESP IDF example it causes the device to boot/crash loop, which I can’t imagine is the preferred behavior if a simple I2C device becomes disconnected.

Your suggestion to not send any I2C if not connected seems absurd? So before every screen refresh or sensor read, I need to check that the actual device is present? That would add 100’s of lines of code to my project. This issue breaks 1000’s of example code found online that doesn’t involve those checks. EDIT: actually this is not possible, since initializing the sensor is what causes the issue.. and we don’t know if it’s present before we try to initialize. I have not been able to progress my own code because I have been trying to deal with this issue.

Unfortunately I feel like in trying initially determine and explain the issue I made things confusing. Now that I know exactly where the issue lies, can you please help me properly explain this to the ESP devs?

[HanksSaloon]

@me-no-dev

Ok, here goes:

Hardware
ESP-module DOIT ESP32 DEVKIT V1 (38 pin version)
I2C channel #0 on ports D21 (SDA) and D22 (SCL)
I2C channel #1 on ports D32 (SDA) and D33 (SCL)

I/O DEVICE ; Type ; address; I2C-ch; Library
MCP23008 ; 8-ports I/O ; 0x20 ; I2C#1 ; https://github.com/RobTillaart/MCP23008
PCF8574 ; 8-ports I/O ; 0x21 ; I2C#0 ; https://skyduino.wordpress.com/2011/06/07/librairie-arduino-i2c-pour-le-pcf8574/
ADS7830 ; ADC-8 ; 0x48 ; I2C#0 ; https://github.com/adafruit/Adafruit_ADS7830
24CSM01 ; EEPROM ; 0x50 ; I2C#0 ; https://github.com/WifWaf/AT24CM01
PCF8563T ; RealTimeClock; 0x51 ; I2C#0 ; https://github.com/orbitalair/Rtc_Pcf8563
47L16 ; RAM ; 0x52 ; I2C#0 ; https://github.com/jerry-magnin/47XXX_EERAM_Arduino_Library
ACS37800 ; Power Monitor; 0x65 ; I2C#0 ; https://github.com/sparkfun/SparkFun_ACS37800_Power_Monitor_Arduino_Library

All libraries have been slightly modified, but only to the extent of adding or removing some interface routines.

Physically, all circuits are on the same 100x100mm PCB, except for the ACS37800 which is located about 8 meters away.
Therefore, on I2C#0 there's a bus buffer P82B96 to extend the bus (https://www.nxp.com/docs/en/data-sheet/P82B96.pdf).
All pull-up resistors are 1k8 ohm, external pull ups on the extended bus are 1k1 ohm (relates to the remote cable type and length).

Standard #includes are
stdlib.h
Wire.h
Arduino.h

To explain the analyzer outputs:
The sketch is divided into two processes, let's call them "S" and "R", S being setup first in the first core. It then cranks up the other process in the second core.
I2C channel #0 is dedicated to the S-process, channel #1 to the R-process.
I2C-channels are initiated by a call to Wire.begin() and Wire1.begin(), respectively. (I have not tried TwoWire, as it seemed to require many more changes to the device libraries).
Both I2C channels run in fast mode (400k), although the problem occurs also @100k.

The RTC-module sends a 1Hz interrupt signal to pin D25. The interrupt handling routine couldn't be simpler, it just increases the value of a semaphore. Takes minimum of time. The semaphore is then processed elsewhere in the S process.

Both processes poll their I2C channel regularly, process S once every second, process R according to it's own internal logic.
This can be seen in the analyzer outputs attached.

Finally, the samples:
A_ESP32-3_1_3(.sr) everything works okay
A_ESP32-3_2_0(.sr) does not work.
(In these samples bus speeds are 400k, sampling rate is 1 Msps.)

At first, the two outputs seem very similar and there is no clear explanation to what the problem is in 3.2.0.
The only major difference can be noted at the end of each command, when the controller should send a Stop, i.e. release the bus.

In 3.2.0 SCL is released much later. Compare these examples:
In 3_1_3: timeline +1715776 -> +1715778 us (SCL is released appx 2 us from end of data)
In 3_2_0: timeline +2646044 -> +2646060 us (SCL release is appx 16 us)

I haven't had time to check the standard if there's a time limit when the SCL must be released after the data is sent, but as a delayed SCL release also delays the ACK/NACK response from the slave it seems that the ESP (ver 3.2.0) doesn't understand where the response comes from and therefore logs an error condition.

A_ESP32-3_.zip

[MarksWizard]

@HanksSaloon awesome work, thanks for helping to get this resolved. Can you post all of the above info in the issue opened on the ESP GitHub? Mythbuster5 has been asking for the logic analyzer stuff which I can’t provide.

espressif/esp-idf#15734 (comment)

[mythbuster5]

@HanksSaloon How to open the DMS file in your zip. Or it's not dma, because I download from mac. So the question is how to open that file? Or what's the correct extention name?

[mythbuster5]

Or just provide key picture is ok I think. Do you mean this part is suspicious？

[HanksSaloon]

The .Zip contains two files, <A_ESP32-3_1_3.sr> and <A_ESP32-3_2_0.sr>. They can be opened in PulseView / Sigrok.
Here are two screenshots of the difference I mention earlier.

[HanksSaloon]

Here are two more screenshots, of I2C channel 2. Highlight the difference.

[mythbuster5]

Got it.

[mythbuster5]

0001-fix-i2c-optimization-performance-on-esp32.txt

@HanksSaloon I have generated patch for shorten the time between last byte and stop. If your every transaction data is smaller than 63 byte (include address byte), you can try this. However, data in one transaction is larger than 63 bytes still have some problem, I still need time for debugging. Thank you.

@me-no-dev PTAL, and please help to share a method how to patch idf code in ardiuno environment if needed. Thanks.

[HanksSaloon]

@mythbuster5 @me-no-dev
Ok, great. I could give the patch a try, if someone can tell how to implement it a patch. (I'm on Linux.)
Currently the longest transaction is net 48 bytes payload, I think. Stays well below that 63 byte limit.

[MarksWizard]

@mythbuster5 here is how the issue is manifesting for me in Arduino IDE

IMG_6624.mov

[HanksSaloon]

@MarksWizard @mythbuster5 That's good to know.
I now realize, that our issues might have the same root cause, or then not.
Your issue seem to be that the CPU is kind of panicking when the I2C bus is disconnected.
The issue I see is that the I2C bus doesn't work at all, because of the errors, but my sketch seem to work, except for the I2C devices.
For logging/debugging purposes I use to insert dummy Serial.println() statements here and there, just to be able to follow where the control is, or where it gets lost. I then undefine debugging when the sketch is ready for "production".
Even though the I2C is continuously logging errors, the CPU is doing what it is supposed to do, and sends the debugging messages every now and then between the error messages.
I haven't had the possibility to test the patch yet, I'm waiting for instructions how to implement it.

[enjoyneering]

library - https://github.com/enjoyneering/SCD30_I2C

The sensor is very slow at startup - it takes about 2 seconds to warm up. It also requires 150 ms of clock stretching at 100 kHz due to self-calibration from time to time.

[HanksSaloon]

@enjoyneering Okay(?)
Do you say that you see the same issue with that sensor?
So far it seems that the issue discussed here is not related to sensor behavior, but to bus timings. However, this opinion might change as we investigate more.
When using a sensor that requires wake up/warm up, you should try to add the required delays in the driver library.
E.g many EEPROM and similar memories need some minimum delay between writes.

[enjoyneering]

I also see strange behavior on the I2c bus. Serial debug is disabled, but I get the message - i2c.master: i2c transaction timeout detected. Full log:

E (1124) i2c.master: I2C transaction timeout detected
E (1125) i2c.master: probe device timeout. Please check if xfer_timeout_ms and pull-ups are correctly set up
E (1138) i2c.master: I2C hardware NACK detected
E (1139) i2c.master: I2C transaction unexpected nack detected
E (1144) i2c.master: s_i2c_synchronous_transaction(924): I2C transaction failed
E (1151) i2c.master: i2c_master_multi_buffer_transmit(1186): I2C transaction failed
E (1161) i2c.master: I2C hardware timeout detected
E (1163) i2c.master: s_i2c_synchronous_transaction(924): I2C transaction failed
E (1170) i2c.master: i2c_master_multi_buffer_transmit(1186): I2C transaction failed
E (1180) i2c.master: I2C hardware timeout detected
E (1182) i2c.master: s_i2c_synchronous_transaction(924): I2C transaction failed
E (1189) i2c.master: i2c_master_multi_buffer_transmit(1186): I2C transaction failed
E (1203) i2c.master: I2C hardware timeout detected
E (1203) i2c.master: s_i2c_synchronous_transaction(924): I2C transaction failed
E (1208) i2c.master: i2c_master_multi_buffer_transmit(1186): I2C transaction failed
E (1217) i2c.master: I2C hardware timeout detected
E (1219) i2c.master: s_i2c_synchronous_transaction(924): I2C transaction failed
E (1226) i2c.master: i2c_master_multi_buffer_transmit(1186): I2C transaction failed
E (1236) i2c.master: I2C hardware timeout detected
E (1238) i2c.master: s_i2c_synchronous_transaction(924): I2C transaction failed
E (1245) i2c.master: i2c_master_multi_buffer_transmit(1186): I2C transaction failed

[MarksWizard]

@mythbuster5 @me-no-dev

I rolled back to Arduino 2.3.4 and installed the esp board manager [v3.1.3 based on ESP-IDF v5.3] As you can see the issue doesn’t exist here. THIS IS HOW I WOULD PREFER THE CODE TO WORK!

IMG_6627.mov