Supporting latest Arduino Core v3 - issues with fixed Network IP address, settings lost after upgrade etc

[tp1de]

Being in my holiday home with netfit thermostat moduline 2050 (ems 2.0) I recognized that the thermostat_hc1_mode does not work anymore (neither dashboard nor mqtt). No changes possible - no error reported.

EMS-ESP version 3.7.0-dev.10 works ok, but EMS-ESP version 3.7.0-dev.12 and 13 are not working.
On my EMS+ system with RC310 versions EMS-ESP version 3.7.0-dev.12 and 13 are working fine.

What has been changed?

[tp1de]

.... and while installing 3.7.0-dev.13 both systems do not restart anymore. My home system with E32 needed a hard reboot and was working again.
My S3 system within my holiday home lost all config data and wifi started with access point. This is not good.

[MichaelDvP]

For mode see #1800, should be fixed.

The new dev is compiled with arduino 3.0.1. Maybe the newer littleFS version reformats on an old system. I've seen this a long time ago when frst trying the arduino 3.xx. But since that i can change between 2.xx and 3.xx versions without issue.

[proddy]

This is indeed not good. We'll need to test the migration and see what is causing the FS issue. If it's a problem for others we should perhaps better remove the firmware bin files from the dev repo

[MichaelDvP]

I don't think it'a a general issue, Harvay in #1800 and others don't have FS cleared on update.
I think there was one specific version in the past and systems USB-flashed with this version are not recognized y the arduino-3-LittleFS. I have seen it a very few time, but can not reproduce and find the specific version.

[tp1de]

For mode see #1800, should be fixed.

The fix is still not within latest dev13 .... or?

[mattreim]

Version 3.7.0-dev.13(3) from yesterday has been running for me for 24 hours without any problems.

[tp1de]

I tested again: dev.13 was not working and after uploading bin-file all settings where lost on my S3 gateway again.
So I flashed the gateway from my laptop , loaded the settings and afterwards it was working again.
The thermostat.hc1.mode was changeable. So far so good.

But I recognized that the network settings in respect to fixed ip address and hostname where loaded but not used.
(dhcp instead). Fixed ip was only used after some retries. The defined hostname is not populated on the network.

[mattfro]

I have esp32 s3, after going from dev12 to dev14..I lost wifi, esp-ems went into AP mode.
In AP mode I tried to use wifi scan to search for networks, it did not find anything. Going back to dev12 it worked again.
In the end I even tried to manually install dev14 with ESP flasher and in that point I think all my settings was lost, but still wifi scan did not work. Stupid me, did not even think of setting wifi manually before this :) Because that seems to work in the end. Now running dev14 after setting back all settings like mqtt and wifi. It seems to work as it should

[proddy]

Thanks @mattfro - there are definitely something compatibility issues with the LittleFS file system that stores the EMS-ESP settings between Espressif Core (v2 on dev 12 and before) and the newer builds on Core v3.0.1 (dev 13 + 14). When I'm back home I'll need to run some tests, flashing with dev12 on an S3 and trying the upgrade. It may also be something related to the partitions or how the BBQKees gateways are originally flashed - we'll see. For now I would stick to dev12.

[MichaelDvP]

I don't know why s3 compiles with arduino core 3.xx. There seems to be a bug in platformio.
We use platform = espressif32 which should use the latest stable from https://registry.platformio.org/platforms/platformio/espressif32, which is 6.7.0 with arduino core 2.0.16. Also https://github.com/platformio/platform-espressif32/releases says latest release is 6.7.0.
Using the development branch platform=https://github.com/platformio/platform-espressif32.git compiles with arduino 2.0.17.

In https://github.com/MichaelDvP/EMS-ESP32/releases i have the actual dev14 compiled with arduino core 2 (s3: [email protected], esp32: tasmota@2024-05-00)

With the arduino 3.0.1 i don't see issues with littleFS and fixed ip, but i can confirm that network scanner is not working with arduino 3.0.1 and 3.0.0 builds.

[proddy]

Maybe just explicitly add the 6.7.0 to the platform as we did last time.

For WiFi maybe it's related to the event names https://randomnerdtutorials.com/esp32-migrating-version-2-to-3-arduino/#wifi-event-name-changes

I'm not home so can't test at the moment

[MichaelDvP]

Maybe just explicitly add the 6.7.0 to the platform as we did last time.

That's exactly what i have done for the test: see b175026

BTW: the S3 arduino3.0.1builds are also very small, like the tasmota-arduino without TSL, only 1.6M bin file
But with:

platform_packages=
  platformio/framework-arduinoespressif32 @ https://github.com/espressif/arduino-esp32.git#3.0.1
  platformio/framework-arduinoespressif32-libs @ https://github.com/espressif/esp32-arduino-libs.git#idf-release/v5.1

i get 2.1M bin file. (But also network scan with empty result)

The event names are changed since arduino 2.0.12, in 3.xx the old ones are removed and throw error when using.

[proddy]

I made a dev14 that uses the previous 2.0.17 Arduino espressif32 library

[tp1de]

It is dev 15 and seems to work as before. WiFi scan is ok.

[mattfro]

Same here... everything seems to work now.

[proddy]

thanks @tp1de and @mattfro for quickly testing. I think we'll keep to the old Ardunio library until we see the benefits of moving to v3 of the core and have fully tested the Network and Filesystem changes.

Interesting article: https://www.cnx-software.com/2024/06/01/espressif-releases-arduino-esp32-core-3-0-0-but-platformio-support-in-doubt/

[proddy]

thanks @tp1de and @mattfro for quickly testing. I think we'll keep to the old Ardunio library until we see the benefits of moving to v3 of the core and have fully tested the Network and Filesystem changes.

Interesting article: https://www.cnx-software.com/2024/06/01/espressif-releases-arduino-esp32-core-3-0-0-but-platformio-support-in-doubt/

Related to espressif/platformio's little argument and further development, some of the devs have started creating their own pio fork: https://github.com/pioarduino/platform-espressif32

[tp1de]

@proddy @MichaelDvP while updating to latest dev.26 I lost all settings again.
After re-setting I recognized that within ioBroker adapter all axios get request from devices failed (e.g. .../api/boiler etc.)

What has been changed?

[proddy]

Best to comment in #1806 - this issue is about core v3 and we're still running on v2

[MichaelDvP]

Not in #1806. This is the new message:

EMS-ESP32/src/emsesp.cpp

Lines 758 to 762 in 55476a7

	// if the EMS device was valid, but the cmd not found show an error
	if (found_device) {
	root["message"] = std::string("unknown command ") + cmd;
	return false;
	}

which is shown in api/boiler/values etc. in first place. but is no entity.

[proddy]

oh, yes, I'll fix that

[MichaelDvP]

I've made a PR to clear output if there is a command function.
@tp1de Were all settings cleared? have you changed between espressif-arduino and tasmota-arduino (e.g. 16M/4M version)?
Is it reproducible?

[proddy]

oh, yes, I'll fix that

fixed now with #1879. Sorry about that

[proddy]

I see 6.8.0 is out (still on 2.0.17) so will try that https://github.com/platformio/platform-espressif32/releases/tag/v6.8.0

[proddy]

I'd like to try out https://github.com/pioarduino/platform-espressif32 too

[MichaelDvP]

It's updated very often, don't know if these are bugfixes or enhancements.
I've tested a few times on ESP32S3 with PSRAM. Works.
Comparing arduino 2.0.17 (espressif develop) to 3.04 (pioarduino develop)
filesize 2.3 MB 2.6 MB
free ram 201 198
max alloc 187 183
free caps 8343 8337
free main stack 4720 3852
free TCP stack 936 88

Free TCP stack is very low (setting to 6144), maybe increase to 7 or 8k.

Edit: it was set to 5k for arduino 3, so stack tcp stack usage is same for both versions, setting of 6k is ok.

No issues when switching between the builds, littleFS read all the configs. I only get littleFS issues when switching between tasmota and espressif builds.

Edit: Network scan always returns no results on arduino 3.

[proddy]

There's quite a lively discord channel with all the famous people (Jason2866, me-no-dev, E.M. etc) on it discussing the differences. I sent you an invite, but I know you dislike Discord ;-) Jason was talking about the differences in the Tasmota build compared to this (apparently has MQTT, Rainmaker, WifiProv and BLE all removed)

[proddy]

Edit: Network scan always returns no results on arduino 3.

would be interesting to see if this works on Arduno 3.0.4 https://raw.githubusercontent.com/RuiSantosdotme/Random-Nerd-Tutorials/master/Projects/ESP32/ESP32_WiFiScan.ino

[proddy]

Tried with the latest pioardunio (platform=https://github.com/pioarduino/platform-espressif32/releases/tag/53.03.10+rc1) which is Arduino v3.1.0/IDF v5.3.3 and comparing to Arduino v2.0.17/IDF v4.4.7 and both freemem and max alloc went down by 20% which isn't good.

** edit **
Back with 3.0.4/IDF 5.1 things are better, only 5KB less memory. But WiFi scan is still broken.

[proddy]

Continue in #2108