Sonoff SwitchMan M5-2C crashes

[miwojcik]

Detailed steps to reproduce it
I have 31 Sonoff M5 switches at home - 11 of 1C, 12 of 2C and 8 of 3C. They are all connected to the two UniFi U6 Pro access points with very good signal and are running latest HAA v12.10.3.

About 6-8 of the 2C switches (always the same ones) are randomly crashing every 1-48 hours where they completely stop working for 5-20 minutes:

• button lights (controlled by PWM) are not lighted up
• relays don't react to button presses
• WiFi shuts down - no ping, no setup mode, no HAA-XXX AP broadcasting, no HAA/Home app access

Power cycling by main breaker immediately reboots them and fixes the issue for next random amount of time. If not, the switch will usually reboot itself after 5-20 minutes (screenshot from UniFi Controller attached).

They were also crashing on older HAA v12.8 before I moved in and set up the Wi-Fi at home, so I doubt it's related to connectivity issues or recent changes. On older firmware I also noticed that sometimes the right button light (controlled by GPIO22) was not working after button press, even though the relay was functioning correctly, but I'm not seeing it now - could be related to esphome/issues#5085.

Things I tried:

• disabling WiFI sleep
• enabling frequent (every 3s) gratitous ARP
• switching from hardware to software PWM
• disabling WiFi AP mode
• switching from active roaming to passive roaming
• marking GPIO0 as input (even though it's not used) - the only difference I've found between 1C and 3C switches that are all stable
• EDIT: moving front panels with ESPs between different bases to exclude power source issues

I'm afraid it could be caused by a hardware issue, but I also found this post on Sonoff group that they released official firmware that fixed similar crashes: https://www.facebook.com/groups/ewelinkandsonoff/posts/1380550352511840/

MEPLHAA Script
I started with this config that on some of the 2C switches is working perfectly fine:

{
  "c": {
    "io": [
      [[5, 19, 22, 23], 2],
      [[0, 4, 15], 6],
      [[18], 9]
    ],
    "l": 5,
    "i": 1,
    "b": [
      [4, 5]
    ],
    "z": 0
  },
  "a": [
    {
      "0": {
        "q": [
          [18, 5000, 1000, 1000]
        ],
        "r": [
          [23]
        ]
      },
      "1": {
        "r": [
          [23, 1]
        ]
      },
      "b": [
        [4, 1]
      ],
      "es": [
        {
          "0": {
            "r": [
              [19],
              [22]
            ]
          },
          "1": {
            "r": [
              [19, 1],
              [22, 1]
            ]
          },
          "b": [
            [15, 1]
          ]
        }
      ]
    }
  ]
}

And after trying various tweaks I ended up with:

{
  "c": {
    "io": [
      [[5, 19, 22, 23], 2],
      [[0, 4, 15], 6 ],
      [[18], 7, 0, 5000]
    ],
    "d": 0,
    "l": 5,
    "i": 1,
    "b": [
      [4, 5]
    ],
    "z": 0,
    "m": 600,
    "e": 3
  },
  "a": [
    {
      "0": {
        "r": [
          [23]
        ]
      },
      "1": {
        "r": [
          [23, 1]
        ]
      },
      "b": [
        [4, 1]
      ],
      "es": [
        {
          "0": {
            "r": [
              [19],
              [22]
            ]
          },
          "1": {
            "r": [
              [19, 1],
              [22, 1]
            ]
          },
          "b": [
            [15, 1]
          ]
        }
      ]
    }
  ]
}

Used device
Sonoff SwitchMan M5-2C

HAA version
12.8-12.10.3

iOS/iPadOS version
17.1.2

Logs
Enabled network logs on affected switches. I hope it will catch anything before a crash.

EDIT: There is nothing useful in the network log. It just disappeared and appeared again:

Jan 13 20:22:13 192.168.2.55-71325C [51] Get CH
Jan 13 20:23:18 192.168.2.55-71325C [51] Get CH
Jan 13 20:24:24 192.168.2.55-71325C [51] Get CH
Jan 13 20:24:41 192.168.2.55-71325C mDNS prob 1
Jan 13 20:25:16 192.168.2.55-71325C mDNS prob 2
Jan 13 20:25:23 192.168.2.55-71325C mDNS TTL 4500/2115s
Jan 13 20:25:29 192.168.2.55-71325C [51] Get CH
Jan 13 20:25:53 192.168.2.55-71325C NTP 192.168.2.1 (-1)
Jan 13 20:25:58 192.168.2.55-71325C NTP 192.168.2.1 (-1)
Jan 13 20:26:02 192.168.2.55-71325C NTP pool.ntp.org (0)
Jan 13 20:26:34 192.168.2.55-71325C [51] Get CH
Jan 13 20:27:39 192.168.2.55-71325C [51] Get CH
Jan 01 00:00:26 192.168.2.55-71325C Free Heap 222444
Jan 01 00:00:26 192.168.2.55-71325C Start HK
Jan 01 00:00:26 192.168.2.55-71325C HK ID: D9:15:69:ED:19:D4
Jan 01 00:00:26 192.168.2.55-71325C Found 891B27FD-B64E-4C62-B7A9-E3D1BDC5EEE6
Jan 01 00:00:26 192.168.2.55-71325C Free Heap 211260
Jan 01 00:00:26 192.168.2.55-71325C <0> Run A2
Jan 01 00:00:26 192.168.2.55-71325C mDNS prob 1
Jan 01 00:00:26 192.168.2.55-71325C mDNS Name=HAA-71325C md=RavenSystem HAA Merlinpv=1.1id=D9:15:69:ED:19:D4c#=10s#=35sf=0ci=8
                                                                                                                              sh=3YwYOQ== Port=5556 TTL=4500
Jan 01 00:00:26 192.168.2.55-71325C [51] New 192.168.2.209:7108 1/12 HEAP 207540
Jan 01 00:00:26 192.168.2.55-71325C [51] Verify 1/2
Jan 01 00:00:26 192.168.2.55-71325C [51] Verify 2/2
Jan 01 00:00:26 192.168.2.55-71325C [51] Pair found
Jan 01 00:00:26 192.168.2.55-71325C [51] Verify OK
Jan 01 00:00:26 192.168.2.55-71325C [51] Get ACC
Jan 01 00:00:26 192.168.2.55-71325C [51] Upd CH
Jan 01 00:00:26 192.168.2.55-71325C [51] Get CH
Jan 01 00:00:27 192.168.2.55-71325C [51] Get CH
Jan 01 00:00:28 192.168.2.55-71325C NTP 192.168.2.1 (-1)
Jan 01 00:00:28 192.168.2.55-71325C [51] Get CH
Jan 01 00:00:33 192.168.2.55-71325C NTP 192.168.2.1 (-1)
Jan 13 20:37:51 192.168.2.55-71325C NTP pool.ntp.org (0)
Jan 13 20:37:54 192.168.2.55-71325C mDNS TTL 4500/2186s
Jan 13 20:38:42 192.168.2.55-71325C [51] Get CH
Jan 13 20:39:43 192.168.2.55-71325C [51] Get CH

Screen-shots or videos showing issue

[miwojcik]

I compared one of the broken and one of the working switches I have. They probably came from different sellers as they are different HW revisions. The biggest difference is the ESP chip itself! The affected one uses ESP32-DOWD R2-V3 while the one that is working uses ESP32-DOWD V3 (no R2 here!). From what I understand they differ in the PSRAM size.

Pictures:

• affected
  
  
• working
  
  

[RavenSystem]

Try adding [[16],1,1] to "io": array.

Source: https://www.espressif.com/sites/default/files/documentation/esp32_datasheet_en.pdf

[miwojcik]

Interesting find. Thank you @RavenSystem, I'll try that!

[miwojcik]

I applied it to all affected switches and updated everything to v12.10.4. I'll report back in 24 hours.

[miwojcik]

I'll wait another 24h to be sure as my APs updated overnight and I don't have full uptime, but so far so good!

[miwojcik]

Thank you very much @RavenSystem! You saved me a lot of trouble.
None of the switches disconnected in last 40 hours and the status lights are working as they should.
Do you think GPIO16 input pull-up could be set automatically if HAA detects it's running on ESP32-DOWD R2-V3? That way we could avoid hacking it in the config.

[RavenSystem]

About gpio reset and disable, check code, because you are mixing codes from different SDKs.

[miwojcik]

Check code again.

As you see it's hard to properly analyze the code without knowing the flags and config that is used to compile it. I see the old build instructions are gone. Do you plan to post updated ones?

I definitely agree with the idea of "open firmware where user can do anything", but the firmware should also prevent its user from shooting in the foot and work properly out of the box. This discussion will be quickly buried and it's hard to expect that someone who just got their board with ESP32-D0WDR2-V3 will know that it requires a HAA config specific hack not to crash randomly. If you think that not all boards with this CPU are affected, then this workaround should be at least mentioned in the wiki in some visible space. However:

other device with same CPU could crash with this because uses GPIO 16 connected to an external sensor that needs GPIO with mode output low

In current state, I'm not sure if it would work at all with this CPU. Both GPIO 16 and 17 are marked red in the datasheet as "allocated for communication with in-package flash/PSRAM and NOT recommended for other uses.". So unless PSRAM support is disabled in SDK config, there is a high chance it would be crashing with anything connected to it. If it's disabled, then we may looking at a different HW/SW bug that crashes the device even though PSRAM is not used.

About gpio reset and disable, check code, because you are mixing codes from different SDKs.

I'm looking at ESP-ID 5.1.2 that you updated to in v12.10.0. What am I missing?
https://github.com/espressif/esp-idf/blob/482a8fb2d78e3b58eb21b26da8a5bedf90623213/components/driver/gpio/gpio.c#L433

[RavenSystem]

All flags are declared in the current code or in the used SDK. I removed old build instructions because they were useless. No special instructions are needed, only official build from used SDK.

I amo not going to begging a fight to include all exceptions and all special configurations for each device in the market. Default GPIO configuration is the default used by each SDK; except for ESP8266 chips, because esp-open-rtos-rsf SDK does not apply any default configuration to GPIOs, but since this SDK is developed by myself, I can move initial GPIO configuration from user code to SDK code at any moment.

gpio_reset_pin(i) is a function used in ESP-IDF SDK, and gpio_disable(gpio) is a function used in esp-open-rtos-rsf SDK. First check standard ESP_PLATFORM flag and how it works at official ESP-IDF SDK documentation: https://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-guides/build-system.html?highlight=esp_platform#preset-component-variables

However, if you are talking about two lines:

gpio_reset_pin(i);
gpio_set_direction(i, GPIO_MODE_DISABLE);

These lines are between /* and */. These symbols are used to comment code, and all this code is not used to the compiler in any way. You can learn about this here, for example: https://en.wikipedia.org/wiki/Comment_(computer_programming)

[miwojcik]

All flags are declared in the current code or in the used SDK. I removed old build instructions because they were useless. No special instructions are needed, only official build from used SDK.

Thank you for clarification. If it's that simple, then why not put those few commands and a short note there? It would save us half of the discussion and ensured that everyone else who wish to build and flash the firmware has the same, reproducible build.

These lines are between /* and */. These symbols are used to comment code, and all this code is not used to the compiler in any way. You can learn about this here, for example: https://en.wikipedia.org/wiki/Comment_(computer_programming)

I know what they are. I simply didn't notice that the whole 39 lines are commented out. The wiki article you mentioned says: "They are added with the purpose of making the source code easier for humans to understand". Leaving big chunks of the code commented out in the published version is the opposite of that and it's simply not a good practice.

[RavenSystem]

Thank you for clarification. If it's that simple, then why not put those few commands and a short note there? It would save us half of the discussion and ensured that everyone else who wish to build and flash the firmware has the same, reproducible build.

Simple? I did say nothing about simple. I said that no special instruction are needed. But it is not simple. It is necessary to study and learn for several years about hardware and software, and then study thousands of pages from official documentation, adding some other years of experience with professional and commercial devices.

I know what they are. I simply didn't notice that the whole 39 lines are commented out. The wiki article you mentioned says: "They are added with the purpose of making the source code easier for humans to understand". Leaving big chunks of the code commented out in the published version is the opposite of that and it's simply not a good practice.

If you did not notice that the whole 39 lines are commented out, you should be sure of that before saying erroneous things. The wiki I mentioned if the Wikipedia, and like it says, I use comments in my code making my source code easier for my understand. HAA source code is public only for transparency. HAA is not a firmware developed by the community; it is a personal firmware that I share for free, and I use the comments needed by me. But the purpose of the comments is irrelevant: commented code is not compiled.

To summarize: I am not going to write documentation explaining step by step how to build firmwares using ESP-IDF, or detailed instructions about how to firmware and hardware works. And I am not going to change HAA code to satisfy your personal needs. Remember that you had an issue with your devices (not a firmware bug), and solution was only read official documentation; something that you did not do. Please, feel free to code your own firmware instead using HAA.