Awtrix 0.96 & 0.94 on two Ulanzi TC001 reboot up to two times a day

[UCaGk]

Bug report

I use two different Awtrix versions on two Ulanzi TC0001 clocks.
The clocks are under control of Home Assistant MQTT, each clock only shows time and on different events ICON, Sound and Text, but mostly time in format HH:MM:SS or HH:MM in color white, brithness 1-20, depending on some events.
Every clock reboots at a purely random time, without an event, just like that. Sometimes a colored dot appears, sometimes part of the matrix turns bright white, then the firmware restarts and clock run again.
Looks like the display "freeze" (sometimes a few leds flicker) and a watchdog reboots...
Both clocks has accumulator removed, some changes to pcb and a 3.3V stepdown regulator.
I'm not sure if the clocks reboot more than 2 times, because I'm not sitting in front of them the whole day :-)

Devices involved:

• Model: 2x Ulanzi (TC001)
• awtrix3 version: v0.94 & v0.96

Steps to reproduce the behavior:

• not possible, because this happens randomly

A clear and concise description of what you expected to happen.

• It would be nice to know if there ist problem in software, if so, a new version would be great :-)

[Blueforcer]

Check the RAM usage.
If your using the Homeassistant discovery entities very often the RAM usage increases because of a memory leak in the used library.
Switch to raw MQTT API commands and you're fine.

[UCaGk]

Oh, that was fast :-)

Sorry for my misunderstanding, but what do you mean with "Switch to raw MQTT API commands"

This is a code snippet: Is this raw MQTT ? If not, how looks it like ?

  - alias: Zeitformat HH:MM Hell
    service: mqtt.publish
    metadata: {}
    data:
      qos: 0
      retain: false
      topic: UhrWZ/settings
      payload: >-
        {    "TIME_COL": [255,255,255],  "WD": false,  
        "TFORMAT": "%H:%M",   "TMODE": 0, "ABRI": false,  "BRI":
        20 }
    enabled: true

[Blueforcer]

Thats correct.
i mean using the awtrix entities like

[UCaGk]

I checked the free ram usage of both clocks, each has about 135000 Bytes of free RAM.
HA reboot each clock in the middle of the night, so hopefully the memory should be free after that. Not sure how fast the free memory space shrinks.
But if my code is correct and memory looks like much enough, what could I do to prevent random rebooting ?

[Jeandhom]

if the clocks reboot more than 2 times, because I'm not sitting in front of them the whole day

You can monitor the device's uptime.

[Blueforcer]

Not sure how fast the free memory space shrinks.

Just use Homeassistants history, you can see the ram over time in a graph 📉

[UCaGk]

Thanks for the hints, i made some entities, here are the results.

The other clock looks similar, only the battery history is a line at 100%.
It`s a strange battery plot, but the uptime only restarts at ~ 4:30 h this morning. (I reboot the clock at 3 am every day)
If there is a power problem the uptime counter should restart more than once in this history, right ?
I think it's not a memory problem.

[Blueforcer]

RAM usage looks good, and there was no other cause of reboots in the past.
I have read your issue again carefully
I assume that something is wrong with the DIY power supply.
When the display freezes but some leds flickers, its not a software issue.

[UCaGk]

Thanks for your help, I was also worried that it was the power supply, I was already thinking about whether I would only use the RGB LEDs with housing from the Ulanzi and replace the hardware with a simple ESP, will see.

[Trychlik]

i have same behaviour - once a day it freezes to half display working and after few seconds/minutes it reboots and works to next day ok. it crashes about at 40MB free ram. connected to 5V 2A power. battery is always 100%. maybe memory is a problem.

[Blueforcer]

40kb is too low. Check wich automation cause the memory drop

[UCaGk]

Sorry for writing again, maybe it helps, if there is a bug.
After a better look onto the clock I saw that the pixel do not flicker. Often there is a dot on the uper left, the clock is frozen and about 10 s the colored AWTRIX - IP - MQTT startup comes up, after that the clock is running like before, if display was HH:MM
than ist shows HH:MM, was the display HH:MM:SS before, HH:MM:SS is shown and everything look like before.
Situation is like before, memory always >110kb

[Blueforcer]

You should read the whole issue. Don't control the HA entities often, instead use raw mqtt commands

[Blueforcer]

Did you find the error?

[UCaGk]

Hello Stephan, thanks for asking.
The last few weeks have been a bit stressful, I haven't had time, but I'm currently checking the voltages and wanted to solder a capacitor to the 3.3V voltage converter in case of short-term current spikes.
In the meantime, I also had the situation where the clock restarted and switched to AP mode, and once it didn't start automatically at all.

[UCaGk]

Update:

After adding the capacitor to both clocks, things getting more and more strange...
Before adding the capacitor, the "uptime counter" only increases to about 16000 (see diagram)
After adding the capacitor, the counter continues to increase without resetting at about 16000 (see diagram after 12.6./9:00) ????
What is the bit size of the uptime counter?

[UCaGk]

Update after one day running with the capacitor the clock didn't reset, the uptime value increase.
But, sometimes the display switch off (fully dark) for about 1 sec, after that the display switch on again as if there was nothing, without restarting (uptime still increasing). Think this is the reason why the voltage of DC-DC stepdown regulator without cap has a voltage spike the restarts the esp inside.
Now it would be nice to know why the display switch off for a second ?

[Blueforcer]

There is no software automation to switch off the display.
Maybe one of your automation send the power request.
But for hardware related things, this repo isnt the right place. unfortunately i can't help if someone modifies the hardware. i only do software :)
I recommend you join the discrod, a few have done it there. And there's also the problem that the Ulanzi will eventually switch itself off after the mod.

[vogtmh]

I already mentioned my "RAM problem" in another issue, but it might apply here as well:

I had a similar issue, could successfully reproduce and fix it. Long story short: I didn't know about the blink or fade functionality included in Awtrix, so I used a loop in a Home Assistant automation to turn the lights on and off every second, using a custom component which defaulted to QoS of 1.

The solution was to remove the custom component and use the built-in mqtt.publish instead, defining a QoS of 0 for all commands. I also replaced the loops with single mqtt.publish commands, where I set the indicators to blink or fade.

All problems are gone since then. Thanks for this AWesome custom firmware! :)

[eku]

I observed the following yesterday.

During the display of a custom app, its value was changed as a result of an update via MQTT. The value of the display still changed, but it was no longer switched to the next app. The software was frozen and was forced to restart by the watchdog.

[UCaGk]

FYI
After a few days of using the capacitor, the operating time no longer resets.
What I see is that the clock display sometimes "turns off" for about 1 second. One time, the top left LED was green while the rest of the display was black. After the second, the display turns on, the green dot is gone, and the time continues as if nothing happened. The memory remains almost constant except for the usual fluctuations.
Because this short "turn off" is very rare and the clock does not restart, I am fine with it.

[popy2k14]

@UCaGk I am the guy which posted the battery removal guide on reddit back than. Also replaced the power supply of my two ulanzis with an 3V3 converter. I am running https://github.com/lubeda/EspHoMaTriXv2 (complete other software).

And you know what...

I have the exact same issue, the esp reboots a few times a day (sometimes with the first led/dot in the corner).
Programmed my yaml the way, so it resumes where it was, so no big deal for me ... But also annoyed me since I have those clocks. I have two of them, and one is worse than the other. Sadly had no time to dig deeper from an HW PoV.

Interesting that an Cap solved the issue for you.
It seems that internally some circuit is switching power of at certain events (maybe some external WDT ic...).
As stated, I have not gone deeper into this.

But sure I want it as stable as possible without spending to much time.

So here are my questions:

Does this fix work for you also after using it longer?
(It's been months now since you posted your solution)

Can you please tell us which value you have soldered into the device and exactly where?
(At the Input, output or both sides of the DCDC? Photo would be cool)

Thx

[UCaGk]

@popy2k14
Nice to hear that i'm not alone with this problem, even this is not a nice problem.
And, is good to hear that is not a problem of the firmware, because we are using different once. In te past i also though to install EspHoMaTriX, but after your feedback i do not need to switch now.

It is exactly the same behavior, you discribe, with my two clocks. One clock reboots (not really a full reboot) raher rare, one reboot more often. With the caps it is not a full reboot, it is something like this: The display gets black for, not sure, about 2-3 seconds, sometimes there is a green dot in the upper left part of the display before it get black sometimes not. Because the second clock is in a really dark display mode, i'm not sure if the behavior is the same.
Before useing the caps the clocks made a full reboot with new connectiong to wifi.

Afteri made this changes it was and is ok for me, even if it would be better the clock would work without such "black-out features".

Have you ever thought about replacing the whole Ulanzi hardware by a simple esp connected to the display of the Ulanzi, and use the EspHoMaTriXv2 firmware with it ? This was something i thought about in the past.

[popy2k14]

@UCaGk yeah, i had those issues since i made my post on reddit back than: https://www.reddit.com/r/homeassistant/comments/131okf7/comment/jwudxjk/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button

Have you done your mod from my picture / guide?

I have now added an 1000µF / 10V cap on each clock directly on the ESP32 GND & VCC Pins (3V3 rail) and will see how it goes. Also activated the uptime sensor logging.

Yeah i have thought of ditching the original pcb board and just using an ESP32 (which i also have laying around).
But the ulantzi pcb has some features which i dont want to miss or replicate (ex.: the light sensor, buttons, ...).
Sure we can develop our own pcb, but hell yeah it always the same ... lack of time... :-)

If you think again to switch to EspHoMaTriXv2, be aware that it has some bugs and is not that much maintained because lack of other devs. Dont get me wrong, lubeda and andrewjswan doing an amazing job!!!, but if you have other use cases than the devs (like me :-) ) you may face some bugs. Ex.: there is an looping bug which was not found up until now, where i just have made an workaround in my yaml for (some sort of wdt). But overall i am pretty happy with it. This here: lubeda/EspHoMaTriXv2#245

Will keep you updated if my reboots of the ESP32 is gone.

[UCaGk]

@popy2k14
I'm sorry, i forgot the position and value of the cap, it is also a 1000 uF, in my case a 35V because it was in my "junk box".
I put it on the 3,3V side of the converter.
Thought abot a second one on the 5V side.
And yes I made it from you pictures.

It would be nice too hear from you if the reboots are gone.
When finding time to replace the Ulanzi board and build my own hardware i will post it.

[popy2k14]

@UCaGk ok thx for confirming your used cap value, so i should see the same result.

If i had time i would make my own PCB (i am an HW engineer for ~20 years) but sadly dont have much time for that.
Would be nice if you share replacement hardware, if you find some time to make one :-)

But until that happens HUGHE thanks to bring this issue back to my attention and also post a workaround!!!
To be hohnest, i just forgot about it :-)
Really appreciate that!

If that also works for me, i will update my reddit post.

[Blueforcer]

I have direct contact to the Ulanzi hardware engineer because of the next version 🤫. I will ask him how to Bypass the battery.

[popy2k14]

I have direct contact to the Ulanzi hardware engineer. I will ask him how to Bypass the battery.

That would be awesome!!!!
Maybe he can share the circuit also, that would help a lot.

Thx

[UCaGk]

This would be the BEST SOLUTION ever !
Hope he could help.

Thanks Stephan !

[popy2k14]

Here are my results with the additional 1000µF/16V cap on the 3V3 rail.
Working as expected, and the ESP32 of the clocks are stable for an day now:

And yes, something is switching the matrix off/on. Also when the esp32 is stable now, the matrix sometimes glitches (black or bright dot in the top left corner for about 2-3 secs) like @UCaGk stated above also on my clocks.

@Blueforcer
Would be really nice if we can get help of the hw dev, whats going on here.

[eku]

PR #564 might help to shade a light in restarts.

[popy2k14]

@eku thanks for the hint. the ESP restart is because "something" on the board is switching power on certain events.
This was mitigated and solved by adding an 3V3 capacitor + confirmed by @UCaGk and me.
Sure this feature would have helped to get a reset reason. In our case it probably would be something like brown out or power on reset.

As stated the esp is now stable, but not the LED matrix, which get's reset in an interval (sadly sometimes one led is at 100% white, when this happens).

So there mus be something on the PCB which triggers those events which we don't know or have investigated.
When @Blueforcer (Stephan) would get infos from the ulantzi dev, maybe we know more :-)

[eku]

@eku thanks for the hint. the ESP restart is because "something" on the board is switching power on certain events.

As mentioned in my first comment here, I had observed a software (dead)lock that triggered the watchdog.

[popy2k14]

@eku sorry, missed that. Sure for software related issues, knowing why it restarted may help.

[popy2k14]

@Blueforcer any news from the hw engineer of ulanzi tc001?

PS.: Since adding of the cap, they are much more stable!!!!!

[Blueforcer]

@eku thanks for the hint. the ESP restart is because "something" on the board is switching power on certain events.

As mentioned in my first comment here, I had observed a software (dead)lock that triggered the watchdog.

but it also happens with other completely different firmwares. At the exact same time.

[popy2k14]

hi guys. Because the original reddit thread was archived, i have created an new one here: https://www.reddit.com/r/homeassistant/comments/1grzksj/ulanzi_tc001_always_on_mod_battery_removal/
with updated instructions.

[UCaGk]

@popy2k14
👍

Hope there will be some information on the “power problem” in the near future, from whoever,

[Blueforcer]

No message from Ulanzi so far. I think they used discord only for contacting me...

[popy2k14]

thx, for trying.
Maybe you get an answer.

As stated, for me two clocks are much more stable now with the added cap.

[UCaGk]

@popy2k14
I added a second 1000uF cap to the input of the step down in both clocks.

It's only been a few hours and I don't always check the clocks but it seems stable at the moment.
If you have a second cap in stock, it would be nice if you could plug it in too so we can get better stats :-)

It would be nice to have a method to catch the error without watching...

[popy2k14]

@UCaGk thx for the hint
I had spare caps laying around and also added those to the step down input. I was already really happy with the added caps on the output, I think that did the stable trick for me.
But yeah, caps on the Input could not harm.

Will see how it goes...

Yeah, since esp32 is stable since adding the output caps (checked with uptime sensor), it's hard to tell if there was an led matrix outage. Both clocks are under TV's in my appartement, so every day some family member is seeing the watch. And maybe catching an led matrix outage. But since I have added the output caps, no member complained about an issue with the clock, nor have I seen an issue.
Fingers 🤞🏻

P.s: can I use your picture for the reddit thread I created?

[UCaGk]

@popy2k14
Good to hear you have some spare parts !

Yes, you can use the picture, no problem. Hope the cap has some benefit.
One of my clock is also near by the TV the other under my monitor. While working the last days i not sure if i saw a short flicker or if it was only the seconds overflow.
Yes, the biggest problem is that there is no indicator were we can see if the display make some "mistakes".
Think have to watch some boring films on TV so there is enough time to check the clock :-)

[popy2k14]

@UCaGk thx, i have now used your picture (forgot to make some).
Yeah, let's binge something. Also the weather here says -> you must binge :-)