Home Assistant hangs when the inverters aren't available #344
Comments
|
I got the same issue, when inverters are disconnected, HA gets slow and i got reconnects from the website. As a workaround i disabled the integrations and HA is running correct again. I have a deye sun1600G3-230 with the 4mppt profile connected and 3x the sun800G3-230 with the 2mppt profile. I don't have any logging yet from the issue, but just want to tell you are not the only one with this behavior :-) when the inverters are going offline, i noticed the issue yesterday after upgrading to the v24.12.22 version |
|
Yeah, that's unfortunate, but I'm not able to replicate your problem, and since there are many users with microinverters, it must be in combination with something else, or there must be more to it than just running the integration. You can for example try to run clean install of HA and just this integration added as a starting point. What is the type of the installation of your HA anyway? |
|
I created a new HA in Proxmox with the following setup ( see scrip webpage ) https://community-scripts.github.io/ProxmoxVE/scripts?id=haos-vm and i deleted all default integrations and only installed this solarman integration with my 4 inverters, i will check tonight when they are offline the status, i also enabled the debug logging fingers crossed :-) |
|
I'm also running haos but just in qemu (which shouldn't make any difference) and I think it's the most reliable way. 😉 BTW run it with debug logging enabled nonstop so we have as much data as possible. |
|
Hello, Attached please find some debug logging. FYI: I hope this will help you. |
It's used by HA for docker. |
|
Another thing to try:
|
|
I am facing also problem when inverters go offline. I have DEYE M80G3 and M80G4. |
This exception is to be "expected" when is the device unreachable. |
|
@radektkacik, it's a stretch but changes in feat: Improve inverter load method exception resolution could potentially help. |
Unfortunately it didn't help. I think there must be some synchronous call which hangs whole HA for a while. After disable integration it behaves normally. |
davidrapan
changed the title
This would mean it would also happen w/ my installation too which is not the case. Did at least the exception change? |
Maybe you can try add entities on the main dashboard, and also make automation which will use entities of the inverter. I can confirm the problem is the same as lenwar described. |
Not sure what do you mean by that. Even so it can't affect the integration in any way of course I'm using entities on my dashboards and have plenty of automations which do interact w/ inverter (some of which even starts alongside every HA start). |
|
I have template sensors that use the entities. Maybe that is something? |
|
Me too and again this just can't in any way affect the integration. |
Maybe it is related specifically to microinverters only ? Just guessing. |
My idea was that the integration sets the entities on ‘available’, but it won’t actually fill them or lock them somehow, which causes it to hang. I haven’t checked anything, but that was something that came in to mind. (I have no idea how the internals of the template sensors work, so maybe that behaviour is covered) |
The integration has no idea what the device on the other end is until it successfully reads some modbus registers.
Both.
There are no entities until after successful connection. |
|
In your case everything what is called/processed is inside this method and any entities are created only if this action is successful. This method inside calls (and depends on successful call) of which tries to discover and connect to your devices and read registers 0-22. |
|
Maybe it is something with Home Assistant itself? I can`t see any big CPU load when it happens at my system, but it took like 10 seconds before the new page loads if I click something in HA. It feels like a waiting for a timeout. Also if I click on a sensor the graph will load after ~10 seconds. My setup: (Proxmox) 2 vCPU cores with 8 GB RAM and HA OS |
|
I can confirm the same behaviour as Skarabaen described. HA waits for something in case of unavailable inverter. |
|
Attached the debug log file, this is from a fresh HA installation with only the Solarman integration added with 4 deye microinverters. At about 16.21 the inverter(s) got offline because of the darkness, then you can see the disconnects in the error log. At that moment HA gets slow with loading the pages and i got reconnect messages from the HA webpage home-assistant_solarman_2024-12-25T16-13-16.155Z.zip (because the txt file is 28 mb big, i needed to zip it ) |
|
I don't know what more debug logs i need to enable in HA itself, but i have this test instance still runing, so just tell me if more is needed then i can collect the logs if needed |
|
Would it be useful to maybe write out specific aspects of our network setups? Even though they seem or are irrelevant for the behaviour? Possibly, it's a combination of network topology/hardware/setups? For myself:
I don't see how the items above could be relevant or explain the behaviour, but maybe we can find something. |
|
I don`t think it is something with the network or hardware, because as soon as the inverters have enough sunlight to power on there is no issue. As soon as the sun goes down = inverters go offline the issue starts. So in my opinion it is something with the plugin or interface to HA. Can we put something in the code like ping device and if it is available then use the code, otherwise wait x seconds to ping again and so on? A question about the profile. it is set to "auto" at my setup. How can I see which profile the addon is using? I now set it to "deye_micro.yaml" for my MI600 Inverter. Under additional options there was also number of MPPT set to 4 (I moved it now to 2 MPPT) and number of Phase was set to 3 and i moved it to 1. Maybe this will help. |
|
So in that case it has to happen w/ fake inverter too, so? Did someone test it? |
|
Maybe some library is causing it (modbus..) ? |
You need to remove the |
|
I removed the await this afternoon. The sun is down, and everything is fine, now! Everything keeps working snappy. I’ll check in tomorrow when the inverters are back up again. |
|
I got the same issue once when inverter wasn't unreachable due to one mesh instance was off so there was no connection. HA freezed, started to work only after restart. |
|
It seems to work now. Install the currently latest available version 24.12.22 and change the api.py and discover.py with the version of the repo. No lags anymore. |
|
Hi, A few minutes ago, my last inverter came back online. From my angle, this resolves the issue. For completeness: I replaced api.py and discover.py with the current 'Main' version. By removing the 'await' and 'discover()' phrases and commenting out line 110 and 111, everything is good. The interface is snappy (for as much as we can expect from the HA-interface ;) ) and the wall-buttons respond 'instantly'. |
I have not very good wifi signal at inverter. After this change couple of hours ago i had sensors not available for 9 mins and than another time for 11mins. But its better than HA freeze 🙂
|
|
Hi @lenwar, can you please test for me which on of those edits has bigger impact? |
|
Hi, I was able to test in the evening (restarting Home Assistant with the inverters offline) If you like, I can also test tomorrow when the Integrations are running and try to firewall my inverters. (so that HA can't connect to them any more, to mimic the behaviour of them going offline after sundown) |
|
Maybe you could try to measure runtime of both code blocks? So we can distil specific parts of the code which are causing the delays. |
|
I would like to try, but I wouldn’t have a clue on how to do that. Could you provide info on that? |
|
Hi all I just just installed it, I had the other integration… without any uninstall or other… it blocked my HA with big lags and reboot. I needed to delete the configuration of my micro inverter that looks to have been transfered here. As they are offline maybe its the reason … hope it will works tomorrow for first configuration and will run when they will be offline but « known ». I didnt get this problem with other integration… but came here for up to date support 😉 and looking for integration of smartmeter DDZY422-D2 |
|
I have got into the habit of disabling this integration if I need to restart HA whilst the inverter is offline and then manually enabling the next day. I haven't noticed the slow down but I do get a huge log output if I forget. Fingers crossed for #72. |
I will try to think about it before restarting… begining in HA I restart very often 😂 |
I was able to perform some tests with debugging on. What exactly does 'broadcasting on ' do? (network-wise) I'm asking because my inverters are in another subnet (as you can see later on, in the logging. My inverter is on 192.168.60.41 and my Home Assistant is in 192.168.60.16/29, which is another subnet.) I only have TCP ports 80 and 8899 open towards the subnet of my inverters. When I look at the logging. Is it correct that the integrations always wait for one another? (They don't seem to work completely parallel to one-another. But I may be mistaking. I thought I noticed that by how they write their debug-logging towards the home-assistant log file.) |
|
I'm not sure if I should post this here or start a new issue but with the current 24.12.14 variant it no longer handles the inverter going off line cleanly, in fact it's the worst it's ever been!
you can see from the message in logs the issue 3420 times!! It's not happy for an event that happens every day. I dare not turn on debug logging as I don't think my server will cope with the volume. |
|
Hi, If that is the case, then that concept is 'uncommon' at best and that won't work across networks. (UDP-broadcasting isn't forwarded on routers by default, as it is a common DoS-method.) Normally you would send a Udp-broadcast towards the broadcast address (255.255.255.255) to hit every possible node within a subnet. (Which is the purpose of a broadcast.) You loop through all nodes that respond and perform relevant actions for the responses. I'm not entirely familiar with HA's network stack/hygiene but I can imagine that this can causes issues. If you would remain within the same subnet, then technically it would work, but it's still an uncommon way of working. What is the purpose of the discovery-packet towards the configured ip-address? Edit: If my assumption is correct, you can replicate the behaviour yourself be defining 5 (or more) (fake) integrations with target ip-addresses outside HA's own subnet, where no UDP-broadcasting is forwarded. The more integrations you define the stronger the effect. |
|
In fact when the inverter goes offline it never stops trying, now at over 19,000 attempts!! I'm reverting back to a previous version....
|
|
I'm gonna need full debug log from that stack to be able to do something about it. |
|
@gedger Do you have any specific non-standard (for consumers) network layout? Is your inverter in another network segment? Or do you have any type of firewalling/proxying going on around Home Assistant and/or your inverters? So anything which is not "having HA in the same network segment than your inverter(s) without any firewall/proxy/etc" |
|
No, everything is on the same subnet, the inverter just goes off-line as it does every night. It's something that has been introduced in the last few releases as I reverted back to an earlier version and everything is fine. I'm rarely in a position to grab a debug log at sunset at the moment but I'll see what I can do. I can't leave debug logging on as when it happens it already swamps the system with normal logging and I suspect debug may well bring it to its knees. Once I can get a debug log I'll create a new issue as it is obviously something different to your issues. I think my slowdown is caused by the sheer mass of logging rather than anything else? |
|
There really was not introduced anything new and this type of an error should not even happen in usual course of actions. That being said it could be thrown out I guess. |
|
I did some tcpdumping: For completeness: Network: 192.168.60.16/29 My inverters are in: tcpdump -nn -i eth0.616 'ip and src host 192.168.60.18 and (ether broadcast)' -vvvvv -X Looking on Home Assistant, I see the same packages: (the bad udp chksum are normal on Linux when broadcasting UDP) tcpdump -i end0 'ip and (ether broadcast)' -nn -vvvvv -X So this looks okay. The nic of HA is actually forwarding the packages to my router. (so I think we can rule out some low-level thing in the operating system) Is there any reason why you are trying to 'broadcast' this, instead of 'multicasting' it, or even better unicasting it directly to the configured node? Is that something specific for the inverters? When I look at the behaviour of the iOS-app (on my phone). I also get one broadcast-message when I start it up, but with the payload 'tryme', but it only tries once and then never again. (( No broadcast-traffic is happening in the subnet in which my inverters live )). Been staring at it for a few minutes now. )) What exactly is the purpose of the UDP broadcast? I have commented it out as per your suggestion, and everything works fine then. It is impossible that it ever received any form of response ever, because of how my network is set up (it never reaches my inverters). |
Yes, it's specific to the hw of the stick.
Yes.
I thought it's obvious. Mainly inverter discovery but can also react to IP change or detect incorrect serial number configured. 😉 |
Ah... Far too obvious for me to have thought of that 😃 duh... (I'm a Linux engineer, not a software developer, obviously 😉 )
If you can make the discovery optional, then it would resolve the issue. (Give it a checkmark for detecting IP-changes or something like that, make a suggestion to give the inverter a static address or something like that, and so on) Technically, it would only be a workaround, of course. I was just thinking. Could it be a bug in HA itself? (As broadcasting isn't common, it may not happen often.) |
|
That wasn't meant to be an insult or anything. 😆 |
|
I agree it seems unnecessary to do a full discovery every time you connect. In my view discovery should be an option at configuration time and once the inverter is discovered the address hard coded. 99% of routers will always give the same address to a device but it always make sense to me to fix the address. |
I know. It probably got lost in translation somewhere. 😄 If you can make the (re)discovery optional, then the issue will be gone at least. It may be worth it to investigate further (outside of this Issue), as it could be a bug in HA itself? |
|
Same problem here. My home-assistant will become very slow and unresponsive when my inverter is offline. When I disable the addon, it's butter smooth again. |
Hi, I have the same issues, editing this file solves them for me, it seems stable so far. |
Description
When using the ha-solarman integration and my micro-inverters aren't available, my entire Home Assistant instance 'hangs' periodically (automations don't run, the web interface hangs and so on). I'm not sure to troubleshoot this, as there is no specific logging.
What I notice is that everything hangs when the ha-solarman integration tries to connect to the inverters that are switched off. When it disconnects, everything works fine. (which is about 4 seconds before it tries to reconnect again) So the 4 seconds between the 5 top entries and the 5 bottom entries in the logging below, everything is fine.)
So when the last log-lines say 'Connecting', then everything hangs.
I'm using 5 dual-port Solarman/Deye micro inverters using the 2MPPT profile.
Firmware: MW3_16U_5408_5.0C-S
I made a similar bug-report some time ago, but I didn't have the opportunity to respond and troubleshoot, so I closed the issue. The behaviour was different back then
#101
Reproduction Steps
Have the ha-solarman integration running when my micro-inverters aren't available. (due to the sun being under for example)
Log
Version
v24.12.22
Home Assistant Version
2024.12.5
The text was updated successfully, but these errors were encountered: