Stopped working midnight 28th/29th jan #238
Comments
|
Here is the detailed log for just one of the many errors: This error originated from a custom integration. Logger: custom_components.solcast_solar.sensor Unable to get sensor value: 'pv_estimate': Traceback (most recent call last): File "/config/custom_components/solcast_solar/sensor.py", line 465, in _handle_coordinator_update self._sensor_data = self._coordinator.get_sensor_value(self.entity_description.key) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/config/custom_components/solcast_solar/coordinator.py", line 479, in get_sensor_value return self.__get_value[key][0]"method" ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^ File "/config/custom_components/solcast_solar/solcastapi.py", line 1912, in get_forecast_remaining_today self.__get_forecast_pv_remaining( ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^ start_utc, ^^^^^^^^^^ ...<2 lines>... forecast_confidence=forecast_confidence, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ), ^ File "/config/custom_components/solcast_solar/solcastapi.py", line 2208, in __get_forecast_pv_remaining result = self.__get_remaining(site, forecast_confidence, (start_utc - day_start).total_seconds()) File "/config/custom_components/solcast_solar/solcastapi.py", line 2171, in __get_remaining return self._forecasts_remaining["all" if site is None else site][ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^ self._use_forecast_confidence if forecast_confidence is None else forecast_confidence ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ][int(n_min / 300)] ^ KeyError: 'pv_estimate' |
|
My Solcast integration is working fine, also with two arrays so I don't think its a problem with the integration. Try shutting down and restarting Home Assistant |
|
I restarted ha and forced an update to a sensor, and am still getting errotlrs like: This error originated from a custom integration. Logger: custom_components.solcast_solar.solcastapi Exception in __get_forecast_pv_remaining(): 'pv_estimate': Traceback (most recent call last): File "/config/custom_components/solcast_solar/solcastapi.py", line 2208, in __get_forecast_pv_remaining result = self.__get_remaining(site, forecast_confidence, (start_utc - day_start).total_seconds()) File "/config/custom_components/solcast_solar/solcastapi.py", line 2171, in __get_remaining return self._forecasts_remaining["all" if site is None else site][ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^ self._use_forecast_confidence if forecast_confidence is None else forecast_confidence ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ][int(n_min / 300)] ^ KeyError: 'pv_estimate' I attached the integration .json debug |
|
Those log entries are a symptom, not the root, and just noise. And they are highly un-readable (please use back-ticks ``` to enclose posted logs, @GSV3MiaC). This is a symptom of remaining and momentary splines not being available for a given pv_estimate type (10/50/90). I believe this emanates from a bug in v4.2.7 where logs explode like this after messing with attribute breakdown options. pv_estimate50 I believe. It is highly likely that you changed options, or upgraded to this version (I have no context in this regard), and then midnight rolled over and the splines were recalculated. Is this what happened? If so, turn the pv_estimate options back on for now and re-load the integration. Fixed in v4.2.8 (coming). |
|
Thanks Steve, yes I upgraded to the 4.2.7 version 4 days ago. I changed no options, just forced to do various integration updates since BMW integration broke (by BMW). Will try what you suggest. |
|
Still does not produce most of the entities, with same log errors. Guess I have to wait for 4.2.8. The daily forecast for today is there, everything else (remaining, next 6 days, etc) comes up null. |
|
This is most odd then. It is spline-related, so please post the log from the beginning of start (in back-ticks ```) up to where the log spews. Debug logging enabled would also be of benefit (#38 (comment)). Please also post your options set, ensuring your API key is not in the snip. |
|
(Or simply attach the log. At debug level the set options are logged, so if it's simpler just attach a debug log alone.) |
|
Here is the last part of the log, hope it contains all you need. home-assistant_2025-01-29T09-05-28.901Z.log and here is options
|
|
These are the sensors it can/can't get:
|
|
Enable all of these. Well, try and just ENABLE estimate 50 first, not disable. That is the cause of the log explosion. Then reload integration. 
|
|
I had already tried changing them both ways, no joy. I just ticked 50, and reloaded, and this is the log:
Maybe is easier to delete the integration completely and reload from scratch? I have no idea what version I was at before I upgraded. |
|
You could try turning all options on except the site breakdown, (and of course don't tick the modify dampening factors). This is the default options settings. Rather than scorched-earth remove/re-install, I would rather get to the bottom of this to understand whether it is a known issue with 4.2.7. |
|
From your attached log the logged issue is now with Enable the options as I asked, then re-load the integration. |
|
OK here is the current options (less api key):
reloaded integration asked for an update of entities, got same errors in log and missing entity data. I see in the ha /config folder there is a solcast.json (and a couple of others) which have not been up[dated since the version change on 22/Jan .. should those be rewritten? The solcast-sites.json was rewritten today. |
|
-sites.json gets written every startup. I am out of ideas. What does "asked for an update of entities" mean? The integration will update all related entities on start... |
|
Only sites.Json is being written I guess the other writes don't happen because the integration never finishes? |
Can you please save a backup of the solcast*.* files? We might get you to share them. First, remove the integration. If you updated from a very old version, it doesn't have much history, so the best bet is back up those files - so we can have a look, then delete them. Then, with the files removed, add the integration again, and run through the basic configuration. It should just work. |
|
@GSV3MiaC do you have a backup prior to doing the upgrade? If so, the solcast*.* files from that backup will help us understand what the situation was like. And do you know what version you upgraded from? |
|
Hello Logger: custom_components.solcast_solar.solcastapi Exception in load_saved_data(): Traceback (most recent call last): File "/config/custom_components/solcast_solar/solcastapi.py", line 1364, in adds_moves_changes cache_sites = list(self._data["siteinfo"].keys()) ~~~~~~~~~~^^^^^^^^^^^^ KeyError: 'siteinfo' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/config/custom_components/solcast_solar/solcastapi.py", line 1446, in load_saved_data await adds_moves_changes() File "/config/custom_components/solcast_solar/solcastapi.py", line 1383, in adds_moves_changes raise f"Exception while adding new sites: {e}" from e TypeError: exceptions must derive from BaseException Thanks very much |
|
Totally different issue, @timmo65. KeyError 'siteinfo' would indicate that you have come from a VERY old integration, probably from the early Oziee days or before. This is not considered an "upgrade" in my opinion, however v4.2.8 when it is released will cope with this. I consider it a "migration" because the code is so far removed from that version that even Oziee did not code in a migration from v3 to v4. My suggestion? Scorched earth re-installation. v3 did not retain history, so you are losing nothing if this is the case. Delete everything If you want to double check before the "scorched earth" kill-it-with-fire approach, then post your And welcome. This integration is much evolved. |
|
Hello and thanks for the welcome and super quick response. I had deleted the custom components folder but hadnt considered the json files. I removed those and it sprung into life. I must have meddled with this a couple of years ago and gave up on it - so now time to revisit. |
|
v4.2.8 cannot come quickly enough for the "old Oziee gang" who keep getting smacked in the face by legacy. 😉 But I am not going to rush it. My aim is a slowly delivered assurance of quality and reliability. v4.2.7 is good, so @timmo65 you're good to go. Just please do not turn off the options for estimate 10/50/90 attributes until v4.2.8. 😬 4.2.8 carries way more than superficial bugs fixed. Behind the scenes it incorporates much unit/integration testing of the Python source (100% goal, 99% achieved), and also carries strict type checking for all variables. This has picked up many sleeping bugs that us humans have not hit yet. Achieving this has taken countless hours, and almost as many thousands of lines of pytest code as there are lines of integration. But it's been fun. |
|
Steve, here are the solcast custom component files from backups. Not sure which one of the two sets was active when it last worked, I suspect maybe the antique 2024 version. Hopefully you can handle .zip files. let me know if you need more. I will also save the current solcast components folder before I slash and burn. I suspect I may have similar problem to the one above - 'upgrading' via HACS some old installation which was not put there that way. Just took a week for the problem to work thru the caching and surface. ========== Update .. slash and burn and redownload worked just fine. I think I may be a belated ozzie victim. Btw, I do appreciate your efforts. Do I gather you are involved in the tesla messes too? [There is some timing issue with the custom tesla integration, where almost all the sensors go 'unavailable' for several seconds before assuming a sensible value. I guess there is some issue with the new asynch stuff - creating the entity and assigning it a value are not happening at the same instant. Seen similar on some other old integrations too] |
Thanks, @GSV3MiaC - that's really useful. It looks like you've gone from The big change in 4.1.8 was Full spread of attributes breakdown by siteAbsolutely all sensor attributes may now be broken down by site, including the detailed half-hourly and hourly forecasts. For why? I dunno. They're there for the taking, and for you to decide what you want to do with the data. How about build an Apex chart representation of each site forecast? Or build an automated per-site dampening automation? Your install, your things. (And tell us if you can't do what you want because the data is missing. We may have missed exposing something.)
The big attributes, like per-site detailed forecasts are not sent to recorder, so your Home Assistant database size will not suffer even with everything turned on. So you've upgraded from a non-attribute breakdown version to an attribute breakdown version - albeit one with a bug that has hit 2 other people (so we're not rushing the fix - but @autoSteve and I are both running the code that will be 4.2.8. I still manage to break it every now and again, and Steve still manages to fix it. solcast-undampened was created 2025-01-22T18:18:52+00:00 and indicates that at least one update worked but has a unix-epoch (1 Jan 1970) date for It looks like it got your forecasts once around the 22nd, and that's why you had forecasts available until the 29th. solcast.json indicates Whereas solcast(1).json indicates: , So, it kind of feels like a v4 to v5 solcast.json format update gone wrong. Having said that, it does look like the integration managed to migrate, because it has written new files. If I was to guess - and that's all it is - you might have set some of the configuration options at the same time you set up the new version in such a way to allow it to upgrade, but not to then calculate splines (due to the bug in 4.2.7). You didn't notice until the 29th because it had fetched forecasts. @autoSteve I think this is the attributes issue already fixed, just with confusing timing. It occurred around the 22nd, but wasn't noticed until the 29th when the forecasts ran out is my guess. |
|
Pretty sure that this is: Fixed by #226 |
That code is dumb as a bagful of hammers, so I'd think unlikely. I recently reviewed it because v4.2.7 -> v4.2.8+ transitions to v6 schema. I did not have to alter the upgrade sequence much, and the PyTest code tested fine for v4->v5. It still tests fine for v?->v5/v6. My theory is that the integration was set in motion, performed one update because the cached data was desperately out of date (thanks, @gcoan for that feature), and then an automation to upgrade the forecasts was either disabled or non-existing. Auto-update is not mentioned in the .json files, and the default for an upgrade from an old version that did not do auto-update is auto-update off. And then time passed. And the data available got less and less. And then, "surprise!" Log spew when data covering "today" ran out. (Splined data is only built from a little over 24-hours worth of forecasts covering "today" and an hour into "tomoz".) |
Ahh, that makes a lot of sense. Would need lots of logs - and the stamina to want to review them for something that is known to be working again. |
Which is actually a fascinating thought. We track the number of days ahead that have valid data available. At "~five days until the tank is empty" should the integration raise that a repair is required? I do check for this situation now in v4.2.8, but the response is a rather benign only spew a few logs instead of filling the disk with them repeatedly every five minutes. But the user is unaware of this until they scratch their head wondering where solar forecasting went, and maybe have a gander at the logs. |
And correct. Issuing an action "Solcast PV Forecast: Update" would have revealed fruitful things. Until the forecasts ran out again, hence the need for an automation to call this, or simply enable auto-update and simplify things. We don't get new data from Solcast on entity update because one prime reason. API calls are just way too scarce given the super-small daily quota of ten for new hobbyist users. |
|
(Thank you, @GSV3MiaC for the couple of pots of ale.) |
|
Well the link was to the. Png image, but I founfld you anyway.
Rgds, Dave Storey
…On 30 Jan 2025, 07:47, at 07:47, Steve Saunders ***@***.***> wrote:
> Do you have a buy a beer link??
It's actually a pleasure. I want this to work for me so...
I do have a pat-me-on-the-bum link with a one-off or subscription on
GitHub sponsors...
<img width="695" alt="Image"
src="https://github.com/user-attachments/assets/f2d6f33e-13b1-4081-bd14-17d9847bd0e6"
/>
--
Reply to this email directly or view it on GitHub:
#238 (comment)
You are receiving this because you were mentioned.
Message ID:
***@***.***>
|
Good idea, if we know what the problem is.
Or...
Yeah, this seems like the go. I won't attempt to mark it up in github, but a message along the lines of "No updates detected in the last five days - would you like to enable automatic updates? Choose Yes to enable automatic updates, Dismiss to dismiss this message and not enable updates". Fire after 3 days as part of the stale check - so if they dismiss after the first warning, they get notifications for the next three days before it explodes. |
|
Ah so 'update entity day 3 forecast' is basically a nop, unless the date has changed.
I am an antique user, got 50 api calls. 🙂
Rgds, Dave Storey
…On 30 Jan 2025, 07:55, at 07:55, Steve Saunders ***@***.***> wrote:
> Ah, when I used ha action 'update entity' after trying the fixes,
maybe that just goes to look at the cached data (which is still not
there?) rather than forcing a new api call?
And correct. Issuing an action "Solcast PV Forecast: Update" would have
revealed fruitful things. Until the forecasts ran out again, hence the
need for an automation to call this, or simply enable auto-update and
simplify things.
We don't get new data from Solcast on entity update because one prime
reason. API calls are just way too scarce given the super-small daily
quota of ten for new hobbyist users.
--
Reply to this email directly or view it on GitHub:
#238 (comment)
You are receiving this because you were mentioned.
Message ID:
***@***.***>
|
I was lookin' to do something this evening. It's either this or Rambo, pt 4. Rambo is actually winning for the time being, as one cannot get enough of Stallone's, "Mmmf, ya know whaaaa? Heard he bought it back in Nammm, but sclywg sez hm gfmmm bhurff and ta-da, a fkn ROCKET!" He's so articulate. |
|
Super disappointed no Andy Vajna producing "Rambo: Last Blood". (Vagna. Tee-hee.) He died in 2019, the same year this last missive was released upon the world, only receiving a critics "meh" and barely a cap dothing to the closure. "It is the streak of flabby sentimental self-importance that makes Rambo: Last Blood unwatchable." Righto. I think you kind of missed the point that Rambo is now an old bastard who fought like a man possessed for one last time for a real and personal "human" reason, not a senseless war and now done has no fight left in him, and shall not fight again. Kind of symbolically going down with the Vagna trilogy ship. Ende. I thought it was a perfectly unwatchable book-end. It never wanted writing, but it deffo wanted watching. |
So am I - but I only use 20 of them. 30 are for testing this integration 🤣
If you want to update forecasts, you need to call the action force_update_forecasts. It can be done through developer tools, or the simplest automation which you run when you want to update them: alias: Solcast Force Update
description: Solcast Force Update
mode: single
triggers: []
conditions: []
actions:
- action: solcast_solar.force_update_forecasts
data: {}This will cause the integration to request new data from Solcast (and, providing that Solcast is playing nice and doesn't 429 and time out or refuse to serve you), will fetch that data, and update all the sensors.
After calling the automation:
|
|
If it is 'going to explode' , just do an update whether the user wants one or not. You are just using one api call, and avoiding a major support furore. I mean one api call, come on, not like you are wiping their hard drive, or abducting their wife. 🙂 |
Auto-update by stealth. A possibility, although not without its weaknesses, too. A networking issue that prevents forecast updates over a period of time will also result in the days ahead dwindling.
|
|
Yes it can still fail, and then you crater as before (but most of those failures won't even USE the one api call, because you get nowhere near the website) |
|
Simple is best, here:
As the repo owner, and having been in this (software) game for a while, this looks like a scale perspective issue:
|
|
Simple is best @BJReplay. Read the doco is good, but being alerted when the future forecasts are starting to run out is a step better, and a low risk change. I shall also attempt to add a link in the repair dialogue to the doco... |
|
Pressing "Learn more" takes the user to the GitHub README.md section regarding updating forecasts. 
|
|
I was calling manual update about once an hour iirc. The updated integration had the auto update box checked (see screenshot from 2 data ago) , but seems like (even after ha restart) that was not happening, and I guess the updated integration was not accepting the previous hourly calls? After slash and burn the auto update is working fine as far as I can tell, and yes, it is bleating about manual update calls which I had failed to disable. |
|
Fascinating, @GSV3MiaC. I had "tunnel vision" for the attribute options, and did not spot the auto-update setting change. Changing the setting for auto-update will result in an integration re-start. That it did not auto-update is disturbing. And way-out-of-date forecast should also result in an immediate fetch on load, but sadly I could not see if that had attempted because not in log. The earlier zipped log was partial, and not from the start of HA load. What were the actual settings at that time? Not sure because almost certainly not at I got nothing but the currently committed Band-aid. |
|
(Soz. IPL = Initial Programme Load. IBM speak. Soz. I hate acronyms, yet I used an acronym... 🤦🏻) |
|
@BJReplay, statistics be damned, we may hit this one again given the latest from Dave. Hope not 🤞, but there would appear to be an elusive hole that wants plugging given he/they/it (can't be sure of pronouns these days, @GSV3MiaC 😂) switched to auto-updates. This should have got things back on the straight and level, and yet... Yeah, nah. Boom. |
|
(Then again, maybe not enough time had elapsed to allow an auto-update to occur. This was all in the same evening I think. Any configured automation doing a simple "update" would have been refused because auto-update set, requiring a "force update" instead...) |
|
(But then, ignore me. On re-start an "ancient" update check would occur [thanks to @gcoan], and an update should be forced. So again, back to I got nothin'. Unless that update was blocked in v4.2.7, which is plausible.) |
Yeah, it is interesting. Replying from my phone, but, from memory I thought auto-update was off from the zipped configs shared. Not gonna attempt to open zip the Json from phone, but even if I were to, it won't recreate the history or sequence of events. At least one (and possibly at most one) fetch was done. solcast.json and solcast-usage.json were written. And I have been typing wss so much today that the phone corrected was to wss. |
|
Sundry replies .. Dave = GSV3MiaC (GSV 3 Minds in a Can .. you have to be an Iain M Banks fan) I updated via HACS from an ancient version, which I may not have fully cleaned up. I did not change any config options. I may have logs back that far, but not debug log. Should have defaulted to 'auto' but apparently something was never set up because .. I continued to run hourly " Perform action 'Solcast PV Forecast: Update' " automation, and got no bleating about it being configured for auto (though it did that after I slashed and burned). It obviously was NOT going getting the data, though I don't think I have api usage data back that far to see. Nor was the auto update getting any, and I also had several restarts over the period iirc. the log shows lots of 'gee the data is ancient' complaints, right before it then spewed the spline errors. The HA config shows that the .json files were not apparently updated (though whether that happens early in the fetch I don't know .. I would suspect those would be written early??) Let me know if I can did out any more. I guess the logs may also be in the backups which I still have, but debug won't be on. |
|
Two distinct repair flows now, @BJReplay... First if auto-update is already enabled and forecast data is running out: 
Second if updates are manual and forecast data is running out: 
|
|
That is brilliant. I have just spent 4 hours as an electrician's apprentice - 7.3kW charger installed. And water skiing this morning. I am stuffed. |
Usually involves getting sent under the floorboards with the white tail spiders or into the roof cavity with mice, rats and possum piss. Hope not. 😂 Rest. Beer. Revive. |
And some work on a ladder with a yellow-tongue. Really neat install, impressed with the sparky. Charger is on OCPP, automations are drafted for charge at full noise from 11am to 1:55pm, and excess solar the rest of the time. Yet to be tested on real hardware, but seemed to work on the simulator. |
As of 00:00 29th Jan the integration is producing a slew of errors and failing to generate most forecast sensors. I can only assume solcast must have changed the format. I updated to latest version 4 days ago.
Sample log file attached.
I do have 2 arrays defined for solcast, which may be unusual
I am running 4.2.7 under HA 2025.1.3, on a headless pc running HA OS
The text was updated successfully, but these errors were encountered: