-
|
hi again, I have now 2 sites setup in solcast. I can see them in attributes but the second site never seems to have any data. See below for an example 
The data is only from one site. Previously I had just one site but due to having multiple rooftops, I added another. I've also rotated the API key which made the integration pull down the new site attribute but yet the data is missing. Any help is appreciated! Thanks. |
Beta Was this translation helpful? Give feedback.
Replies: 10 comments 52 replies
-
|
Can you follow the instructions at #38 to turn debug mode on, restart HA and share what you see in the logfile when your automation does a solcast retrieve |
Beta Was this translation helpful? Give feedback.
-
|
I see. I added the second site three days ago. I would have assumed that it would've picked up by now at least once. I understand the 10 call limit being split across two sites but I assume that it is in the following round robin order: site 1, site 2, site 1, site 2... |
Beta Was this translation helpful? Give feedback.
-
|
Good advice re. logging and five vs. ten calls, @gcoan. Damn. You replied while I was typing!
Let's see what happens in a few hours for midnight UTC reset, and after your next poll. I'm quite interested to work out whether this is a bug, as this is an infrequently encountered scenario, hence my desire to know the timing of site addition. |
Beta Was this translation helpful? Give feedback.
-
|
If it was added three days ago @simar7 then I am leaning towards convertng this discussion to an issue for further triage. |
Beta Was this translation helpful? Give feedback.
-
|
This is a latter version running. v4.1.6? Platform specifics? i.e. HASS/Docker? HA version? |
Beta Was this translation helpful? Give feedback.
-
|
@autoSteve see below for the timings on when it was added 
As for specifics, this is running v4.1.3 (can try to update it). Home assistant in a VM. HA versions below: |
Beta Was this translation helpful? Give feedback.
-
|
v4.1.6 is available. And yes, you could try that, but not before the midnight UTC API reset please. I want to see the logs of next forecast fetch. |
Beta Was this translation helpful? Give feedback.
-
|
So I have the logs but they get rotated often enough that I think we lost the midnight UTC messages Out of this, this interested me: But strangely enough the data for both the sites is exactly the same, for every single data point. This isn't correct as it can be seen on the Solcast graphs 
|
Beta Was this translation helpful? Give feedback.
-
|
I have updated it to the latest release v4.1.6 as well. |
Beta Was this translation helpful? Give feedback.
-
|
Just for clarity I'm graphing the |
Beta Was this translation helpful? Give feedback.
-
|
No debug logging. Gah. Set it up in configuration.yaml rather than setting "Enable debug logging" in the integration. That will get reset after a restart. (Logs will also rotate on restart.) Checking @bajarrr's logs... |
Beta Was this translation helpful? Give feedback.
-
|
ah bummer - sorry about that I've enabled via configuration.yaml now. Let's see. |
Beta Was this translation helpful? Give feedback.
-
|
I'm having the same issue as @simar7, but I have 4 sites (2 API tokens), only 1 shows in the site breakdown within attributes. I updated to v4.1.6 before doing an update with debug logging on and it shows an interesting result, some sensors show 0 for 3/4 sites, some show duplicate data: Debug logs below (hopefully I retrieved these correctly): |
Beta Was this translation helpful? Give feedback.
-
|
Sorry, I'm new to github, how do I format the log message dump more nicely than I did above? |
Beta Was this translation helpful? Give feedback.
-
|
You can wrap your text with triple ticks, ```
|
Beta Was this translation helpful? Give feedback.
-
|
These lines indicate that you have just added a second API key. Is this correct? |
Beta Was this translation helpful? Give feedback.
-
You can also edit a post to add the back-ticks. |
Beta Was this translation helpful? Give feedback.
-
|
I have similar oddness with the site breakdown attributes. The main sensor value appears to be fine. Checking further. The forecast fetch itself in this case looks perfectly fine. This may just be an instrumentation problem with building the attributes. 
|
Beta Was this translation helpful? Give feedback.
-
|
I just went back through my ha history and I added the second API key on Sept 2, is there something iffy with my config if the message is still coming up this long afterwards? I did update the version to 4.1.16 just before this reboot, could that cause the message? |
Beta Was this translation helpful? Give feedback.
-
|
Yep. That was added most recently, post v4.1.3. What was happening is that when a second API key got added there would exist both a solcast-sites.json, as well as the new solcast-sites-myapikeyhere.json. The added code cleans up the solcast-sites.json because not used. The drama with adding an API key was that it caused the sites usage to be reset to zero. It is now migrated from the solcast-usage.json into the new caches. |
Beta Was this translation helpful? Give feedback.
-
|
There is an issue at line 1080 (v4.1.6). Oops. If one or both of you could please replace this one line in With this: Then restart HA all should be well with the sites breakdown with the 'power now' sensor attributes (and others that utilise splined data). This does not explain the zero attribute values for some sensors like `peak forecast today' (which I can't replicate) so I'm looking into that. 
|
Beta Was this translation helpful? Give feedback.
-
|
@simar7 I'm interested in how you produced this chart in HA. I tried to graph my own multiple arrays against the forecast and failed. I'm guessing this wasn't an apex chart? |
Beta Was this translation helpful? Give feedback.
-
|
This is Grafana for data viz. I collect the metrics with the plugin and forward them to influxDB. Grafana is the front end. One caveat though with this is I need to figure out the "offsets" for forecast data still. I'm still exploring is this is possible with Grafana but it might just be easier to do so with influxDB directly. Correct me if I'm wrong but when the plugin collects forecast data it doesn't have future timestamps with it right? |
Beta Was this translation helpful? Give feedback.
-
Not sure I completely understand. By definition a forecast is future, so they are future tampstamps. The raw forecasts from Solcast have forward estimates for half-hourly average periods. These are held in the sensors forecast today, tomorrow, day_3, day_4, day_5, day_6, day_7 in the detailedForecast attribute. This is a summation of all Solcast sites. This integration runs those raw numbers through a spline calc for 'today' only, building per-five minute estimates (which is the same as the sensor update period), and these are fifteen-minute time shifted (because half-hourly average input data). These splines are not exposed outwardly, rather used to maintain things like 'power now' / 'power in 30' / 'power in 60' / 'power next X hours' sensors (with X-hour reverting to a linear interp. beyond the current day data). |
Beta Was this translation helpful? Give feedback.
-
|
That's right in the detailedForecast attribute there's this: The plugin sends this as-is. In influxDB it is stored as a string which makes it not very useable out of the box in a data viz layer like Grafana, at least I haven't figured that out just yet. 
Is it possible to "flatten" this data in a way as other metrics such as dayname are sent? 
If it's flattened it's easily queryable as a field 
|
Beta Was this translation helpful? Give feedback.
-
|
The existing attributes will not be messed with because many folks like @gcoan and I use them for use cases like Apex charts. We do this like so: Could you adopt this kind of approach to pulling data into a 'flattened' template that influxdb could pull? I have had issues recently with the Given this, if you can come up with a simple and viable sensor attribute structure that influxdb would grok nicely, then do call it out for consideration, as in the near future adding even more optional attributes to this integration will not cause recorder pain. (Just testing and code maintenance pain for the likes of me. And on that subject, if the proposal is too complex then with respect it will be rejected.) |
Beta Was this translation helpful? Give feedback.
-
|
Thinking out loud here. Generally when a zero is returned from a calculation function, where a far more positive response would be expected it is because an exception is occurring. These exceptions are logged as 'error' level. I see none of this in the logs shared. Updating the code as instructed at #166 (comment) should yield an immediate thumbs up or down for the attributes after an HA restart. What I expect to see is that the 'duplication' of results is now gone because this duplication was not actually a site breakdown, but rather the total for all sites duplicated across the site breakdown sensors, which was wrong. Sorry. What is breaking my head is the initial report by @simar7, where the 'power now' sensor sees zeros. This should have seen duplication, because it gets its value from splined data, yet instead it 'kind of' worked well but for one site only, which makes little sense. My head is further broken by @bajarrr reporting zeros for 'peak forecast today', which has nothing to do with the broken spline site breakdown that is fixed by the previously mentioned patch. There must be a futher issue lurking, but I haven't found it yet, and my primary install does not exhibit it making it harder to spot. Any and all sensor/attribute/logging data that I can get post applying the patch above will be greatly appreciated. Don't hold back. @bajarrr, your four-site set up is not something that I do, so I will particularly value experience. |
Beta Was this translation helpful? Give feedback.
-
|
This can be proven, @bajarrr, by adding this debug line after the first in buildforecastdata() and restarting HA. If this shows zero length for three sites then I have my smoking gun. If it does not, then the corruption is elsewhere. Yields for me: |
Beta Was this translation helpful? Give feedback.
-
|
@bajarrr, given I cannot replicate this it would be quite helpful if you could share your current There's no personally identifying infromation in that file. The integration is choking on something in it, given no forecasts have been retrieved from Solcast on startup. I can run it through an offline dev instance to replicate what is happening using reconstructed sites data from your logs, and stop bothering you with requesting code logging tweaks and useless restarts. |
Beta Was this translation helpful? Give feedback.
-
|
I have looked at my solcast pv forecast today and tomorrow, both of my sites are populated correctly for both estimate and estimate 10, and the Solcast totals are correct (sum of the sites) Peak time today and tomorrow, again I have total, total 10%, both sites and both sites 10%, all populated with what appear to be sensible values and all with today/tomorrow's date Anything else you want me to look at @autoSteve Like you I am leaning to this being caused by a corrupt solcast.json file, but how? |
Beta Was this translation helpful? Give feedback.
-
|
IKR?, @gcoan. But it has happened to two wonderful humans, so that cannot be a coincidence! Well, it could be, but we're talking Python here... There will be a smoking gun. Code is code, and this code did it. They didn't. (@bajarrr, @simar7, you aren't AI who have done this on purpose are you??? 😂) It sounds like you're in the "I'm fine" camp with me @gcoan, so no, nothing further bloke. Thanks for sharing your uneventful experience. |
Beta Was this translation helpful? Give feedback.
-
|
I've grabbed the solcast.json and attached below. And no, I am 100% a human with the normal 7 fingers on each hand encased in skin just like other normal humans. :) |
Beta Was this translation helpful? Give feedback.
-
|
Hate to be the bearer of bad news (and feel free to tell me if we need a separate discussion) but it seems the data isn't really matching up to what solcast report on their dashboard. See below for details 
While as solcast has much lower values (which are correct because today is rainy). 
One data point to compare is 12pm. 0.51kW at solcast vs 1242 W at the integration. Maybe I'm not looking at the right metric? This is graphing |
Beta Was this translation helpful? Give feedback.
-
|
This looks to be related to the earlier thing fixed where sites total was duplicated across the breakdown not being applied @simar7. I can't find evidence of this disparity so far, but I will look at that specific sensor tomorrow (the interpolated per-five minute power now, yes?). In a to-be-merged version.next of the code over at autoSteve/ha-solcast-solar, when configured I can get a detailedForecast half-hourly attribute broken down by site by looking at attributes in developer tools. These values exactly align to the Solcast web site for an individual site of mine. Switch to the updated |
Beta Was this translation helpful? Give feedback.
-
|
I have news. I know what is going wrong. I hacked together a set of fictional sites/usage caches to match your The cause is a bear trap that was laid by the former author, Oziee of which I have wandered into in the woods. (The underlying reason is calculation of what is called a 'forecast list slice'. The end-time basis for this is the length of all forecasts, which is questionable logic. It works great if every site timeline lines up, but not so much if sites have been added at a later date, which is absolutely the case here. The site breakdown arrays are of differing length, so the end index is dead wrong for newly added sites, hence the zero values. It also explains why a bajillion other folks are not chiming in with "Me too!", because they, like me set all sites up on the same day.) What I need to do is change the logic to something more sensible. I should be able to do that tonight, but it is not going to be a single-line replacement / move on / know that it will be fixed in the next release for good. It may involve downloading a whole new Are you two up for field testing in that way? Then again, it might just end up being a whole method replaced, with other bits changed not being important. I'll let you know the state of play in a while. |
Beta Was this translation helpful? Give feedback.
-
|
So this is great timing, because I had a theory that the problem could be that I added the 2nd 3rd and 4th sites after running with just one site for a while, so i thought to test this on a fresh install. Once I got a chance, I quickly spun up the dev home assistant instance I haven't used in a while, setup solcast-solar and lo and behold it works: I was part way through getting things setup when you posted this response :) Would a workaround be to just delete the integration and reinstall? or delete the broken solcast.json? Thanks heaps for all your help :) |
Beta Was this translation helpful? Give feedback.
-
|
Brill. Don't scorched earth existing... I have pushed a fix, and want a real world test, not done on my Frankenstein's monster of a test rig! Deets in a min. |
Beta Was this translation helpful? Give feedback.
-
FYI, this would change nothing. I'm mentioning it for the future, and for others here for posterity... It's probably worth me writing a troubleshooting FAQ post about. The solcast.json and sites/usage caches would not get removed on integration removal. It's designed this way so that one can do unnatural things like switch integration completely to the repo at autoSteve/ha-solcast-solar to check out an alpha release, or switch from the Oziee days and still have all the good data in place without going back to square one. Under my watch this integration will always be data file backwards compatible. Maybe not forward-compatible, but I would also consider that. The only way to go scorched earth / day0 is to delete the solcast files from /config and restart. All of these would get created shiny fresh and new, with all history lost, with double the normal API quota use than a simple forecast fetch being chewed. With a limit of ten that would hurt. |
Beta Was this translation helpful? Give feedback.
-
|
If you switch to the post 4.1.6 branch at BJReplay (https://github.com/BJReplay/ha-solcast-solar/tree/Post-v4.1.6) you can grab the This includes the earlier fix, plus this one. Enjoy, and report back! I look forward to hearing that all is brilliant with sites breakdown. (I haven't overlooked your prior comment @simar7. It's on the to-do list to check out.) |
Beta Was this translation helpful? Give feedback.
-
|
Absolutely great to hear. Happy Friday. (Soz. Work got in the way of fixing this earlier.) |
Beta Was this translation helpful? Give feedback.
-
|
Just tried this @autoSteve - I'm hopefully that this is looking much better now (and correct). 
I will have the full forecast graphs by the end of the day to compare with the real production. |
Beta Was this translation helpful? Give feedback.
-
|
As promised, it is looking good! 
Right around the mark of the annotation is when I switched over to the new version. |
Beta Was this translation helpful? Give feedback.
-
|
Would you release a new version @autoSteve? |
Beta Was this translation helpful? Give feedback.
-
|
That is the plan, @simar7. A PR for v4.1.7 is approved by BJ and I intend to merge it soon, after my final code review. Thanks for highlighting this issue. |
Beta Was this translation helpful? Give feedback.
-
|
v4.1.7 is released, fixing all this. Closing. |
Beta Was this translation helpful? Give feedback.
There is an issue at line 1080 (v4.1.6). Oops.
If one or both of you could please replace this one line in
/config/custom_components/solcast_solar/solcastapi.py:With this:
Then restart HA all should be well with the sites breakdown with the 'power now' sensor attributes (and others that utilise splined data).
This does not explain the zero attribute values for some sensors like `peak forecast today' (which I can't replicate) so I'm lo…