first pass #127

Hellowlol · 2022-03-21T12:24:51Z

This PR is an attempt to make the integration more resilient against Nordpool API errors, by retrying on HTTP errors and missing/wrong data with exponential retries.

custom_components/nordpool/__init__.py

yozik04 · 2022-03-23T10:53:14Z

With the version from yesterday there is a bug that it keeps the same prices for tommorow.
Today
0.171, 0.179, 0.175, 0.173, 0.175, 0.116, 0.241, 0.32, 0.348, 0.329, 0.252, 0.234, 0.223, 0.209, 0.148, 0.132, 0.093, 0.132, 0.27, 0.306, 0.306, 0.277, 0.267, 0.277
Tomorrow
0.171, 0.179, 0.175, 0.173, 0.175, 0.116, 0.241, 0.32, 0.348, 0.329, 0.252, 0.234, 0.223, 0.209, 0.148, 0.132, 0.093, 0.132, 0.27, 0.306, 0.306, 0.277, 0.267, 0.277

Raw today and Raw tomorrow have both same data, same timestamps, same values.

custom_components/nordpool/__init__.py

yozik04 · 2022-04-02T07:42:50Z

2022-04-02 10:37:35 WARNING (MainThread) [homeassistant.components.sensor] Setup of sensor platform nordpool is taking over 10 seconds.
....
2022-04-02 10:41:26 WARNING (MainThread) [homeassistant.bootstrap] Waiting on integrations to complete setup: sensor.nordpool

Starts already for 4 minutes. something is wrong there...

Hellowlol · 2022-04-02T08:50:17Z

Please enable the debug log and the log. The sensor can take a long time depending on the issues it has

yozik04 · 2022-04-02T12:56:18Z

The integration made 69 requests to finally get data. i think it fails if prices are requested before midday (when new prices are published)
See log attached.
nordpool-fix-1_full.log

Hellowlol · 2022-04-02T14:15:16Z

@yozik04 Thanks for the debug log! It was really helpful, the problem was that the retry method considers all "falsy" as a request for retrying. Ill fix that :)

yozik04 · 2022-04-02T14:29:18Z

The method you used to make nordpool library async is very scary. I'd better throw a PR to add async to that library. =)

yozik04 · 2022-04-02T14:31:19Z

BTW. Take a look to the latest version of that library. It already has async support.

yozik04 · 2022-04-02T14:53:34Z

Ahh, you have just made a copy of that async version and modified... Ok, your code, your rules...

Hellowlol · 2022-04-02T15:04:15Z

@yozik04 The API wasn't async when I made this, I was the one that sent an upstream PR to make it async. I kept my code to make sure that it worked. See PR kipe/nordpool#8 and kipe/nordpool#11 If you have a better way to make this async while supporting multiple async HTTP libraries, feel free to send a PR

Hellowlol · 2022-04-02T17:31:51Z

@yozik04 I think it's ready to be merged. Can you test and leave it running until the prices for Monday should be available?

yozik04 · 2022-04-02T18:55:36Z

@Hellowlol I will. Updating

yozik04 · 2022-04-02T19:05:04Z

It still does 6 requests to nordpool on HA restart.

yozik04 · 2022-04-02T19:06:38Z

Updating todays prices and Updating tomorrows prices. 3 requests each.

Hellowlol · 2022-04-02T19:27:53Z

@yozik04 thats intended. The api stores/get the data in cet. So if you requested data area in another time zone you wouldn’t get the correct prices. That’s why we request data for 3 days and extract what we need.

yozik04 · 2022-04-02T19:44:38Z

It is ok to fetch 3. But why it fetches 3 requests two times.

Hellowlol · 2022-04-02T19:53:28Z

Its one request for today and one for tomorrow.

yozik04 · 2022-04-03T13:17:38Z

Found infinty invalid data in area SYS
Why it checks SYS if I am interested in EE only?

Looks like in my region prices appear at 15:00 UTC+3. For some reason it started requesting at 14:00, which is 11 UTC.

~~But overall it works.~~ Actually not. See below.

yozik04 · 2022-04-03T13:20:57Z

Ahh, my father has restarted HA in the morning. He told it showed today and tomorrow same prices...

Hellowlol · 2022-04-03T13:21:08Z

@yozik04 can you post the log? I want to check why it started requesting data then

Hellowlol · 2022-04-03T19:31:11Z

I will drop an idea here how I would do the whole thing.
`
I'd create a class that would hold all fetched valid values (yesterday, today, tomorrow) in a single dictionary (datetime as a key). Let's name it PriceData That class would have methods:

Thanks for your suggestion, that is pretty much how this is designed today with some small changes to be able to handle currencies and we also need to clear the sensors data in addition to NordpoolData as we can't do IO in a property.

get_todays_data - would scan the dictionary and output todays's prices today

get_tomorrows_data - would scan the dictionary and output tomorrow's prices tomorrow

has_todays_data - would tell if dictionary has todays data | Does not exist

has_tomorrows_data - would tell if dictionary has tomorrows data | Does not exist

clear_expired - would scan the dictionary and drop data that is already expired new_day_cb

store_data - injects the data to the dictionary (actually a nice place to validate the data as well, and inject only valid values) _update

get_current_price - probably it is also needed, you know better. | Cant be handle here, must be handled in sensor

Code out of PriceData class would query nordpool for today's data only if PriceData.has_todays_data returns False. Same with tomorrow but should also verify that time is adequate for the request. If fetch was successful and data is valid we can still reuse has_todays_data, has_tomorrows_data

Every day something needs to call PriceData.clear_expired as well (probably run it before every request to nordpool). new_day_cb

That would be my design. I would even cover PriceData with some unit tests. But I do not really want to go and change your code. =) No offence.

I'll accept every PR that makes the sensor more reliable as long as none of the current features are missed. 😊

Hellowlol · 2022-04-03T21:40:18Z

Ill add some tests

yozik04 · 2022-04-04T05:40:28Z

At night both day prices became unavailable with new version. Will send logs later.

yozik04 · 2022-04-04T06:18:59Z

Todays log: nordpool-2022-04-04.log

yozik04 · 2022-04-04T06:20:20Z

yesterday's logs were lost unfortunately.

Hellowlol · 2022-04-04T15:29:24Z

Thanks, that was helpful. 👍 The server was restarted after midnight so there isn't any available data for tomorrow, it will try one then, stop. However, it also did a new HTTP call to get today's data. There were invalid values in FI results, so the requests for retried over and over until the setup of the senor failed. We validate on currency as the NordpoolData don't know what data the user wants

yozik04 · 2022-04-05T11:24:54Z

nordpool_2022-04-04_05.log
Worked fine today on version before tests. Posting log for reference.

yozik04 · 2022-04-06T06:13:01Z

I can confirm the last one is working as expected.

and only makes noice.

Hellowlol · 2022-04-06T19:39:50Z

@yozik04 Can you do one last test? I was thinking of making a new release this weekend.

yozik04 · 2022-04-06T21:42:04Z

Pulled latest changes.
I have 00:37 on the clock now and it failed to download/update the data. Found infinity inside FI... Will see if it will recover during the night.
As before, at the night time it fails. Due to Nordpool returning junk or an other reason. Anyway I will send logs tomorrow. I have not digged what is in the returned XML yet. There are some valid values for sure.

if the user has multiple regions all of them have to be valid

Hellowlol · 2022-04-06T23:05:13Z

The latest update should only verify the area that the user requested. If the user has multiple sensors using the same currency, then all of the data for the selected areas have to be valid.

yozik04 · 2022-04-07T15:12:00Z

Logs from version 34be90f
nordpool-2022-04-07_night.log

JSON files that were fetched at night:
08-04-2022.txt
07-04-2022.txt
06-04-2022.txt

# 2022-04-07 00:33:08 DEBUG (MainThread) [custom_components.nordpool.aio_price] requested https://www.nordpoolgroup.com/api/marketdata/page/10?currency=EUR&endDate=06-04-2022 {'currency': 'EUR', 'endDate': '06-04-2022'}
cat 06-04-2022.txt | jq -c '[.data.Rows[].Columns | map(select(.Name == "EE"))[].Value]'
["81,73","72,17","69,23","71,48","74,79","74,13","87,07","136,61","151,41","95,96","90,42","87,16","86,68","84,67","84,90","86,91","87,68","87,63","87,91","104,52","111,75","54,68","45,08","27,35","27,35","151,41","85,08","94,65","83,40","59,72"]

# 2022-04-07 00:33:08 DEBUG (MainThread) [custom_components.nordpool.aio_price] requested https://www.nordpoolgroup.com/api/marketdata/page/10?currency=EUR&endDate=07-04-2022 {'currency': 'EUR', 'endDate': '07-04-2022'}
cat 07-04-2022.txt | jq -c '[.data.Rows[].Columns | map(select(.Name == "EE"))[].Value]'
["81,73","72,17","69,23","71,48","74,79","74,13","87,07","136,61","151,41","95,96","90,42","87,16","86,68","84,67","84,90","86,91","87,68","87,63","87,91","104,52","111,75","54,68","45,08","27,35","27,35","151,41","85,08","94,65","83,40","59,72"]

# 2022-04-07 00:33:08 DEBUG (MainThread) [custom_components.nordpool.aio_price] requested https://www.nordpoolgroup.com/api/marketdata/page/10?currency=EUR&endDate=08-04-2022 {'currency': 'EUR', 'endDate': '08-04-2022'}
cat 08-04-2022.txt | jq -c '[.data.Rows[].Columns | map(select(.Name == "EE"))[].Value]'
["-","-","-","-","-","-","-","-","-","-","-","-","-","-","-","-","-","-","-","-","-","-","-","-","-","-","-","-","-","-"]

As you can see 6th April's data is absolutely equal to 7th April, what is bullshit. 8th is not available.

Hellowlol · 2022-04-07T19:14:30Z

Thanks again! Clearly it’s bug in the retry. However I have no idea why the values for today and tomorrow was the same. How if you get the json? Did you do manual request and download it? The raw json isn’t logged in this integration.

yozik04 · 2022-04-07T19:19:00Z

I have copied urls from the HA log and used wget at that time.

poffe · 2022-05-02T09:50:08Z

Hi! Just a friendly poke to see how this is progressing? I've seen this issue a couple of times so that's the reason. Great work btw.

Hellowlol · 2022-05-02T11:40:41Z

I haven't forgotten this. I just got frustrated with the API returning junk :P It will get merged sometime in the future :D

first pass

c730117

yozik04 reviewed Mar 22, 2022

View reviewed changes

custom_components/nordpool/__init__.py Outdated Show resolved Hide resolved

Hellowlol added 2 commits March 22, 2022 21:43

Use backoff

4348707

Update __init__.py

21639e4

janneojala reviewed Mar 30, 2022

View reviewed changes

custom_components/nordpool/__init__.py Outdated Show resolved Hide resolved

alexschwantes mentioned this pull request Apr 1, 2022

Values for tomorrow are null #125

Closed

Hellowlol added 3 commits April 1, 2022 23:12

remove unused functions

c97421f

fix typo

4b40dd1

Update __init__.py

ace8129

Hellowlol added 2 commits April 2, 2022 19:20

Minor cleanup

f0185a4

Remove redundant code

a68a53f

Hellowlol added 6 commits April 4, 2022 20:46

First pass on tests

91e4f16

damn yaml

3f79f0c

Update push.yml

ab87f8e

Update push.yml

9b98760

Update requirements_dev.txt

f49952f

MOAR

de1bbb0

Update test_init.py

87c00a3

Hellowlol added 4 commits April 6, 2022 21:04

summer time fix and slow api response/retries on startup

9199470

Formatting

bc080f8

Disable test and it dont work

e17b86e

and only makes noice.

Merge branch 'master' into fix-1

34be90f

only verify the region the user asks for,.

0aa8592

if the user has multiple regions all of them have to be valid

Floppe mentioned this pull request Aug 9, 2022

Prices not always updated after day change #178

Closed

Hellowlol mentioned this pull request Jul 4, 2023

Fix issue with late published prices #333

Merged

first pass #127

Are you sure you want to change the base?

first pass #127

Conversation

Hellowlol commented Mar 21, 2022 • edited Loading

yozik04 commented Mar 23, 2022 • edited Loading

yozik04 commented Apr 2, 2022

Hellowlol commented Apr 2, 2022

yozik04 commented Apr 2, 2022 • edited Loading

Hellowlol commented Apr 2, 2022

yozik04 commented Apr 2, 2022

yozik04 commented Apr 2, 2022

yozik04 commented Apr 2, 2022

Hellowlol commented Apr 2, 2022 • edited Loading

Hellowlol commented Apr 2, 2022

yozik04 commented Apr 2, 2022

yozik04 commented Apr 2, 2022

yozik04 commented Apr 2, 2022 • edited Loading

Hellowlol commented Apr 2, 2022

yozik04 commented Apr 2, 2022

Hellowlol commented Apr 2, 2022

yozik04 commented Apr 3, 2022 • edited Loading

yozik04 commented Apr 3, 2022

Hellowlol commented Apr 3, 2022

Hellowlol commented Apr 3, 2022 • edited Loading

Hellowlol commented Apr 3, 2022

yozik04 commented Apr 4, 2022 • edited Loading

yozik04 commented Apr 4, 2022

yozik04 commented Apr 4, 2022

Hellowlol commented Apr 4, 2022 • edited Loading

yozik04 commented Apr 5, 2022

yozik04 commented Apr 6, 2022

Hellowlol commented Apr 6, 2022

yozik04 commented Apr 6, 2022

Hellowlol commented Apr 6, 2022

yozik04 commented Apr 7, 2022 • edited Loading

Hellowlol commented Apr 7, 2022

yozik04 commented Apr 7, 2022

poffe commented May 2, 2022

Hellowlol commented May 2, 2022

Hellowlol commented Mar 21, 2022 •

edited

Loading

yozik04 commented Mar 23, 2022 •

edited

Loading

yozik04 commented Apr 2, 2022 •

edited

Loading

Hellowlol commented Apr 2, 2022 •

edited

Loading

yozik04 commented Apr 2, 2022 •

edited

Loading

yozik04 commented Apr 3, 2022 •

edited

Loading

Hellowlol commented Apr 3, 2022 •

edited

Loading

yozik04 commented Apr 4, 2022 •

edited

Loading

Hellowlol commented Apr 4, 2022 •

edited

Loading

yozik04 commented Apr 7, 2022 •

edited

Loading