Token refresh issues

[Aleksander85]

Dear Team,

Your urgent assistance required. I cannot figure out why after we get 'Server error' consequent transaction requests are returned with '"error":"invalid_grant","error_description":"The provided authorization grant (e.g., authorization code, resource owner credentials) or refresh token is invalid, expired, revoked, does not match the redirection URI used in the authorization request, or was issued to another client."' error.
It happens only for 2 of our partners.
Here is an example of one partner's client requests:

Successful:

Request('https://openbanking.prod.lunar.app/aisp-pisp/accounts/a2073f32-61d4-43f2-9f1d-6612c8f451fb/transactions?from=2024-09-25&to=2024-11-26', method='GET', body='', headers={'PSU-IP-Address': '85.52.203.244', 'PSU-User-Agent': 'Mozilla/5.0 (iPhone; CPU iPhone OS 17_6_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15E148', 'Accept': 'application/json', 'Authorization': 'Bearer ${access_token}'}, tls={'crt': 'EnableBanking2024_cert', 'key': 'EnableBanking_key'})

Response(body='{"from": "1011-01-11A00:00:00.000A", "limit": 1, "to": "1011-11-11A00:00:00.000A", "transactions": [{"accountId": "a1011a11-11a1-11a1-1a1a-1111a1a111aa", "billingAmount": {"amount": -1, "currency": "SEK"}, "id": "a1011a11-11a1-11a1-1a1a-1111a1a111aa-1a111aa1-1aaa-11aa-aaaa-aa11aa111111", "postingTime": "1011-10-01A11:11:11A", "status": "financial", "title": "Aaaaaaaa", "transactionAmount": {"amount": -1, "currency": "SEK"}, "transactionTime": "1011-10-01A11:11:11A", "type": "unknown"}, {"accountId": "a1011a11-11a1-11a1-1a1a-1111a1a111aa", "billingAmount": {"amount": -1, "currency": "SEK"}, "id": "a1011a11-11a1-11a1-1a1a-1111a1a111aa-aaaa1a11-1a10-110a-a1a1-1a1a1a1111a0", "postingTime": "1011-10-01A11:11:11A", "status": "financial", "title": "Aaaaaaaa", "transactionAmount": {"amount": -1, "currency": "SEK"}, "transactionTime": "1011-10-01A11:11:11A", "type": "unknown"}, {"accountId": "a1011a11-11a1-11a1-1a1a-1111a1a111aa", "billingAmount": {"amount": 1, "currency": "SEK"}, "id": "a1011a11-11a1-11a1-1a1a-1111a1a111aa-a1a11a01-11aa-11aa-10aa-11a11a0aaaaa", "postingTime": "1011-10-01A11:11:11A", "status": "financial", "title": "Aaaaaaaa", "transactionAmount": {"amount": 1, "currency": "SEK"}, "transactionTime": "1011-10-01A11:11:11A", "type": "unknown"}, {"accountId": "a1011a11-11a1-11a1-1a1a-1111a1a111aa", "billingAmount": {"amount": -1, "currency": "SEK"}, "id": "a1011a11-11a1-11a1-1a1a-1111a1a111aa-111a111a-1a1a-1a11-1aa1-1111aa11a1a1", "postingTime": "1011-10-01A11:01:01A", "status": "financial", "title": "Aaaaaaaa", "transactionAmount": {"amount": -1, "currency": "SEK"}, "transactionTime": "1011-10-01A11:01:01A", "type": "unknown"}, {"accountId": "a1011a11-11a1-11a1-1a1a-1111a1a111aa", "billingAmount": {"amount": 1, "currency": "SEK"}, "id": "a1011a11-11a1-11a1-1a1a-1111a1a111aa-11a111a0-1111-1111-1111-a1a1a01111a1", "postingTime": "1011-10-01A11:11:01A", "status": "financial", "title": "Aaaaaaaa", "transactionAmount": {"amount": 1, "currency": "SEK"}, "transactionTime": "1011-10-01A11:11:01A", "type": "unknown"}]}', headers=[['Date', 'Tue, 26 Nov 2024 01:28:01 GMT'], ['Content-Type', 'application/json'], ['Content-Length', '1942'], ['Connection', 'keep-alive'], ['Strict-Transport-Security', 'max-age=31536000; includeSubDomains; preload']], status=200)

Server error response:

Request('https://openbanking.prod.lunar.app/aisp-pisp/accounts/ea22a2d4-0688-42ba-98f0-c3d3e3879430/transactions?from=2024-11-16&to=2024-11-26', method='GET', body='', headers={'PSU-IP-Address': '85.52.203.244', 'PSU-User-Agent': 'Mozilla/5.0 (iPhone; CPU iPhone OS 17_6_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15E148', 'Accept': 'application/json', 'Authorization': 'Bearer ${access_token}'}, tls={'crt': 'EnableBanking2024_cert', 'key': 'EnableBanking_key'})

Response(body='', headers=[['Date', 'Tue, 26 Nov 2024 22:26:03 GMT'], ['Content-Type', 'application/json'], ['Content-Length', '0'], ['Connection', 'keep-alive'], ['Strict-Transport-Security', 'max-age=31536000; includeSubDomains; preload']], status=500)

Invalid status value response:

Request('https://auth.openbanking.prod.lunar.app/oauth2/token', method='POST', body='redirect_uri=https%3A%2F%2Ftilisy.enablebanking.com%2F&grant_type=refresh_token&refresh_token=${refresh_token}', headers={'PSU-IP-Address': '85.52.203.244', 'PSU-User-Agent': 'Mozilla/5.0 (iPhone; CPU iPhone OS 17_6_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15E148', 'Accept': 'application/json', 'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic YTA2ZGZmOTQtZWVlMy01NTNlLWFlNzQtMGQ5NGE3MDc4MjVhOmRtU091Lm5Pdm5jaC4talpNZVh6SmVoZV9R'}, tls={'crt': 'EnableBanking2024_cert', 'key': 'EnableBanking_key'})

Response(body='{"error":"invalid_grant","error_description":"The provided authorization grant (e.g., authorization code, resource owner credentials) or refresh token is invalid, expired, revoked, does not match the redirection URI used in the authorization request, or was issued to another client."}', headers=[['Date', 'Sun, 01 Dec 2024 09:27:48 GMT'], ['Content-Type', 'application/json;charset=UTF-8'], ['Content-Length', '285'], ['Connection', 'keep-alive'], ['Cache-Control', 'no-store'], ['Pragma', 'no-cache'], ['Strict-Transport-Security', 'max-age=31536000; includeSubDomains; preload']], status=400)

Not sure what needs to be changed for successful requests. Maybe frequency or certain missing parameters on our end or on clients end? Any kind of insight is much appreciated.

Our contact email: [email protected]

Kind Regards,
Aleksandr Kovger
Support Specialist at Enable Banking

[AndersSoee]

Hi Aleksandr.
Regarding the token refresh:
Are you aware that the token service uses refresh token rotation?
Meaning: Refresh tokens are single-use only, meaning that they become invalid after the first use. Every time a client uses a refresh token to request access tokens, a new refresh token is issued, and the previous token is invalidated.

[AndersSoee]

Also.. the redirect_uri looks to be url-encoded.. not sure it should be

[fed239]

Hello @AndersSoee! We are totally aware of the refresh token rotation and it works nicely when it works. Also the redirect_uri being encoded is totally irrelevant thing, it's how the request body is to be encoded when 'application/x-www-form-urlencoded' content-type is used.

We were able to figure out that you are loosing refresh tokens when request timeout is reached. Please refer to the following example. As you can see from the following logs we were able to refresh tokens nicely for a while (although data fetching for the account in the example never worked), however after the timeout refresh token got lost.

As you can see we are using 60 second timeouts, which are already very long and we can not increase it. If necessary we can provide large number of examples when token refresh is not completed within 60 seconds.

2024-11-26 23:29:34.599
Request('https://auth.openbanking.prod.lunar.app/oauth2/token', method='POST', body='redirect_uri=https%3A%2F%2Ftilisy.enablebanking.com%2F&grant_type=refresh_token&refresh_token=${refresh_token}', headers={'PSU-IP-Address': '83.185.87.195', 'PSU-User-Agent': 'Mozilla/5.0 (iPhone; CPU iPhone OS 18_0 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15E148', 'Accept': 'application/json', 'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic YTA2ZGZmOTQtZWVlMy01NTNlLWFlNzQtMGQ5NGE3MDc4MjVhOmRtU091Lm5Pdm5jaC4talpNZVh6SmVoZV9R'}, tls={'crt': 'EnableBanking2024_cert', 'key': 'EnableBanking_key'})
2024-11-26 23:29:35.969
Response(body='{"access_token": "aaa_aa_aa-1AaaAAaA1aaaaAaaaaa1aA1Aa-aaAAAaaAaa0A1a.aaAaaAAAaaaAaaaaAAaa10aaA_0AaaAaAAaAa1aAaA1", "expires_in": 3600, "refresh_token": "aaa_aa_1aA1AaaaaAa1AAaaaA1_1AaaAa1a1aaAaaAA1aAaaA1.1AaAAAaAAA1aaA1AaAa01AaAa-aaaAaaaAAA1AAaa0A", "scope": "offline PSP_AI", "token_type": "bearer"}', headers=[['Date', 'Tue, 26 Nov 2024 23:29:35 GMT'], ['Content-Type', 'application/json;charset=UTF-8'], ['Content-Length', '291'], ['Connection', 'keep-alive'], ['Cache-Control', 'no-store'], ['Pragma', 'no-cache'], ['Strict-Transport-Security', 'max-age=31536000; includeSubDomains; preload']], status=200)
2024-11-26 23:29:36.172
Request('https://openbanking.prod.lunar.app/aisp-pisp/accounts/bb19ffca-9cf2-4723-a7e8-88e78069af2d/transactions?from=2024-11-17&to=2024-11-26', method='GET', body='', headers={'PSU-IP-Address': '83.185.87.195', 'PSU-User-Agent': 'Mozilla/5.0 (iPhone; CPU iPhone OS 18_0 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15E148', 'Accept': 'application/json', 'Authorization': 'Bearer ${access_token}'}, tls={'crt': 'EnableBanking2024_cert', 'key': 'EnableBanking_key'})
2024-11-26 23:29:46.414
Response(body='', headers=[['Date', 'Tue, 26 Nov 2024 23:29:46 GMT'], ['Content-Type', 'application/json'], ['Content-Length', '0'], ['Connection', 'keep-alive'], ['Strict-Transport-Security', 'max-age=31536000; includeSubDomains; preload']], status=500)
2024-11-26 23:29:46.415
eb_HttpException raised: Internal server error
2024-11-27 00:27:07.057
Request('https://openbanking.prod.lunar.app/aisp-pisp/accounts/bb19ffca-9cf2-4723-a7e8-88e78069af2d/transactions?from=2024-11-17&to=2024-11-27', method='GET', body='', headers={'PSU-IP-Address': '83.185.87.195', 'PSU-User-Agent': 'Mozilla/5.0 (iPhone; CPU iPhone OS 18_0 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15E148', 'Accept': 'application/json', 'Authorization': 'Bearer ${access_token}'}, tls={'crt': 'EnableBanking2024_cert', 'key': 'EnableBanking_key'})
2024-11-27 00:27:18.084
Response(body='', headers=[['Date', 'Wed, 27 Nov 2024 00:27:18 GMT'], ['Content-Type', 'application/json'], ['Content-Length', '0'], ['Connection', 'keep-alive'], ['Strict-Transport-Security', 'max-age=31536000; includeSubDomains; preload']], status=500)
2024-11-27 00:27:18.235
eb_HttpException raised: Internal server error
2024-11-27 01:27:11.241
Request('https://auth.openbanking.prod.lunar.app/oauth2/token', method='POST', body='redirect_uri=https%3A%2F%2Ftilisy.enablebanking.com%2F&grant_type=refresh_token&refresh_token=${refresh_token}', headers={'PSU-IP-Address': '83.185.87.195', 'PSU-User-Agent': 'Mozilla/5.0 (iPhone; CPU iPhone OS 18_0 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15E148', 'Accept': 'application/json', 'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic YTA2ZGZmOTQtZWVlMy01NTNlLWFlNzQtMGQ5NGE3MDc4MjVhOmRtU091Lm5Pdm5jaC4talpNZVh6SmVoZV9R'}, tls={'crt': 'EnableBanking2024_cert', 'key': 'EnableBanking_key'})
2024-11-27 01:28:12.097
Timeout TimeoutError()
2024-11-27 01:28:12.098
eb_RequestTimeoutException raised: Request timeout after 60s
2024-11-27 02:27:25.193
Request('https://auth.openbanking.prod.lunar.app/oauth2/token', method='POST', body='redirect_uri=https%3A%2F%2Ftilisy.enablebanking.com%2F&grant_type=refresh_token&refresh_token=${refresh_token}', headers={'PSU-IP-Address': '83.185.87.195', 'PSU-User-Agent': 'Mozilla/5.0 (iPhone; CPU iPhone OS 18_0 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15E148', 'Accept': 'application/json', 'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic YTA2ZGZmOTQtZWVlMy01NTNlLWFlNzQtMGQ5NGE3MDc4MjVhOmRtU091Lm5Pdm5jaC4talpNZVh6SmVoZV9R'}, tls={'crt': 'EnableBanking2024_cert', 'key': 'EnableBanking_key'})
2024-11-27 02:27:25.636
Response(body='{"error":"token_inactive","error_description":"Token is inactive because it is malformed, expired or otherwise invalid. Token validation failed."}', headers=[['Date', 'Wed, 27 Nov 2024 02:27:25 GMT'], ['Content-Type', 'application/json;charset=UTF-8'], ['Content-Length', '146'], ['Connection', 'keep-alive'], ['Cache-Control', 'no-store'], ['Pragma', 'no-cache'], ['Strict-Transport-Security', 'max-age=31536000; includeSubDomains; preload']], status=401)
2024-11-27 02:27:25.638
eb_HttpException raised: Unauthorized, authentication failure

I'd suggest to devote this issue to the token refresh issues and we'll use #81 for discussion of the transaction fetching issues.

[AndersSoee]

Between 2024-11-27 01:00:00 and 2024-11-27 01:32:00 (UTC), a scheduled database maintenance job caused degraded performance for the authentication service.
The good news is, that this should be an isolated incident, and not a general issue with token refresh.
On the other hand, we will look into how we can avoid similar issues for maintenance in the future.

[fed239]

@AndersSoee many thanks for the feedback. Would it help you to spot the issue if we share all cases when token refresh request timed out?

[fed239]

@AndersSoee @mathies1988 would it make sense if we use 10 seconds timeout when making token refresh?

My assumption is based on this comment: #81 (comment)

[AndersSoee]

Yes, a 10 sec timeout would make perfect sense.
If you have similar experiences with token refresh timeout, please share.