JWT is not well formed, there are no dots

[RogerSelwyn]

I have a few people who use the Home Assistant O365 integration that are getting the errors as shown below (they are encapsulated in HA login, but you can see what is being surfaced by O365). I've only seen reports in the last few months, but I have no idea what is causing them. Unfortunately I don't get the problem myself (or just don't notice the error in my logs, so I don't really know what triggers it).

Any ideas as to what is causing the problem? The integration sets up a FileSystemTokenBackend, which it uses I the Account setup call, beyond that it leaves token management to O365 (apart from reading the token once to check granted permissions). Code as below:

def _try_authentication(perms, credentials, main_resource):
    _LOGGER.debug("Setup token")
    token_backend = FileSystemTokenBackend(
        token_path=perms.token_path,
        token_filename=perms.token_filename,
    )
    _LOGGER.debug("Setup account")
    account = Account(
        credentials,
        token_backend=token_backend,
        timezone=CONST_UTC_TIMEZONE,
        main_resource=main_resource,
    )

    try:
        return account, account.is_authenticated

    except json.decoder.JSONDecodeError:
        return account, False

Logger: custom_components.o365.calendar
Quelle: custom_components/o365/calendar.py:655
Integration: Office 365 (Dokumentation, Probleme)
Erstmals aufgetreten: 08:48:54 (1 Vorkommnisse)
Zuletzt protokolliert: 08:48:54


Error getting calendar events - 401 Client Error: Unauthorized for url: https://graph.microsoft.com/v1.0/me/calendars/AQMkADAwATNiZmYAZC04M2ZhLTkwNDYtMDACLTAwCgBGAAADxUpIqt1PQ02qEe37a0cywQcAe9JyutQAp0a3sg19Hw9tFgAAAgEGAAAAe9JyutQAp0a3sg19Hw9tFgAF1RvVAwAAAA==/calendarView?%24top=999&startDateTime=2024-11-08T07%3A48%3A44.399251%2B00%3A00&endDateTime=2024-11-09T07%3A48%3A44.399274%2B00%3A00&%24select=sensitivity%2Cstart%2Ccategories%2Cbody%2CisAllDay%2Csubject%2Cend%2Clocation%2CseriesMasterId%2Cattendees%2CshowAs | Error Message: IDX14100: JWT is not well formed, there are no dots (.). The token needs to be in JWS or JWE Compact Serialization Format. (JWS): 'EncodedHeader.EncodedPayload.EncodedSignature'. (JWE): 'EncodedProtectedHeader.EncodedEncryptedKey.EncodedInitializationVector.EncodedCiphertext.EncodedAuthenticationTag'.
Logger: O365.connection
Quelle: /usr/local/lib/python3.12/site-packages/O365/connection.py:838
Erstmals aufgetreten: 08:48:54 (1 Vorkommnisse)
Zuletzt protokolliert: 08:48:54

Client Error: 401 Client Error: Unauthorized for url: https://graph.microsoft.com/v1.0/me/calendars/AQMkADAwATNiZmYAZC04M2ZhLTkwNDYtMDACLTAwCgBGAAADxUpIqt1PQ02qEe37a0cywQcAe9JyutQAp0a3sg19Hw9tFgAAAgEGAAAAe9JyutQAp0a3sg19Hw9tFgAF1RvVAwAAAA==/calendarView?%24top=999&startDateTime=2024-11-08T07%3A48%3A44.399251%2B00%3A00&endDateTime=2024-11-09T07%3A48%3A44.399274%2B00%3A00&%24select=sensitivity%2Cstart%2Ccategories%2Cbody%2CisAllDay%2Csubject%2Cend%2Clocation%2CseriesMasterId%2Cattendees%2CshowAs | Error Message: IDX14100: JWT is not well formed, there are no dots (.). The token needs to be in JWS or JWE Compact Serialization Format. (JWS): 'EncodedHeader.EncodedPayload.EncodedSignature'. (JWE): 'EncodedProtectedHeader.EncodedEncryptedKey.EncodedInitializationVector.EncodedCiphertext.EncodedAuthenticationTag'. | Error Code:
The user has checked their access token. There are no dots inside. They've also checked it with jwt.io and get the following error:

Error: Looks like your JWT Header is not encoded correclty using base64url (https://tools.ietf.org/html/rfc4648#section-5). Note that padding ("=") must be omitted as per https://tools.ietf.org/html/rfc7515'section-2
The error does not occur permanently, but only occasionally. It then seems to heal itself, since the synchronization of tasks and calendar generally works.

[alejcas]

Hi, I’ll have some time next week to investigate. I’ll report back. Never happened this to me before nor anyone reported this earlier.

[RogerSelwyn]

I’ve previously seen this issue logged, similar error, different situation I think - #998

[alejcas]

The problem seems to be that oauthlib is no longer caching the correct error codes MS Graph gives and thus giving apps error codes like 401 (unauthorised) when it should raise TokenExpiredError.
I have already exposed this but I don't have the time not the expertise to go into this.

From my end this is difficult to solve as O365 don't know if a 401 is returned because the token is expired and oauthlib didn't do it's work raising the TokenExpiredError or it's actually a unauthorised error that is correct.

So, either anyone solves this inside oauthlib (which I think should be very difficult) or O365 drops it's custom auth methods to use msal.

I would go wih msal, but this is a BIG change that touches many 'moving parts'.

[RogerSelwyn]

I’m not sure I have the knowledge to rework the auth method, but I can take a look when I get back home next week.

In the interim, presumably I can trap the errors in my integration and trigger a token refresh?

[alejcas]

The current quick solution involves the following (other users reported this several times here):

1. Manually catch the exception
2. Call account.connection.refresh_token()
3. Try the request again

[RogerSelwyn]

Thanks

[alejcas]

And please remember that the refresh should ask before to the token backed:

should_refresh_token = self.token_backend.should_refresh_token(self)
if should_refresh_token is True:
    # The backend has checked that we can refresh the token
    if self.refresh_token() is False:
        raise RuntimeError('Token Refresh Operation not working')

Although should_refresh_token is barely used by anyone...

[alejcas]

I have analysed what it means to implement authentication with MSAL.

• Authentication:

1. Modify methods from Connection object: request_token, refresh_token and get_authorization_url
2. Modify methods from Account object: authenticate
3. Modify object Token: msal tokens are somehow different. I need to figure out how to adapt the token methods so O365 can know when a token is expired.
4. Modify methods: _internal_requestand oauth_request

• Readme: Modify the readme to show the change. Many methods change and auth is done differently

• Optional changes: Configure a msal TokenCache to work with O365 Token Backends

I have been able to authenticate and use a O365 with tokens retrieved using msal.

I think I will drop custom auth in favor of msal as soon as possible
Many work to do and testing...

[RogerSelwyn]

That would be brilliant. I'd be happy to test as soon as you have something (preferably not before the 23rd, if it requires code change at my end).

[alejcas]

I will be working in the msal branch

https://github.com/O365/python-o365/tree/msal