object_store Azure Government using OAuth

[zadeluca]

Which part is this question about
object_store Azure

Describe your question
I have reviewed this issue #4853 which also pertains to using Azure Government. That example uses shared key authentication. For OAuth (ClientSecretOAuthProvider), I believe it is also necessary to set authority_host to authority_hosts::AZURE_GOVERNMENT in order for the token_url to be correct.

Without this setting, I am currently getting this error "AADSTS900382: Confidential Client is not supported in Cross Cloud request. I understand it is possible to override using MicrosoftAzureBuilder::with_authority_host(), however I am using delta-rs which requires that all configuration is provided though storage_options with keys defined in AzureConfigKey. This currently does not allow setting AuthorityHost.

Does that sound correct? If so, is it possible to expose AuthorityHost through AzureConfigKey?

Additional context
I have only tried in Python using delta-rs. I am going to attempt to use arrow-rs directly in order to verify that setting AuthorityHost does indeed resolve the error mentioned above. I am a Rust beginner so it may take some effort.

[tustvold]

I'm not very familiar with Azure's government offering, but we should expose the option as a config key if it isn't

[zadeluca]

I have confirmed that including both of these:

            .with_authority_host("https://login.microsoftonline.us")
            .with_endpoint("https://<ACCOUNT>.blob.core.usgovcloudapi.net".to_string())

on the builder allows OAuth to succeed. I will try to submit a PR to add AuthorityHost to AzureConfigKey.