Cannot configure dlt for Azure GovCloud

[jeff-tilton]

dlt version

1.4.0

Describe the problem

I cannot configure dlt to run with Azure GovCloud, it returns an error and the error shows it pointing to commercial cloud.

I was able to fix this after updating the adlfs/spec.py to point specifically to the blob.core.usgovcloudapi.net endpoint instead of windows.net, but I could not explicitly set this in the configuration. I was able to land the data, but the streamlit app fails to connect to the data because it is once again pointing to the commercial cloud somewhere within the codebase.

DatabaseTransientException: Fail to get a new connection for: https://myresource.blob.core.windows.net. Couldn't resolve host name

It should be pointing to usgovcloudapi.net instead of windows.net.

Expected behavior

I should be able to configure dlt to run on Azure commercial or USGovCloud.

Steps to reproduce

This is not possible to reproduce without a USGovCloud account.

Operating system

Windows

Runtime environment

Local

Python version

3.11

dlt data source

sql_database

dlt destination

Filesystem & buckets

Other deployment details

No response

Additional information

I was able to run the pipeline by updating adlfs/spec.py, but not able to run the streamlit app.

[rudolfix]

hey @jeff-tilton
it seems that the newest adlfs (2024.7.0) supports configurable hosts. I was able to successfully pass your gov url and make a connection. You are able to authorize users (I can't - don't have access to gov cloud) so it should work, You can configure host in your secrets.toml or config.toml:

[destination.filesystem]
bucket_url="..."
[destination.filesystem.kwargs]
account_host="blob.core.usgovcloudapi.net"
[destination.filesystem.credentials]
...

if you use env variabes you need to pass a serialized kwargs dict in DESTINATION__FILESYSTEM__KWARGS='{....}'

lmk if that worked. then we'll update documentation.