feat(python): add capability to read unity catalog (uc://) uris

[omkar-foss]

Description

This adds capability to read directly from uc:// uris using the local catalog-unity crate. This also exposes the UC temporary credentials in storage_options of the DeltaTable instance so polars or similar readers can use it.

Related Issue(s)

• closes Add Python-side support for Unity Catalog #3046

Documentation

N/A

[github-actions]

ACTION NEEDED

delta-rs follows the Conventional Commits specification for release automation.

The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification.

[omkar-foss]

This allows users to work with a local (non-https) Unity Catalog REST API with delta-rs.

[omkar-foss]

@ion-elgreco I've updated this PR to include the temp credentials functionality from PR #3078. Cheers.

[codecov]

Codecov Report

Attention: Patch coverage is 12.90323% with 108 lines in your changes missing coverage. Please review.

Project coverage is 72.16%. Comparing base (2f1a5ac) to head (0e6c77f).
Report is 18 commits behind head on main.

Files with missing lines	Patch %	Lines
crates/catalog-unity/src/lib.rs	10.83%	106 Missing and 1 partial ⚠️
python/src/lib.rs	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3113      +/-   ##
==========================================
- Coverage   72.31%   72.16%   -0.16%     
==========================================
  Files         138      138              
  Lines       45398    45518     +120     
  Branches    45398    45518     +120     
==========================================
+ Hits        32831    32846      +15     
- Misses      10489    10591     +102     
- Partials     2078     2081       +3     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[ion-elgreco]

@omkar-foss looks good from my side!

@hntd187 can you please also take a look. Specifically on whether the credentials need to be refreshed as part of the objectstore? Does UC also support writing temp credentials?

[omkar-foss]

I'll need to add a few python tests for this flow, will add tomorrow and then if all good, I suppose we can merge it.

Also please note, the current UC REST integration that we have is more fine-tuned for Databricks Unity Catalog. The Unity Catalog OSS has slightly different response payloads for some APIs. e.g. Generate Temp Credentials API returns r2_temp_credentials and url in the JSON response for Databricks UC but not for OSS UC. So this causes json decoding errors if you try out the current UC client implementation with an OSS UC instance.

I can look into supporting OSS UC smoothly in a separate PR after this one. Cheers.

[omkar-foss]

@omkar-foss can you add an integration page to the docs? :)

Yes sure, is it okay if I add it in the next PR?

@ion-elgreco On a second thought, I've added basic UC docs to this PR itself. It looks as follows:

In the next PR, I'll also add some code examples and all, along with the UC OSS payload support.

Looks good, would be slightly easier to find under integrations section but can be done in follow up

[ion-elgreco]

@omkar-foss the UC integration test is failing

[omkar-foss]

@omkar-foss the UC integration test is failing

Yes I checked, it's passing on my local. Seems like it's an intermittent failure. Would it be possible for you to retrigger that particular failing job, let's see if it passes?

[ion-elgreco]

@omkar-foss can you try to make it not flaky?

[omkar-foss]

@omkar-foss can you try to make it not flaky?

Seems like it gets stuck intermittently due to a deadlock, checking on it now.

[hntd187]

Nice, I appreciate you doing all the doc work with this as well, everything looks great to me.

[omkar-foss]

@omkar-foss can you try to make it not flaky?

Seems like it gets stuck intermittently due to a deadlock, checking on it now.

I've added a dedicated function to execute futures pertaining to UC, based on the function we have for AWS S3 storage. This is helping to prevent the deadlocks by using an appropriate block_on(). A deadlock can still occur, but very low chances (less than 2% probability as per my sampling of 100 runs, each 5 seconds apart).

@ion-elgreco @hntd187 guys, please check it out and let me know if this makes sense. Thank you :)