Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Simplify paged queries - Python Generators #74

Open
santhoshramaraj opened this issue Oct 25, 2024 · 3 comments · May be fixed by #75
Open

Simplify paged queries - Python Generators #74

santhoshramaraj opened this issue Oct 25, 2024 · 3 comments · May be fixed by #75
Labels
enhancement New feature or request

Comments

@santhoshramaraj
Copy link
Member

santhoshramaraj commented Oct 25, 2024

Some client methods return paged responses. The user is required to handle the paged responses using continuation_token appropriately in the application.

Example - DataFrameClient().query_tables() , SpecClient().query_specs()

These paged responses can be simplified by using Generators in Python for easy use with loops and extracting the underlying paged data.

@santhoshramaraj santhoshramaraj added the enhancement New feature or request label Oct 25, 2024
@santhoshramaraj santhoshramaraj linked a pull request Oct 25, 2024 that will close this issue
1 task
@ChrisStrykesAgain
Copy link

I think there could be value here to implementing a decorator and just using it any time there's an API call that uses a continuation token. It seems like that would be cleaner and reduce code duplication.

@santhoshramaraj
Copy link
Member Author

The decorator concept satisfies the continuation token handling, but in most cases, users use these in some sort of loop, and making them a generator and exposing the underlying data intuitively is better than requiring the user to extract the data of interest.

For instance, query_tables() can directly return an iterable of TableMetadata instead of the user needing to invoke PagedTables.tables

@ChrisStrykesAgain
Copy link

I see your point; I was oversimplifying. I do feel there's some kind of reusable approach though. I'll think it through and post something more intelligent later if I can get my thoughts aligned.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants