Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fetching database records with streaming approach #107

Open
salamonpavel opened this issue Dec 14, 2023 · 0 comments · May be fixed by #108
Open

Fetching database records with streaming approach #107

salamonpavel opened this issue Dec 14, 2023 · 0 comments · May be fixed by #108
Labels
enhancement New feature or request

Comments

@salamonpavel
Copy link
Contributor

Background

At the moment if we're fetching a large number of records from a database without using streaming, we can potentially run into an OutOfMemoryError. This is because all the records are loaded into memory at once.

Feature

Streaming, on the other hand, loads data in chunks, keeping only a portion of the data in memory at any given time. This is more memory-efficient and can handle large datasets that would not fit into memory all at once.

Both Doobie and Slick support streaming. Moreover streaming integration with ZIO is possible. It would be great to explore the space and find out if/how we could introduce streaming capabilities into this library.

@salamonpavel salamonpavel added the enhancement New feature or request label Dec 14, 2023
@salamonpavel salamonpavel changed the title Fetching database records with streaming approach [RESEARCH] Fetching database records with streaming approach Dec 14, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
Status: 🆕 To groom
Development

Successfully merging a pull request may close this issue.

1 participant