Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement shuffle using Ray object store #55

Open
andygrove opened this issue Dec 14, 2024 · 0 comments
Open

Implement shuffle using Ray object store #55

andygrove opened this issue Dec 14, 2024 · 0 comments

Comments

@andygrove
Copy link
Member

andygrove commented Dec 14, 2024

We want to reimplement the shuffle mechanism and have it use the Ray object store to store the shuffle output.

Here are some notes on what this may look like.

ShuffleWriterExec and ShuffleReaderExec should be re-implemented in Python rather than Rust (although they can still call Rust code where needed).

Planning

The reader needs to know how to find the output from the writer, so we perhaps need to add some extra data to the distributed plan, such as a UUID for each query stage.

Writer

The new ShuffleWriterExec should execute its child plan to get a stream of record batches and iterate over them and repartition them according to the shuffle partitioning schema.

The smaller repartitioned batches should be written to the object store (although we may want to buffer them in memory first until they are a certain size).

When all batches have been processed, we should be able to write a final object that contains references to the other objects.

Reader

The reader will need to find the final object stored by the writer so that it can then load the other batches, We probably need to remove the objects once they have been read.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant