[RFC] Rust refactors

[jaychia]

[RFC] Rust refactors

RFC Summary

This RFC scopes out the work for porting portions of the Daft codebase into Rust for the 0.1.0 release.

RFC Details

Motivation

Python as a language is a great user-interface for data scientists and users due to its dynamic typing and ecosystem of open-sourced libraries. However for implementing Daft itself, Python has several limitations.

Problem

1. Lack of pattern matching: until Python 3.10, Python lacks good structural matching. This makes several modules difficult to implement and extend (query optimizer, expression optimization, operator type resolution).
2. Kernels: Daft currently relies heavily on PyArrow kernels, some custom C++ kernels and sometimes casting to types such as Pandas/Polars where PyArrow is buggy or insufficient. Having an easy way to define our own kernels is key to unlock future project velocity.
3. Packaging: packaging is a nightmare - relying on Polars/Pandas/Numpy/PyArrow for kernel execution makes Daft very heavy to install

Solution

1. Rust's support for pattern matching will give us robust tooling for building out more complex query optimizations and type resolution.
2. Moving to Rust gives us access to the excellent Arrow2 (https://github.com/jorgecarleitao/arrow2) library, which we can further extend with our own Rust kernels as required.
3. With Rust, we can build and package our code in a self-contained wheel and remove many of Daft's external dependencies

This is a WIP to be scoped out into issues for the refactoring of the following modules:

1. Execution (Blocks and vPartitions)
2. LogicalPlan
3. Expressions
4. Expression Types and Operators
5. Query optimizer

User Experience

The Daft interface from a user perspective is unchanged except:

1. Building Daft from source will now require the Rust compiler
2. Daft no longer builds against the Arrow C++ codebase, which fixes a lot of packaging and distribution issues

Blocking

This issue blocks the following:

1. (Arrow + ComplexArrow) / Python Types #367 expression typing system issue