Skip to content

Commit

Permalink
Update to DataFusion 42, Arrow 53.1 (RC), remove DataFrameDataSource …
Browse files Browse the repository at this point in the history
…Python logic (#513)

* Initial update to DataFusion 42 and Arrow 53.1

* Remove protobuf python code

* Remove python protobuf instructions

* fix jsonwriter tests

* fix select tests

* more sort test fixes

* more test fixes

* test / clippy fixes

* more test/lint fixes

* clippy/fmt fixes

* Remove order col after evaluation

* clippy fix

* fixes

* update baselines

* fix sql tests

* update baselines

* sort

* fmt

* use published arrow crates
  • Loading branch information
jonmmease authored Oct 6, 2024
1 parent 9ea6441 commit 65022d3
Show file tree
Hide file tree
Showing 68 changed files with 1,651 additions and 3,027 deletions.
13 changes: 0 additions & 13 deletions BUILD.md
Original file line number Diff line number Diff line change
Expand Up @@ -103,16 +103,3 @@ pixi add my-new-package --build
```

This will install the package and update the pixi.lock file with the new environment solution.

## Updating DataFusion protobuf
When updating the version of DataFusion that VegaFusion depends on, the protobuf definition file at `python/vegafusion/proto/datafusion.proto` must be updated to match. This file should be downloaded from GitHub from https://raw.githubusercontent.com/apache/arrow-datafusion/27.0.0/datafusion/proto/proto/datafusion.proto, replacing `27.0.0` with the matching version of DataFusion.

Then Python bindings should be generated using `protoc`

```
pixi shell
cd vegafusion-python
protoc --python_out=vegafusion ./proto/datafusion.proto
```

This will update the `vegafusion-python/vegafusion/proto/datafusion_pb2.py` file.
Loading

0 comments on commit 65022d3

Please sign in to comment.