Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Experimental s3 and parquet support #417

Merged
merged 15 commits into from
Nov 15, 2023
Merged

Experimental s3 and parquet support #417

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
2c90bc1
pixi job to launch minio with test data
jonmmease Nov 11, 2023
22f8e5c
Test loading movies.json from s3
jonmmease Nov 11, 2023
3578c14
Also support loading csv and arrow from s3
jonmmease Nov 11, 2023
8621921
Add initial parquet support for DataFusion connection
jonmmease Nov 12, 2023
d85d702
Run minio server in the background on CI
jonmmease Nov 12, 2023
c158359
debug ci
jonmmease Nov 12, 2023
9df43e8
don't import vegafusion from start_minio.py
jonmmease Nov 12, 2023
e5d759c
Revert "debug ci"
jonmmease Nov 12, 2023
cf3155b
Try background-action
jonmmease Nov 12, 2023
2ab7731
try it
jonmmease Nov 12, 2023
453baa5
Don't use tempfile for csv upload
jonmmease Nov 13, 2023
c70bdb0
another
jonmmease Nov 13, 2023
51fad69
terminate
jonmmease Nov 13, 2023
993d9a8
again
jonmmease Nov 13, 2023
1d86632
Flag s3 tests behind VEGAFUSION_S3_TESTS environment variable
jonmmease Nov 13, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .github/workflows/build_test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,8 @@ jobs:
~/.cargo
target
.pixi
- name: start minio server in the background
run: pixi run start-minio &
- name: Test rust
run: |
pixi run test-rs --release
Expand Down
1 change: 1 addition & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,3 +44,4 @@ __pycache__
/java/.idea/
/.pixi/
/python/vegafusion-jupyter/MANIFEST
/minio_data/
7 changes: 7 additions & 0 deletions BUILD.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,13 @@ Then restart your shell.
For more information on installing Pixi, see https://prefix.dev/docs/pixi/overview.

## Build and test Rust

Start the test minio server in a dedicated terminal

```
pixi run start-minio
```

Build and test the VegaFusion Rust crates with:

```
Expand Down
Loading
Loading