chore: upgrade to DataFusion 46.0.0 (WORK IN PROGRESS) #3261
Conversation
alamb
requested review from
wjones127,
roeap,
rtyler,
hntd187 and
ion-elgreco
as code owners
|
ACTION NEEDED delta-rs follows the Conventional Commits specification for release automation. The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification. |
alamb
changed the title
| @@ -44,6 +44,7 @@ impl<'a, S: ContextProvider> DeltaSqlToRel<'a, S> { | |||
| enable_ident_normalization: self.options.enable_ident_normalization, | |||
| support_varchar_with_length: false, | |||
| enable_options_value_normalization: false, | |||
| collect_spans: false, | |||
There was a problem hiding this comment.
The source code locations for expressions:
https://docs.rs/sqlparser/latest/sqlparser/tokenizer/struct.Span.html
We are starting to gather / plumb this into DataFusion. This particular setting means the sql planner won't try and pass the span information along. The only thing the spans are used for now is debug / help messages
There was a problem hiding this comment.
Ah ok thanks for explaining :) I learned something new about datafusion!
There was a problem hiding this comment.
FWIW this is a new feature that was starting to be added in DataFusion 46
|
Looks like from CI we also need to update rust to be 1.82 https://github.com/delta-io/delta-rs/actions/runs/13506223447/job/37736308124?pr=3261
|
|
I have a few cleanup PRs I would plan to make as I work on this. Here is the first one: |
|
I also started breaking this PR up into some smaller ones: |
Description
chore: upgrade to DataFusion 46.0.0
This is a work in progress. I am testing the effects of upgrading to DataFusion 46
Related Issue(s)
46.0.0apache/datafusion#14123Documentation