Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

apply a schema to fix column names #331

Merged
merged 54 commits into from
Oct 21, 2024
Merged
Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
54 commits
Select commit Hold shift + click to select a range
5a7c419
apply a schema to fix column names
Sep 10, 2024
31e1977
remove an unwrap
Sep 10, 2024
5acbdb2
cleanup
Sep 10, 2024
1ea7554
checkpoint with working rename
Sep 12, 2024
54493aa
cleanup unneeded complexity, fmt
Sep 12, 2024
532a870
add map types
Sep 12, 2024
a5feb17
fmt
Sep 12, 2024
91ca99c
only apply if needed
Sep 12, 2024
29b09f7
fmt+clippy
nicklan Oct 8, 2024
451a76d
Merge branch 'main' into fix-cm-name-take1
nicklan Oct 10, 2024
ada2f7b
checkpoint
nicklan Oct 10, 2024
1633e49
Merge branch 'main' into fix-cm-name-take1
nicklan Oct 11, 2024
4af1bbd
checkpoint
nicklan Oct 11, 2024
e0e3ed6
refactor for helpers
nicklan Oct 12, 2024
ce4d3f3
small bug fix
nicklan Oct 15, 2024
9d3b49d
arg order
nicklan Oct 15, 2024
69c1e27
checkpoint
nicklan Oct 15, 2024
e232c77
Merge branch 'main' into fix-cm-name-take1
nicklan Oct 15, 2024
40833db
match main
nicklan Oct 15, 2024
f7652d5
use transform
nicklan Oct 15, 2024
4113603
match main
nicklan Oct 15, 2024
7ca2705
golden test not ready
nicklan Oct 15, 2024
39cfe0f
also align nullability
nicklan Oct 16, 2024
07c56ca
Also fix metadata
nicklan Oct 16, 2024
f2f75b8
use as_string instead
nicklan Oct 16, 2024
3dda376
ok_or_else all the things
nicklan Oct 16, 2024
e67360c
address comments
nicklan Oct 16, 2024
085bd19
remove unneeded arg
nicklan Oct 16, 2024
88b4db3
Adjust flow for comment
nicklan Oct 16, 2024
09bc974
also update metadata
nicklan Oct 16, 2024
29645a7
validate type on non-struct exprs
nicklan Oct 16, 2024
927bbaf
Update doc comment
nicklan Oct 16, 2024
b605e5e
fmt
nicklan Oct 16, 2024
2d00469
make column mapping test work :)
nicklan Oct 17, 2024
0b80c0e
Merge branch 'main' into fix-cm-name-take1
nicklan Oct 17, 2024
a4c6de6
Merge branch 'main' into fix-cm-name-take1
nicklan Oct 18, 2024
38456d1
address some comments
nicklan Oct 18, 2024
5e65247
address comments
nicklan Oct 18, 2024
9020a25
add comment
nicklan Oct 18, 2024
9cbd992
fix comment
nicklan Oct 18, 2024
17a5303
comment
nicklan Oct 18, 2024
ae67479
more consistent naming
nicklan Oct 18, 2024
b2e143a
Merge branch 'main' into fix-cm-name-take1
nicklan Oct 18, 2024
0b48b4c
add tests
nicklan Oct 18, 2024
65b6218
address some comments
nicklan Oct 18, 2024
479758d
ArrayRef all the things
nicklan Oct 18, 2024
efc5193
refactor to new_field_with_metadata
nicklan Oct 18, 2024
e02357a
move ensure_data_types to own module
nicklan Oct 21, 2024
eb786d8
avoid feature flag repetition
nicklan Oct 21, 2024
d9af456
switch to struct for ensure
nicklan Oct 21, 2024
622c03f
address final comments, and make feature flags at least consistent
nicklan Oct 21, 2024
dd65781
Merge branch 'main' into fix-cm-name-take1
nicklan Oct 21, 2024
ecbd668
Apply suggestions from code review
nicklan Oct 21, 2024
929a187
final comments
nicklan Oct 21, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 5 additions & 6 deletions kernel/src/engine/arrow_expression.rs
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rescuing #331 (comment):

We're still only at 9% patch coverage. We might need to take @zachschuermann advice and make a schema just for this?

Maybe you could reuse this one from the schema depth checker test?
https://github.com/delta-incubator/delta-kernel-rs/blob/main/kernel/src/schema.rs#L1005
It has a wide variety of nested structures, at least.

Down to 7% now...

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, I've added some more tests and it's up now

Original file line number Diff line number Diff line change
Expand Up @@ -25,8 +25,8 @@ use itertools::Itertools;
use super::arrow_conversion::LIST_ARRAY_ROOT;
use super::arrow_utils::make_arrow_error;
use crate::engine::arrow_data::ArrowEngineData;
use crate::engine::arrow_utils::ensure_data_types;
use crate::engine::arrow_utils::prim_array_cmp;
use crate::engine::ensure_data_types::ensure_data_types;
use crate::error::{DeltaResult, Error};
use crate::expressions::{BinaryOperator, Expression, Scalar, UnaryOperator, VariadicOperator};
use crate::schema::{ArrayType, DataType, MapType, PrimitiveType, Schema, SchemaRef, StructField};
Expand Down Expand Up @@ -131,7 +131,7 @@ impl Scalar {
}

fn wrap_comparison_result(arr: BooleanArray) -> ArrayRef {
Arc::new(arr) as ArrayRef
Arc::new(arr) as _
}

trait ProvidesColumnByName {
Expand Down Expand Up @@ -400,7 +400,7 @@ fn transform_struct(
let target_field = target_field.borrow();
let transformed_col = apply_schema_to(&sa_col, target_field.data_type())?;
let transformed_field = new_field_with_metadata(
target_field.name.as_str(),
&target_field.name,
transformed_col.data_type(),
target_field.nullable,
Some(target_field.metadata_with_string_values()),
Expand Down Expand Up @@ -445,11 +445,10 @@ fn apply_schema_to_list(
let (field, offset_buffer, values, nulls) = la.clone().into_parts();

let transformed_values = apply_schema_to(&values, &target_inner_type.element_type)?;
let transformed_field = new_field_with_metadata(
let transformed_field = ArrowField::new(
field.name(),
transformed_values.data_type(),
transformed_values.data_type().clone(),
target_inner_type.contains_null,
None,
);
Ok(ListArray::try_new(
Arc::new(transformed_field),
Expand Down
Loading
Loading