fix: Handle predicates on non-nullable columns without stats

[adamreeve]

Fixes #698

What changes are proposed in this pull request?

Updates the DataSkippingFilter to treat all columns as nullable for the purpose of parsing stats, as suggested in #698 (comment).

This is particularly important for partition columns, which won't have values present in stats. But stats are also only usually stored for the first 32 columns, so we shouldn't rely on stats being present for non-partition fields either.

How was this change tested?

I've added a new unit test.

I've also tested building duckdb-delta with this change (cherry-picked onto 0.6.1) and verified that the code in #698 now works.

[adamreeve]

But stats are also only usually stored for the first 32 columns, so we shouldn't rely on stats being present for non-partition fields either.

I haven't added a separate test for this, but I verified this is also a problem with DuckDB-Delta. The following code also crashes without this fix:

import duckdb
from deltalake import write_deltalake
import pyarrow as pa
import numpy as np


delta_dir = "./delta_table"
num_cols = 33

schema = pa.schema([
    pa.field(f"x{i}", pa.int32(), nullable=False) for i in range(num_cols)
])

rng = np.random.default_rng(0)
for p in range(10):
    num_rows = 1_000
    table = pa.Table.from_pydict({
        f'x{i}': pa.array(rng.integers(p * 100, (p + 1) * 100, num_rows), type=pa.int32()) for i in range(num_cols)
    })
    write_deltalake(
            table_or_uri=delta_dir,
            data=table,
            mode='append',
            schema=schema)

results = duckdb.query(f"SELECT * FROM delta_scan('{delta_dir}') WHERE x32 = 101")
print(results)

[codecov]

Codecov Report

Attention: Patch coverage is 93.10345% with 2 lines in your changes missing coverage. Please review.

Project coverage is 84.26%. Comparing base (16d2557) to head (ba67db8).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
kernel/src/scan/data_skipping.rs	90.00%	0 Missing and 2 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #700      +/-   ##
==========================================
+ Coverage   84.22%   84.26%   +0.04%     
==========================================
  Files          77       77              
  Lines       17926    17950      +24     
  Branches    17926    17950      +24     
==========================================
+ Hits        15098    15126      +28     
+ Misses       2110     2109       -1     
+ Partials      718      715       -3     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[scovich]

Thanks for tackling this, the approach looks good. Couple questions tho.

[scovich]

Very nice, thanks for fixing this!

[zachschuermann]

LGTM thanks for catching + fixing this!!

[zachschuermann]

nice, thanks for adding test!