Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Implicitly plan UNNEST as lateral #13695

Merged
merged 8 commits into from
Dec 13, 2024

Conversation

rkrishn7
Copy link
Contributor

@rkrishn7 rkrishn7 commented Dec 9, 2024

Which issue does this PR close?

Partial fix for #13659

Rationale for this change

Fixes support for implicitly planning table functions in the FROM clause as lateral.

What changes are included in this PR?

This PR adds support for implicit lateral planning of UNNEST table factors.

Are these changes tested?

Yes, sqllogictest added.

Are there any user-facing changes?

Potentially - because UNNEST is now implicitly lateral, like Postgres. However, since outer references still fail in the query planning/optimization phase, I'm not sure if documentation updates are required.

@github-actions github-actions bot added sql SQL Planner sqllogictest SQL Logic Tests (.slt) labels Dec 9, 2024
@rkrishn7 rkrishn7 changed the title Fix/unnest implicit lateral fix: Implicitly plan UNNEST as lateral Dec 9, 2024
# Issue: https://github.com/apache/datafusion/issues/13659
# TODO: https://github.com/apache/datafusion/issues/10048
query error DataFusion error: This feature is not implemented: Physical plan does not support logical expression OuterReferenceColumn\(List\(Field \{ name: "item", data_type: Int64, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: \{\} \}\), Column \{ relation: Some\(Bare \{ table: "u" \}\), name: "column1" \}\)
select * from unnest_table u, unnest(u.column1);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's better to add a logical plan test like

explain select * from unnest_table u, unnest(u.column1);

You can refer to how other LATERAL tests do it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added!

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's good to keep this one and another planning one in the sqllogictest (for end-to-end SQL test).

Just like the following case:

# Test CROSS JOIN LATERAL syntax (planning)
query TT
explain select t1_id, t1_name, i from join_t1 t1 cross join lateral (select * from unnest(generate_series(1, t1_int))) as series(i);

and

# Test CROSS JOIN LATERAL syntax (execution)
# TODO: https://github.com/apache/datafusion/issues/10048
query error DataFusion error: This feature is not implemented: Physical plan does not support logical expression OuterReferenceColumn\(UInt32, Column \{ relation: Some\(Bare \{ table: "t1" \}\), name: "t1_int" \}\)
select t1_id, t1_name, i from join_t1 t1 cross join lateral (select * from unnest(generate_series(1, t1_int))) as series(i);

@github-actions github-actions bot added sqllogictest SQL Logic Tests (.slt) and removed sqllogictest SQL Logic Tests (.slt) labels Dec 9, 2024
Copy link
Contributor

@alamb alamb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @rkrishn7

Can you please add some slt tests that actually run this kind of query to show them working and generating correct results?

Thank you 🙏

@alamb
Copy link
Contributor

alamb commented Dec 11, 2024

BTW, here are the instructions for sqllogictests: https://github.com/apache/datafusion/tree/main/datafusion/sqllogictest

Ideally you should be able to extend one of the existing test files in https://github.com/apache/datafusion/tree/main/datafusion/sqllogictest/test_files

@rkrishn7 rkrishn7 force-pushed the fix/unnest-implicit-lateral branch from 4e8f62c to 5ba8907 Compare December 13, 2024 04:04
@rkrishn7
Copy link
Contributor Author

@alamb @goldmedal

Added a few slt tests! Apologies for the delay!

@rkrishn7 rkrishn7 requested review from goldmedal and alamb December 13, 2024 04:05
Copy link
Contributor

@goldmedal goldmedal left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @rkrishn7 it looks good to me.

@alamb alamb merged commit 03e39da into apache:main Dec 13, 2024
25 checks passed
@alamb
Copy link
Contributor

alamb commented Dec 13, 2024

Thanks @rkrishn7 and @goldmedal !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
sql SQL Planner sqllogictest SQL Logic Tests (.slt)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants