Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Migrate optimizer tests to insta #15445

Closed
wants to merge 37 commits into from
Closed

Migrate optimizer tests to insta #15445

wants to merge 37 commits into from

Conversation

qstommyshu
Copy link
Contributor

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Migrated tests in datafusion/core/tests/optimizer to use insta.

Are these changes tested?

Yes, I manually tested the before/after changes.

Are there any user-facing changes?

No

@github-actions github-actions bot added the optimizer Optimizer rules label Mar 27, 2025
alamb and others added 27 commits March 26, 2025 21:47
@alamb @qstommyshu
Update datafusion-testing pin to fix extended tests (apache#15368)
eabf514
@PokIsemaine @ZhouSiLe
@jonahgao @qstommyshu
chore(deps): Update sqlparser to 0.55.0 (apache#15183)
56000b9 
* chore(deps): Update sqlparser to 0.55.0

* merge upstream

* style: cargo fmt

* fix: import

* style: cargo fmt

* fix: struct.slt, update.slt and lateral join

* chore: cargo fmt & remove unwrap

* fix: remove meaningless comment for rustfmt

* refactor: plan_datafusion_err, remove clone

* remove println

---------

Co-authored-by: silezhou <[email protected]>
Co-authored-by: jonahgao <[email protected]>
@blaginin @qstommyshu
Only unnest source for EmptyRelation (apache#15159)
0c39c13 
* Only unnest source for `EmptyRelation`

* Add a note on new condition

* Remove new test

* Put all unnest assumptions into one function
@dependabot @qstommyshu
chore(deps): bump rust_decimal from 1.37.0 to 1.37.1 (apache#15378)
d8e3e31 
Bumps [rust_decimal](https://github.com/paupino/rust-decimal) from 1.37.0 to 1.37.1.
- [Release notes](https://github.com/paupino/rust-decimal/releases)
- [Changelog](https://github.com/paupino/rust-decimal/blob/master/CHANGELOG.md)
- [Commits](paupino/rust-decimal@1.37.0...1.37.1)

---
updated-dependencies:
- dependency-name: rust_decimal
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
@dependabot @qstommyshu
chore(deps): bump chrono-tz from 0.10.1 to 0.10.2 (apache#15377)
9f799f1 
Bumps [chrono-tz](https://github.com/chronotope/chrono-tz) from 0.10.1 to 0.10.2.
- [Release notes](https://github.com/chronotope/chrono-tz/releases)
- [Commits](https://github.com/chronotope/chrono-tz/commits)

---
updated-dependencies:
- dependency-name: chrono-tz
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
@goldmedal @qstommyshu
remove duplicate test (apache#15385)
fb4f4b1
@zhuqi-lucas @qstommyshu
Minor: add average time for clickbench benchmark query (apache#15381)
9c474a3
@waynexia @qstommyshu
feat: include some BinaryOperator from sqlparser (apache#15327)
33e8f89 
* include some BinaryOperator from sqlparser

Signed-off-by: Ruihang Xia <[email protected]>

* slt for new operators

Signed-off-by: Ruihang Xia <[email protected]>

* add document about availability

Signed-off-by: Ruihang Xia <[email protected]>

---------

Signed-off-by: Ruihang Xia <[email protected]>
@jsai28 @qstommyshu
added test (apache#15362)
ac31578
@Jiashu-Hu @alamb @qstommyshu
Documentation: Plan custom expressions (apache#15353)
32c043a 
* Added Custom Expression Planning Section

* improve syntax

* Update docs/source/library-user-guide/adding-udfs.md

Co-authored-by: Andrew Lamb <[email protected]>

* Update docs/source/library-user-guide/adding-udfs.md

Co-authored-by: Andrew Lamb <[email protected]>

* Update docs/source/library-user-guide/adding-udfs.md

Co-authored-by: Andrew Lamb <[email protected]>

* Update docs/source/library-user-guide/adding-udfs.md

Co-authored-by: Andrew Lamb <[email protected]>

* Update docs/source/library-user-guide/adding-udfs.md

Co-authored-by: Andrew Lamb <[email protected]>

* Update adding-udfs.md

added assert for CI process

* formatted language request by prettier

---------

Co-authored-by: Andrew Lamb <[email protected]>
@Shreyaskr1409 @qstommyshu
Migrate physical plan tests to insta (Part-2) (apache#15364)
1a12356 
* Migrated tests to insta in aggregate/topk/heap.rs

* Migrated tests to insta in joins/nested_loop_join.rs

* Migrated tests to insta in joins/sort_merge_join.rs

* Remove snapshot uploaded by mistake

* Migrated tests to insta in repartition/mod.rs

* Fix batches_to_sort_string problem and formatting
@christophermcdermott @qstommyshu
fix: write hive partitions for any int/uint/float (apache#15337)
0b0e308 
Co-authored-by: christophermcdermott <masked>
@shruti2522 @alamb @qstommyshu
fix: core_expressions feature flag broken, move overlay into `cor…
aeccc85 
…e` functions (apache#15217)

* fix: remove core_expressions, move overlay to core

* fix license header

* fix header

* update doc

* Remove CI reference

---------

Co-authored-by: Andrew Lamb <[email protected]>
@2010YOUY01 @qstommyshu
fix: Redundant files spilled during external sort + introduce `SpillM…
1dd5f31 
…anager` (apache#15355)

* implement SpillManager

* more comments
@Shreyaskr1409 @qstommyshu
Migrate physical plan tests to insta (Part-3 / Final) (apache#15399)
11a3c53 
* Migrated tests to insta in sorts/partial_sort.rs

* Migrated tests to insta in sorts/sort.rs and sorts/sort_preserving_merge.rs

* Migrated tests to insta in windows/bounded_window_agg_exec.rs

* Removed unused imports

* Fixes for failing tests
@findepi @qstommyshu
Restore lazy evaluation of fallible CASE (apache#15390)
5f07e4a 
* Restore lazy evaluation of fallible CASE

Commit 0f4b8b1 introduced a new
optimized evaluation mode for CASE expression with exactly one WHEN-THEN
clause and ELSE clause being present. Apparently, the new mode did not
take into account expensive or fallible expressions, as it
unconditionally evaluated both branches. This is definitely incorrect
for fallible expressions. For these, fallback to the original execution
mode.

* Workaround for running sqllogictests in RustRover
@dependabot @qstommyshu
chore(deps): bump log from 0.4.26 to 0.4.27 (apache#15410)
e68b6d1 
Bumps [log](https://github.com/rust-lang/log) from 0.4.26 to 0.4.27.
- [Release notes](https://github.com/rust-lang/log/releases)
- [Changelog](https://github.com/rust-lang/log/blob/master/CHANGELOG.md)
- [Commits](rust-lang/log@0.4.26...0.4.27)

---
updated-dependencies:
- dependency-name: log
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
@dependabot @qstommyshu
chore(deps): bump chrono-tz from 0.10.2 to 0.10.3 (apache#15412)
0219043 
Bumps [chrono-tz](https://github.com/chronotope/chrono-tz) from 0.10.2 to 0.10.3.
- [Release notes](https://github.com/chronotope/chrono-tz/releases)
- [Commits](chronotope/chrono-tz@v0.10.2...v0.10.3)

---
updated-dependencies:
- dependency-name: chrono-tz
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
@zhuqi-lucas @qstommyshu
Perf: Support Utf8View datatype single column comparisons for SortPre…
8d8b275 
…servingMergeStream (apache#15348)

* Perf: Support Utf8View datatype single column comparisons for SortPreservingMergeStream

* Add safety and bench sql

* fix

* Fix

* Add benchmark testing
@goldmedal @qstommyshu
Enforce JOIN plan to require condition (apache#15334)
c002b0c 
* add check for missing join condition

* modify the tests

* handle the cross join case

* remove invald sql case

* fix sqllogictests

* fix for cross join case

* revert the change for limit.slt

* add cross join test
@Omega359 @qstommyshu
Fix type coercion for unsigned and signed integers (Int64 vs `UInt6…
eb74863 
…4`, etc) (apache#15341)

* type coercion fix for uint/int's.

* Refactored common numerical coercion logic into a single function.

* Cargo fmt.
@davidhewitt @qstommyshu
simplify array_has UDF to InList expr when haystack is constant (a…
4515be5 
…pache#15354)

* simplify `array_has` UDF to `InList` expr when haystack is constant

* add `.slt` tests, also simplify with `make_array`

* tweak comment

* add test for `make_array` arg simplification
@xudong963 @qstommyshu
Refactor: add FileGroup structure for Vec<PartitionedFile> (apach…
a7113dc 
…e#15379)

* Refactor: add FileGroup structure

* update

* address comments

* address comments from alamb

* fix clippy
@Dandandan @qstommyshu
[Minor] Remove/reorder logical plan rules (apache#15421)
6d6db98 
* Remove rules that execute twice

* Move optimization rule around
@UBarney @qstommyshu
Improve performance of first_value by implementing special `GroupsA…
8a254f3 
…ccumulator` (apache#15266)

* Improve speed of first_value by implementing special GroupsAccumulator

* rename and other improvements

* `append_n` -> `resize`

* address comment

* use HashMap::entry

* remove hashMap in get_filtered_min_of_each_group
@jayzhan-synnada @berkaysynnada @qstommyshu
Move DataSink to datasource and add session crate (apache#15371)
e23bc17 
* session

* clippy

* fmt

* session

* minor

* Update README.md

* doc

* fmt

* doc

---------

Co-authored-by: berkaysynnada <[email protected]>
@Weijun-H @qstommyshu
refactor: SpillManager into a separate file (apache#15407)
82eedad 
* refactor: SpillManager into a separate file

* chore
Dandandan and others added 8 commits March 26, 2025 21:50
@Dandandan @qstommyshu
Always use PartitionMode::Auto in planner (apache#15339)
9dc3264 
* Always use PartitionMode::Auto

* Always use PartitionMode::Auto

* Fix test

* Fix test

* Fix tests

* Update plans

* Add changed splans
@chenkovsky @qstommyshu
fix typo (apache#15434)
fca0ecd
@JackKelly @qstommyshu
Fix link to Volcano paper (apache#15437)
fae993f
@berkaysynnada @qstommyshu
Update concepts-readings-events.md (apache#15440)
53ccad2 
* Update concepts-readings-events.md

* Update concepts-readings-events.md
@logan-keede @qstommyshu
minor: Add new crates to labeler (apache#15426)
2a0be17 
* Add new crates to labeler

* testing labeler

* testing labeler

* testing labeler

* testing labeler

* testing labeler

* testing labeler

* testing labeler

* testing labeler

* fixing

* adding to functions and catlog

* checking for path regex

* restore fixed
@2010YOUY01 @qstommyshu
refactor: Use SpillManager for all spilling scenarios (apache#15405)
0bf1415 
* Use SpillManager in all spilling scenarios

* resolve conflict

* fix ci format
@ctsk @Weijun-H @qstommyshu
refactor(hash_join): Move JoinHashMap to separate mod (apache#15419)
bc6ac3f 
* refactor(hash_join): Move JoinHashMap to separate mod

* Add description to join_hash_map module

Co-authored-by: Alex Huang <[email protected]>

---------

Co-authored-by: Alex Huang <[email protected]>
@gabotechs @qstommyshu
Add support for DISTINCT + ORDER BY in ARRAY_AGG (apache#14413)
c6d7d35 
* Add support for DISTINCT and ORDER BY in ARRAY_AGG

* Add DISTINCT + ORDER BY docs

* Add some more sqllogictests

* Update aggregate_functions.md
@github-actions github-actions bot added documentation Improvements or additions to documentation sql SQL Planner development-process Related to development process of DataFusion logical-expr Logical plan and expressions physical-expr Changes to the physical-expr crates core Core DataFusion crate sqllogictest SQL Logic Tests (.slt) substrait Changes to the substrait crate catalog Related to the catalog crate common Related to common crate proto Related to proto crate functions Changes to functions implementation datasource Changes to the datasource crate labels Mar 27, 2025
@qstommyshu
Copy link
Contributor Author

Ah, I accidentally merged all bunch of code from the main branch... I think it is easier for me to resolve all these by just creating another branch and PR

@qstommyshu qstommyshu closed this Mar 27, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
catalog Related to the catalog crate common Related to common crate core Core DataFusion crate datasource Changes to the datasource crate development-process Related to development process of DataFusion documentation Improvements or additions to documentation functions Changes to functions implementation logical-expr Logical plan and expressions optimizer Optimizer rules physical-expr Changes to the physical-expr crates proto Related to proto crate sql SQL Planner sqllogictest SQL Logic Tests (.slt) substrait Changes to the substrait crate
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Migrate optimizer tests to insta