DataFusion weekly project plan (Andrew Lamb) - Nov 27, 2023

[alamb]

The idea of this ticket is make my plans for DataFusion visible, largely for my own personal organizational needs, but also to:

1. Try some different ways to communicate / coordinate in the community
2. Help provide an interesting summary of what is happening in DataFusion this week

It would be great if anyone else who has plans like this for DataFusion could try to make them visible somehow as well

Plans for this week personally:

• sqlparser release: Release version 0.40.0 datafusion-sqlparser-rs#1030
• Epic: Statistics improvements #8227 (specifically, Consolidate statistics aggregation #8229 as a step towards Introduce a way to represent constrained statistics / bounds on values in Statistics #8078)
• Final edits + draft submission for Write DataFusion paper for (SIGMOD / VLDB / ICDE) #6782 with @ozankabak @Dandandan @yjshen and @JayjeetAtGithub @sunchao and @viirya
• File follow on tickets towards making function packages more modular: [EPIC] Unify Function Interface (remove BuiltInScalarFunction) #8045

My plans for next week

• FIFO JoinHashMap for HashJoin #8130, in support of memory explosion in HashJoin outputs: HashJoinStream memory tracking insufficient #7848 with @korowa

Major projects I intend to help review and coordinate

• Next steps towards function packages more modular (make all BuiltInScalarFunctions ScalarUDFs): [EPIC] Unify Function Interface (remove BuiltInScalarFunction) #8045 with @2010YOUY01
• ARRAY function hardening: [Epic] General ticket for the concept of the practical implementation of ARRAY #6980 / https://github.com/apache/arrow-datafusion/issues/7988with @jayzhan211 @Veeupup @edmondop and @xudong963
• Review any new ticket / bug reports

Algorithm for (my) prioritizing PRs

Note there are many committers who can review and merge PRs as well, so this is not the priorities of the project as a whole, just the approximate algorithm I am using

Priority:

1. Bug fixes (where something is just incorrect), especially regressions (where it used to work and now does not)
2. Improvements directly related to stuff needed for InfluxDB (as I am employed by them)
3. Documentation and test improvements (I view these as very strategically important)
4. PRs that I think are strategically important
5. Other new features / additions to functionality

The current strategically important projects in my head are:

• Support User Defined Table Functions / Table Value Functions #7926
• [EPIC] Unify Function Interface (remove BuiltInScalarFunction) #8045
• Improving performance of high cardinality aggregation

Note that adding new array functionality (e.g. #6980 and its children) is very low on my priority list, not because I don't think they are valuable, but because I think they are less valuable than the other projects going on

Thus, if you are interested in contributing to DataFusion and are interested in a fast turn around time I would recommend looking into bug fixes / test improvements.

If you propose adding new functionality, the review cycle will be longer. You can make it a shorter cycle by looking at the comments on other recent PRs and following the same model (e.g. ensure there are tests in sqllogictest for example)

[alamb]

Arrow-rs

• Improve cast docs arrow-rs#5114

DataFusion

• Support User Defined Table Function #8306
• Materialize dictionaries in group keys #8291
• Detect when filters on unique constraints make subqueries scalar #8312
• Support array_distinct function. #8268
• move array function unit_tests to sqllogictest #8332
• Introduce Boolean Coercion #8331 (review)
• refactor: output ordering #8304
• Fix incorrect results for NOT IN subqueries with nulls #8271
• Add support for recursive CTEs #7581
• Projection Pushdown over StreamingTableExec #8299
• Improve type coercion and casting #8302

sqlparser-rs

• add {pre,post}_visit_query to Visitor datafusion-sqlparser-rs#1044
• Update sqlparser-derive to use syn 2.0 datafusion-sqlparser-rs#1040

[alamb]

I am very backed up on reviews. I will try and work through them over the next few days

[alamb]

DataFusion

• Library Guide: Add Using the DataFrame API #8319 (review)
• Support User Defined Table Function #8306
• Port tests in aggregates.rs to sqllogictest #8316
• NTH_VALUE reverse support #8327 (review)
• Introduce Boolean Coercion #8331 (review)
• Port tests in limit.rs to sqllogictest #8315
• Materialize dictionaries in group keys #8291
• Port tests in csv_files.rs to sqllogictest #8251

DataFusion: As time permits (lower priority)

• Detect when filters on unique constraints make subqueries scalar #8312
• Support array_distinct function. #8268
• Fix incorrect results for NOT IN subqueries with nulls #8271
• Add support for recursive CTEs #7581
• Projection Pushdown over StreamingTableExec #8299
• Improve type coercion and casting #8302

[alamb]

I also want to make my algorithm for prioritizing PR reviews more visible, so adding it here (and will add to descriptions too). I want to point out we have many committers who can review and merge PRs as well, so this is not the priorities of the project as a whole, just the approximate algorithm I am using

Priority:

1. Bug fixes (where something is just incorrect), especially regressions (where it used to work and now does not)
2. Improvements directly related to stuff needed for InfluxDB (as I am employed by them)
3. PRs for documentation and test improvements (I view these as very strategically important)
4. PRs that I think are strategically important
5. Other new features / additions to functionality

The current strategically important projects I am tracking are listed in the description of this ticket

Thus, if you are interested in contributing to DataFusion and are interested in a fast turn around time I would recommend looking into bug fixes / test improvements.

If you propose adding new functionality, the review cycle will be longer. You can make it a shorter cycle by looking at the comments on other recent PRs and following the same model (e.g. ensure there are tests in sqllogictest for example)

[alamb]

Arrow

• Support nested schema projection (#5148) arrow-rs#5149

DataFusion

• [MINOR]: Move merge projections tests to under optimize projections #8352
• Minor: Move some datafusion-optimizer::utils down to datafusion-expr::utils #8354
• Add quote and escape attributes to create csv external table #8351 (review)
• MINOR: Change display of Window and Aggregate Functions #8353 (review)
• Implement Aliases for ScalarUDF #8360 (review)
• Refactor aggregate function handling #8358
• Support User Defined Table Function #8306
• Materialize dictionaries in group keys #8291
• Introduce Boolean Coercion #8331

DataFusion: As time permits (lower priority)

• Support array_distinct function. #8268
• Fix incorrect results for NOT IN subqueries with nulls #8271
• Add support for recursive CTEs #7581
• Improve type coercion and casting #8302

[alamb]

DataFusion

• Materialize dictionaries in group keys #8291
• Document timestamp input limits #8369
• Support LargeUtf8 to Temporal Coercion #8357 (review)
• feat: support LargeList in array_empty #8321 (review)
• Introduce Boolean Coercion #8331
• Double type argument for to_timestamp function #8159
• Support User Defined Table Function #8306
• fix: make ntile work in some corner cases #8371 (review)
• Add PRIMARY KEY Aggregate support to dataframe API #8356

DataFusion: As time permits (lower priority)

• Support array_distinct function. #8268
• Fix incorrect results for NOT IN subqueries with nulls #8271
• Add support for recursive CTEs #7581
• Improve type coercion and casting #8302

[alamb]

DataFusion

• Make filter selectivity for statistics configurable #8243 (review)
• Introduce Boolean Coercion #8331
• Allow default value on lag/lead if they are coerceable to value #8308
• Rename expr::window_function::WindowFunction to WindowFunctionDefinition, make structure consistent with ScalarFunction #8382 (review)
• Minor: Refactor function argument handling in ScalarFunctionDefinition #8387
• Refactors on TreeNode Implementations #8395
• Rewrite array_ndims to fix List(Null) handling #8320 (review)
• Support named query parameters #8384
• Add PRIMARY KEY Aggregate support to dataframe API #8356

DataFusion: As time permits (lower priority)

• Avoid concat for array_replace #8337
• Support array_distinct function. #8268
• Fix incorrect results for NOT IN subqueries with nulls #8271
• Add support for recursive CTEs #7581
• Improve type coercion and casting #8302

[jackwener]

Thanks @alamb !

Many people have asked me how to get involved in the DataFusion community and what they can do for the project (Because I've previously promoted DataFusion on the Chinese social media). This summary is very helpful for them.

I have pinned this issue to make it easy for everyone to view.

Express gratitude once again for all that you have done for the community.

[alamb]

Thanks @alamb !

Many people have asked me how to get involved in the DataFusion community and what they can do for the project (Because I've previously promoted DataFusion on the Chinese social media). This summary is very helpful for them.

Thank you for this feedback. It is very helpful. Other than writing up more detailed tickets, do you know of other things that would help?

I have pinned this issue to make it easy for everyone to view.

I saw this -- that was very cool. Thank you. I will pin future such issues.

Express gratitude once again for all that you have done for the community.

Thank you very much @jackwener - I am always amazed by all the work we do together.

[alamb]

Next Week: #8420