[EPIC] Automatically generate all function documentation from code

[alamb]

Is your feature request related to a problem or challenge?

Currently, when we add a new function to DataFusion library we have to remember to document that function in the documentation, but currently we have to remember this during code review.

This has a few downsides:

1. we have forgotten to document some functions
2. the documentation has drifted over time,
3. help text for various functions can only be found on the DataFusion website, and not, for example within the function itself.

@Omega359 has created the basic framework for automatically generating content in #12668

This ticket tracks the work required to port the rest of the documentation to programatic form

Tasks

• Automatically generate function documentation from comments / code #12432
• Improve description of function migration #12743
• Remove redundant aggregate/window/scalar function documentation #12745
• feat: Add alternative_syntax function for docs #13139
• Remove old static documentation function pages (Aggregate/Scalar) #12741
• Remove old static Window Functions page #13171
• Add support for DESCRIBE xyz; where xyz is a function. Ideally this would support not just the core functions in DF but also any added externally (for example those in https://github.com/datafusion-contrib/datafusion-functions-extra) and verify it works in the CLI
• Look into generating the function sections in expressions.md (it's a bit out of date)
• Make generate_function_docs.sh error if there is a function without documentation

Aggregate Function Documentation Migration

• Migrate documentation for aggregate functions from aggregate_functions.md to code #12827
• Port / Add Documentation for BitwiseOperation
• Port / Add Documentation for VarianceSample Port / Add Documentation for VarianceSample and VariancePopulation #12742
• Port / Add Documentation for VariancePopulation Port / Add Documentation for VarianceSample and VariancePopulation #12742

Scalar Function Documentation Migration

• [EPIC] Improve examples to make them easier to navigate #11172
• Migrate documentation for all string functions from scalar_functions.md to code  #12775
• Migrate documentation for all array functions from scalar_functions.md to code #12927
• Migrate documentation for all crypto functions from scalar_functions.md to code #12828
• Migrate documentation for all datetime functions from scalar_functions.md to code #12859
• Migrate documentation for all encoding functions from scalar_functions.md to code #12858
• Migrate documentation for all math functions from scalar_functions.md to code #12867
• Migrate documentation for all regex functions from scalar_functions.md to code #12868
• Migrate documentation for all string functions from scalar_functions.md to code #12934
• Migrate documentation for all unicode functions from scalar_functions.md to code #12935

Window Function Documentation Migration

• Port / Add Documentation for Lead
• Port / Add Documentation for RowNumber
• Migrate documentation for remaining window functions to window_functions.md #12936

[alamb]

Update is I have a few follow on PRs

• Remove old static documentation function pages (Aggregate/Scalar) #12741
• Automatically generate function documentation from comments / code #12432

Then I figure I will file one or two other tickets to see how easy it is to port other functions

And if all goes well I'll make a ticket storm to port the docs

[Omega359]

FYI I've finished up the string expression migration locally - just waiting on the updated code from #12742 to land before pushing a PR.

[alamb]

FYI I've finished up the string expression migration locally - just waiting on the updated code from #12742 to land before pushing a PR.

Thanks, I'll merge those as soon as they get a review (I can't merge them until they get an approval from another committer)

[Omega359]

No rush I just wanted to make sure no one was duplicating work needlessly

[jonathanc-n]

@alamb I think it should be mentioned here that ./dev/update_function_docs.sh should be ran, as it wasn't immediately obvious, and when I did my first migration I took a bit of time to find it.

[alamb]

@alamb I think it should be mentioned here that ./dev/update_function_docs.sh should be ran, as it wasn't immediately obvious, and when I did my first migration I took a bit of time to find it.

Thank you @jonathanc-n -- I added it to the description of #12859 and will add it on subsequent tickets

[alamb]

Thanks to @jonathanc-n @Omega359 and others we are cranking right along with this. I expect it to be done sometime later this week

[alamb]

I think we are pretty close here

After we merge these PRs from @jonathanc-n and myself

• Minor: Add documentation for cot #13069
• feat: Migrate Map Functions #13047

We then have:

• Migrate all documentation for all map functions from scalar_functions.md to code #13021

Then we have just a few more functions to document (lead, lag) and we can turn on a CI check to ensure all new functions are documented

(venv) andrewlamb@Andrews-MacBook-Pro-2:~/Software/datafusion$ ./dev/update_function_docs.sh
/Users/andrewlamb/Software/datafusion
Inserting header
Running CLI and inserting aggregate function docs table
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.34s
     Running `target/debug/print_functions_docs aggregate`
Running prettier
docs/source/user-guide/sql/aggregate_functions_new.md 59ms
'docs/source/user-guide/sql/aggregate_functions_new.md' successfully updated!
Inserting header
Running CLI and inserting scalar function docs table
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.09s
     Running `target/debug/print_functions_docs scalar`
INFO: The following functions do not have documentation:
  - map_values
  - map_keys
  - map
  - map_extract
Running prettier
docs/source/user-guide/sql/scalar_functions_new.md 208ms
'docs/source/user-guide/sql/scalar_functions_new.md' successfully updated!
Inserting header
Running CLI and inserting window function docs table
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.09s
     Running `target/debug/print_functions_docs window`
INFO: The following functions do not have documentation:
  - lag
  - lead
Running prettier
docs/source/user-guide/sql/window_functions_new.md 26ms
'docs/source/user-guide/sql/window_functions_new.md' successfully updated!

[Omega359]

lead/lag PR @ #13095

Other functions that still exist on scalar functions page - #13036

[alamb]

Here is a PR to make sure no new functions are added without documentation: #12938

[alamb]

🎉 I think all we need is #13171 and we can claim this project is complete. Thanks again @Omega359 @buraksenn and @jonathanc-n for pushing it along

[alamb]

🎉 epic