-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Document SQL dialect guidance #13706
base: main
Are you sure you want to change the base?
Changes from all commits
6bb2aa8
5af035c
49edb03
b674a42
8fd7f44
e9f5da1
c6a83f3
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,53 @@ | ||
<!--- | ||
Licensed to the Apache Software Foundation (ASF) under one | ||
or more contributor license agreements. See the NOTICE file | ||
distributed with this work for additional information | ||
regarding copyright ownership. The ASF licenses this file | ||
to you under the Apache License, Version 2.0 (the | ||
"License"); you may not use this file except in compliance | ||
with the License. You may obtain a copy of the License at | ||
|
||
http://www.apache.org/licenses/LICENSE-2.0 | ||
|
||
Unless required by applicable law or agreed to in writing, | ||
software distributed under the License is distributed on an | ||
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
KIND, either express or implied. See the License for the | ||
specific language governing permissions and limitations | ||
under the License. | ||
--> | ||
|
||
# SQL Dialect | ||
|
||
The included SQL supported in Apache DataFusion mostly follows the [PostgreSQL | ||
SQL dialect], including: | ||
|
||
- The SQL parser and [SQL planner] | ||
- Type checking, analyzer, and type coercions | ||
- Semantics of functions bundled with DataFusion | ||
|
||
Notable exceptions: | ||
|
||
- Array/List functions and semantics follow the [DuckDB SQL dialect]. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Not quite? DuckDB array seems to be fixed size. Is this saying DF array follows DuckDB's list semantics? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. yes DuckDB's "list" not "array". We prefer to convert fixed-size lists into regular lists, so we don’t make a distinction between the two There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. is this going to be obvious to the reader of this document? |
||
- DataFusion's type system is based on the [Apache Arrow type system], and the mapping to PostgreSQL types is not always 1:1. | ||
- DataFusion has its own syntax (dialect) for certain operations (like [`CREATE EXTERNAL TABLE`]) | ||
|
||
As Apache DataFusion is designed to be fully customizable, systems built on | ||
DataFusion can and do implement different SQL semantics. Using DataFusion's APIs, | ||
you can provide alternate function definitions, type rules, and/or SQL syntax | ||
that matches other systems such as Apache Spark or MySQL or your own custom | ||
semantics. | ||
|
||
[postgresql sql dialect]: https://www.postgresql.org/docs/current/sql.html | ||
[sql planner]: https://docs.rs/datafusion/latest/datafusion/sql/planner/struct.SqlToRel.html | ||
[duckdb sql dialect]: https://duckdb.org/docs/sql/functions/array | ||
[apache arrow type system]: https://arrow.apache.org/docs/format/Columnar.html#data-types | ||
[`create external table`]: ddl.md#create-external-table | ||
|
||
## Rationale | ||
|
||
SQL Engines have a choice to either use an existing SQL dialect or define their | ||
own. Using an existing dialect may not fit perfectly as it is hard to match | ||
semantics exactly (need bug-for-bug compatibility), and is likely not what all | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. More reasons for using pogstgres specifically: it is
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
mostly true (but i know of some deviations) i kind of assumed PostgreSQL ship has sailed and we're just retro-documenting. But if the ball (choice) is still in play, my vote goes to Trino as a good reference implementation. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
That was my assumption too, but as others like @jayzhan211 (and yourself) have pointed out, I don't think DataFusion is (or can be) 100% consistent with this point |
||
users want. However, it avoids the (very significant) effort of defining | ||
semantics as well as documenting and teaching users about them. |
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -21,6 +21,7 @@ SQL Reference | |
.. toctree:: | ||
:maxdepth: 2 | ||
|
||
dialect | ||
data_types | ||
select | ||
subqueries | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
it would be good to say this section pertains the SQL frontend:
see also #13704 (comment)
it's also worth noting that we are not going to align with PostgreSQL's dialect fully
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reworded and expanded in b674a42