-
Notifications
You must be signed in to change notification settings - Fork 3.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Use statistics in Faker CTAS #24585
Use statistics in Faker CTAS #24585
Conversation
@raunaqmorarka this is the last one, I promise :-) |
plugin/trino-faker/src/main/java/io/trino/plugin/faker/FakerMetadata.java
Outdated
Show resolved
Hide resolved
plugin/trino-faker/src/main/java/io/trino/plugin/faker/FakerMetadata.java
Outdated
Show resolved
Hide resolved
8af4d29
to
c40b9fa
Compare
@raunaqmorarka and @losipiuk this is ready for a review. It's the last one about Faker, I don't have anything else planned for it. |
@raunaqmorarka @losipiuk a gentle reminder |
plugin/trino-faker/src/main/java/io/trino/plugin/faker/FakerMetadata.java
Outdated
Show resolved
Hide resolved
plugin/trino-faker/src/main/java/io/trino/plugin/faker/FakerMetadata.java
Outdated
Show resolved
Hide resolved
plugin/trino-faker/src/main/java/io/trino/plugin/faker/FakerMetadata.java
Outdated
Show resolved
Hide resolved
plugin/trino-faker/src/main/java/io/trino/plugin/faker/FakerMetadata.java
Outdated
Show resolved
Hide resolved
plugin/trino-faker/src/main/java/io/trino/plugin/faker/FakerMetadata.java
Outdated
Show resolved
Hide resolved
plugin/trino-faker/src/main/java/io/trino/plugin/faker/FakerMetadata.java
Outdated
Show resolved
Hide resolved
plugin/trino-faker/src/main/java/io/trino/plugin/faker/FakerMetadata.java
Outdated
Show resolved
Hide resolved
plugin/trino-faker/src/main/java/io/trino/plugin/faker/FakerMetadata.java
Outdated
Show resolved
Hide resolved
plugin/trino-faker/src/main/java/io/trino/plugin/faker/FakerMetadata.java
Outdated
Show resolved
Hide resolved
plugin/trino-faker/src/main/java/io/trino/plugin/faker/FakerMetadata.java
Show resolved
Hide resolved
plugin/trino-faker/src/test/java/io/trino/plugin/faker/TestFakerQueries.java
Outdated
Show resolved
Hide resolved
plugin/trino-faker/src/test/java/io/trino/plugin/faker/TestFakerQueries.java
Outdated
Show resolved
Hide resolved
plugin/trino-faker/src/main/java/io/trino/plugin/faker/FakerConnector.java
Outdated
Show resolved
Hide resolved
c40b9fa
to
c2f5400
Compare
plugin/trino-faker/src/main/java/io/trino/plugin/faker/FakerMetadata.java
Outdated
Show resolved
Hide resolved
plugin/trino-faker/src/main/java/io/trino/plugin/faker/FakerMetadata.java
Outdated
Show resolved
Hide resolved
plugin/trino-faker/src/main/java/io/trino/plugin/faker/FakerMetadata.java
Outdated
Show resolved
Hide resolved
plugin/trino-faker/src/main/java/io/trino/plugin/faker/FakerMetadata.java
Outdated
Show resolved
Hide resolved
plugin/trino-faker/src/main/java/io/trino/plugin/faker/FakerMetadata.java
Outdated
Show resolved
Hide resolved
plugin/trino-faker/src/main/java/io/trino/plugin/faker/FakerMetadata.java
Outdated
Show resolved
Hide resolved
plugin/trino-faker/src/main/java/io/trino/plugin/faker/FakerMetadata.java
Outdated
Show resolved
Hide resolved
plugin/trino-faker/src/main/java/io/trino/plugin/faker/FakerConfig.java
Outdated
Show resolved
Hide resolved
plugin/trino-faker/src/main/java/io/trino/plugin/faker/FakerConnector.java
Outdated
Show resolved
Hide resolved
plugin/trino-faker/src/main/java/io/trino/plugin/faker/FakerConfig.java
Outdated
Show resolved
Hide resolved
c2f5400
to
c7cf0bb
Compare
c7cf0bb
to
3d035e3
Compare
plugin/trino-faker/src/main/java/io/trino/plugin/faker/FakerMetadata.java
Outdated
Show resolved
Hide resolved
plugin/trino-faker/src/main/java/io/trino/plugin/faker/FakerConnector.java
Outdated
Show resolved
Hide resolved
plugin/trino-faker/src/main/java/io/trino/plugin/faker/FakerConfig.java
Outdated
Show resolved
Hide resolved
plugin/trino-faker/src/main/java/io/trino/plugin/faker/FakerConfig.java
Outdated
Show resolved
Hide resolved
68f71d4
to
254ad19
Compare
plugin/trino-faker/src/main/java/io/trino/plugin/faker/FakerMetadata.java
Outdated
Show resolved
Hide resolved
plugin/trino-faker/src/main/java/io/trino/plugin/faker/FakerConfig.java
Outdated
Show resolved
Hide resolved
please squash the fixups |
254ad19
to
626364a
Compare
I messed up squashing the fixups yesterday, it was too late in the day. I'll clean up the commits and let you know when it's ready. |
626364a
to
9d7baf1
Compare
When creating a table in the Faker connector from an existing table, gather column statistics to determine range constraints, set them as column properties.
When creating a table in the Faker connector from an existing table, using column statistics determine low cardinality columns, and generate values from a randomly generated set.
When creating tables in the Faker connector using CREATE TABLE AS SELECT, use the NUMBER_OF_NON_NULL_VALUES column statistic to set the null_probability column property.
9d7baf1
to
c44f57f
Compare
Should we somehow document this? I kinda think yes, assuming it affects the shape of the random data generated .. maybe in a specific section about CTAS in the SQL support section of the connector docs |
Yes. The usage section only shows create table like, and we can add a paragraph about create table as select and explain what's the difference. |
Will you send a PR @nineinchnick ? Also .. if you add a little section for dedicated for CTAS we should link to it from the SQL support statement list, see also... |
Description
Use statistics when using
CREATE TABLE AS SELECT
in the Faker connector to:default_limit
table property to the estimated number of rows from the source tablemin
andmax
column properties based on the statisticsAdditional context and related issues
Previous attempt #24098 was abandoned after #24147 was reported. This time we only use views for sequence columns, and if this is not very useful, we can avoid creating the views automatically. Or this could be yet another column property.
Release notes
( ) This is not user-visible or is docs only, and no release notes are required.
( ) Release notes are required. Please propose a release note for me.
(x) Release notes are required, with the following suggested text: