[EPIC] Sample Mode

[QMalcolm]

Sample mode in dbt will be a "special" running mode to based on some criteria build models with a subset of the input data (i.e. a "sample"). This is important because a lot of us deal with incredibly large data sets on a frequent basis. During development, when we want to be iterating quickly, these large models take a while to run and slow us down. In its simplest form, at the end of this epic, one will be able to do the following dbt run --sample 3 days. The resulting run execution will create "sample" models for any model with an event_time with at least one input that also has an event_time defined.

P0

Preview Give feedback

1. Gate using sample mode with dbt_experimental_sample_mode environment variable #11227
   enhancement sample_mode +2
   
2. Add --sample flag to CLI invocation for run command #11230
   enhancement sample_mode +2
   
3. Allow for specification of a "relative" time window for sample mode #11231
   enhancement sample_mode +2
   
4. allow for the building of "normal" models as "samples" #11248
   enhancement sample_mode +2
   
5. allow for the building of "incremental" models as "samples" #11254
   enhancement sample_mode +2
   
6. allow for the building of "microbatch" models as "samples" #11252
   enhancement sample_mode +2
   
7. allow for specification of sample time window in targeted manner #11258
   enhancement sample_mode +2
   
8. collapse --sample and --sample-window into just --sample #11299
   enhancement sample_mode +2
   
9. allow for sample mode during build command #11298
   enhancement sample_mode +2
   
10. ability to sample seeds #11300
    enhancement sample_mode +2
    
11. ability to sample snapshots #11301
    enhancement sample_mode +2
    
12. ability for sample mode to work with python models #11302
    enhancement sample_mode +2
    

P1

Preview Give feedback

1. Remove dbt_experimental_sample_mode environment variable gate #11228
   enhancement sample_mode +2

P2

Preview Give feedback

No tasks being tracked yet.

bugs 👀

Preview Give feedback

No tasks being tracked yet.

--empty bugs 👀

Preview Give feedback

1. [CT-3310] Support --empty flag for dbt seed #8981
   Impact: Adapters empty unit tests user docs +4
2. DBT External tables don't work with the --empty flag #10464
   awaiting_response empty unit tests +3
3. [Bug] --empty flag fails with dbt.ref in snowpark #10517
   bug empty python_models unit tests +4
4. [Feature] Support contextual table aliasing dbt-adapters#199
   type:enhancement +1
5. [Bug] dbt build --empty results in a SQL compilation error when joining unaliased models dbt-adapters#693
   pkg:dbt-snowflake type:bug +2
6. [Bug] Model reference and AS with empty run fails dbt-adapters#660
   pkg:dbt-postgres type:bug +2

triage

Preview Give feedback

1. allow for opting out specific models from sample mode #11264
   +0
2. Allow for specifying sample time window in project configs #11232
   enhancement sample_mode +2
   
3. allow for specifying sample time window at model level #11297
   +0
4. allow sampling for representative segments

[chrishiste]

@QMalcolm This is awesome, excited about this!

[QMalcolm]

@QMalcolm This is awesome, excited about this!

@chrishiste I'm excited that you're excited 🙂

[QMalcolm]

Also for people who wind up here, but haven't seen the discussion post, here's the discussion post 🙂