Code

Microbatch first last batch serial#11072

Code

Merged

QMalcolm merged 21 commits intomaindbt-labs/dbt-core:mainfrom microbatch-first-last-batch-serialdbt-labs/dbt-core:microbatch-first-last-batch-serialCopy head branch name to clipboardDec 7, 2024

+575-219

Commits

Commits on Nov 28, 2024

• microbatch: split out first and last batch to run in serial

  MichelleArk

  committed
  0d61609

  Copy full SHA for 0d61609

• use Task.get_runner

  MichelleArk

  committed
  bec5d57

  Copy full SHA for bec5d57

• only run pre_hook on first batch, post_hook on last batch

  MichelleArk

  committed
  32002ea

  Copy full SHA for 32002ea

• refactor: internalize parallel to RunTask._submit_batch

  MichelleArk

  committed
  37dbd11

  Copy full SHA for 37dbd11

Commits on Dec 4, 2024

• Add optional force_sequential to _submit_batch to allow for skipping parallelism check

  QMalcolm

  committed
  23271b9

  Copy full SHA for 23271b9

• Force last batch to run sequentially

  QMalcolm

  committed
  9c903f7

  Copy full SHA for 9c903f7

• Force first batch to run sequentially

  QMalcolm

  committed
  e624057

  Copy full SHA for e624057

• Fixup

  QMalcolm

  committed
  0c2e327

  Copy full SHA for 0c2e327

• Remove batch_idx check in should_run_in_parallel

  Show description for a37f3a6
  QMalcolm

  committed
  a37f3a6

  Copy full SHA for a37f3a6

Commits on Dec 6, 2024

• Begin skipping batches if first batch fails

  QMalcolm

  committed
  cfe1dcf

  Copy full SHA for cfe1dcf

• Write custom on_skip for MicrobatchModelRunner to better handle when batches are skipped

  Show description for 8424209
  QMalcolm

  committed
  8424209

  Copy full SHA for 8424209

• Add microbatch pre-hook, post-hook, and sequential first/last batch tests

  QMalcolm

  committed
  50abb45

  Copy full SHA for 50abb45

• Fix/Add tests around first batch failure vs latter batch failure

  QMalcolm

  committed
  9bc3816

  Copy full SHA for 9bc3816

• Fix MicrobatchModelRunner.on_skip to handle skipping the entire node

  Show description for 7699f52
  QMalcolm

  committed
  7699f52

  Copy full SHA for 7699f52

• Fix conditional logic for setting pre and post hooks for batches

  Show description for 74a76c4
  QMalcolm

  committed
  74a76c4

  Copy full SHA for 74a76c4

• Revert back to using the MicrobatchModelRunner initializer directly

  Show description for 0c16d07
  QMalcolm

  committed
  0c16d07

  Copy full SHA for 0c16d07

Commits on Dec 7, 2024

• Add two new event types LogStartBatch and LogBatchResult

  QMalcolm

  committed
  3fa7bbc

  Copy full SHA for 3fa7bbc

• Update MicrobatchModelRunner to use new batch specific log events

  QMalcolm

  committed
  f4315ef

  Copy full SHA for f4315ef

• Fix event testing

  QMalcolm

  committed
  bec65b8

  Copy full SHA for bec65b8

• Update microbatch integrationt tests to catch batch specific event types

  QMalcolm

  committed
  c0743e0

  Copy full SHA for c0743e0

• Add changie doc

  QMalcolm

  committed
  0198741

  Copy full SHA for 0198741