Code

Less contention in tokenization#893

Code

Merged

dlwh merged 2 commits intomainstanford-crfm/levanter:mainfrom less_contention_tokenizationstanford-crfm/levanter:less_contention_tokenizationCopy head branch name to clipboardFeb 26, 2025

+106-39

Commits

Commits on Feb 25, 2025

• factor out a metadata copier class to reduce contention

  dlwh

  committed
  02ae385

  Copy full SHA for 02ae385

Commits on Feb 26, 2025

• less spam

  dlwh

  committed
  d8928d7

  Copy full SHA for d8928d7