Code

Add support for MoE models#60

Code

Merged

epwalsh merged 61 commits intomainallenai/OLMo-core:mainfrom epwalsh/moeallenai/OLMo-core:epwalsh/moeCopy head branch name to clipboardOct 21, 2024

+1,281-205

Commits

Commits on Oct 2, 2024

• add placeholder for MoE module

  epwalsh

  committed
  f5a96e0

  Copy full SHA for f5a96e0

Commits on Oct 7, 2024

• start adding MoE components

  epwalsh

  committed
  936b6e8

  Copy full SHA for 936b6e8

Commits on Oct 8, 2024

• fix megablocks commit SHA

  epwalsh

  committed
  12f40a4

  Copy full SHA for 12f40a4

• clean up

  epwalsh

  committed
  346dc62

  Copy full SHA for 346dc62

• fix

  epwalsh

  committed
  4527d34

  Copy full SHA for 4527d34

Commits on Oct 9, 2024

• updates

  epwalsh

  committed
  ee44f7d

  Copy full SHA for ee44f7d

• pin numpy

  epwalsh

  committed
  f9b08bb

  Copy full SHA for f9b08bb

• updates

  epwalsh

  committed
  3fe143c

  Copy full SHA for 3fe143c

Commits on Oct 10, 2024

• Merge branch 'main' into epwalsh/moe

  epwalsh

  committed
  4095321

  Copy full SHA for 4095321

• pin to fork of megablocks

  epwalsh

  committed
  217fa7e

  Copy full SHA for 217fa7e

• fix docker build

  epwalsh

  committed
  31f6db1

  Copy full SHA for 31f6db1

• show image size

  epwalsh

  committed
  3128526

  Copy full SHA for 3128526

• fix

  epwalsh

  committed
  8ecfdef

  Copy full SHA for 8ecfdef

• clear up some disk space

  epwalsh

  committed
  85bed9b

  Copy full SHA for 85bed9b

• fix

  epwalsh

  committed
  0366d7c

  Copy full SHA for 0366d7c

• use larger runner

  epwalsh

  committed
  47785d6

  Copy full SHA for 47785d6

• updates

  epwalsh

  committed
  43dab58

  Copy full SHA for 43dab58

• Build separate dev image

  epwalsh

  committed
  8f00ac8

  Copy full SHA for 8f00ac8

• use dev image for testing

  epwalsh

  committed
  0d3e025

  Copy full SHA for 0d3e025

• fix other image builds

  epwalsh

  committed
  a195ee5

  Copy full SHA for a195ee5

• revert nightly version

  epwalsh

  committed
  f539821

  Copy full SHA for f539821

• updates

  epwalsh

  committed
  ebc82ef

  Copy full SHA for ebc82ef

• Add MoE test

  epwalsh

  committed
  7a34742

  Copy full SHA for 7a34742

• Adjust tests

  epwalsh

  committed
  8879118

  Copy full SHA for 8879118

Commits on Oct 11, 2024

• fixes

  epwalsh

  committed
  4d1b5d8

  Copy full SHA for 4d1b5d8

• fix

  epwalsh

  committed
  ba95e86

  Copy full SHA for ba95e86

• increase dims

  epwalsh

  committed
  5fe437a

  Copy full SHA for 5fe437a

• comment

  epwalsh

  committed
  e7e75cb

  Copy full SHA for e7e75cb

• Merge branch 'main' into epwalsh/moe

  Show description for d5bc650
  epwalsh

  committed
  d5bc650

  Copy full SHA for d5bc650

• add 9.0 arch list

  epwalsh

  committed
  c3176a4

  Copy full SHA for c3176a4

• fix

  epwalsh

  committed
  5a0432b

  Copy full SHA for 5a0432b

• Use wrapper around megablocks classes

  epwalsh

  committed
  aefc85f

  Copy full SHA for aefc85f

• Fixes, add callback to collect loss

  epwalsh

  committed
  f3b485d

  Copy full SHA for f3b485d

• fix test

  epwalsh

  committed
  97a6ab5

  Copy full SHA for 97a6ab5

• add training script for OLMoE-1B-7B

  epwalsh

  committed
  6e77e4b

  Copy full SHA for 6e77e4b

• fix init

  epwalsh

  committed
  738eed6

  Copy full SHA for 738eed6

• fix

  epwalsh

  committed
  6105be1

  Copy full SHA for 6105be1

• another fix

  epwalsh

  committed
  881f6e1

  Copy full SHA for 881f6e1

• decrease mbz

  epwalsh

  committed
  a5840de

  Copy full SHA for a5840de

• fix?

  epwalsh

  committed
  baa0f89

  Copy full SHA for baa0f89

• fix

  epwalsh

  committed
  2ffa147

  Copy full SHA for 2ffa147

Commits on Oct 12, 2024

• Fix Z-loss calculation

  epwalsh

  committed
  c8bd786

  Copy full SHA for c8bd786

• Load MoE losses separately

  epwalsh

  committed
  8975115

  Copy full SHA for 8975115

• clean up API

  epwalsh

  committed
  8d65d68

  Copy full SHA for 8d65d68

• fix

  epwalsh

  committed
  b8c84c4

  Copy full SHA for b8c84c4

• test different MoE MLP implementations

  epwalsh

  committed
  912dc06

  Copy full SHA for 912dc06

• Another fix to loss buffer reset

  epwalsh

  committed
  12f13a2

  Copy full SHA for 12f13a2

• Set test input dtype to BF16

  epwalsh

  committed
  68f2c79

  Copy full SHA for 68f2c79

• Set dtype correctly

  epwalsh

  committed
  81541c6

  Copy full SHA for 81541c6

• improve how test dtype is logged

  epwalsh

  committed
  f5706a9

  Copy full SHA for f5706a9

• specify when docker builds should run

  epwalsh

  committed
  046560f

  Copy full SHA for 046560f

Commits on Oct 14, 2024

• try compile again

  epwalsh

  committed
  2394ba7

  Copy full SHA for 2394ba7

• improve image builds

  epwalsh

  committed
  f58430f

  Copy full SHA for f58430f

• Add memory-optimized variant

  epwalsh

  committed
  edcadb3

  Copy full SHA for edcadb3

• Add fine-grained wrapping strategy

  epwalsh

  committed
  a1ba50e

  Copy full SHA for a1ba50e

• fix

  epwalsh

  committed
  a189795

  Copy full SHA for a189795

• fix example

  epwalsh

  committed
  b949489

  Copy full SHA for b949489

• fix

  epwalsh

  committed
  ac09e91

  Copy full SHA for ac09e91

Commits on Oct 17, 2024

• Merge branch 'main' into epwalsh/moe

  epwalsh

  committed
  1af7c8e

  Copy full SHA for 1af7c8e

Commits on Oct 18, 2024

• Explicitly mark beta features

  epwalsh

  committed
  f2ed966

  Copy full SHA for f2ed966

• Update README.md

  epwalsh

  authored
  f44a110

  Copy full SHA for f44a110