Code

OLMo 70B official training run#507

Code

Merged

dirkgr merged 199 commits intomainallenai/OLMo:mainfrom train-olmo-largeallenai/OLMo:train-olmo-largeCopy head branch name to clipboardMay 6, 2024

+5,090-137

Commits

Commits on Mar 19, 2024

• finalize config

  epwalsh

  committed
  227ce5f

• no openbook_qa I guess

  epwalsh

  committed
  255e7ef

• comment out more datasets

  epwalsh

  committed
  3962a10

• Fix openbook_qa

  epwalsh

  committed
  cb8e4d8

• Add openbook_qa back

  epwalsh

  committed
  e86879e

• new datasets cache

  epwalsh

  committed
  4660b68

• Fix RPJ arxiv paths

  epwalsh

  committed
  904bdbe

• fix BasicArithmetic task

  epwalsh

  committed
  2e5b4f3

• No basic_arithmetic for now

  epwalsh

  committed
  f281be4

• add some debugging info

  epwalsh

  committed
  33f8f99

• update debugging info

  epwalsh

  committed
  edfda94

• clean up

  epwalsh

  committed
  fea700a

• update debug info

  epwalsh

  committed
  7651045

• fix

  epwalsh

  committed
  00d7726

• more debugging

  epwalsh

  committed
  637a0db

• update

  epwalsh

  committed
  a9dcfd1

• fix paths

  epwalsh

  committed
  6cc1428

• put things on CPU!

  epwalsh

  committed
  d7974a3

• fix

  epwalsh

  committed
  654981b

• clean up

  epwalsh

  committed
  4329470

• Merge remote-tracking branch 'origin/epwalsh/flatten-osd' into train-olmo-large

  epwalsh

  committed
  a867769

• update

  epwalsh

  committed
  344430b

• Increase duration to 3T tokens

  epwalsh

  committed
  2e939cb

• Adds a config to train the 7B in the same way

  dirkgr

  committed
  1236894

• scheduler too

  epwalsh

  committed
  788d0b1

• Run in my own workspace

  dirkgr

  committed
  fc9e21d

• Use full name of the image

  dirkgr

  committed
  37281e1

Commits on Mar 20, 2024

• Disable ipv6

  dirkgr

  committed
  73df78e

• Microbatch of 2

  dirkgr

  committed
  6bbf456

• Save overwrite

  dirkgr

  committed
  2343828

• That did not work.

  dirkgr

  committed
  7f1d2bb

• Two-stage launch scripts

  dirkgr

  committed
  bae3ac7

• Executable permissions

  dirkgr

  committed
  a330546

• set speed monitor window size to 1

  epwalsh

  committed
  84e51a3

• Try to save some more memory

  dirkgr

  committed
  452d1ac

• export

  dirkgr

  committed
  5b8b750

• Another way of passing NUM_NODES

  dirkgr

  committed
  04789de

• We have to pass in more stuff.

  dirkgr

  committed
  6b12512

• Like this?

  dirkgr

  committed
  b5b3901

• Bringing back bin bash

  dirkgr

  committed
  4d2c614

• Merge branch 'main' into train-olmo-large

  epwalsh

  committed
  a40e7fc

• This is not the most elegant launcher.

  dirkgr

  committed
  a1cf51e

• No activation checkpointing

  dirkgr

  committed
  4f838f8

• Revert "No activation checkpointing"

  Show description for 5e2a51a
  dirkgr

  committed
  5e2a51a

• Merge pull request #511 from allenai/train-olmo-medium

  Show description for cebf110
  dirkgr

  authored
  cebf110

Commits on Mar 21, 2024

• increase job time limit

  epwalsh

  committed
  66ae6ca

Commits on Mar 22, 2024

• No time limit by default

  dirkgr

  committed
  3eaefa8

• Load from checkpoint

  dirkgr

  committed
  de5f5ca

• Make tar more robust

  dirkgr

  committed
  6190cbc

• Yes I know how tar works. Why are you asking?

  dirkgr

  committed
  bb37a7b

Commits on Mar 23, 2024

• Automatically find the latest checkpoint

  dirkgr

  committed
  931bc67

• I that did wrong.

  dirkgr

  committed
  3cca4bd

Commits on Mar 24, 2024

• increase time limit to 7 days

  epwalsh

  committed
  916bdbb

Commits on Mar 25, 2024

• double global batch size

  epwalsh

  committed
  ec2ec79

Commits on Mar 26, 2024

• Load test on beaker

  dirkgr

  committed
  2cb8b36

• Needs more secrets

  dirkgr

  committed
  22cca0c

• Config for running the 7B on MosaicML

  dirkgr

  committed
  bfedb82

• This is running on an older version of torch.

  dirkgr

  committed
  37b5f20

Commits on Mar 27, 2024

• More nodes

  dirkgr

  committed
  b2a9c74

• Start from unsharded checkpoint

  dirkgr

  committed
  3235dbf

• 128 GPUs terminates for some reason

  dirkgr

  committed
  ecc0fd3

• Try 128 nodes again

  dirkgr

  committed
  2b6fe88

• New load path

  dirkgr

  committed
  806d6ef

• add my unshard script

  epwalsh

  committed
  b8d2297

• Have to reshard again

  dirkgr

  committed
  60b9280

• 32 GPUs :-(

  dirkgr

  committed
  8362893

• Fix the mitchich70 launch scripts for Beaker

  dirkgr

  committed
  0236ceb

• Merge pull request #529 from allenai/70b-load-test

  Show description for 0f96e2f
  dirkgr

  authored
  0f96e2f

• update config

  epwalsh

  committed
  ea0532b

• fix merge conflicts

  epwalsh

  committed
  7edf135

• New HF cache

  dirkgr

  committed
  f67d9e7

Commits on Mar 28, 2024

• add more MMLU variants to config

  epwalsh

  committed
  c0a638a

• All the GPUs, new checkpoint

  dirkgr

  committed
  d41527d

• Merge branch 'train-olmo-large' of https://github.com/allenai/LLM into train-olmo-large

  dirkgr

  committed
  f50c375

• Add more MMLU evals

  dirkgr

  committed
  47ab679

• Unclear why it is reading the "default" one? Where does it even get that?

  dirkgr

  committed
  85d77d2

• no piqa I guess

  epwalsh

  committed
  6aabcd1

• Just keep on trucking

  dirkgr

  committed
  f25833d

• Merge branch 'train-olmo-large' of https://github.com/allenai/LLM into train-olmo-large

  dirkgr

  committed
  63830d9

• piqa is back!

  epwalsh

  committed
  2ba18ba

• Fewer GPUs

  dirkgr

  committed
  a74e7b0

Commits on Mar 29, 2024

• update load path

  epwalsh

  committed
  298b3d6

Commits on Apr 2, 2024

• clean up

  epwalsh

  committed
  38f03f6

• Merge branch 'main' into train-olmo-large

  epwalsh

  committed
  e784af3

• upload local files from each rank

  epwalsh

  committed
  a807f90

• fall back to train rank 0

  epwalsh

  committed
  4dea838

• New cluster who dis?

  dirkgr

  committed
  d6ff1d4

• Merge branch 'train-olmo-large' of https://github.com/allenai/LLM into train-olmo-large

  dirkgr

  committed
  da47718

• Load olmo_core style checkpoint and see if we can chase down the slow batch problem

  dirkgr

  committed
  d289272

• Back to a reasonable number of checkpoints

  dirkgr

  committed
  64af0cf

• update for new cluster + checkpointer

  epwalsh

  committed
  8791ca1

• Do GC as normal

  dirkgr

  committed
  40b5abb

• Much more prefetch

  dirkgr

  committed
  6acb011

• Merge branch 'train-olmo-large' of https://github.com/allenai/LLM into train-olmo-large

  dirkgr

  committed
  80843c4

• Activation checkpointing is back

  dirkgr

  committed
  0ef892b

Commits on Apr 3, 2024

• Enable compile

  dirkgr

  committed
  d49a91d

Commits on Apr 4, 2024

• Update version of torch

  dirkgr

  committed
  3144e93

• Revert "Enable compile"

  Show description for 437dd18
  dirkgr

  committed
  437dd18

• config updates

  epwalsh

  committed
  d996cee

• Merge branch 'train-olmo-large' of https://github.com/allenai/LLM into train-olmo-large

  dirkgr

  committed
  4fbabe6

• add unshard method for OlmoCoreCheckpointer

  epwalsh

  committed
  8d035b1

Commits on Apr 6, 2024

• Switch to a different Docker hub

  dirkgr

  committed
  2a87785

• Revert "Switch to a different Docker hub"

  Show description for 3df3dca
  dirkgr

  committed
  3df3dca

• Try on fewer GPUs so we're not totally idle.

  dirkgr

  committed
  3c77c35

• Back to full node count

  dirkgr

  committed
  374b2b2

• Mitchish 1B

  dirkgr

  committed
  7d48d9e

• Go faster

  dirkgr

  committed
  e848ac1

• Work harder, not smarter

  dirkgr

  committed
  410b106

• Less GC

  dirkgr

  committed
  1e3b551

• Mcli script to get hostnames

  dirkgr

  committed
  3d7d803

• Whitelist for nodes

  dirkgr

  committed
  591b14f

Commits on Apr 7, 2024

• New epoch

  dirkgr

  committed
  0608033

• Less GC

  dirkgr

  committed
  7794004

• Looks like we have a bug when running more than one but less than two epochs

  dirkgr

  committed
  24df0a1

Commits on Apr 8, 2024

• relax assertion

  epwalsh

  committed
  79b389f

• Unshard S3 checkpoint and convert to Hf.

  dwadden

  committed
  b664ac4

• omit bad nodes

  epwalsh

  committed
  bce98bf

• 1B goes to 2 epochs

  dirkgr

  committed
  ff78e95

• Update dockerfile for PyTorch 2.2

  dirkgr

  committed
  bb07d50

• Merge branch 'train-olmo-large' of https://github.com/allenai/LLM into train-olmo-large

  dirkgr

  committed
  5cc5484

• Add unsharding note.

  dwadden

  committed
  f800c55

• Get rid of safetensors option.

  dwadden

  committed
  bb2c39e

Commits on Apr 9, 2024

• Some more Dockerfile shenanigans

  dirkgr

  committed
  a2b5c1a

• Merge branch 'train-olmo-large' of https://github.com/allenai/LLM into train-olmo-large

  dirkgr

  committed
  c87b7bf

• More radical Docker approach

  dirkgr

  committed
  b1ec637

• Merge from AkshitaB-tokenizer-patch

  

  Oyvind Tafjord

  committed
  23dc804

• More bad nodes

  dirkgr

  committed
  1921d43

• Let's run mitchish7 on the rest

  dirkgr

  committed
  f5c5a0b

• fix merge conflicts

  epwalsh

  committed
  a964b13

• yet another bad node

  epwalsh

  committed
  125e721

• configure for 912 GPUs

  epwalsh

  committed
  9cec01b

• Run on 32 nodes

  dirkgr

  committed
  cefe730

• Run the 1B as well

  dirkgr

  committed
  b60e0fd

Commits on Apr 10, 2024

• Different nodes

  dirkgr

  committed
  f179721

• Amazon's container registry is no better

  dirkgr

  committed
  08a7b11

• update

  epwalsh

  committed
  ae9c3b4

• switch to Google Cloud registry

  epwalsh

  committed
  d34d817

Commits on Apr 11, 2024

• add run manager script

  epwalsh

  committed
  39382db

• 16 nodes

  dirkgr

  committed
  03c2dd4

• Merge branch 'train-olmo-large' of https://github.com/allenai/LLM into train-olmo-large

  dirkgr

  committed
  d310965

• clean up

  epwalsh

  committed
  3c47b5a

• clean up

  epwalsh

  committed
  df6f1d9

• tweaks

  epwalsh

  committed
  1eafd6b

Commits on Apr 12, 2024

• hack to work around mcli bug

  epwalsh

  committed
  5660857

Commits on Apr 15, 2024

• More GPUs for the 7B

  dirkgr

  committed
  7abbd32

Commits on Apr 18, 2024

• update config

  epwalsh

  committed
  5f13793

• clip to 0.5

  epwalsh

  committed
  ae77217

• back to 912 GPUs

  epwalsh

  committed
  a21e59d

Commits on Apr 19, 2024

• Merge branch 'main' into train-olmo-large

  epwalsh

  committed
  046d952

• hybrid shard

  epwalsh

  committed
  c3dbb31

Commits on Apr 25, 2024

• Revert "clip to 0.5"

  Show description for 169e2af
  dirkgr

  committed
  169e2af

• Decay embeddings

  dirkgr

  committed
  948febd

• Bigger batch size

  dirkgr

  committed
  64d6cc8

• Fix run name, so we don't overwrite our existing checkpoints

  dirkgr

  committed
  cc5eaca

• Load from 1T

  dirkgr

  committed
  de1e99d

Commits on Apr 26, 2024

• Batch size that works

  dirkgr

  committed
  b022c23

• Can't make the top secret image work

  dirkgr

  committed
  263cf09

• Better names

  dirkgr

  committed
  87a2c80

• Load hacked checkpoint (with the extra parameter group for WD)

  dirkgr

  committed
  404d9e8

• Load optimizer state in turns

  dirkgr

  committed
  9f77341

• Don't report progress

  dirkgr

  committed
  8c84ba8

• Run aws s3 cp with --no-progress

  2015aroras

  committed
  59a2ca8

• Merge branch 'train-olmo-large' of https://github.com/allenai/LLM into train-olmo-large

  dirkgr

  committed
  71678c4

• Ron on many nodes again

  dirkgr

  committed
  b549388

Commits on Apr 29, 2024

• 7B model with Llama initialization

  dirkgr

  committed
  b8d0c44

Commits on May 1, 2024

• Try the old rdzv

  dirkgr

  committed
  c0b4634

• New beaker settings

  dirkgr

  committed
  ba71aa8

• This doesn't work yet in gantry.

  dirkgr

  committed
  13d40f6

• Better logging during startup

  dirkgr

  committed
  dc7bedd

• Init barrier

  dirkgr

  committed
  f476b14

• Try running on Pluto instead

  dirkgr

  committed
  7ef41aa

• Revert "Try the old rdzv"

  Show description for f024c75
  dirkgr

  committed
  f024c75

• prepare_cli_environment() relies on the process group

  dirkgr

  committed
  22b4e29

Commits on May 2, 2024

• Load from checkpoint

  dirkgr

  committed
  c034cd4

• Synchronized start

  dirkgr

  committed
  3bc591f

• Maybe I can specify this in seconds?

  dirkgr

  committed
  9d7a395

• Nope, still broken

  dirkgr

  committed
  e0863ac

• Put back the synchronized start

  dirkgr

  committed
  5b0196b

Commits on May 3, 2024

• mask instances with too many repetitions

  epwalsh

  committed
  41477cb

• make configurable

  epwalsh

  committed
  ef3162d

• log number of instances masked

  epwalsh

  committed
  9938909

• update hparams

  epwalsh

  committed
  5371205

• add

  epwalsh

  committed
  ab6f365

Commits on May 4, 2024

• Make mitchish7 configs similar

  dirkgr

  committed
  dc0fc26

• Better Beaker settings

  dirkgr

  committed
  ee0bf19

• Wrong image

  dirkgr

  committed
  cd9e691

• Do rendezvous the way Shane figured out

  dirkgr

  committed
  c88d94e

• We have to export this for omegaconf

  dirkgr

  committed
  50efbd6

• Has to be a dataclass

  dirkgr

  committed
  da15cfe

• Works even if the whole microbatch is masked

  dirkgr

  committed
  7146473

• We want to see grad norms

  dirkgr

  committed
  6b2a8f3

• Bigger batch, reset

  dirkgr

  committed
  c8d4cf9

• Plan C

  dirkgr

  committed
  4e2d6a8

• Load from an earlier checkpoint, so we can see some differences

  dirkgr

  committed
  ddebc72

• Let's call this run something else, since it's a fork

  dirkgr

  committed
  43f9b1d

• Fix branch name

  dirkgr

  committed
  58ffdab

Commits on May 5, 2024

• Load checkpoints from the right folder

  dirkgr

  committed
  fe68d1b

• Save and eval more often

  dirkgr

  committed
  1f32fb7

Commits on May 6, 2024

• Merge pull request #568 from allenai/epwalsh/train-olmo-large

  Show description for 72a60a2
  dirkgr

  authored
  72a60a2