Code

muP implementation#637

Code

Open

AkshitaB wants to merge 151 commits intomainallenai/OLMo:mainfrom akshitab-scaleallenai/OLMo:akshitab-scaleCopy head branch name to clipboard

+7,682-64

Commits

Commits on Feb 21, 2024

• mup olmo

  AkshitaB

  committed
  52f3d53

Commits on Mar 20, 2024

• wip

  AkshitaB

  committed
  4d715ad

• Merge branch 'main' into akshitab-scale

  AkshitaB

  committed
  43d674d

Commits on Mar 21, 2024

• MuOLMo, coord check (wip)

  AkshitaB

  committed
  675290f

Commits on Apr 10, 2024

• coord check runs

  AkshitaB

  committed
  e4a80bc

• remove extra args, add readme

  AkshitaB

  committed
  2354425

• requirements

  AkshitaB

  committed
  07df33e

• Update README.md

  AkshitaB

  authored
  2dd327b

• Update README.md

  AkshitaB

  authored
  74f56a9

• coord checks

  AkshitaB

  committed
  325cad1

• legend for coord checks

  AkshitaB

  committed
  a31b698

Commits on Apr 11, 2024

• updates to mup implementation

  AkshitaB

  committed
  d1533b1

• coord checks

  AkshitaB

  committed
  c4dcbda

• Merge branch 'akshitab-scale' of https://github.com/allenai/OLMo into akshitab-scale

  AkshitaB

  committed
  ff3778a

• coord checks

  AkshitaB

  committed
  0f39e6e

• smaller model

  AkshitaB

  committed
  c180a8e

• fixes for network output

  AkshitaB

  committed
  d498385

• smaller bs, larger model

  AkshitaB

  committed
  e566b72

• lr=0.005, bs=2

  AkshitaB

  committed
  12016bd

• confirm normal init

  AkshitaB

  committed
  a5807a1

• scale by d

  AkshitaB

  committed
  4285275

• scale by d

  AkshitaB

  committed
  85b23fd

• remove unnecessary code

  AkshitaB

  committed
  fe5c71d

• extra cleanup

  AkshitaB

  committed
  c875dcc

• more width

  AkshitaB

  committed
  27a4eef

• Update README.md

  AkshitaB

  authored
  1a35e1c

Commits on May 7, 2024

• Update README.md

  AkshitaB

  authored
  9aa8e66

• don't use readout

  AkshitaB

  committed
  ccbcd08

• init scale

  AkshitaB

  committed
  0ea01e3

• rope

  AkshitaB

  committed
  29eaf76

• readme

  AkshitaB

  committed
  d4ab24f

Commits on May 8, 2024

• remove plots

  AkshitaB

  committed
  520fd72

• formatting

  AkshitaB

  committed
  ea58502

• get_batch_loss func

  AkshitaB

  committed
  d3abad1

• wip: train, eval funcs

  AkshitaB

  committed
  d61de42

• gitignore

  AkshitaB

  committed
  acd5032

Commits on May 9, 2024

• bug fix

  AkshitaB

  committed
  505f46e

• coords file

  AkshitaB

  committed
  5a73a1a

Commits on May 10, 2024

• easy file name

  AkshitaB

  committed
  cd326aa

• script for examining coords

  AkshitaB

  committed
  efcd37b

Commits on Jun 23, 2024

• Merge branch 'main' into akshitab-scale

  AkshitaB

  committed
  f7f7f34

Commits on Jun 24, 2024

• add mup to main model

  AkshitaB

  committed
  a9ea54b

• tests

  AkshitaB

  committed
  06101ad

• easier testing

  AkshitaB

  committed
  6b65600

• simplified coord check

  AkshitaB

  committed
  d088220

• 30 iter coord checks

  AkshitaB

  committed
  c2595ea

• rename

  AkshitaB

  committed
  b055d41

• refactor, add tests

  AkshitaB

  committed
  424a361

• fix save_base_shapes

  AkshitaB

  committed
  effe212

• Revert "fix save_base_shapes"

  Show description for 67158cb
  AkshitaB

  committed
  67158cb

• Revert "refactor, add tests"

  Show description for 5c02f3f
  AkshitaB

  committed
  5c02f3f

• rename output files

  AkshitaB

  committed
  f6722b5

• remove extra code

  AkshitaB

  committed
  f0775f8

• function for save_base_shapes

  AkshitaB

  committed
  febca85

• make plot optional, for testing purposes

  AkshitaB

  committed
  41d293d

• make widths configurable

  AkshitaB

  committed
  d060fb5

• split out the args

  AkshitaB

  committed
  e7b6f66

• rename plotdir

  AkshitaB

  committed
  14db801

• add tests

  AkshitaB

  committed
  e001879

• rename to coordinates

  AkshitaB

  committed
  eb2f3ee

• remove dict_in_out

  AkshitaB

  committed
  6bbca2d

Commits on Jun 25, 2024

• remove unused args

  AkshitaB

  committed
  5213a21

• always compute loss

  AkshitaB

  committed
  258d8d3

• add bigger config

  AkshitaB

  committed
  c44c8f1

• updated outputs

  AkshitaB

  committed
  282fbe6

• reorganize

  AkshitaB

  committed
  81b331f

• update paths in readme

  AkshitaB

  committed
  954f761

• add scaffolding for scaling laws

  AkshitaB

  committed
  48fa137

Commits on Jun 28, 2024

• Merge branch 'main' into akshitab-scale

  AkshitaB

  committed
  1d5eaf2

• fix lint, etc

  AkshitaB

  committed
  ea900ee

• correct mup version

  AkshitaB

  committed
  1302614

Commits on Jun 29, 2024

• minor updates

  AkshitaB

  committed
  7fb6839

• fix pyproject for testing

  AkshitaB

  committed
  96a61b1

Commits on Jun 30, 2024

• update changelog

  AkshitaB

  committed
  59e9e67

• ensure config runs

  AkshitaB

  committed
  e3b7253

Commits on Jul 1, 2024

• orig_params with fsdp

  AkshitaB

  committed
  26d1abc

• Merge branch 'akshitab-scale' of https://github.com/allenai/OLMo into akshitab-scale

  AkshitaB

  committed
  dc53949

• remove unnecessary code

  AkshitaB

  committed
  288e4a2

• Merge branch 'akshitab-scale' of https://github.com/allenai/OLMo into akshitab-scale

  AkshitaB

  committed
  e8ea0fa

• updated coord checks

  AkshitaB

  committed
  f70ffa4

• simplified base config

  AkshitaB

  committed
  b0dfe0a

• use mup

  AkshitaB

  committed
  561b813

• scripts for running check

  AkshitaB

  committed
  16c935e

• rename runs

  AkshitaB

  committed
  e9193cb

• fixes

  AkshitaB

  committed
  823ec37

• no need with 1 node

  AkshitaB

  committed
  1b83d56

• no weka

  AkshitaB

  committed
  fb76dd0

• install on the fly

  AkshitaB

  committed
  137deaf

• bug fix

  AkshitaB

  committed
  35c9a88

• debug

  AkshitaB

  committed
  9ff73f6

• bug fix again

  AkshitaB

  committed
  fe9fb4d

• full runs

  AkshitaB

  committed
  078c459

• no warmup

  AkshitaB

  committed
  c7f6820

• linear warmup, same base shape

  AkshitaB

  committed
  016c3c5

• use correct optimizer

  AkshitaB

  committed
  50073c0

• fix mup optimizer

  AkshitaB

  committed
  3ef14dc

• fix

  AkshitaB

  committed
  9299f82

• lower priority

  AkshitaB

  committed
  1f15eeb

• base shapes

  AkshitaB

  committed
  96aed44

• fix

  AkshitaB

  committed
  15a2390

• save progress

  AkshitaB

  committed
  ff8520b

• run on jupiter

  AkshitaB

  committed
  019daf7

• more cluster options

  AkshitaB

  committed
  59ef245

• priority

  AkshitaB

  committed
  77d6d6d

• Revert "priority"

  Show description for 7cbf520
  AkshitaB

  committed
  7cbf520

• ensure that correct lr is used

  AkshitaB

  committed
  9b02f7d

• simplify config further

  AkshitaB

  committed
  23a6986

• update base shapes

  AkshitaB

  committed
  f3218c1

• stability at init

  AkshitaB

  committed
  e27d806

Commits on Jul 2, 2024

• run sp baseline

  AkshitaB

  committed
  09925d1

• fix readout init

  AkshitaB

  committed
  a4a5b6e

• Merge branch 'akshitab-scale' of https://github.com/allenai/OLMo into akshitab-scale

  AkshitaB

  committed
  5dc7210

• updated paths

  AkshitaB

  committed
  d4b395b

• sp scripts

  AkshitaB

  committed
  1adea44

• run for remaining

  AkshitaB

  committed
  5cf94df

Commits on Jul 9, 2024

• base config for experiments

  AkshitaB

  committed
  10d93cf

Commits on Jul 18, 2024

• Merge branch 'main' into akshitab-scale

  AkshitaB

  committed
  9251161

• Merge branch 'main' into akshitab-scale

  AkshitaB

  committed
  4fe7b3f

• black

  AkshitaB

  committed
  a750ca8

• 7B width should be multiple of 64

  AkshitaB

  committed
  61ffe77

• move general functions to utils

  AkshitaB

  committed
  91d32fb

• mup ladder

  AkshitaB

  committed
  23bbed8

• new scales

  AkshitaB

  committed
  73a6346

• remove cluster

  AkshitaB

  committed
  0b1994d

• remove cluster

  AkshitaB

  committed
  10dc91d

• install mup

  AkshitaB

  committed
  51810df

• fix device

  AkshitaB

  committed
  60b4bf5

• flash attn

  AkshitaB

  committed
  a3b1e81

• debug

  AkshitaB

  committed
  7e7d596

• more debug

  AkshitaB

  committed
  42c1d14

Commits on Jul 19, 2024

• test with more layers

  AkshitaB

  committed
  702afc8

• less layers

  AkshitaB

  committed
  b600ba9

• update shape

  AkshitaB

  committed
  9f539b4

• fsdp

  AkshitaB

  committed
  c56aff1

• device batch size

  AkshitaB

  committed
  a19f5f6

• hp sweep

  AkshitaB

  committed
  a5c6747

• make it divisible

  AkshitaB

  committed
  014606b

• run with my key

  AkshitaB

  committed
  9a3c2df

• don't rescale when loading from checkpoint

  AkshitaB

  committed
  4d15c81

• more checkpoints for debugging

  AkshitaB

  committed
  908463c

• checkpoint less often

  AkshitaB

  committed
  ba2c6bf

Commits on Jul 22, 2024

• run with standard parametrization

  AkshitaB

  committed
  de2cf97

• no weka, expand cluster options

  AkshitaB

  committed
  d80702e

• increase priority

  AkshitaB

  committed
  b5d040b

• revert

  AkshitaB

  committed
  4f2dc47

• increase timeout

  AkshitaB

  committed
  dacaa95

• make mup optional

  AkshitaB

  committed
  32bf956

• fix

  AkshitaB

  committed
  447ab47

• bug fix

  AkshitaB

  committed
  b3d871c

• test with cosine

  AkshitaB

  committed
  8cc17c9

• fix

  AkshitaB

  committed
  57ef19f