Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

OLMo 70B official training run #507

Merged
merged 199 commits into from
May 6, 2024
Merged

OLMo 70B official training run #507

merged 199 commits into from
May 6, 2024

Conversation

epwalsh
Copy link
Member

@epwalsh epwalsh commented Mar 19, 2024 •
edited
Loading

Quick links

Babysitting instructions

Restarting the run

Update the MCLI run config fields compute.gpus and compute.node_names as well as the OLMo trainer options --global_train_batch_size and --device_train_microbatch_size as needed.

Then just run:

mcli run -f configs/mcli/mitchish70.yaml --follow

epwalsh and others added 30 commits March 19, 2024 08:55
@epwalsh
finalize config
227ce5f
@epwalsh
no openbook_qa I guess
255e7ef
@epwalsh
comment out more datasets
3962a10
@epwalsh
Fix openbook_qa
cb8e4d8
@epwalsh
Add openbook_qa back
e86879e
@epwalsh
new datasets cache
4660b68
@epwalsh
Fix RPJ arxiv paths
904bdbe
@epwalsh
fix BasicArithmetic task
2e5b4f3
@epwalsh
No basic_arithmetic for now
f281be4
@epwalsh
add some debugging info
33f8f99
@epwalsh
update debugging info
edfda94
@epwalsh
clean up
fea700a
@epwalsh
update debug info
7651045
@epwalsh
fix
00d7726
@epwalsh
more debugging
637a0db
@epwalsh
update
a9dcfd1
@epwalsh
fix paths
6cc1428
@epwalsh
put things on CPU!
d7974a3
@epwalsh
fix
654981b
@epwalsh
clean up
4329470
@epwalsh
Merge remote-tracking branch 'origin/epwalsh/flatten-osd' into train-…
a867769 
…olmo-large
@epwalsh
update
344430b
@epwalsh
Increase duration to 3T tokens
2e939cb
@dirkgr
Adds a config to train the 7B in the same way
1236894
@epwalsh
scheduler too
788d0b1
@dirkgr
Run in my own workspace
fc9e21d
@dirkgr
Use full name of the image
37281e1
@dirkgr
Disable ipv6
73df78e
@dirkgr
Microbatch of 2
6bbf456
@dirkgr
Save overwrite
2343828
dirkgr and others added 27 commits May 1, 2024 14:30
@dirkgr
prepare_cli_environment() relies on the process group
22b4e29
@dirkgr
Load from checkpoint
c034cd4
@dirkgr
Synchronized start
3bc591f
@dirkgr
Maybe I can specify this in seconds?
9d7a395
@dirkgr
Nope, still broken
e0863ac
@dirkgr
Put back the synchronized start
5b0196b
@epwalsh
mask instances with too many repetitions
41477cb
@epwalsh
make configurable
ef3162d
@epwalsh
log number of instances masked
9938909
@epwalsh
update hparams
5371205
@epwalsh
add
ab6f365
@dirkgr
Make mitchish7 configs similar
dc0fc26
@dirkgr
Better Beaker settings
ee0bf19
@dirkgr
Wrong image
cd9e691
@dirkgr
Do rendezvous the way Shane figured out
c88d94e
@dirkgr
We have to export this for omegaconf
50efbd6
@dirkgr
Has to be a dataclass
da15cfe
@dirkgr
Works even if the whole microbatch is masked
7146473
@dirkgr
We want to see grad norms
6b2a8f3
@dirkgr
Bigger batch, reset
c8d4cf9
@dirkgr
Plan C
4e2d6a8
@dirkgr
Load from an earlier checkpoint, so we can see some differences
ddebc72
@dirkgr
Let's call this run something else, since it's a fork
43f9b1d
@dirkgr
Fix branch name
58ffdab
@dirkgr
Load checkpoints from the right folder
fe68d1b
@dirkgr
Save and eval more often
1f32fb7
@dirkgr
Merge pull request #568 from allenai/epwalsh/train-olmo-large
72a60a2 
mask instances with too many repetitions
@dirkgr dirkgr merged commit 467adcc into main May 6, 2024
8 of 11 checks passed
@dirkgr dirkgr deleted the train-olmo-large branch May 6, 2024 18:40
@epwalsh epwalsh restored the train-olmo-large branch May 6, 2024 19:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@epwalsh @dirkgr @dwadden @2015aroras