Code

merge: microsoft-main into main#62

Code

Merged

saforem2 merged 112 commits intomainargonne-lcf/Megatron-DeepSpeed:mainfrom microsoft-mainargonne-lcf/Megatron-DeepSpeed:microsoft-mainCopy head branch name to clipboardNov 12, 2024

+7,339-2,537

Commits

Commits on Feb 1, 2024

• Track additional metrics with W&B in megatron/training.py

  saforem2

  committed
  b87a03c

  Copy full SHA for b87a03c

Commits on Feb 21, 2024

• Merge branch 'microsoft:main' into main

  saforem2

  authored
  51ce9f7

  Copy full SHA for 51ce9f7

• Update megatron/training.py

  saforem2

  committed
  1302c61

  Copy full SHA for 1302c61

Commits on Feb 26, 2024

• Merge branch 'microsoft:main' into main

  saforem2

  authored
  3472f34

  Copy full SHA for 3472f34

• Remove assert num_datasets < 255 in megatron/data/blendable_dataset.py

  saforem2

  committed
  de60d86

  Copy full SHA for de60d86

• [format] megatron/data/indexed_dataset.py

  saforem2

  committed
  c11589d

  Copy full SHA for c11589d

• Add debug logic (+ formatting fixes) in megatron/data/gpt_dataset.py

  saforem2

  committed
  df5c92b

  Copy full SHA for df5c92b

• Update .gitignore

  saforem2

  committed
  1906d31

  Copy full SHA for 1906d31

• Merge branch 'microsoft:main' into main

  saforem2

  authored
  8c8c260

  Copy full SHA for 8c8c260

• Update .gitignore

  saforem2

  committed
  ee7ce6f

  Copy full SHA for ee7ce6f

Commits on Jun 26, 2024

• Add steps and results for running ZeRO stage 3 withUniversal Checkpointing (microsoft#383)

  xylian86

  authored
  634e37b

  Copy full SHA for 634e37b

Commits on Jun 27, 2024

• Add Zero Bubble Pipeline Parallelism H1 Schedule (microsoft#396)

  Show description for 527957e
  

  3 peopleauthored
  527957e

  Copy full SHA for 527957e

• Fix ParallelMLP and enable accelerator test (microsoft#403)

  xinyu-intel

  authored
  f2d7589

  Copy full SHA for f2d7589

• Fix test_deallocate_output_tensor (microsoft#404)

  xinyu-intel

  authored
  ea4b67a

  Copy full SHA for ea4b67a

Commits on Jul 1, 2024

• Fixed missing BookCorpus dataset. (microsoft#407)

  Show description for 08f5a99
  

  costin-eseanu

  and

  Costin Eseanu

  authored
  08f5a99

  Copy full SHA for 08f5a99

• Set proper arguments when constructing models in unit tests (microsoft#408)

  xinyu-intel

  authored
  c3a13be

  Copy full SHA for c3a13be

Commits on Jul 8, 2024

• use split/squeeze instead of slice for performance (microsoft#409)

  Show description for 330f9f2
  polisettyvarma

  authored
  330f9f2

  Copy full SHA for 330f9f2

• improve performance by keeping attention_mask on device and run ops further on device (microsoft#411)

  Show description for af06d14
  polisettyvarma

  authored
  af06d14

  Copy full SHA for af06d14

Commits on Jul 11, 2024

• Improve RoPE perf by using cached sin/cos tensors (microsoft#410)

  Show description for ec3f1f4
  polisettyvarma

  authored
  ec3f1f4

  Copy full SHA for ec3f1f4

Commits on Jul 12, 2024

• Extend test utilities to support more accelerators (microsoft#418)

  Show description for 354e420
  xinyu-intel

  authored
  354e420

  Copy full SHA for 354e420

• clear document (microsoft#395)

  inkcherry

  authored
  73252c0

  Copy full SHA for 73252c0

Commits on Jul 15, 2024

• add PyTorch profiler support (microsoft#414)

  Show description for 0971e68
  polisettyvarma

  authored
  0971e68

  Copy full SHA for 0971e68

Commits on Jul 16, 2024

• [Wandb] Refine wandb logging function (microsoft#416)

  Show description for 73029ed
  billishyahao

  authored
  73029ed

  Copy full SHA for 73029ed

Commits on Jul 17, 2024

• add kill switch file support to gracefully exit training at runtime (microsoft#412)

  Show description for fc989b8
  polisettyvarma

  authored
  fc989b8

  Copy full SHA for fc989b8

Commits on Jul 18, 2024

• add support to run custom Hf tokenizer for training and dataset pre-processing (microsoft#421)

  Show description for 7d23e33
  polisettyvarma

  authored
  7d23e33

  Copy full SHA for 7d23e33

Commits on Jul 19, 2024

• improve repeat_kv GQA perf (microsoft#419)

  polisettyvarma

  authored
  13f2673

  Copy full SHA for 13f2673

• acquire device when required (microsoft#420)

  Show description for 3af2e25
  polisettyvarma

  authored
  3af2e25

  Copy full SHA for 3af2e25

• Add basic compilation test (microsoft#426)

  loadams

  authored
  08b9376

  Copy full SHA for 08b9376

• Update yml to be valid (microsoft#427)

  loadams

  authored
  3afd267

  Copy full SHA for 3afd267

Commits on Jul 29, 2024

• Update/add GPT/Llama universal checkpointing scripts (microsoft#391)

  Show description for 8822a5c
  lekurile

  authored
  8822a5c

  Copy full SHA for 8822a5c

Commits on Aug 1, 2024

• fixing the bug of flash_attn import and the wrong gather index when using flash_attn_cuda in sequence parallel (microsoft#406)

  Show description for 1bfc35c
  

  YJHMITWEB

  and

  Jinghan Yao

  authored
  1bfc35c

  Copy full SHA for 1bfc35c

Commits on Aug 4, 2024

• add fused_rms_norm support on XPU device (microsoft#431)

  ys950902

  authored
  53b241f

  Copy full SHA for 53b241f

Commits on Aug 7, 2024

• pass batch_dim_idx to deepspeed sequence parallel distributed attention for supporting batch size larger than 1 (microsoft#433)

  Show description for 61350c5
  

  YJHMITWEB

  and

  Jinghan Yao

  authored
  61350c5

  Copy full SHA for 61350c5

Commits on Aug 10, 2024

• [LLaMa] Adding support converting checkpoint from mds to hf (microsoft#432)

  Show description for f132876
  billishyahao

  authored
  f132876

  Copy full SHA for f132876

Commits on Aug 14, 2024

• add device check when import ipex (microsoft#436)

  ys950902

  authored
  cdf5194

  Copy full SHA for cdf5194

Commits on Aug 19, 2024

• fix TFLOPs calculation (microsoft#371)

  Show description for b7b2d5e
  polisettyvarma

  authored
  b7b2d5e

  Copy full SHA for b7b2d5e

Commits on Aug 24, 2024

• fix nan issue when running megatron-deepspeed (microsoft#434)

  ys950902

  authored
  4f9f1f6

  Copy full SHA for 4f9f1f6

Commits on Aug 26, 2024

• enable empty cache on XPU device (microsoft#438)

  ys950902

  authored
  8e9d973

  Copy full SHA for 8e9d973

Commits on Aug 27, 2024

• [wandb] disable wandb more gracefully (microsoft#422)

  Show description for 543543a
  

  billishyahao

  and

  loadams

  authored
  543543a

  Copy full SHA for 543543a

• [Bug] Fix crash when logging optimizer state to tb (microsoft#417)

  billishyahao

  authored
  1280f59

  Copy full SHA for 1280f59

Commits on Aug 30, 2024

• fixed dftracer compatibility

  zhenghh04

  committed
  ea0c3c7

  Copy full SHA for ea0c3c7

Commits on Sep 3, 2024

• added requirements.txt

  zhenghh04

  committed
  a0ac750

  Copy full SHA for a0ac750

Commits on Sep 4, 2024

• Enable Sequence Parallelism (microsoft#429)

  polisettyvarma

  authored
  0d6e379

  Copy full SHA for 0d6e379

• Update utils.py

  zhenghh04

  authored
  de7f22f

  Copy full SHA for de7f22f

Commits on Sep 12, 2024

• merge: Create microsoft-main

  saforem2

  committed
  8bc5313

  Copy full SHA for 8bc5313

• Remove duplicate --profile arg

  saforem2

  committed
  a1ede68

  Copy full SHA for a1ede68

• debug: sequence_parallel issue in RMSNorm ??

  saforem2

  committed
  6b32cff

  Copy full SHA for 6b32cff

• fix check

  zhenghh04

  committed
  12f6f8e

  Copy full SHA for 12f6f8e

• Update megatron/training_log_alcf.py

  saforem2

  committed
  5ac877a

  Copy full SHA for 5ac877a

Commits on Sep 13, 2024

• Update megatron/training.py

  saforem2

  committed
  b3e0f6f

  Copy full SHA for b3e0f6f

• Update megatron/utils.py

  saforem2

  committed
  2113dbc

  Copy full SHA for 2113dbc

• Update megatron/training_log.py

  saforem2

  committed
  7f71572

  Copy full SHA for 7f71572

Commits on Sep 15, 2024

• Update pretrain_gpt_alcf.py

  saforem2

  committed
  7cb9c11

  Copy full SHA for 7cb9c11

• Update megatron/training_log.py

  saforem2

  committed
  e83de19

  Copy full SHA for e83de19

• Warn if mismatch b/w iters in megatron/checkpointing.py

  saforem2

  committed
  29756d6

  Copy full SHA for 29756d6

Commits on Sep 16, 2024

• fix: try/except for non tensors in megatron/training_log.py

  saforem2

  committed
  1a7f03b

  Copy full SHA for 1a7f03b

Commits on Sep 17, 2024

• fix: Correctly draw grad_acc_steps batches of data when skipping step

  saforem2

  committed
  828f6a9

  Copy full SHA for 828f6a9

• Update pretrain_gpt_alcf.py

  Show description for 295fcb3
  saforem2

  committed
  295fcb3

  Copy full SHA for 295fcb3

Commits on Sep 20, 2024

• grad_wei can't be NoneType when running with DeepSpeed, for zero3 will divided the gradient (microsoft#428)

  ys950902

  authored
  598c092

  Copy full SHA for 598c092

Commits on Sep 23, 2024

• added sophia

  

  Marieme Ngom

  committed
  cf80e6b

  Copy full SHA for cf80e6b

Commits on Sep 30, 2024

• Merge pull request #59 from mngom2/spike-skipper

  Show description for 09accde
  saforem2

  authored
  09accde

  Copy full SHA for 09accde

Commits on Oct 4, 2024

• fix init issue for rms_norm in squence_parallel (microsoft#448)

  ys950902

  authored
  8be7f48

  Copy full SHA for 8be7f48

Commits on Oct 8, 2024

• enable profiler for specific ranks (microsoft#451)

  ranzhejiang

  authored
  4448492

  Copy full SHA for 4448492

• Merge pull request #58 from argonne-lcf/spike-skipper

  Show description for cef3fc7
  saforem2

  authored
  cef3fc7

  Copy full SHA for cef3fc7

• merge: Resolve merge conflicts pulling in from Microsoft upstream

  saforem2

  committed
  fd94b37

  Copy full SHA for fd94b37

Commits on Oct 11, 2024

• Merge branch 'microsoft:main' into main

  saforem2

  authored
  ecc248a

  Copy full SHA for ecc248a

• merge: argonne-lcf-microsoft-main into main

  saforem2

  committed
  9b5be12

  Copy full SHA for 9b5be12

Commits on Oct 12, 2024

• shuffle concate dataset index

  zhenghh04

  committed
  5394156

  Copy full SHA for 5394156

• fixed bugs

  zhenghh04

  committed
  573b668

  Copy full SHA for 573b668

• Update ALCF/helpers.sh, train_aGPT_7B.sh

  saforem2

  committed
  41ff059

  Copy full SHA for 41ff059

• merge: feature/profile with data fix into microsoft-main

  saforem2

  committed
  89db92a

  Copy full SHA for 89db92a

• Fix shuffle_idx in megatron/data/gpt_dataset.py

  saforem2

  committed
  9de83a9

  Copy full SHA for 9de83a9

• Fix shuffle_idx in megatron/data/gpt_dataset.py

  saforem2

  committed
  d7a2594

  Copy full SHA for d7a2594

Commits on Oct 13, 2024

• Update ALCF/helpers.sh, train_aGPT_7B.sh

  saforem2

  committed
  3e33a6a

  Copy full SHA for 3e33a6a

• Update pretrain_gpt_alcf.py

  saforem2

  committed
  43cde2b

  Copy full SHA for 43cde2b

• Update megatron/data/{blendable,gpt,indexed}_dataset.py

  saforem2

  committed
  9f09733

  Copy full SHA for 9f09733

• Update ALCF/requirements/requirements.txt

  saforem2

  committed
  2b31b44

  Copy full SHA for 2b31b44

• Update megatron/utils.py

  saforem2

  committed
  5e9eed0

  Copy full SHA for 5e9eed0

Commits on Oct 14, 2024

• fixed bugs and added commandline option

  zhenghh04

  committed
  3dcb297

  Copy full SHA for 3dcb297

• Merge branch 'debug-logging' into feature/profile

  saforem2

  authored
  bec9b7a

  Copy full SHA for bec9b7a

• fixed typo

  zhenghh04

  committed
  43fc2fe

  Copy full SHA for 43fc2fe

• Merge branch 'feature/profile' of github.com:argonne-lcf/Megatron-DeepSpeed into feature/profile

  zhenghh04

  committed
  94d5337

  Copy full SHA for 94d5337

• Merge pull request #67 from argonne-lcf/feature/profile

  saforem2

  authored
  bb55e97

  Copy full SHA for bb55e97

• added support for blending samples across different files in the same corpus

  zhenghh04

  committed
  d50239f

  Copy full SHA for d50239f

• Merge pull request #64 from argonne-lcf/debug-logging

  saforem2

  authored
  9b4f510

  Copy full SHA for 9b4f510

Commits on Oct 15, 2024

• Merge branch 'alcf-hzheng-data-fix' into hzheng-data-fix

  saforem2

  authored
  324ef11

  Copy full SHA for 324ef11

• Discard changes to megatron/data/gpt_dataset.py

  saforem2

  authored
  45ff652

  Copy full SHA for 45ff652

• Consistent logging in megatron/data/*.py

  saforem2

  committed
  52a406c

  Copy full SHA for 52a406c

Commits on Oct 16, 2024

• Update megatron/data/gpt_dataset.py

  saforem2

  committed
  63b1901

  Copy full SHA for 63b1901

• Use time.perf_counter in megatron/data/blendable_dataset.py

  saforem2

  committed
  7ef26bf

  Copy full SHA for 7ef26bf

Commits on Oct 17, 2024

• fix init issue for silently ignoring the deepspeed config (microsoft#452)

  xylian86

  authored
  deb95cd

  Copy full SHA for deb95cd

• Update ALCF/helpers.sh

  saforem2

  committed
  68da2db

  Copy full SHA for 68da2db

Commits on Oct 18, 2024

• Merge branch 'main' of https://github.com/microsoft/Megatron-DeepSpeed into hzheng-data-fix

  saforem2

  committed
  ab3a8ec

  Copy full SHA for ab3a8ec

• Merge branch 'hzheng-data-fix' of https://github.com/argonne-lcf/Megatron-DeepSpeed into hzheng-data-fix

  saforem2

  committed
  ed21bd9

  Copy full SHA for ed21bd9

• fix moe tflops (microsoft#445)

  ranzhejiang

  authored
  6acc370

  Copy full SHA for 6acc370

• Merge 'upstream/main' into hzeng-data-fix

  Show description for 467279b
  saforem2

  committed
  467279b

  Copy full SHA for 467279b

• Remove duplicate gradient_accumulation_steps in DS config

  saforem2

  committed
  9e015cc

  Copy full SHA for 9e015cc

Commits on Oct 21, 2024

• Update default EVAL args

  saforem2

  committed
  58dc2d7

  Copy full SHA for 58dc2d7

• Catch eval metrics in megatron/training.py

  saforem2

  committed
  277d308

  Copy full SHA for 277d308

• Save git branch to env in train_aGPT_7B.sh

  saforem2

  committed
  af4cba1

  Copy full SHA for af4cba1

• fixed print out bug

  zhenghh04

  committed
  8a8472c

  Copy full SHA for 8a8472c

• Merge pull request #68 from argonne-lcf/feature/blending_corpus

  saforem2

  authored
  dfd0643

  Copy full SHA for dfd0643

• Fix args.shuffle in megatron/data/gpt_dataset.py

  saforem2

  committed
  6cb727d

  Copy full SHA for 6cb727d

Commits on Oct 24, 2024

• Update --{shuffle,blend}-sample-in-corpus arg in ALCF/helpers.sh

  saforem2

  committed
  5d10179

  Copy full SHA for 5d10179

Commits on Oct 31, 2024

• fix: GRAD_ACC_STEPS when NHOSTS == 256

  saforem2

  committed
  160d6a6

  Copy full SHA for 160d6a6

Commits on Nov 5, 2024

• Merge pull request #63 from argonne-lcf/hzheng-data-fix

  Show description for 40db8c2
  saforem2

  authored
  40db8c2

  Copy full SHA for 40db8c2

Commits on Nov 7, 2024

• 🚧 ALCF/ds_to_universal.py

  saforem2

  committed
  ce7d553

  Copy full SHA for ce7d553

• docs: Add ALCF/notes/checkpoints.md

  saforem2

  committed
  8e0bff8

  Copy full SHA for 8e0bff8

• feat: Enable --use-flash-attn-builder by default on Aurora

  saforem2

  committed
  bd8c246

  Copy full SHA for bd8c246

• Update python.yml

  saforem2

  authored
  26f2e71

  Copy full SHA for 26f2e71

• Update python.yml

  saforem2

  authored
  48b3c81

  Copy full SHA for 48b3c81

• Update python.yml

  saforem2

  authored
  0a997bb

  Copy full SHA for 0a997bb