Train large scale model on Single GPU

[lifeiteng]

I'm trying to train large scale model on my GPU (with 12G memory).

DeepSpeed

• https://github.com/microsoft/DeepSpeed

PR #19
error microsoft/DeepSpeed#2818

ColossalAI

• https://github.com/hpcaitech/ColossalAI

• TODO

can anyone help?

[lifeiteng]

deepspeed error have been solved #19

[lifeiteng]

GPU 12G memory

# the paper --num-decoder-layers 12
deepspeed bin/trainer.py --max-duration 24 --use-fp16 true \
      --decoder-dim 1024 --nhead 16 --num-decoder-layers 6 \
      --deepspeed --deepspeed_config configs/ds_zero2.config \
      --exp-dir exp/valle_ds_zero2_CPUAdam_Layer6


2023-02-18 12:54:52,919 INFO [trainer.py:974] Sanity check -- see if any of the batches in epoch 1 would cause OOM.
2023-02-18 12:55:16,603 INFO [trainer.py:1004] Maximum memory allocated so far is 6093MB
2023-02-18 12:55:16,960 INFO [trainer.py:1004] Maximum memory allocated so far is 6094MB
2023-02-18 12:55:17,304 INFO [trainer.py:1004] Maximum memory allocated so far is 6094MB
2023-02-18 12:55:17,643 INFO [trainer.py:1004] Maximum memory allocated so far is 6094MB
2023-02-18 12:55:17,886 INFO [trainer.py:1004] Maximum memory allocated so far is 6094MB
2023-02-18 12:55:18,126 INFO [trainer.py:1004] Maximum memory allocated so far is 6094MB
2023-02-18 13:00:04,684 INFO [trainer.py:684] Epoch 1, batch 100, train_loss[loss=6.812, over 1032.00 frames. ], tot_loss[loss=6.975, over 108216.87 frames. ], batch size: 1, lr: 2.22e-05,
2023-02-18 13:04:49,349 INFO [trainer.py:684] Epoch 1, batch 200, train_loss[loss=6.383, over 1346.00 frames. ], tot_loss[loss=6.814, over 175826.38 frames. ], batch size: 2, lr: 2.55e-05,
2023-02-18 13:09:34,472 INFO [trainer.py:684] Epoch 1, batch 300, train_loss[loss=6.315, over 1298.00 frames. ], tot_loss[loss=6.599, over 219340.66 frames. ], batch size: 3, lr: 2.75e-05,
2023-02-18 13:14:18,852 INFO [trainer.py:684] Epoch 1, batch 400, train_loss[loss=6.597, over 1405.00 frames. ], tot_loss[loss=6.427, over 243305.81 frames. ], batch size: 6, lr: 2.89e-05,
2023-02-18 13:19:03,288 INFO [trainer.py:684] Epoch 1, batch 500, train_loss[loss=5.879, over 1561.00 frames. ], tot_loss[loss=6.302, over 254497.07 frames. ], batch size: 2, lr: 3.00e-05,
2023-02-18 13:23:47,974 INFO [trainer.py:684] Epoch 1, batch 600, train_loss[loss=6.054, over 1471.00 frames. ], tot_loss[loss=6.203, over 263684.29 frames. ], batch size: 6, lr: 3.08e-05,
2023-02-18 13:28:32,754 INFO [trainer.py:684] Epoch 1, batch 700, train_loss[loss=6.118, over 1070.00 frames. ], tot_loss[loss=6.141, over 271613.76 frames. ], batch size: 1, lr: 3.16e-05,

[lifeiteng]

DeepSpeed doesn't help, as we randomly sample training stage 4d2aaa2

# 12G GPU --max-duration 24 --num-decoder-layers 6
# the paper --num-decoder-layers 12
bash run.sh --stage 4 --stop-stage 4 \
    --num-decoder-layers 6 \
    --max-duration 24 --use-fp16 true

[FluffyDiscord]

Quick questions

• can I stop training for example in epoch 4 with CTRL+C then continue later?
• does it work with non English speech? Do I need different dataset? Where would I find that?