Quick Start

	Megatron-LM-Dense	Megatron-Core-Dense	Megatron-Core-MoE	MegaBlocks-MoE
Qwen2-VL	N/A	ReadMe	N/A	N/A
LLaVA	N/A	ReadMe	N/A	N/A
Qwen2.5	N/A	ReadMe	N/A	N/A
LLama3.1	N/A	ReadMe	N/A	N/A
LLama3	ReadMe	ReadMe	N/A	N/A
LLama2	ReadMe	ReadMe	N/A	N/A
Mistral	ReadMe	ReadMe	ReadMe	N/A
Qwen2	N/A	ReadMe	ReadMe	N/A
Qwen1.5	ReadMe	ReadMe	ReadMe	ReadMe
DeepSeek-V2	N/A	N/A	ReadMe	N/A

Introduction

English | 简体中文

Pai-Megatron-Patch (https://github.com/alibaba/Pai-Megatron-Patch) is a deep learning training toolkit built for developers to train and predict LLMs & VLMs by using Megatron framework easily. With the continuous development of LLMs, the model structure and scale are rapidly evolving. Although these models can be conveniently manufactured using Transformers or DeepSpeed training framework, the training efficiency is comparably low. This phenomenon becomes even severer when the model scale exceeds 10 billion. The primary objective of Pai-Megatron-Patch is to effectively utilize the computational power of GPUs for LLM. This tool allows convenient training of commonly used LLM with all the accelerating techniques provided by Megatron-LM.

What's New:

• Support training Qwen2-VL models by using Megatron-Core. [🔥🔥 2024.11.27]
• Support training LLaVA models by using Megatron-Core. [🔥🔥 2024.11.20]
• Add llm auto configurator and apply per seq sft loss for qwen2/2.5 models. [🔥🔥 2024.10.30]
• Upgrade deepseek-v2-moe models to support MLA via transformer engine and pipeline ckpts conversion. [🔥🔥 2024.09.26]
• Support training Qwen2.5 models by using Megatron-Core. [🔥🔥 2024.09.20]
• Support Sequence Packing in SFT for Qwen2 and LLaMA 3.1 models. [🔥🔥 2024.09.13]
• Upgrade qwen2 dense and moe models to support Flash-Attention 3, Offloading, Comm-Overlapping features. [🔥🔥 2024.08.26]
• Support training LLaMA 3.1 dense models with Flash-Attention 3 backend. [🔥🔥 2024.08.23]
• Support training LLaMA 3.1 dense models by using Megatron-Core. [🔥🔥 2024.08.23]
• Support auto optimizer offloading in OffloadDistributedOptimizer. [🔥🔥 2024.07.25]
• Support static optimizer offloading in OffloadDistributedOptimizer. [🔥🔥 2024.07.15]
• Support training qwen2 moe models by using Megatron-Core. [🔥🔥 2024.06.19]
• Support training qwen2 dense models by using Megatron-Core. [🔥🔥 2024.06.12]
• Support training deepseek-v2-moe models by using Megatron-Core. [🔥🔥 2024.05.30]
• Support training qwen1.5-moe models by using Megatron-Core. [🔥🔥 2024.05.13]
• Support training llama3 models by using Megatron-LM and Megatron-Core. [🔥🔥 2024.04.21]
• Support training qwen1.5 models by using Megatron-Core. [🔥🔥 2024.03.20]
• Support training qwen1.5 models by using Megatron-LM. [🔥🔥 2024.02.28]
• Support training mixtral-8x7b moe model by using Megatron-Core. [🔥🔥 2024.01.26]
• Support training qwen-vl multimodel by using Megatron-LM. [🔥🔥 2023.12.15]
• Support training LLava multimodel by using Megatron-LM. [🔥🔥 2023.12.01]
• Support training deepseek model by using Megatron-LM. [🔥🔥 2023.11.24]
• Support training qwen-72B model by using Megatron-LM. [🔥🔥 2023.11.23]
• Support training Mistral-7B, Yi-6B and Codellama-34B [🔥🔥 2023.11.16]
• Upgrade Megatron-LM for Llama2, qwen and baichuan2 to use transformer engine and fp8. [🔥🔥 2023.10.19]
• Support training qwen-14B and baichuan2-13B model by using Megatron-LM. [🔥🔥 2023.10.08]

Highlights

Pai-Megatron-Patch is developed by the Alibaba Cloud Machine Learning Platform (PAI) algorithm team. The tool aims to assist developers in quickly getting started with Lingjun products and completing the entire development pipeline for LLM, including efficient distributed training, supervised fine-tuning, and offline model inference or verification. It has several merits as follows:

• Support for multiple commonly used LLM such as llama, llama-2, codellama, deepseek, baichuan, qwen, Falcon, GLM, Starcoder, Bloom, chatglm, etc.
• Support for model weight conversion: Mapping operator namespaces between Huggingface, Megatron, and Transformer Engine.
• Support for FP8 training acceleration in Flash Attention 2.0 and Transformer Engine modes, ensuring training convergence.
• Rich and user-friendly usage examples, offering best practices for the entire workflow of LLM pre-training, fine-tuning, evaluation, and inference, as well as reinforcement learning.

Framework

The design philosophy of Pai-Megatron-Patch is to avoid invasive modifications to the source code of Megatron-LM. In other words, it does not add new modules directly to Megatron-LM. Instead, the functions that need expansion and improvement are presented in the form of patch. This decoupling ensures that users can continue to embrace the best practices of LLM without being affected by upgrades of Megatron-LM.

Pai-Megatron-Patch includes key components for building LLM training, such as model library, tokenizers, model convertors, reinforcement learning , offline text generation, usages examples, and toolkits. The model library provides popular LLMs implemented in Megatron, such as baichuan, bloom, chatglm, falcon, galactica, glm, llama, qwen, and starcoder. More Megatron-based implementations of LLMs will be added as needed in the future. Additionally, the patch provides bidirectional conversion between Huggingface and Megatron model weights. This allows users to easily utilize Huggingface pretrained models for continued pre-training or fine-tuning in Megatron, as well as evaluating model quality using Huggingface's evaluation/inference pipelines on trained Megatron models.

In the reinforcement learning section, the patch offers PPO training workflows, enabling users to perform reinforcement learning with SFT models and RM models. Finally, the patch provides numerous usage examples to help users quickly start LLMs training and offline inference. For specific usage processes within Alibaba Cloud Lingjun products, please refer to the following link: PAI-Lingjun Intelligent Computing Service LLM solution.

Technical Reports

• Pai-Megatron-Patch：围绕Megatron-Core打造大模型训练加速生态
• Meta Llama3.1模型在PAI-Megatron-Patch的最佳实践
• 基于Megatron-Core的稀疏大模型训练工具：阿里云MoE大模型最佳实践
• Mixtral-8x7B在PAI灵骏的训练指南
• 通义千问开源模型在PAI灵骏的最佳实践
• 阿里云机器学习PAI开源AI大模型训练工具Pai-Megatron-Patch, 助力大模型技术落地
• 基于单机最高能效270亿参数GPT模型的文本生成与理解
• 中文稀疏GPT大模型落地 — 通往低成本&高性能多任务通用自然语言理解的关键里程碑
• 预训练知识度量比赛夺冠！阿里云PAI发布知识预训练工具
• 阿里云PAI获得FewCLUE基于大模型的小样本学习双料冠军

Contact

Use Dingtalk to scan blow QR code.

Note: group 1 is full, please add group 2.

License

This project is licensed under the Apache License (Version 2.0). This toolkit also contains some code modified from other repos under other open-source licenses. See the NOTICE file for more information.