Update model_loader deps and qqq quantization deps #2220
Conversation
HandH1998
requested review from
merrymercy,
Ying1123,
hnyls2002,
zhyncs,
ispobock and
ByronHsu
as code owners
|
There are some failures due to |
|
|
||
| import torch | ||
| from torch.nn.parameter import Parameter | ||
| from torchao.ops import marlin_qqq_gemm |
There was a problem hiding this comment.
ImportError: cannot import name 'marlin_qqq_gemm' from 'torchao.ops'
It should be due to the version, the current release of torchao (v0.6.1) does not include qqq.
There was a problem hiding this comment.
Perhaps we can introduce qqq in the next PR after torchao releases a new version, how about that @HandH1998
| from torchao.ops import marlin_qqq_gemm | ||
| from torchao.quantization.utils import dynamically_quantize_per_channel | ||
| from vllm.model_executor.layers.linear import LinearBase, LinearMethodBase | ||
| from vllm.model_executor.parameter import ( |
There was a problem hiding this comment.
This part also needs to be migrated.
| from typing import Optional | ||
|
|
||
| from torch import nn | ||
| from vllm.config import ( |
There was a problem hiding this comment.
This part also needs to be migrated.
| from torch import nn | ||
| from transformers import AutoModelForCausalLM, PretrainedConfig | ||
| from transformers.utils import SAFE_WEIGHTS_INDEX_NAME | ||
| from vllm.config import ( |
There was a problem hiding this comment.
This part also needs to be migrated.
| get_tensor_model_parallel_rank, | ||
| get_tensor_model_parallel_world_size, | ||
| ) | ||
| from vllm.envs import VLLM_USE_MODELSCOPE |
There was a problem hiding this comment.
L39:46 This part also needs to be migrated.
|
|
||
| import torch | ||
| from torch import nn | ||
| from vllm.config import ModelConfig |
There was a problem hiding this comment.
This part also needs to be migrated.
| from huggingface_hub import HfFileSystem, hf_hub_download, snapshot_download | ||
| from safetensors.torch import load_file, safe_open, save_file | ||
| from tqdm.auto import tqdm | ||
| from vllm.config import LoadConfig, ModelConfig |
There was a problem hiding this comment.
L22:L25 This part also needs to be migrated. (except vllm.distributed)
Motivation
Update the model_loader deps and qqq quantization deps for SGLang.
Modifications
We modified the relevant code primarily according to vLLM. Thanks the vLLM team for their significant contributions. Here we list the main modifications.
model_loadercode from https://github.com/vllm-project/vllm/tree/main/vllm/model_executor/model_loader and modified it adaptively for SGLang. The updatedmodel_loadercode is located atpython/sglang/srt/model_loader.registry.pyatpython/sglang/srt/models/registry.pyand registered all the models into classModelRegistry. Consequently, we removed all monkey patches inpython/sglang/srt/model_executor/model_runner.py.marlin_qqq_gemmfromtorchao. For more details on qqq, please refer to our paper and our code repo.