[Bug] Qwen2-VL-7B with sglang has significant numerical calculation errors compared to HF Transformers

[kritohyh]

Checklist

• 1. I have searched related issues but cannot get the expected help.
• 2. The bug has not been fixed in the latest version.
• 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
• 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
• 5. Please use English, otherwise it will be closed.

Describe the bug

In practice, we found that sglang Qwen2-VL model has numerical calculation errors compared to HF Transformers model in both Qwen2VisionTransformer and Qwen2Model parts.
Our input image has 720 tokens input to Vit encoding, and the lowest embedded cosine similarity in the output is 0.1775. In addition, we directly feed the Vit output and text embedding of Transformers to the LLM part. The cosine similarity of the lowest among the 830 HS outputs in the Prefill stage is reduced to 0.9499, and in the generation stage, as the output increases, the cosine similarity may further decrease to 0.580.

Here are the code blocks we found that caused the differences：
RMSNorm's CUDA Kernel and PyTorch Native
Residual Sum Precision in Transformer Blocks
QKVParallelLinear vs nn.Linear in Transformer Blocks
Silu in Qwen2MLP
Calculation of sin/cos cache in RotaryEmbedding

After eliminating the above differences, we achieved precision alignment. But for performance reasons, is there a repair plan for the above issues？

Reproduction

Qwen2-VL

Environment

Python: 3.10.16 (main, Dec 4 2024, 08:53:37) [GCC 9.4.0]
CUDA available: True
GPU 0: NVIDIA GeForce RTX 4090 D
GPU 0 Compute Capability: 8.9
CUDA_HOME: /usr/local/cuda
NVCC: Cuda compilation tools, release 12.1, V12.1.105
CUDA Driver Version: 550.90.12
PyTorch: 2.5.1+cu124
sglang: 0.4.0.post1
flashinfer: 0.1.6
triton: 3.1.0
transformers: 4.45.2
torchao: 0.7.0
numpy: 1.26.4
aiohttp: 3.11.11
fastapi: 0.115.6
hf_transfer: 0.1.8
huggingface_hub: 0.27.0
interegular: 0.3.3
modelscope: 1.18.1
orjson: 3.10.12
packaging: 24.2
psutil: 6.1.1
pydantic: 2.10.4
multipart: 0.0.20
zmq: 26.2.0
uvicorn: 0.34.0
uvloop: 0.21.0
vllm: 0.6.4.post1
openai: 1.58.1
anthropic: 0.42.0
decord: 0.6.0
NVIDIA Topology:
GPU0 NIC0 NIC1 NIC2 NIC3 NIC4 CPU Affinity NUMA Affinity GPU NUMA ID
GPU0 X SYS SYS SYS SYS SYS N/A
NIC0 SYS X PHB PHB PHB PHB
NIC1 SYS PHB X PHB PHB PHB
NIC2 SYS PHB PHB X PHB PHB
NIC3 SYS PHB PHB PHB X PHB
NIC4 SYS PHB PHB PHB PHB X

Legend:

X = Self
SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
PIX = Connection traversing at most a single PCIe bridge
NV# = Connection traversing a bonded set of # NVLinks

NIC Legend:

NIC0: mlx5_0
NIC1: mlx5_1
NIC2: mlx5_2
NIC3: mlx5_3
NIC4: mlx5_4

Hypervisor vendor: KVM
ulimit soft: 1048576

[zhaochenyang20]

Thanks. We have noticed this for long. This is urgent to us.

[yizhang2077]

FYI, The vit encoding accuracy problem may be alleviated once #2977 has been merged, but we still need do some tests.

[zhaochenyang20]

@yizhang2077 should we close it now?

[yizhang2077]

I think we can close it, we need implement this #3142 to have a baseline early.

[dmakhervaks]

@zhaochenyang20 @yizhang2077 I am still seeing inconsistency in Qwen2-VL models when running on SGlang versus directly via HF. The outputs are considerably shorter and worse. I even tested the latest version of SGlang

[yizhang2077]

@zhaochenyang20 @yizhang2077 I am still seeing inconsistency in Qwen2-VL models when running on SGlang versus directly via HF. The outputs are considerably shorter and worse. I even tested the latest version of SGlang

@dmakhervaks Do you add correct chat template like --chat-template qwen2-vl? If you had add it, could you provide some method to reproduce results? Thanks.

[dmakhervaks]

@yizhang2077 yes I did.

Here is some reproduction logic

Sglang inference
Launch server
python -m sglang.launch_server --model-path Qwen/Qwen2-VL-7B-Instruct --chat-template qwen2-vl --port 30000 --host 0.0.0.0

Submit request

image_path = 'fhcb0001_page0134.png'
 prompt = 'Generate a detailed caption for this image.'
 url = "http://localhost:30000/v1/chat/completions"
 messages = []
 messages.append({"role":"user", "content": [
     {"type": "text", "text": prompt},
     {"type": "image_url","image_url": {"url": image_path, "detail": "high"}}]})

 data = {
     "model": "qwen2-vl",
     "messages": messages,
     "temperature": 0.01,
     "max_tokens": 4000
 }
 response = requests.post(url, json=data)
 response = response.json()
 print(response["choices"][0]["message"]["content"])

Get result:
The image is a scanned page from a document, specifically page 9 of a study titled "DENNIS AND COMPANY, INC." The page contains a series of questions and answer options related to marketing research studies. The questions are numbered and include options for "Yes," "No," and "1" or "2," indicating a binary or multiple-choice format. The text is printed in a standard font, and the page appears to be part of a larger research document or questionnaire.

Huggingface inference

 prompt = 'Generate a detailed caption for this image.'
   image_path = 'fhcb0001_page0134.png'
from qwen_vl_utils import process_vision_info
    from transformers import Qwen2VLForConditionalGeneration
    model_path = 'Qwen/Qwen2-VL-7B-Instruc'
    model = Qwen2VLForConditionalGeneration.from_pretrained(
        model_path,
        torch_dtype=torch.bfloat16,
        attn_implementation="flash_attention_2",
        device_map="cuda:1",
    )

    # default processer
    processor = AutoProcessor.from_pretrained(model_path)
        messages = [
            {
                "role": "user",
                "content": [
                    {
                        "type": "image",
                        "image": image_path,
                    },
                    {"type": "text", "text": prompt},
                ],
            }
        ]
        print(image_path)
        # Preparation for inference
        text = processor.apply_chat_template(
            messages, tokenize=False, add_generation_prompt=True
        )
        if image_path is not None:
            image_inputs, video_inputs = process_vision_info(messages)
            inputs = processor(
                text=[text],
                images=image_inputs,
                videos=video_inputs,
                padding=True,
                return_tensors="pt",
            )
        else:
            inputs = processor(
                text=[text],
                images=None,
                videos=None,
                padding=True,
                return_tensors="pt",
            )
        inputs = inputs.to("cuda:1")

        # Inference: Generation of the output
        generated_ids = model.generate(**inputs, max_new_tokens=1000)
        generated_ids_trimmed = [
            out_ids[len(in_ids) :] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
        ]
        output_text = processor.batch_decode(
            generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False
        )
        print(f'output: {output_text}')

Get result:
'The image is a page from a document titled "DENNIS AND COMPANY, INC." It appears to be a questionnaire or survey form, specifically Study #2520, and it is on page 9 of the document. The page is numbered as "CD 2" at the top right corner.\n\nThe page contains several questions and instructions for the respondent. Here is a breakdown of the content:\n\n1. **Question 11a**: "Have you participated in any marketing research study in the last 2 years at a research facility, mall, or hotel? (Circle one answer. If Safety Harbor, ask for 4 years.)"\n - Yes: 1 (Continue)\n - No: 2 (Skip to invitation)\n\n2. **Question 11b**: "What was the subject of the study you participated in? (Probe for product category.)"\n - (If cigarettes, continue. If non-cigarette, skip to invitation.)\n\n3. **Question 11c**: "What type of cigarette study did you participate in? (Probe for details/brands discussed.)"\n\n4. **Instructions**: \n - If Eclipse, terminate. If respondent cannot remember brand but mentioned smokeless, ashless, or cigarette that does not burn down, terminate in box below.\n\n5. **Term: GTC/Eclipse Cigarette Research**:\n - 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 (30,31)\n\n6. **Instructions**: \n - If Atlanta or Safety Harbor, ask Q.11d. Otherwise, skip to Q.12.\n\n7. **Question 11d**: "By chance, have you attended a presentation at a research facility or hotel within the past two years that explained all about a new brand of cigarettes? (Circle one answer.)"\n - Yes: a (Terminate)\n - No: 1 (Continue)\n\n8. **Term: Attended Eclipse Presentation**:\n - 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 (35,36)\n\n9. **Question 12**: "Do you have any vision problems which cause you difficulty in reading or which prevents you from reading? (If can read with glasses, circle no and remind to bring glasses with them during invitation. Circle one answer.)"\n - Yes: a (Terminate)\n - No: 1 (Continue)\n\n10. **Term: Vision Problems**:\n - 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 (38,39)\n\nThe page also contains a series of numbers and terms that seem to be part of the study\'s coding or categorization system.'

[yizhang2077]

@dmakhervaks currently we still have no formal benchmark to check our VLM models accuracy (only depends on developer run benchmark in local env), we will build it in #3562 soon, after that we will fix our VLM accuracy easier.

[ymzhang0303]

@yizhang2077 Hi, I met the same issue, the vllm inferences result is same with huggingface inference result, while sglang is different. And I found the issue comes from that sglang didn't use correct 'positions' in the forward function of Qwen2VLForConditionalGeneration. I solved the issue and created a pull request here:
#3605