Refactor turbomind attention

[lzhangzz]

TODO

• Fix attention/decoding on sm_75/sm_70
• Fix dispatch heuristic for split-k decoding
• BF16 dispatch

[zhyncs]

Hi @lzhangzz Maybe you could merge the latest main branch to fix the win build error.

[lvhan028]

• LLM Evaluation with turbomind
  • llama2
  • internlm2
  • qwen
  • llama2-4bit
  • internlm2-4bit
  • qwen-4bit
• CLI chat
• pipeline api
• long-context inference
• deepseek linear rope scaling
• v100
• T4

[lvhan028]

@irexyc May help verify the VL models

[zhyncs]

Hi @lzhangzz I used the model lmsys/vicuna-13b-v1.3 to compare performance with the latest main branch, and found that the improvement in throughput is not significant. The following is the reproduction method, maybe pay attention. Thanks.

# convert
python3 -m lmdeploy convert llama /workdir/vicuna-13b-v1.3
# server
python3 -m lmdeploy serve api_server /workdir/workspace
# client
python3 benchmark/profile_restful_api.py --server_addr 127.0.0.1:23333 --tokenizer_path /workdir/vicuna-13b-v1.3 --dataset /workdir/ShareGPT_V3_unfiltered_cleaned_split.json --concurrency 128 --num_prompts 5000
# ShareGPT_V3_unfiltered_cleaned_split.json
https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json

# turbomind 3
concurrency: 128
elapsed_time: 641.477s

number of prompt tokens: 1239680
number of completion tokens: 1162390
token throughput (completion token): 1812.053 token/s
token throughput (prompt + completion token): 3744.594 token/s
RPS (request per second): **7.795 req/s**
RPM (request per minute): 467.671 req/min

# latest main
concurrency: 128
elapsed_time: 660.434s

number of prompt tokens: 1239680
number of completion tokens: 1162390
token throughput (completion token): 1760.039 token/s
token throughput (prompt + completion token): 3637.107 token/s
RPS (request per second): **7.571 req/s**
RPM (request per minute): 454.247 req/min

7.795/7.571-1=3%

[zhyncs]

In addition to improving readability, what are the main benefits of refactoring attention. In which usage scenarios will performance improvements be more significant?

[lvhan028]

This PR doesn't target at improving the performance. Instead, it focuses on cleaning the messy attention codes to firm the foundation of supporting more models.
This is the first step. Later on, @lzhangzz will work on decoupling the continuous batching and model implementation.

[zhyncs]

This PR doesn't target at improving the performance. Instead, it focuses on cleaning the messy attention codes to firm the foundation of supporting more models. This is the first step. Later on, @lzhangzz will work on decoupling the continuous batching and model implementation.

When I discussed with @lzhangzz earlier, I learned that there would be performance improvements, but I did not inquire specifically about the scenarios and models. Therefore, I believe there is a performance improvement. If the current test results meet your expectations, then there should be no doubt.

[lzhangzz]

For 7B models with ShareGPT_V3, attention takes roughly 1/3 of total GPU time (13B model should be similar). I guess 3% RPS improvement is significant enough.

Current dispatching strategy is to maximize bandwidth utilization when there is enough data. There are faster configs for the data distribution of ShareGPT_V3 (which have some other limitations).

[zhyncs]

For 7B models with ShareGPT_V3, attention takes roughly 1/3 of total GPU time (13B model should be similar). I guess 3% RPS improvement is significant enough.

Current dispatching strategy is to maximize bandwidth utilization when there is enough data. There are faster configs for the data distribution of ShareGPT_V3 (which have some other limitations).

make sense

[zhyncs]

Hi @lzhangzz, please merge the latest changes from the main branch to address the documentation workflow issue.