[benchmark] Add fused_moe_triton benchmark and tuning tools

[BBuf]

The tuning scripts is related to this pr, and it successfull produced efficient fused_moe_triton kernel config for qwen2-57b and mixtral 8x7b in both tp4 fp8_w8a8 condition.

In GTX 4090, I have checked benchmark script in Qwen/Qwen2-57B-A14B-Instruct-FP8 model's fused_moe_triton kernel, the result:

python benchmark/kernels/fused_moe_triton/fused_moe_triton/benchmark_vllm_vs_sglang_fused_moe_triton.py --model /mnt/bbuf/Qwen2-57B-A14B-Instruct-FP8 --tp-size 4

python benchmark/kernels/fused_moe_triton/fused_moe_triton/benchmark_vllm_vs_sglang_fused_moe_triton.py --model /mnt/bbuf/Qwen2-57B-A14B-Instruct-FP8 --tp-size 4 --use-fp8

[BBuf]

@merrymercy The pr need review, thanks.

[HaiShaw]

@BBuf
Thanks for your great work!
Can you also refer to https://github.com/sgl-project/sglang/blob/main/3rdparty/amd/tuning/benchmark_moe_rocm.py?
One goal is to come up with an uniformed tuning scripts.
But if you can't do in this PR due to access limit, probably fine, we can try to unify later too.

[BBuf]

@BBuf Thanks for your great work! Can you also refer to https://github.com/sgl-project/sglang/blob/main/3rdparty/amd/tuning/benchmark_moe_rocm.py? One goal is to come up with an uniformed tuning scripts. But if you can't do in this PR due to access limit, probably fine, we can try to unify later too.

Alright, looking at the diffs between the two tuning scripts, there seem to be much differences. If you want to unify them, you can work on that. Thanks!

[BBuf]

@HaiShaw I had fix your comment, have another look when you are free, thanks.

[HaiShaw]

@BBuf Thanks for your PR!

[HaiShaw]

@merrymercy please have a look too.