Faster than llama.cpp on CUDA

[EricLBuehler]

To reproduce with mistral.rs:

cargo run --release --features cuda -- --log output.txt --port 1234 gguf -t mistralai/Mistral-7B-Instruct-v0.1 -m TheBloke/Mistral-7B-Instruct-v0.1-GGUF -f mistral-7b-instruct-v0.1.Q4_K_M.gguf

python3 example/server/chat.py

To reproduce with llama.cpp

make GGML_CUDA=1
./llama-cli -m mistral-7b-instruct-v0.1.Q4_K_M.gguf -ngl 33 -n 1024 -p "Graphene is the best"

Mistral 7b Q4K medium, BS=1

Hardware	mistral.rs T/s (decode)	llama.cpp (decode)
A10	86	82
A100	127	137
A6000	113	100
RTX6000	103	96

[Skarian]

Have any benchmarks been ran for hybrid inference (CPU + partial offload to GPU)?