[Feature]: SwiftKV cache compression

[pranjalst]

🚀 The feature, motivation and pitch

SwiftKV is a cache compression technique which reuses KV cache across layers with reduction in computation, latency and improving throughput. They already have a forked VLLM implementation at this repo.make it available to run on Gaudi hardware to test its speed.

Alternatives

No response

Additional context

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[PatrykWo]

Hi @pranjalst , thank you for the proposition. We will review that feature.