CPU accelerate solution

[Aisuko]

Hugging Face Transformers load gguf into fp32

After I check the code. The Hugging Face Transformers support gguf isn't what we think. According to the PR. This PR offers the ability to load .gguf files within transformers, dequantizing them to float32. This isn't we want to. So, we need to move on to the other solutions.

Based on the currently researching. Here are two paths.

• Python binding

• Llama.cpp server

There are also have some other bindings. However, I use Python as backend. So, we want to use Python binding. Furthermore, here is my thinking.

We can import Python binding through kimchima. And support CPU accelerate. And we also what already done. This is breaking change, because we will use Llama.cpp rather than transformers. And currently, we need to do investigation on how to support our Conversation history mechanism.

We can import Llama.cpp server as a seperate stateless micro-service. Our backend can be an orchestration system. It means we can also let kimchima become a micro-service.

• https://github.com/SkywardAI/llama.cpp
• https://hub.docker.com/r/gclub/llama.cpp/tags

Reference

• https://jalammar.github.io/illustrated-gpt2/
• https://huggingface.co/docs/transformers/v4.41.3/gguf
• https://huggingface.co/docs/hub/en/gguf#quantization-types
• Loading GGUF files support huggingface/transformers#30391
• https://github.com/abetlen/llama-cpp-python