Skip to content

Latest commit

 

History

History
126 lines (89 loc) · 4.64 KB

llama4.md

File metadata and controls

126 lines (89 loc) · 4.64 KB

🦙 Tutorial: LLaMA 4 Multi-Concurrency Support with KTransformers (Balance Serve Backend)

📌 Overview

We are pleased to announce that KTransformers now provides experimental support for LLaMA 4 models through the powerful balance_serve backend introduced in v0.2.4. This update is available under the dedicated development branch: support-llama4, specifically targeting the newly released Meta LLaMA 4 model architecture.

⚠️ This support is currently not available on the main branch due to dependencies on newer versions of transformers, and compatibility limitations with inference of currently supported models. Work is underway to integrate this into the mainline once broader stability and compatibility are validated.

💡 If you already have an environment based on the main branch, it is strongly recommended to create a new environment to avoid potential dependency conflicts.

🔗 Model & Resource Links

🧪 Demo

2025-04-09.16-52-43.mov

Resource Requirements

The Scout model running with 16 Experts requires approximately 65 GB of memory and 10 GB of GPU memory, while the Maverick model with 128 Experts requires approximately 270 GB of memory and 12 GB of GPU memory.

⚙️ Usage Instructions

1. Clone support-llama4 Branch

git clone https://github.com/kvcache-ai/ktransformers.git
cd ktransformers
git checkout support-llama4
git submodule update --init --recursive

2. Set Up Environment

# Download Miniconda
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh

# Create environment
conda create --name ktransformers python=3.11
conda activate ktransformers

# Install required libraries
conda install -c conda-forge libstdcxx-ng

# Verify GLIBCXX version (should include 3.4.32)
strings ~/anaconda3/envs/ktransformers/lib/libstdc++.so.6 | grep GLIBCXX

sudo apt install libtbb-dev libssl-dev libcurl4-openssl-dev libaio1 libaio-dev libfmt-dev libgflags-dev zlib1g-dev patchelf
pip3 install packaging ninja cpufeature numpy openai
pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu126

3. Build with Balance Serve Support

# Install single NUMA dependencies
USE_BALANCE_SERVE=1  bash ./install.sh
# For those who have two cpu and 1T RAM（Dual NUMA）:
USE_BALANCE_SERVE=1 USE_NUMA=1 bash ./install.sh

4. Use our custom config.json

Currently, you need to copy the content of our custom config file into the config.json under your --model_path.

Please make sure to replace the content of config.json with the appropriate one accordingly.

5. Run LLaMA 4 Inference Server

Make sure you have:

  • --model_path pointing to a local config directory (not a Hugging Face name).
  • --gguf_path pointing to the folder containing quantized .gguf weights.
python ktransformers/server/main.py \
  --port 10002 \
  --model_path <path_to_safetensor_config> \
  --gguf_path <path_to_gguf_files> \
  --optimize_config_path ktransformers/optimize/optimize_rules/Llama4-serve.yaml \
  --max_new_tokens 1024 \
  --cache_lens 32768 \
  --chunk_size 256 \
  --max_batch_size 4 \
  --backend_type balance_serve \

5. Access server

curl -X POST http://localhost:10002/v1/chat/completions \
  -H "accept: application/json" \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      {"role": "user", "content": "hello"}
    ],
    "model": "Llama4",
    "temperature": 0.3,
    "top_p": 1.0,
    "stream": true
  }'

📌 Limitations

  • ✅ Only balance_serve backend is supported for LLaMA 4 models in this version.
  • ⚠️ Requires transformers==4.51.0 or newer. Due to potential compatibility issues with older toolchains, we have not merged this branch to main yet.
  • ❌ Multimodal models are not supported yet in this version. Support will be added in future releases.