Code

Use kernels from the kernel hub#2988

Code

Merged

danieldk merged 25 commits intomainhuggingface/text-generation-inference:mainfrom kernel-hubhuggingface/text-generation-inference:kernel-hubCopy head branch name to clipboardFeb 10, 2025

+7,206-257

Commits

Commits on Feb 7, 2025

• Use Hub kernels for Marlin and cutlass quantization kernels

  danieldk

  committed
  aab6141

  Copy full SHA for aab6141

• Use hub kernels for MoE/GPTQ-Marlin MoE

  danieldk

  committed
  758ff3c

  Copy full SHA for 758ff3c

• Use attention kernels from the Hub

  danieldk

  committed
  b267caa

  Copy full SHA for b267caa

• Cache the kernels in the Docker image

  danieldk

  committed
  c9191f3

  Copy full SHA for c9191f3

• Update moe kernels

  danieldk

  committed
  b35ab54

  Copy full SHA for b35ab54

• Support loading local kernels for development

  danieldk

  committed
  d39f896

  Copy full SHA for d39f896

• Support latest moe kernels

  danieldk

  committed
  c1a564e

  Copy full SHA for c1a564e

• Update to moe 0.1.1

  danieldk

  committed
  dcb3731

  Copy full SHA for dcb3731

• CI: download locked kernels for server tests

  danieldk

  committed
  a60d1e6

  Copy full SHA for a60d1e6

• Fixup some imports

  danieldk

  committed
  f25a7aa

  Copy full SHA for f25a7aa

• CI: activate venv

  danieldk

  committed
  00af6ef

  Copy full SHA for 00af6ef

• Fix unused imports

  danieldk

  committed
  ca1067f

  Copy full SHA for ca1067f

• Nix: add attention/moe/quantization kernels

  danieldk

  committed
  4c8ced2

  Copy full SHA for 4c8ced2

• Update hf-kernels to 0.1.5

  danieldk

  committed
  e038497

  Copy full SHA for e038497

• Update kernels

  danieldk

  committed
  520420a

  Copy full SHA for 520420a

• Update tgi-nix flake for hf-kernels

  danieldk

  committed
  371668e

  Copy full SHA for 371668e

• Fix EOF

  danieldk

  committed
  875ce6d

  Copy full SHA for 875ce6d

• Take load_kernel out of a frequently-called function

  danieldk

  committed
  f74a50d

  Copy full SHA for f74a50d

• Hoist another case of kernel loading out of a somewhat hot function

  danieldk

  committed
  8aecc59

  Copy full SHA for 8aecc59

• marlin-kernels -> quantization

  danieldk

  committed
  8ad383c

  Copy full SHA for 8ad383c

• attention -> paged-attention

  danieldk

  committed
  96a4d4d

  Copy full SHA for 96a4d4d

• EOF fix

  danieldk

  committed
  219b8b1

  Copy full SHA for 219b8b1

• Update hf-kernels, fixup Docker

  danieldk

  committed
  8ae7bc3

  Copy full SHA for 8ae7bc3

• ipex fix

  danieldk

  committed
  df582a1

  Copy full SHA for df582a1

• Remove outdated TODO

  danieldk

  committed
  fc3ac80

  Copy full SHA for fc3ac80