Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add llama.cpp backend #2723

Open
wants to merge 74 commits into
base: main
Choose a base branch
from
Open

Add llama.cpp backend #2723

wants to merge 74 commits into from

Conversation

mfuntowicz
Copy link
Member

@mfuntowicz mfuntowicz commented Nov 4, 2024 •
edited
Loading

This PR is an initial implementation of llama.cpp as potential backend for TGI.

It mostly targets CPU inference in a single/multi stream scheduling fashion, potentially spawning multiple instances of the same model over a non-overlapping subset of the CPU cores.

The current implementation only allows a single request to be running on a working, this constraint will be removed later on.
The current implementation also dupplicate the weights for each worker, this constraint can potentially be removed later on.

@mfuntowicz
feat(llamacpp): initial commit
aa1fcba 
# Conflicts:
#	Cargo.lock
@mfuntowicz
feat(llamacpp): correctly handle CMAKE_BUILD_TYPE for spdlog macros
7d1f8a2
@mfuntowicz
feat(llamacpp): initial end2end build
52d57dc
@mfuntowicz
misc(cmake): add parameter to build specific cuda arch
e4432d3
@mfuntowicz
misc(cmake): wut
fa89d1e
@mfuntowicz
feat(llamacpp): enable cuda
05ad684
@mfuntowicz
feat(backend): correctly load llama.cpp model from llama api and not …
0911076 
…gpt2
@mfuntowicz
feat(backend): tell cmake to build llama-common and link to it
098c669
@mfuntowicz
feat(backend): add some initial decoding steps
45d5a6a
@mfuntowicz
feat(backend): use llama_token as TokenId type
92bb113
@mfuntowicz
feat(backend): minor refactor
d4b5be1
@mfuntowicz
feat(backend): expose frequency and repetition penalties
37faeb3
@mfuntowicz
chore(backend): minor formatting
f9c2486
@mfuntowicz
feat(backend): wip Rust binding
355d8a5
@mfuntowicz
feat(backend): build and link through build.rs
e4d803c
@mfuntowicz
misc(build): handle different lib destination folder lib/lib64
f0859c2
@mfuntowicz
misc(build): refactor build type detection in cmake
179309b
@mfuntowicz
feat(llamacpp): expose number of threads for the backend when constru…
a316c53 
…cting the model
@mfuntowicz
feat(llamacpp): wip explosion
0c1dd0e
@mfuntowicz
misc(offline): link correctly
dbc5b7a
@mfuntowicz
misc(offline): expose more parameters for generate
6115904
@mfuntowicz
feat(backend): entirely rewrite backend
b98c635
@mfuntowicz
misc(offline): update offline tester
6a5f6b0
@mfuntowicz
feat(backend): full rework of the backend internal to safer c++
d52b4c4
@mfuntowicz
misc(offline): match rework
3af2c68
@mfuntowicz
feat(backend): add mapping for ignore_eos_token stopping criteria
f39edc7
@mfuntowicz
feat(backend): add logit parameter in the callback fn
d4aee42
@mfuntowicz
feat(backend): bind incoming request to the server
612f2f9
@mfuntowicz
feat(backend): avoid dropping the boxed stream at the end of the call…
b50dcdd 
…back
@mfuntowicz
feat(backend): somewhat generates the final infer response
3e82f14
@mfuntowicz mfuntowicz marked this pull request as ready for review November 22, 2024 13:47
@mfuntowicz mfuntowicz requested a review from angt November 22, 2024 15:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@co42 co42 Awaiting requested review from co42

@Hugoch Hugoch Awaiting requested review from Hugoch

@OlivierDehaene OlivierDehaene Awaiting requested review from OlivierDehaene

@angt angt Awaiting requested review from angt

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@mfuntowicz