Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add llama.cpp backend #2723

Open
wants to merge 74 commits into
base: main
Choose a base branch
from
Open

Add llama.cpp backend #2723

wants to merge 74 commits into from
+1,716 −2

Commits on Nov 14, 2024

  1. feat(llamacpp): initial commit 

    # Conflicts:
#	Cargo.lock
    @mfuntowicz
    mfuntowicz committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    aa1fcba View commit details
    Browse the repository at this point in the history

  2. feat(llamacpp): correctly handle CMAKE_BUILD_TYPE for spdlog macros

    @mfuntowicz
    mfuntowicz committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    7d1f8a2 View commit details
    Browse the repository at this point in the history

  3. feat(llamacpp): initial end2end build

    @mfuntowicz
    mfuntowicz committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    52d57dc View commit details
    Browse the repository at this point in the history

  4. misc(cmake): add parameter to build specific cuda arch

    @mfuntowicz
    mfuntowicz committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    e4432d3 View commit details
    Browse the repository at this point in the history

  5. misc(cmake): wut

    @mfuntowicz
    mfuntowicz committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    fa89d1e View commit details
    Browse the repository at this point in the history

  6. feat(llamacpp): enable cuda

    @mfuntowicz
    mfuntowicz committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    05ad684 View commit details
    Browse the repository at this point in the history

  7. feat(backend): correctly load llama.cpp model from llama api and not … 

    …gpt2
    @mfuntowicz
    mfuntowicz committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    0911076 View commit details
    Browse the repository at this point in the history

  8. feat(backend): tell cmake to build llama-common and link to it

    @mfuntowicz
    mfuntowicz committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    098c669 View commit details
    Browse the repository at this point in the history

  9. feat(backend): add some initial decoding steps

    @mfuntowicz
    mfuntowicz committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    45d5a6a View commit details
    Browse the repository at this point in the history

  10. feat(backend): use llama_token as TokenId type

    @mfuntowicz
    mfuntowicz committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    92bb113 View commit details
    Browse the repository at this point in the history

  11. feat(backend): minor refactor

    @mfuntowicz
    mfuntowicz committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    d4b5be1 View commit details
    Browse the repository at this point in the history

  12. feat(backend): expose frequency and repetition penalties

    @mfuntowicz
    mfuntowicz committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    37faeb3 View commit details
    Browse the repository at this point in the history

  13. chore(backend): minor formatting

    @mfuntowicz
    mfuntowicz committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    f9c2486 View commit details
    Browse the repository at this point in the history

  14. feat(backend): wip Rust binding

    @mfuntowicz
    mfuntowicz committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    355d8a5 View commit details
    Browse the repository at this point in the history

  15. feat(backend): build and link through build.rs

    @mfuntowicz
    mfuntowicz committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    e4d803c View commit details
    Browse the repository at this point in the history

  16. misc(build): handle different lib destination folder lib/lib64

    @mfuntowicz
    mfuntowicz committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    f0859c2 View commit details
    Browse the repository at this point in the history

  17. misc(build): refactor build type detection in cmake

    @mfuntowicz
    mfuntowicz committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    179309b View commit details
    Browse the repository at this point in the history

  18. feat(llamacpp): expose number of threads for the backend when constru… 

    …cting the model
    @mfuntowicz
    mfuntowicz committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    a316c53 View commit details
    Browse the repository at this point in the history

  19. feat(llamacpp): wip explosion

    @mfuntowicz
    mfuntowicz committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    0c1dd0e View commit details
    Browse the repository at this point in the history

  20. misc(offline): link correctly

    @mfuntowicz
    mfuntowicz committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    dbc5b7a View commit details
    Browse the repository at this point in the history

  21. misc(offline): expose more parameters for generate

    @mfuntowicz
    mfuntowicz committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    6115904 View commit details
    Browse the repository at this point in the history

  22. feat(backend): entirely rewrite backend

    @mfuntowicz
    mfuntowicz committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    b98c635 View commit details
    Browse the repository at this point in the history

  23. misc(offline): update offline tester

    @mfuntowicz
    mfuntowicz committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    6a5f6b0 View commit details
    Browse the repository at this point in the history

  24. feat(backend): full rework of the backend internal to safer c++

    @mfuntowicz
    mfuntowicz committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    d52b4c4 View commit details
    Browse the repository at this point in the history

  25. misc(offline): match rework

    @mfuntowicz
    mfuntowicz committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    3af2c68 View commit details
    Browse the repository at this point in the history

  26. feat(backend): add mapping for ignore_eos_token stopping criteria

    @mfuntowicz
    mfuntowicz committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    f39edc7 View commit details
    Browse the repository at this point in the history

  27. feat(backend): add logit parameter in the callback fn

    @mfuntowicz
    mfuntowicz committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    d4aee42 View commit details
    Browse the repository at this point in the history

  28. feat(backend): bind incoming request to the server

    @mfuntowicz
    mfuntowicz committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    612f2f9 View commit details
    Browse the repository at this point in the history

  29. feat(backend): avoid dropping the boxed stream at the end of the call… 

    …back
    @mfuntowicz
    mfuntowicz committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    b50dcdd View commit details
    Browse the repository at this point in the history

  30. feat(backend): somewhat generates the final infer response

    @mfuntowicz
    mfuntowicz committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    3e82f14 View commit details
    Browse the repository at this point in the history

  31. feat(backend): fix invalid reference to ctx instead of context in rel… 

    …ease build
    @mfuntowicz
    mfuntowicz committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    bd8f0f1 View commit details
    Browse the repository at this point in the history

  32. feat(backend): correctly link to shared fmt and spdlog instead of static

    @mfuntowicz
    mfuntowicz committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    2cdfed9 View commit details
    Browse the repository at this point in the history

  33. chore: unsued variables

    @mfuntowicz
    mfuntowicz committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    86a2ae6 View commit details
    Browse the repository at this point in the history

  34. feat(backend): fix memory leaking on llama_sampler when the decode ends

    @mfuntowicz
    mfuntowicz committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    7b0a56f View commit details
    Browse the repository at this point in the history

  35. feat(backend): remove static from inner_fw visitor as it leads to inv… 

    …alid memory locations
    @mfuntowicz
    mfuntowicz committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    31d9254 View commit details
    Browse the repository at this point in the history

  36. misc(lint): make clippy happier

    @mfuntowicz
    mfuntowicz committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    188442f View commit details
    Browse the repository at this point in the history

  37. feat(backend): add number of generated tokens in the callback

    @mfuntowicz
    mfuntowicz committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    05ff551 View commit details
    Browse the repository at this point in the history

  38. feat(backend): correctly handle the max_new_tokens case for is_eos

    @mfuntowicz
    mfuntowicz committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    06424aa View commit details
    Browse the repository at this point in the history

  39. feat(backend): make eog clearer on c++ side

    @mfuntowicz
    mfuntowicz committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    11c593d View commit details
    Browse the repository at this point in the history

  40. feat(backend): refactor the callback to handle intermediate and end i… 

    …nference message
    @mfuntowicz
    mfuntowicz committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    5b7a951 View commit details
    Browse the repository at this point in the history

  41. misc(ffi): remove unused ffi mapping

    @mfuntowicz
    mfuntowicz committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    958c72a View commit details
    Browse the repository at this point in the history

  42. feat(backend): add early stopping criteria from TGI stream callback

    @mfuntowicz
    mfuntowicz committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    1473259 View commit details
    Browse the repository at this point in the history

  43. feat(backend): expose tokenizer to the GenerationContext to decode token

    @mfuntowicz
    mfuntowicz committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    1149186 View commit details
    Browse the repository at this point in the history

  44. misc(backend): decrease log verbosity in callback

    @mfuntowicz
    mfuntowicz committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    52208f5 View commit details
    Browse the repository at this point in the history

  45. misc(cmake): use url deps and not git repo

    @mfuntowicz
    mfuntowicz committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    62dba1a View commit details
    Browse the repository at this point in the history

  46. misc(backend): missing header <variant>

    @mfuntowicz
    mfuntowicz committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    5884218 View commit details
    Browse the repository at this point in the history

  47. feat(backend): avoid copy constructor

    @mfuntowicz
    mfuntowicz committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    a1154b1 View commit details
    Browse the repository at this point in the history

  48. chore(backend): minor fixes mostly format

    @mfuntowicz
    mfuntowicz committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    7eec0f7 View commit details
    Browse the repository at this point in the history

  49. feat(backend): dockerfile

    @mfuntowicz
    mfuntowicz committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    a7afde4 View commit details
    Browse the repository at this point in the history

  50. feat(dockerfile): build process

    @mfuntowicz
    mfuntowicz committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    2065282 View commit details
    Browse the repository at this point in the history

  51. feat(backend): handle all the tokenization failure and send back to t… 

    …he client
    @mfuntowicz
    mfuntowicz committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    26d0266 View commit details
    Browse the repository at this point in the history

  52. misc(cmake): remove dependency on fmt

    @mfuntowicz
    mfuntowicz committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    cf17928 View commit details
    Browse the repository at this point in the history

  53. misc(cmake): use URL base llama.cpp repo

    @mfuntowicz
    mfuntowicz committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    4f5397c View commit details
    Browse the repository at this point in the history

  54. feat(backend): simplify overall cpp structure

    @mfuntowicz
    mfuntowicz committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    86d30ae View commit details
    Browse the repository at this point in the history

  55. feat(backend): remove reinterpret_cast converting from uint32_t to ll… 

    …ama_token(int32_t)
    @mfuntowicz
    mfuntowicz committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    6915fa3 View commit details
    Browse the repository at this point in the history

  56. feat(backend): remove unused function

    @mfuntowicz
    mfuntowicz committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    7e2890f View commit details
    Browse the repository at this point in the history

  57. feat(backend): fix invalid reference to context in release mode

    @mfuntowicz
    mfuntowicz committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    488ba93 View commit details
    Browse the repository at this point in the history

  58. feat(backend): use std::ranges to map uint32_t to llama_token

    @mfuntowicz
    mfuntowicz committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    363d5e4 View commit details
    Browse the repository at this point in the history

  59. chore(backend): minor improvements

    @mfuntowicz
    mfuntowicz committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    02cd6fe View commit details
    Browse the repository at this point in the history

  60. dockerfile(backend): initial working version of llama.cpp container

    @mfuntowicz
    mfuntowicz committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    daf1631 View commit details
    Browse the repository at this point in the history

  61. feat(backend): simplify Rust callback

    @mfuntowicz
    mfuntowicz committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    57b2154 View commit details
    Browse the repository at this point in the history

  62. feat(backend): wrap Arc tokenizer to avoid duplicating

    @mfuntowicz
    mfuntowicz committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    6f059c4 View commit details
    Browse the repository at this point in the history

  63. feat(backend): update llamacpp to 4077

    @mfuntowicz
    mfuntowicz committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    70c90ad View commit details
    Browse the repository at this point in the history

  64. misc(build): improve build process

    @mfuntowicz
    mfuntowicz committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    23d2bcf View commit details
    Browse the repository at this point in the history

Commits on Nov 20, 2024

  1. feat(backend): multistream inference on CPU

    @mfuntowicz
    mfuntowicz committed Nov 20, 2024
    Configuration menu
    Copy the full SHA
    5335bf9 View commit details
    Browse the repository at this point in the history

Commits on Nov 21, 2024

  1. feat(backend): bind thread and memory affinity for thread

    @mfuntowicz
    mfuntowicz committed Nov 21, 2024
    Configuration menu
    Copy the full SHA
    50c3766 View commit details
    Browse the repository at this point in the history

  2. feat(backend): correctly setup llama_context providing n_threads and … 

    …n_ubatch
    @mfuntowicz
    mfuntowicz committed Nov 21, 2024
    Configuration menu
    Copy the full SHA
    84eead2 View commit details
    Browse the repository at this point in the history

Commits on Nov 22, 2024

  1. feat(backend): rely on multi consumer queue to scheduler workers

    @mfuntowicz
    mfuntowicz committed Nov 22, 2024
    Configuration menu
    Copy the full SHA
    5a85661 View commit details
    Browse the repository at this point in the history

  2. misc(docker): add numa lib as dependency

    @mfuntowicz
    mfuntowicz committed Nov 22, 2024
    Configuration menu
    Copy the full SHA
    30ae996 View commit details
    Browse the repository at this point in the history

  3. misc(backend): allow rebinding numa core affinity

    @mfuntowicz
    mfuntowicz committed Nov 22, 2024
    Configuration menu
    Copy the full SHA
    2d9465d View commit details
    Browse the repository at this point in the history

  4. misc(license): update LICENSE

    @mfuntowicz
    mfuntowicz committed Nov 22, 2024
    Configuration menu
    Copy the full SHA
    4ee2ee5 View commit details
    Browse the repository at this point in the history

  5. misc(doc): c++ documentation

    @mfuntowicz
    mfuntowicz committed Nov 22, 2024
    Configuration menu
    Copy the full SHA
    b9c04b9 View commit details
    Browse the repository at this point in the history

  6. misc(doc): rust documentation

    @mfuntowicz
    mfuntowicz committed Nov 22, 2024
    Configuration menu
    Copy the full SHA
    862a519 View commit details
    Browse the repository at this point in the history

  7. chore: remove unrelated change to trtllm

    @mfuntowicz
    mfuntowicz committed Nov 22, 2024
    Configuration menu
    Copy the full SHA
    9025a26 View commit details
    Browse the repository at this point in the history