Skip to content

Commit

Permalink
[Docs] Clarifying the number of additional model (#2841)
Browse files Browse the repository at this point in the history
This PR updates the documentation for the clarity of the supported number
of additional models. As of now we only support one
additional model for speculative decoding
  • Loading branch information
MasterJH5574 authored Aug 22, 2024
1 parent fd06a42 commit c5dbae2
Showing 1 changed file with 4 additions and 3 deletions.
7 changes: 4 additions & 3 deletions docs/deploy/rest.rst
Original file line number Diff line number Diff line change
Expand Up @@ -129,9 +129,10 @@ MODEL The model folder after compiling with MLC-LLM build proce
than the main model).
When engine is enabled with speculative decoding, additional models are needed.
The way of specifying additional models is:
``--additional-models model_path_1 model_path_2 ...`` or
``--additional-models model_path_1,model_lib_1 model_path_2 ...``.
**We only support one additional model for speculative decoding now.**
The way of specifying the additional model is:
``--additional-models model_path_1`` or
``--additional-models model_path_1,model_lib_1``.
When the model lib of a model is not given, JIT model compilation will be activated
to compile the model automatically.
Expand Down

0 comments on commit c5dbae2

Please sign in to comment.