[Community CTransformers] Ability to use stream and astream #19262
QuentinBriand
started this conversation in
Ideas
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Checked
Feature request
The feature is quite simple. Actually the CTransformers class doesn't integrate the methods _stream and _astream, what it does actually is switching silently (it seems weird for me that there isn't a proper log on the base LLM class to warn for such switches) to the invoke method.
Motivation
For business purpose i would want to be able to load gguf files with CTransformers while still being able to stream the responses via Langserve automatically or via custom SSE's and not get only a huge block at the end.
Proposal (If applicable)
I already added both of the functions into my own scripts but i don't know if i should post the code embed here as is.
Beta Was this translation helpful? Give feedback.
All reactions