Added Streaming Capability to SageMaker LLMs

[dazajuandaniel]

This PR adds the ability to declare a Streaming response in the SageMaker LLM by leveraging the invoke_endpoint_with_response_stream capability in boto3. It is heavily based on the AWS Blog Post announcement linked here.

It does not add any additional dependencies since it uses the existing boto3 version.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
langchain	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Oct 5, 2023 6:14pm

[baskaryan]

cc @3coins

[3coins]

@dazajuandaniel
This is awesome! Thanks for adding this. I don't have a model setup to test, but code looks good. 🚀

One minor suggestion for a follow up PR, would be nice to move the streaming/non-streaming parts to separate private functions for better readability.

[dazajuandaniel]

@dazajuandaniel This is awesome! Thanks for adding this. I don't have a model setup to test, but code looks good. 🚀

One minor suggestion for a follow up PR, would be nice to move the streaming/non-streaming parts to separate private functions for better readability.

Hey @3coins,
Thanks for approving this PR!

Yes, I agree. I am happy to work on the follow up PR, no issues. I tried to follow the pattern that replicate has.

[dazajuandaniel]

@baskaryan - please let me know if anything is missing :)

[baskaryan]

lgtm aside from linting issues. would also be nice to factor out stream implementation into the _stream method defined on parent class, but can also do that in follow up pr

[pinak-p]

@dazajuandaniel, @3coins - Is this good to be merged ?

[3coins]

@baskaryan
This has been open for a while, is there anything needed to merge this?

[dazajuandaniel]

@pinak-p I was missing a mypy fix, I've done it now

[baskaryan]

sorry for the delay, thank you @dazajuandaniel!

[jay2jp]

Is there any documentation on this? I cant seem to find any. @dazajuandaniel