feat: Update AmazonBedrockChatGenerator to use Converse API (BREAKING CHANGE) #1219

vblagoje · 2024-11-26T14:03:33Z

Enhances the integration with Amazon Bedrock by using the new Converse API in AmazonBedrockChatGenerator. The modifications address the removal of outdated adapter structures and simplifies code greatly all while maintaining compatibility as much as possible.

What:

Removed the adapter file: The adapters.py file was deleted to eliminate deprecated structures related to chat model adapters.
Updated chat_generator.py:
- Improved the model selection process and model ID usage.
- Revised how system prompts and messages are prepared for sending to the models.
- Added support for streaming responses and tool configurations in model requests.
Enhanced tests: Significant updates in test_chat_generator.py to include new model tests and validate functionality with the new configurations and expected responses.

How can it be used:

The system now supports:
- Multiple LLMs from Amazon Bedrock with configurations for model-specific parameters.
- Streaming responses during chat interactions.

Example code snippet to run a chat generator:

from haystack_integrations.components.generators.amazon_bedrock import AmazonBedrockChatGenerator

client = AmazonBedrockChatGenerator(model="anthropic.claude-3-5-sonnet-20240620-v1:0")
response = client.run(
    messages=[ChatMessage.from_user("What's Natural Language Processing?")],
    generation_kwargs={"maxTokens": 512}
)

How did you test it:

Conducted integration tests that focused on:

Ensuring the correct integration of the AmazonBedrockChatGenerator with various model configurations.
Validating the model's ability to handle both standard and streaming requests.
Mocked AWS session interactions to simulate real response cycles using the updated chat models.

Notes for the reviewer:

Attention should be given to changes affecting model naming conventions and interaction patterns, especially regarding tool usage in the LLM context.
Review the removal of the adapters.py file to understand how changes in architecture affect existing functionalities.

Breaking Changes

The truncate initialization parameter has been removed from AmazonBedrockChatGenerator

Migration Steps:

Remove any truncate parameter from component initialization.

vblagoje · 2024-11-26T14:13:26Z

@julian-risch @silvanocerza do you remember who was giving us permissions for Amazon Bedrock models, was it @askainet ? We need to ping him to click through all these Amazon Bedrock models in this PR so they become available on our Haystack account...

vblagoje · 2024-11-26T14:14:39Z

integrations/amazon_bedrock/tests/test_chat_generator.py

+
+# so far we've discovered these models support streaming and tool use
+STREAMING_TOOL_MODELS = ["anthropic.claude-3-5-sonnet-20240620-v1:0", "cohere.command-r-plus-v1:0"]
+


These models in MODELS_TO_TEST, MODELS_TO_TEST_WITH_TOOLS and STREAMING_TOOL_MODELS @askainet 🙏 🙏

Paul set us up 🚀

vblagoje · 2024-11-27T13:38:19Z

@anakin87 would you please give it a rough pass first to make sure I'm not forgetting any major parts.

anakin87

I had a cursory look and I like the change and the approach, also because we are deleting a lot of code :-)

Ping me when you need a more in-depth review.

anakin87

I am happy with this refactoring. (Just left some minor comments)

One relevant aspect I noticed involves truncation - we removed it and I am OK with it but we should:

remove the truncate parameter
clearly communicate this in the PR title (which is used for changelog generation)
release this breaking change in a major version once this PR has been merged

anakin87 · 2024-12-09T15:05:25Z

...edrock/src/haystack_integrations/components/generators/amazon_bedrock/chat/chat_generator.py

+
+        messages_list = [{"role": msg.role.value, "content": [{"text": msg.content}]} for msg in messages]
+
+        try:


we can move try closer to the actual client usage
(after callback = streaming_callback or self.streaming_callback)

anakin87 · 2024-12-09T15:20:01Z

...edrock/src/haystack_integrations/components/generators/amazon_bedrock/chat/chat_generator.py

+
+        return {"replies": replies}
+
+    def extract_replies_from_response(self, response_body: Dict[str, Any]) -> List[ChatMessage]:


I think we can easily add an unit test for this method

anakin87 · 2024-12-09T15:21:02Z

...edrock/src/haystack_integrations/components/generators/amazon_bedrock/chat/chat_generator.py

+                        )
+        return replies
+
+    def process_streaming_response(


I would also like to see a unit test for this method, but IDK how tricky it is, so I'll leave it up to you

vblagoje · 2024-12-10T14:10:04Z

Addressed your concerns @anakin87
Not 100% sure about:

from botocore.eventstream import EventStream

direct usage to improve clarity but since we already import botocore I though it was ok - look like stable API. LMK your thoughts here and please have another look!

anakin87

LGTM!

Please incorporate the suggested changes to align with #1236: in short, text has been introduced to smooth transition to new ChatMessage in future but currently it just mirrors content.

Once this PR is merged, please release 2.0.0

...edrock/src/haystack_integrations/components/generators/amazon_bedrock/chat/chat_generator.py

…ts/generators/amazon_bedrock/chat/chat_generator.py Co-authored-by: Stefano Fiorucci <[email protected]>

github-actions bot added integration:amazon-bedrock type:documentation Improvements or additions to documentation labels Nov 26, 2024

vblagoje commented Nov 26, 2024

View reviewed changes

vblagoje marked this pull request as ready for review November 27, 2024 13:37

vblagoje requested a review from a team as a code owner November 27, 2024 13:37

vblagoje requested review from anakin87 and removed request for a team November 27, 2024 13:37

anakin87 reviewed Nov 27, 2024

View reviewed changes

vblagoje added 7 commits December 5, 2024 15:20

Initial commit

1c9b710

Update models tested

35da407

Add tool support

a0a1350

Update Amazon Bedrock model names in tests

d9babb9

Support for tool streaming

2511db3

Format

88bf808

Minot test updates

bdf6c34

vblagoje force-pushed the amazon_bedrock_converse branch from f309b60 to bdf6c34 Compare December 5, 2024 14:25

Lint

6338e74

anakin87 requested changes Dec 9, 2024

View reviewed changes

vblagoje changed the title ~~feat: Update AmazonBedrockChatGenerator to use Converse API~~ feat: Update AmazonBedrockChatGenerator to use Converse API (BREAKING CHANGE) Dec 10, 2024

vblagoje added 7 commits December 10, 2024 14:40

Remove truncate init parameter

fe17cfb

Pull try down

39787b9

Add extract_replies_from_response unit test

2adb50b

Add process_streaming_response unit test

c7f51a1

Lint

aa7c5c0

Small test fix

405865b

Use EventStream from botocore

3990dfd

vblagoje requested a review from anakin87 December 10, 2024 14:10

anakin87 approved these changes Dec 10, 2024

View reviewed changes

...edrock/src/haystack_integrations/components/generators/amazon_bedrock/chat/chat_generator.py Outdated Show resolved Hide resolved

...edrock/src/haystack_integrations/components/generators/amazon_bedrock/chat/chat_generator.py Outdated Show resolved Hide resolved

vblagoje and others added 3 commits December 10, 2024 15:21

Update integrations/amazon_bedrock/src/haystack_integrations/componen…

0a318d1

…ts/generators/amazon_bedrock/chat/chat_generator.py Co-authored-by: Stefano Fiorucci <[email protected]>

Update integrations/amazon_bedrock/src/haystack_integrations/componen…

a16d3d7

…ts/generators/amazon_bedrock/chat/chat_generator.py Co-authored-by: Stefano Fiorucci <[email protected]>

Merge branch 'main' into amazon_bedrock_converse

78617a0

vblagoje merged commit df14a97 into main Dec 10, 2024
10 checks passed

vblagoje deleted the amazon_bedrock_converse branch December 10, 2024 14:40

vblagoje mentioned this pull request Dec 10, 2024

Bedrock Converse API integration #977

Closed

anakin87 mentioned this pull request Dec 10, 2024

feat: bedrock converse #1061

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Update AmazonBedrockChatGenerator to use Converse API (BREAKING CHANGE) #1219

feat: Update AmazonBedrockChatGenerator to use Converse API (BREAKING CHANGE) #1219

vblagoje commented Nov 26, 2024 •

edited

Loading

vblagoje commented Nov 26, 2024

vblagoje Nov 26, 2024

vblagoje Nov 27, 2024

vblagoje commented Nov 27, 2024

anakin87 left a comment

anakin87 left a comment

anakin87 Dec 9, 2024

anakin87 Dec 9, 2024

anakin87 Dec 9, 2024

vblagoje commented Dec 10, 2024

anakin87 left a comment


		# so far we've discovered these models support streaming and tool use
		STREAMING_TOOL_MODELS = ["anthropic.claude-3-5-sonnet-20240620-v1:0", "cohere.command-r-plus-v1:0"]


		messages_list = [{"role": msg.role.value, "content": [{"text": msg.content}]} for msg in messages]

		try:


		return {"replies": replies}

		def extract_replies_from_response(self, response_body: Dict[str, Any]) -> List[ChatMessage]:

feat: Update AmazonBedrockChatGenerator to use Converse API (BREAKING CHANGE) #1219

feat: Update AmazonBedrockChatGenerator to use Converse API (BREAKING CHANGE) #1219

Conversation

vblagoje commented Nov 26, 2024 • edited Loading

What:

How can it be used:

How did you test it:

Notes for the reviewer:

Breaking Changes

Migration Steps:

vblagoje commented Nov 26, 2024

vblagoje Nov 26, 2024

Choose a reason for hiding this comment

vblagoje Nov 27, 2024

Choose a reason for hiding this comment

vblagoje commented Nov 27, 2024

anakin87 left a comment

Choose a reason for hiding this comment

anakin87 left a comment

Choose a reason for hiding this comment

anakin87 Dec 9, 2024

Choose a reason for hiding this comment

anakin87 Dec 9, 2024

Choose a reason for hiding this comment

anakin87 Dec 9, 2024

Choose a reason for hiding this comment

vblagoje commented Dec 10, 2024

anakin87 left a comment

Choose a reason for hiding this comment

vblagoje commented Nov 26, 2024 •

edited

Loading