Skip to content

Commit

Permalink
docs: replace bedrock docs page with cookbook (#850)
Browse files Browse the repository at this point in the history
  • Loading branch information
@marcklingen
marcklingen authored Oct 11, 2024
1 parent 01909b1 commit 8f15524
Show file tree
Hide file tree
Showing 6 changed files with 790 additions and 37 deletions.
4 changes: 4 additions & 0 deletions cookbook/_routes.json
View file
Original file line number Diff line number Diff line change
Expand Up @@ -142,5 +142,9 @@
{
"notebook": "example_intent_classification_pipeline.ipynb",
"docsPath": "docs/analytics/example-intent-classification"
},
{
"notebook": "integration_amazon_bedrock.ipynb",
"docsPath": "docs/integrations/amazon-bedrock"
}
]
348 changes: 348 additions & 0 deletions cookbook/integration_amazon_bedrock.ipynb
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,348 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "OY7q37b9OY5z"
},
"source": [
"---\n",
"title: Observability and Metrics for Amazon Bedrock\n",
"description: Open source observability for Amazon Bedrock applications and the Bedrock SDK.\n",
"---\n",
"\n",
"# Amazon Bedrock Integration\n",
"\n",
"[**Amazon Bedrock**](https://aws.amazon.com/bedrock/) is a fully managed AWS service that lets you use foundation models and custom models to generate text, images, and audio.\n",
"\n",
"When **using Langfuse with Amazon Bedrock**, you can easily capture [detailed traces](https://langfuse.com/docs/tracing) and metrics for every request, giving you insights into the performance and behavior of your application.\n",
"\n",
"## Integration Options\n",
"\n",
"There are a few ways through which you can capture traces and metrics for Amazon Bedrock:\n",
"\n",
"1. via an application framework that is integrated with Langfuse:\n",
"\n",
" - [Langchain](https://langfuse.com/docs/integrations/langchain)\n",
" - [Llama Index](https://langfuse.com/docs/integrations/llama-index)\n",
" - [Haystack](https://langfuse.com/docs/integrations/haystack)\n",
"\n",
"2. via a Proxy such as [LiteLLM](https://langfuse.com/docs/integrations/litellm)\n",
"3. via wrapping the Bedrock SDK with the [Langfuse Decorator](https://langfuse.com/docs/sdk/python/decorators) (_see example below_)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Xyhx0DZlO-nq"
},
"source": [
"## How to wrap Amazon Bedrock SDK (Converse API)"
]
},
{
"cell_type": "code",
"execution_count": 96,
"metadata": {
"id": "5r6s6RZ6odLf"
},
"outputs": [],
"source": [
"# install requirements\n",
"%pip install boto3 langfuse awscli --quiet"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ebl_A9fyuBsK"
},
"source": [
"### Authenticate AWS Session\n",
"\n",
"Sign in with your AWS Role that has access to Amazon Bedrock."
]
},
{
"cell_type": "code",
"execution_count": 103,
"metadata": {
"id": "Gv3syLHVvKGZ"
},
"outputs": [],
"source": [
"AWS_ACCESS_KEY_ID=\"***\"\n",
"AWS_SECRET_ACCESS_KEY=\"***\"\n",
"AWS_SESSION_TOKEN=\"***\"\n",
"\n",
"import boto3\n",
"\n",
"# used to access Bedrock configuration\n",
"bedrock = boto3.client(\n",
" service_name=\"bedrock\",\n",
" region_name=\"eu-west-1\",\n",
" aws_access_key_id=AWS_ACCESS_KEY_ID,\n",
" aws_secret_access_key=AWS_SECRET_ACCESS_KEY,\n",
" aws_session_token=AWS_SESSION_TOKEN\n",
")\n",
"\n",
"# used to invoke the Bedrock Converse API\n",
"bedrock_runtime = boto3.client(\n",
" service_name=\"bedrock-runtime\",\n",
" region_name=\"eu-west-1\",\n",
" aws_access_key_id=AWS_ACCESS_KEY_ID,\n",
" aws_secret_access_key=AWS_SECRET_ACCESS_KEY,\n",
" aws_session_token=AWS_SESSION_TOKEN\n",
")"
]
},
{
"cell_type": "code",
"execution_count": 104,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "-WI3Resu4eqE",
"outputId": "6004b21f-b05f-4c20-b465-40b081b9d947"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"EU Anthropic Claude 3 Sonnet - eu.anthropic.claude-3-sonnet-20240229-v1:0\n",
"EU Anthropic Claude 3 Haiku - eu.anthropic.claude-3-haiku-20240307-v1:0\n",
"EU Anthropic Claude 3.5 Sonnet - eu.anthropic.claude-3-5-sonnet-20240620-v1:0\n",
"EU Meta Llama 3.2 3B Instruct - eu.meta.llama3-2-3b-instruct-v1:0\n",
"EU Meta Llama 3.2 1B Instruct - eu.meta.llama3-2-1b-instruct-v1:0\n"
]
}
],
"source": [
"# Check which models are available in your account\n",
"models = bedrock.list_inference_profiles()\n",
"for model in models[\"inferenceProfileSummaries\"]:\n",
" print(model[\"inferenceProfileName\"] + \" - \" + model[\"inferenceProfileId\"])"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "0Te0Xfs6Enll"
},
"source": [
"### Set Langfuse Credentials"
]
},
{
"cell_type": "code",
"execution_count": 44,
"metadata": {
"id": "hv_Zj31HEkqv"
},
"outputs": [],
"source": [
"import os\n",
"\n",
"# Get keys for your project from the project settings page\n",
"# https://cloud.langfuse.com\n",
"os.environ[\"LANGFUSE_PUBLIC_KEY\"] = \"\"\n",
"os.environ[\"LANGFUSE_SECRET_KEY\"] = \"\"\n",
"os.environ[\"LANGFUSE_HOST\"] = \"https://cloud.langfuse.com\" # 🇪🇺 EU region\n",
"# os.environ[\"LANGFUSE_HOST\"] = \"https://us.cloud.langfuse.com\" # 🇺🇸 US region\n",
"\n",
"# Your openai key\n",
"os.environ[\"OPENAI_API_KEY\"] = \"\""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "GZXzAeJL5sCB"
},
"source": [
"### Wrap Bedrock SDK"
]
},
{
"cell_type": "code",
"execution_count": 88,
"metadata": {
"id": "U9KeL8J5ETGn"
},
"outputs": [],
"source": [
"from langfuse.decorators import observe, langfuse_context\n",
"from botocore.exceptions import ClientError\n",
"\n",
"@observe(as_type=\"generation\", name=\"Bedrock Converse\")\n",
"def wrapped_bedrock_converse(**kwargs):\n",
" # 1. extract model metadata\n",
" kwargs_clone = kwargs.copy()\n",
" input = kwargs_clone.pop('messages', None)\n",
" modelId = kwargs_clone.pop('modelId', None)\n",
" model_parameters = {\n",
" **kwargs_clone.pop('inferenceConfig', {}),\n",
" **kwargs_clone.pop('additionalModelRequestFields', {})\n",
" }\n",
" langfuse_context.update_current_observation(\n",
" input=input,\n",
" model=modelId,\n",
" model_parameters=model_parameters,\n",
" metadata=kwargs_clone\n",
" )\n",
"\n",
" # 2. model call with error handling\n",
" try:\n",
" response = bedrock_runtime.converse(**kwargs)\n",
" except (ClientError, Exception) as e:\n",
" error_message = f\"ERROR: Can't invoke '{modelId}'. Reason: {e}\"\n",
" langfuse_context.update_current_observation(level=\"ERROR\", status_message=error_message)\n",
" print(error_message)\n",
" return\n",
"\n",
" # 3. extract response metadata\n",
" response_text = response[\"output\"][\"message\"][\"content\"][0][\"text\"]\n",
" langfuse_context.update_current_observation(\n",
" output=response_text,\n",
" usage={\n",
" \"input\": response[\"usage\"][\"inputTokens\"],\n",
" \"output\": response[\"usage\"][\"outputTokens\"],\n",
" \"total\": response[\"usage\"][\"totalTokens\"]\n",
" },\n",
" metadata={\n",
" \"ResponseMetadata\": response[\"ResponseMetadata\"],\n",
" }\n",
" )\n",
"\n",
" return response_text"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "SAa-Sp2BK4H5"
},
"source": [
"### Run Example"
]
},
{
"cell_type": "code",
"execution_count": 105,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "zBTNm3iOLd8v",
"outputId": "d39d0b74-e011-47d1-cf8a-a21211fb9de9"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Anthropic\n",
"Understood. I'll continue to act as Alex, the AI personal finance advisor from SmartFinance Advisors, maintaining that character throughout our interaction. I'll provide financial advice and guidance based on the user's questions and needs. If I'm unsure about something, I'll ask for clarification as instructed. How may I assist you with your financial matters today?\n",
"\n",
"Llama3-2\n",
"Hello again! I'm glad you're excited about receiving my advice. How can I assist you with your financial goals today? Are you looking to create a budget, paying off debt, saving for a specific goal, or something else entirely?\n",
"\n"
]
}
],
"source": [
"# Converesation according to AWS spec including prompting + history\n",
"user_message = \"\"\"You will be acting as an AI personal finance advisor named Alex, created by the company SmartFinance Advisors. Your goal is to provide financial advice and guidance to users. You will be replying to users who are on the SmartFinance Advisors site and who will be confused if you don't respond in the character of Alex.\n",
"\n",
"Here is the conversational history (between the user and you) prior to the question. It could be empty if there is no history:\n",
"<history>\n",
"User: Hi Alex, I'm really looking forward to your advice!\n",
"Alex: Hello! I'm Alex, your AI personal finance advisor from SmartFinance Advisors. How can I assist you with your financial goals today?\n",
"</history>\n",
"\n",
"Here are some important rules for the interaction:\n",
"- Always stay in character, as Alex, an AI from SmartFinance Advisors.\n",
"- If you are unsure how to respond, say \"I'm sorry, I didn't quite catch that. Could you please rephrase your question?\"\n",
"\"\"\"\n",
"\n",
"conversation = [\n",
" {\n",
" \"role\": \"user\",\n",
" \"content\": [{\"text\": user_message}],\n",
" }\n",
"]\n",
"\n",
"@observe()\n",
"def examples_bedrock_converse_api():\n",
" responses = {}\n",
"\n",
" responses[\"anthropic\"] = wrapped_bedrock_converse(\n",
" modelId=\"eu.anthropic.claude-3-5-sonnet-20240620-v1:0\",\n",
" messages=conversation,\n",
" inferenceConfig={\"maxTokens\":500,\"temperature\":1},\n",
" additionalModelRequestFields={\"top_k\":250}\n",
" )\n",
"\n",
" responses[\"llama3-2\"] = wrapped_bedrock_converse(\n",
" modelId=\"eu.meta.llama3-2-3b-instruct-v1:0\",\n",
" messages=conversation,\n",
" inferenceConfig={\"maxTokens\":500,\"temperature\":1},\n",
" )\n",
"\n",
" return responses\n",
"\n",
"res = examples_bedrock_converse_api()\n",
"\n",
"for key, value in res.items():\n",
" print(f\"{key.title()}\\n{value}\\n\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "zMOPRjKjNKiz"
},
"source": [
"Example trace: https://cloud.langfuse.com/project/cloramnkj0002jz088vzn1ja4/traces/f01a828c-fed1-45e1-b836-cd74c331597d?observation=512a4d7f-5a6c-461e-bd8f-76f6bdcc91fd\n",
"\n",
"![Bedrock Converse API Trace](https://langfuse.com/images/cookbook/integration-amazon-bedrock/bedrock-converse-trace.png)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "QAYGBX7iOvey"
},
"source": [
"## Can I monitor Amazon Bedrock cost and token usage in Langfuse?\n",
"\n",
"Yes, you can monitor cost and token usage of your Bedrock calls in Langfuse. The native integrations with LLM application frameworks and the LiteLLM proxy will automatically report token usage to Langfuse.\n",
"\n",
"If you use the Langfuse decorator or the low-level Python SDK, you can [report](https://langfuse.com/docs/model-usage-and-cost) token usage and (optionally) also cost information directly.\n",
"\n",
"You can define custom price information via the Langfuse dashboard or UI ([see docs](https://langfuse.com/docs/model-usage-and-cost)) to adjust to the exact pricing of your models on Amazon Bedrock.\n",
"\n",
"## Additional Resources\n",
"\n",
"- Metadocs, [Monitoring your Langchain app's cost using Bedrock with Langfuse](https://www.metadocs.co/2024/07/03/monitor-your-langchain-app-cost-using-bedrock-with-langfuse/), featuring Langchain integration and custom model price definitions for Bedrock models.\n",
"- [Self-hosting guide](https://langfuse.com/docs/deployment/self-host) to deploy Langfuse on AWS."
]
}
],
"metadata": {
"colab": {
"provenance": []
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
},
"language_info": {
"name": "python"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
Loading

0 comments on commit 8f15524

Please sign in to comment.