Skip to content

Commit

Permalink
docs: Add docs on configuring the proxy to talk to azure openai (#177)
Browse files Browse the repository at this point in the history
  • Loading branch information
langchain-infra authored Apr 17, 2024
2 parents a8e26cb + a05f664 commit 84895c1
Show file tree
Hide file tree
Showing 2 changed files with 82 additions and 1 deletion.
81 changes: 81 additions & 0 deletions docs/proxy/azure_openai.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
---
sidebar_label: Use LangSmith Proxy with Azure OpenAI
sidebar_position: 2
---

# Configure LangSmith Proxy to talk to your Azure OpenAI Endpoint

## Prerequisites

1. Docker installed on your local machine
- Instructions for installing Docker can be found [here](https://docs.docker.com/get-docker/)
2. An Azure OpenAI API Key
3. An Azure OpenAI endpoint

## 1. Deploy the LangSmith Proxy

The LangSmith Proxy is available as a Docker container. You can run it in your environment by running the following command:

```bash
docker pull docker.io/langchain/langsmith-proxy:latest # Force pull the latest version of the LangSmith Proxy
docker run -e AZURE_OPENAI_ENDPOINT <YOUR AZURE_OPENAI_ENDPOINT> -p 8080:8080 docker.io/langchain/langsmith-proxy:latest # Run the LangSmith Proxy on port 8080 and publish it to the host
```

You should see the following output:

```bash
2024-03-06 12:59:57,458 CRIT Supervisor is running as root. Privileges were not dropped because no user is specified in the config file. If you intend to run as root, you can set user=root in the config file to avoid this message.
2024-03-06 12:59:57,467 INFO supervisord started with pid 1
2024-03-06 12:59:58,503 INFO spawned: 'nginx' with pid 8
2024-03-06 12:59:58,552 INFO spawned: 'trace-processor' with pid 10
2024-03-06 12:59:59,562 INFO success: nginx entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-03-06 12:59:59,563 INFO success: trace-processor entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
Couldn't create langsmith client: API key must be provided when using hosted LangSmith API, will skip creating runs
Listening for traces at 0.0.0.0:9999
Connection from ('127.0.0.1', 47370)
```
Ignore the `Couldn't create langsmith client` message if you are not configuring tracing.
## 2. Update your app to make requests to the LangSmith Proxy
For this example, we'll be using your local proxy running on `localhost:8080`. You can replace this with the address of your proxy if it's running on a different machine.
You may need to install some packages for this example:
```bash
pip install langchain_openai
```
Now, you can use the LangSmith Proxy to make requests to Azure OpenAI. Here's an example of how you can do this in Python:
Let's create a file called `azure_openai_test.py` and add the following code:
```python
from langchain_openai import AzureChatOpenAI
import os
os.environ["OPENAI_API_VERSION"] = "2023-06-01-preview"
os.environ["OPENAI_API_KEY"] = "YOUR API KEY"
os.environ["AZURE_OPENAI_ENDPOINT"] = "http://localhost:8080/proxy/azure-openai"
llm = AzureChatOpenAI(deployment_name="gpt-35-turbo")
print(llm.invoke("Hello, world!"))
```
Run the script:
```bash
python azure_openai_test.py
```
You should see some output like this
```
content='Hello! How can I assist you today?'
response_metadata={'token_usage': {'completion_tokens': 9, 'prompt_tokens': 11, 'total_tokens': 20}, 'model_name': 'gpt-35-turbo', 'system_fingerprint': 'fp_2f57f81c11',
'prompt_filter_results': [{'prompt_index': 0, 'content_filter_results': {'hate': {'filtered': False, 'severity': 'safe'}, 'self_harm': {'filtered': False, 'severity': 'safe'},
'sexual': {'filtered': False, 'severity': 'safe'}, 'violence': {'filtered': False, 'severity': 'safe'}}}], 'finish_reason': 'stop', 'logprobs': None, 'content_filter_results': {'hate': {'filtered': False, 'severity': 'safe'},
'self_harm': {'filtered': False, 'severity': 'safe'}, 'sexual': {'filtered': False, 'severity': 'safe'}, 'violence': {'filtered': False, 'severity': 'safe'}}} id='run-0884458f-bccd-4444-8357-3f8becc7ea2c-0'
```
Nice! You have successfully configured the LangSmith Proxy to talk to your Azure OpenAI endpoint. You can now use your proxy endpoint as a drop-in replacement for the Azure OpenAI endpoint in your applications.
2 changes: 1 addition & 1 deletion docs/proxy/quickstart.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ sidebar_position: 1

## What is the LangSmith Proxy

The LangSmith Proxy is intended to be a drop-in replacement for your LLM apis with some additional features. It is designed to be a simple, easy-to-use, and easy-to-configure tool that adds minimal overhead to your existing LLM API usage.
The LangSmith Proxy(Alpha) is intended to be a drop-in replacement for your LLM apis with some additional features. It is designed to be a simple, easy-to-use, and easy-to-configure tool that adds minimal overhead to your existing LLM API usage.

- **Easy to use**: The LangSmith LLM Proxy is designed to be easy to use. You can run it as a sidecar to your existing app and start using it immediately. Besides changing the URL of your LLM API, you don't need to make any changes to your app to start using the LangSmith Proxy.
- **Cache support**: The LangSmith LLM Proxy supports caching of requests/responses(with streaming support) from the LLM API. This allows you to cache responses for a configurable amount of time, reducing the number of requests to the LLM API and improving response times. This can be especially useful if you are using the LLM API in a high-traffic environment or running CI(in cases like evals)
Expand Down

0 comments on commit 84895c1

Please sign in to comment.