Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: Add docs on configuring the proxy to talk to azure openai #177

Merged
merged 4 commits into from
Apr 17, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
81 changes: 81 additions & 0 deletions docs/proxy/azure_openai.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
---
sidebar_label: Use LangSmith Proxy with Azure OpenAI
sidebar_position: 2
---

# Configure LangSmith Proxy to talk to your Azure OpenAI Endpoint

## Prerequisites

1. Docker installed on your local machine
- Instructions for installing Docker can be found [here](https://docs.docker.com/get-docker/)
2. An Azure OpenAI API Key
3. An Azure OpenAI endpoint

## 1. Deploy the LangSmith Proxy

The LangSmith Proxy is available as a Docker container. You can run it in your environment by running the following command:

```bash
docker pull docker.io/langchain/langsmith-proxy:latest # Force pull the latest version of the LangSmith Proxy
docker run -e AZURE_OPENAI_ENDPOINT <YOUR AZURE_OPENAI_ENDPOINT> -p 8080:8080 docker.io/langchain/langsmith-proxy:latest # Run the LangSmith Proxy on port 8080 and publish it to the host
```

You should see the following output:

```bash
2024-03-06 12:59:57,458 CRIT Supervisor is running as root. Privileges were not dropped because no user is specified in the config file. If you intend to run as root, you can set user=root in the config file to avoid this message.
2024-03-06 12:59:57,467 INFO supervisord started with pid 1
2024-03-06 12:59:58,503 INFO spawned: 'nginx' with pid 8
2024-03-06 12:59:58,552 INFO spawned: 'trace-processor' with pid 10
2024-03-06 12:59:59,562 INFO success: nginx entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-03-06 12:59:59,563 INFO success: trace-processor entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
Couldn't create langsmith client: API key must be provided when using hosted LangSmith API, will skip creating runs
Listening for traces at 0.0.0.0:9999
Connection from ('127.0.0.1', 47370)
```

Ignore the `Couldn't create langsmith client` message if you are not configuring tracing.

## 2. Update your app to make requests to the LangSmith Proxy

For this example, we'll be using your local proxy running on `localhost:8080`. You can replace this with the address of your proxy if it's running on a different machine.

You may need to install some packages for this example:

```bash
pip install langchain_openai
```

Now, you can use the LangSmith Proxy to make requests to Azure OpenAI. Here's an example of how you can do this in Python:
Let's create a file called `azure_openai_test.py` and add the following code:

```python
from langchain_openai import AzureChatOpenAI
import os

os.environ["OPENAI_API_VERSION"] = "2023-06-01-preview"
os.environ["OPENAI_API_KEY"] = "YOUR API KEY"
os.environ["AZURE_OPENAI_ENDPOINT"] = "http://localhost:8080/proxy/azure-openai"

llm = AzureChatOpenAI(deployment_name="gpt-35-turbo")
print(llm.invoke("Hello, world!"))
```

Run the script:

```bash
python azure_openai_test.py
```

You should see some output like this

```
content='Hello! How can I assist you today?'
response_metadata={'token_usage': {'completion_tokens': 9, 'prompt_tokens': 11, 'total_tokens': 20}, 'model_name': 'gpt-35-turbo', 'system_fingerprint': 'fp_2f57f81c11',
'prompt_filter_results': [{'prompt_index': 0, 'content_filter_results': {'hate': {'filtered': False, 'severity': 'safe'}, 'self_harm': {'filtered': False, 'severity': 'safe'},
'sexual': {'filtered': False, 'severity': 'safe'}, 'violence': {'filtered': False, 'severity': 'safe'}}}], 'finish_reason': 'stop', 'logprobs': None, 'content_filter_results': {'hate': {'filtered': False, 'severity': 'safe'},
'self_harm': {'filtered': False, 'severity': 'safe'}, 'sexual': {'filtered': False, 'severity': 'safe'}, 'violence': {'filtered': False, 'severity': 'safe'}}} id='run-0884458f-bccd-4444-8357-3f8becc7ea2c-0'
```

Nice! You have successfully configured the LangSmith Proxy to talk to your Azure OpenAI endpoint. You can now use your proxy endpoint as a drop-in replacement for the Azure OpenAI endpoint in your applications.
2 changes: 1 addition & 1 deletion docs/proxy/quickstart.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ sidebar_position: 1

## What is the LangSmith Proxy

The LangSmith Proxy is intended to be a drop-in replacement for your LLM apis with some additional features. It is designed to be a simple, easy-to-use, and easy-to-configure tool that adds minimal overhead to your existing LLM API usage.
The LangSmith Proxy(Alpha) is intended to be a drop-in replacement for your LLM apis with some additional features. It is designed to be a simple, easy-to-use, and easy-to-configure tool that adds minimal overhead to your existing LLM API usage.

- **Easy to use**: The LangSmith LLM Proxy is designed to be easy to use. You can run it as a sidecar to your existing app and start using it immediately. Besides changing the URL of your LLM API, you don't need to make any changes to your app to start using the LangSmith Proxy.
- **Cache support**: The LangSmith LLM Proxy supports caching of requests/responses(with streaming support) from the LLM API. This allows you to cache responses for a configurable amount of time, reducing the number of requests to the LLM API and improving response times. This can be especially useful if you are using the LLM API in a high-traffic environment or running CI(in cases like evals)
Expand Down
Loading