diff --git a/docs/docs/integrations/llms/sambanovacloud.ipynb b/docs/docs/integrations/llms/sambanovacloud.ipynb new file mode 100644 index 0000000000000..cffe08553fa95 --- /dev/null +++ b/docs/docs/integrations/llms/sambanovacloud.ipynb @@ -0,0 +1,253 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# SambaNovaCloud\n", + "\n", + "**[SambaNova](https://sambanova.ai/)'s [SambaNova Cloud](https://cloud.sambanova.ai/)** is a platform for performing inference with open-source models\n", + "\n", + ":::caution\n", + "You are currently on a page documenting the use of SambaNovaCloud models as [text completion models](/docs/concepts/text_llms/). We recommend you to use the [chat completion models](/docs/concepts/chat_models).\n", + "\n", + "You may be looking for [SambaNovaCloud Chat Models](/docs/integrations/chat/sambanova/) .\n", + ":::\n", + "\n", + "## Overview\n", + "### Integration details\n", + "\n", + "| Class | Package | Local | Serializable | JS support | Package downloads | Package latest |\n", + "| :--- | :--- | :---: | :---: | :---: | :---: | :---: |\n", + "| [SambaNovaCloud](https://python.langchain.com/api_reference/community/llms/langchain_community.llms.sambanova.SambaNovaCloud.html) | [langchain_community](https://python.langchain.com/api_reference/community/index.html) | ❌ | beta | ❌ | ![PyPI - Downloads](https://img.shields.io/pypi/dm/langchain_community?style=flat-square&label=%20) | ![PyPI - Version](https://img.shields.io/pypi/v/langchain_community?style=flat-square&label=%20) |\n", + "\n", + "This example goes over how to use LangChain to interact with SambaNovaCloud models" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Setup\n", + "\n", + "### Credentials\n", + "To access ChatSambaNovaCloud models you will need to create a [SambaNovaCloud account](https://cloud.sambanova.ai/), get an API key and set it as the `SAMBANOVA_API_KEY` environment variable:" + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": {}, + "outputs": [], + "source": [ + "import getpass\n", + "import os\n", + "\n", + "if \"SAMBANOVA_API_KEY\" not in os.environ:\n", + " os.environ[\"SAMBANOVA_API_KEY\"] = getpass.getpass()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Installation\n", + "\n", + "The integration lives in the `langchain-community` package. We also need to install the [sseclient-py](https://pypi.org/project/sseclient-py/) package this is required to run streaming predictions " + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "%pip install --quiet -U langchain-community sseclient-py" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Instantiation" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": {}, + "outputs": [], + "source": [ + "from langchain_community.llms.sambanova import SambaNovaCloud\n", + "\n", + "llm = SambaNovaCloud(\n", + " model=\"Meta-Llama-3.1-70B-Instruct\",\n", + " max_tokens_to_generate=1000,\n", + " temperature=0.01,\n", + " # top_k = 50,\n", + " # top_p = 1.0\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Invocation\n", + "Now we can instantiate our model object and generate chat completions:" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "\"**Advantages of Open Source Models**\\n\\nUsing open source models can bring numerous benefits to your project or organization. Here are some reasons why you should consider using open source models:\\n\\n### 1. **Cost-Effective**\\n\\nOpen source models are free to use, modify, and distribute. This can significantly reduce the costs associated with developing and maintaining proprietary models.\\n\\n### 2. **Community Support**\\n\\nOpen source models are often maintained by a community of developers and users who contribute to their improvement. This community support can lead to faster bug fixes, new feature additions, and better documentation.\\n\\n### 3. **Transparency and Customizability**\\n\\nOpen source models provide complete transparency into their architecture and implementation. This allows you to customize and fine-tune the model to suit your specific needs.\\n\\n### 4. **Faster Development**\\n\\nBy leveraging pre-trained open source models, you can accelerate your development process. You can focus on fine-tuning the model for your specific use case rather than building one from scratch.\\n\\n### 5. **Improved Security**\\n\\nOpen source models are often reviewed and audited by a large community of developers, which can help identify and fix security vulnerabilities.\\n\\n### 6. **Interoperability**\\n\\nOpen source models can be easily integrated with other open source tools and frameworks, promoting interoperability and reducing vendor lock-in.\\n\\n### 7. **Access to State-of-the-Art Technology**\\n\\nMany open source models are developed by top researchers and institutions, providing access to state-of-the-art technology and techniques.\\n\\n### Example Use Cases\\n\\n* **Computer Vision**: Use open source models like TensorFlow's Object Detection API or OpenCV's pre-trained models for image classification, object detection, and segmentation tasks.\\n* **Natural Language Processing**: Leverage open source models like spaCy or Stanford CoreNLP for text processing, sentiment analysis, and language translation tasks.\\n* **Speech Recognition**: Utilize open source models like Kaldi or Mozilla's DeepSpeech for speech-to-text applications.\\n\\n**Getting Started**\\n\\nTo get started with open source models, explore popular repositories on GitHub or model hubs like TensorFlow Hub or PyTorch Hub. Familiarize yourself with the model's documentation, and experiment with pre-trained models before fine-tuning them for your specific use case.\\n\\nBy embracing open source models, you can accelerate your development process, reduce costs, and tap into the collective knowledge of the developer community.\"" + ] + }, + "execution_count": 3, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "input_text = \"Why should I use open source models?\"\n", + "\n", + "completion = llm.invoke(input_text)\n", + "completion" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "**Advantages of Open Source Models**\n", + "\n", + "Using open source models can bring numerous benefits to your projects. Here are some reasons why you should consider them:\n", + "\n", + "### 1. **Cost-Effective**\n", + "\n", + "Open source models are free to use, modify, and distribute. This can significantly reduce the costs associated with developing and maintaining proprietary models.\n", + "\n", + "### 2. **Community Support**\n", + "\n", + "Open source models are often maintained by a community of developers, researchers, and users. This community can provide support, fix bugs, and contribute to the model's improvement.\n", + "\n", + "### 3. **Transparency and Reproducibility**\n", + "\n", + "Open source models are transparent in their architecture, training data, and hyperparameters. This transparency allows for reproducibility, which is essential in scientific research and development.\n", + "\n", + "### 4. **Customizability**\n", + "\n", + "Open source models can be modified to suit specific use cases or requirements. This customizability enables developers to adapt the model to their needs, which can lead to better performance and accuracy.\n", + "\n", + "### 5. **Faster Development**\n", + "\n", + "Using open source models can accelerate development by providing a pre-trained foundation. This allows developers to focus on fine-tuning the model for their specific task, rather than starting from scratch.\n", + "\n", + "### 6. **Access to State-of-the-Art Models**\n", + "\n", + "Open source models often represent the state-of-the-art in their respective domains. By using these models, developers can leverage the latest advancements in AI research.\n", + "\n", + "### 7. **Reduced Vendor Lock-in**\n", + "\n", + "Open source models are not tied to a specific vendor or platform. This reduces the risk of vendor lock-in and allows developers to switch to alternative solutions if needed.\n", + "\n", + "### Example Use Cases\n", + "\n", + "* **Computer Vision**: Using open source models like YOLO (You Only Look Once) or SSD (Single Shot Detector) for object detection tasks.\n", + "* **Natural Language Processing**: Leveraging open source models like BERT (Bidirectional Encoder Representations from Transformers) or RoBERTa (Robustly Optimized BERT Pretraining Approach) for text classification, sentiment analysis, or language translation.\n", + "* **Speech Recognition**: Utilizing open source models like Kaldi or Mozilla DeepSpeech for speech-to-text applications.\n", + "\n", + "**Getting Started**\n", + "\n", + "To get started with open source models, you can explore popular repositories on GitHub or model hubs like the TensorFlow Model Garden or the PyTorch Model Zoo. These resources provide pre-trained models, documentation, and tutorials to help you integrate open source models into your projects.\n", + "\n", + "By embracing open source models, you can tap into the collective knowledge and expertise of the developer community, accelerate your development process, and create more accurate and efficient AI solutions." + ] + } + ], + "source": [ + "# Streaming response\n", + "for chunk in llm.stream(\"Why should I use open source models?\"):\n", + " print(chunk, end=\"\", flush=True)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Chaining\n", + "We can chain our completion model with a prompt template like so:" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "'The translation of \"I love programming\" in German is:\\n\\n\"Ich liebe das Programmieren.\"\\n\\nHere\\'s a breakdown of the sentence:\\n\\n* \"Ich\" means \"I\"\\n* \"liebe\" is the verb \"to love\" in the first person singular (I love)\\n* \"das\" is the definite article for \"Programmieren\" (programming)\\n* \"Programmieren\" is the verb \"to program\" in the infinitive form, but in this context, it\\'s used as a noun to refer to the activity of programming.\\n\\nSo, \"Ich liebe das Programmieren\" is a common way to express your passion for programming in German.'" + ] + }, + "execution_count": 5, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "from langchain_core.prompts import PromptTemplate\n", + "\n", + "prompt = PromptTemplate.from_template(\"How to say {input} in {output_language}:\\n\")\n", + "\n", + "chain = prompt | llm\n", + "chain.invoke(\n", + " {\n", + " \"output_language\": \"German\",\n", + " \"input\": \"I love programming.\",\n", + " }\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## API reference\n", + "\n", + "For detailed documentation of all `SambaNovaCloud` llm features and configurations head to the API reference: https://python.langchain.com/api_reference/community/llms/langchain_community.llms.sambanova.SambaNovaCloud.html" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "multimodalenv", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.11.3" + } + }, + "nbformat": 4, + "nbformat_minor": 4 +} diff --git a/libs/community/langchain_community/llms/__init__.py b/libs/community/langchain_community/llms/__init__.py index 1bf5e0a7ee76e..9a74a30dce4bd 100644 --- a/libs/community/langchain_community/llms/__init__.py +++ b/libs/community/langchain_community/llms/__init__.py @@ -518,6 +518,12 @@ def _import_sagemaker_endpoint() -> Type[BaseLLM]: return SagemakerEndpoint +def _import_sambanovacloud() -> Type[BaseLLM]: + from langchain_community.llms.sambanova import SambaNovaCloud + + return SambaNovaCloud + + def _import_sambastudio() -> Type[BaseLLM]: from langchain_community.llms.sambanova import SambaStudio @@ -821,6 +827,8 @@ def __getattr__(name: str) -> Any: return _import_rwkv() elif name == "SagemakerEndpoint": return _import_sagemaker_endpoint() + elif name == "SambaNovaCloud": + return _import_sambanovacloud() elif name == "SambaStudio": return _import_sambastudio() elif name == "SelfHostedPipeline": @@ -957,6 +965,7 @@ def __getattr__(name: str) -> Any: "RWKV", "Replicate", "SagemakerEndpoint", + "SambaNovaCloud", "SambaStudio", "SelfHostedHuggingFaceLLM", "SelfHostedPipeline", @@ -1054,6 +1063,7 @@ def get_type_to_cls_dict() -> Dict[str, Callable[[], Type[BaseLLM]]]: "replicate": _import_replicate, "rwkv": _import_rwkv, "sagemaker_endpoint": _import_sagemaker_endpoint, + "sambanovacloud": _import_sambanovacloud, "sambastudio": _import_sambastudio, "self_hosted": _import_self_hosted, "self_hosted_hugging_face": _import_self_hosted_hugging_face, diff --git a/libs/community/langchain_community/llms/sambanova.py b/libs/community/langchain_community/llms/sambanova.py index 4ae6a1e6a2809..7b4badce9b269 100644 --- a/libs/community/langchain_community/llms/sambanova.py +++ b/libs/community/langchain_community/llms/sambanova.py @@ -537,3 +537,329 @@ def _call( response = self._handle_request(prompt, stop, streaming=False) completion = self._process_response(response) return completion + + +class SambaNovaCloud(LLM): + """ + SambaNova Cloud large language models. + + Setup: + To use, you should have the environment variables: + ``SAMBANOVA_URL`` set with SambaNova Cloud URL. + defaults to http://cloud.sambanova.ai/ + ``SAMBANOVA_API_KEY`` set with your SambaNova Cloud API Key. + Example: + .. code-block:: python + from langchain_community.llms.sambanova import SambaNovaCloud + SambaNovaCloud( + sambanova_api_key="your-SambaNovaCloud-API-key, + model = model name, + max_tokens = max number of tokens to generate, + temperature = model temperature, + top_p = model top p, + top_k = model top k + ) + Key init args — completion params: + model: str + The name of the model to use, e.g., Meta-Llama-3-70B-Instruct-4096 + (set for CoE endpoints). + streaming: bool + Whether to use streaming handler when using non streaming methods + max_tokens: int + max tokens to generate + temperature: float + model temperature + top_p: float + model top p + top_k: int + model top k + + Key init args — client params: + sambanova_url: str + SambaNovaCloud Url defaults to http://cloud.sambanova.ai/ + sambanova_api_key: str + SambaNovaCloud api key + Instantiate: + .. code-block:: python + from langchain_community.llms.sambanova import SambaNovaCloud + SambaNovaCloud( + sambanova_api_key="your-SambaNovaCloud-API-key, + model = model name, + max_tokens = max number of tokens to generate, + temperature = model temperature, + top_p = model top p, + top_k = model top k + ) + Invoke: + .. code-block:: python + prompt = "tell me a joke" + response = llm.invoke(prompt) + Stream: + .. code-block:: python + for chunk in llm.stream(prompt): + print(chunk, end="", flush=True) + Async: + .. code-block:: python + response = llm.ainvoke(prompt) + await response + """ + + sambanova_url: str = Field(default="") + """SambaNova Cloud Url""" + + sambanova_api_key: SecretStr = Field(default="") + """SambaNova Cloud api key""" + + model: str = Field(default="Meta-Llama-3.1-8B-Instruct") + """The name of the model""" + + streaming: bool = Field(default=False) + """Whether to use streaming handler when using non streaming methods""" + + max_tokens: int = Field(default=1024) + """max tokens to generate""" + + temperature: float = Field(default=0.7) + """model temperature""" + + top_p: Optional[float] = Field(default=None) + """model top p""" + + top_k: Optional[int] = Field(default=None) + """model top k""" + + stream_options: dict = Field(default={"include_usage": True}) + """stream options, include usage to get generation metrics""" + + class Config: + populate_by_name = True + + @classmethod + def is_lc_serializable(cls) -> bool: + """Return whether this model can be serialized by Langchain.""" + return False + + @property + def lc_secrets(self) -> Dict[str, str]: + return {"sambanova_api_key": "sambanova_api_key"} + + @property + def _identifying_params(self) -> Dict[str, Any]: + """Return a dictionary of identifying parameters. + + This information is used by the LangChain callback system, which + is used for tracing purposes make it possible to monitor LLMs. + """ + return { + "model": self.model, + "streaming": self.streaming, + "max_tokens": self.max_tokens, + "temperature": self.temperature, + "top_p": self.top_p, + "top_k": self.top_k, + "stream_options": self.stream_options, + } + + @property + def _llm_type(self) -> str: + """Get the type of language model used by this chat model.""" + return "sambanovacloud-llm" + + def __init__(self, **kwargs: Any) -> None: + """init and validate environment variables""" + kwargs["sambanova_url"] = get_from_dict_or_env( + kwargs, + "sambanova_url", + "SAMBANOVA_URL", + default="https://api.sambanova.ai/v1/chat/completions", + ) + kwargs["sambanova_api_key"] = convert_to_secret_str( + get_from_dict_or_env(kwargs, "sambanova_api_key", "SAMBANOVA_API_KEY") + ) + super().__init__(**kwargs) + + def _handle_request( + self, + prompt: Union[List[str], str], + stop: Optional[List[str]] = None, + streaming: Optional[bool] = False, + ) -> Response: + """ + Performs a post request to the LLM API. + + Args: + prompt: The prompt to pass into the model. + stop: list of stop tokens + + Returns: + A request Response object + """ + if isinstance(prompt, str): + prompt = [prompt] + + messages_dict = [{"role": "user", "content": prompt[0]}] + data = { + "messages": messages_dict, + "stream": streaming, + "max_tokens": self.max_tokens, + "stop": stop, + "model": self.model, + "temperature": self.temperature, + "top_p": self.top_p, + "top_k": self.top_k, + } + data = {key: value for key, value in data.items() if value is not None} + headers = { + "Authorization": f"Bearer " f"{self.sambanova_api_key.get_secret_value()}", + "Content-Type": "application/json", + } + + http_session = requests.Session() + if streaming: + response = http_session.post( + self.sambanova_url, headers=headers, json=data, stream=True + ) + else: + response = http_session.post( + self.sambanova_url, headers=headers, json=data, stream=False + ) + + if response.status_code != 200: + raise RuntimeError( + f"Sambanova / complete call failed with status code " + f"{response.status_code}." + f"{response.text}." + ) + return response + + def _process_response(self, response: Response) -> str: + """ + Process a non streaming response from the api + + Args: + response: A request Response object + + Returns + completion: a string with model generation + """ + + # Extract json payload form response + try: + response_dict = response.json() + except Exception as e: + raise RuntimeError( + f"Sambanova /complete call failed couldn't get JSON response {e}" + f"response: {response.text}" + ) + + completion = response_dict["choices"][0]["message"]["content"] + + return completion + + def _process_stream_response(self, response: Response) -> Iterator[GenerationChunk]: + """ + Process a streaming response from the api + + Args: + response: An iterable request Response object + + Yields: + GenerationChunk: a GenerationChunk with model partial generation + """ + + try: + import sseclient + except ImportError: + raise ImportError( + "could not import sseclient library" + "Please install it with `pip install sseclient-py`." + ) + + client = sseclient.SSEClient(response) + for event in client.events(): + if event.event == "error_event": + raise RuntimeError( + f"Sambanova /complete call failed with status code " + f"{response.status_code}." + f"{event.data}." + ) + try: + # check if the response is not a final event ("[DONE]") + if event.data != "[DONE]": + if isinstance(event.data, str): + data = json.loads(event.data) + else: + raise RuntimeError( + f"Sambanova /complete call failed with status code " + f"{response.status_code}." + f"{event.data}." + ) + if data.get("error"): + raise RuntimeError( + f"Sambanova /complete call failed with status code " + f"{response.status_code}." + f"{event.data}." + ) + if len(data["choices"]) > 0: + content = data["choices"][0]["delta"]["content"] + else: + content = "" + generated_chunk = GenerationChunk(text=content) + yield generated_chunk + + except Exception as e: + raise RuntimeError( + f"Error getting content chunk raw streamed response: {e}" + f"data: {event.data}" + ) + + def _call( + self, + prompt: Union[List[str], str], + stop: Optional[List[str]] = None, + run_manager: Optional[CallbackManagerForLLMRun] = None, + **kwargs: Any, + ) -> str: + """Call out to SambaNovaCloud complete endpoint. + + Args: + prompt: The prompt to pass into the model. + stop: Optional list of stop words to use when generating. + + Returns: + The string generated by the model. + """ + if self.streaming: + completion = "" + for chunk in self._stream( + prompt=prompt, stop=stop, run_manager=run_manager, **kwargs + ): + completion += chunk.text + + return completion + + response = self._handle_request(prompt, stop, streaming=False) + completion = self._process_response(response) + return completion + + def _stream( + self, + prompt: Union[List[str], str], + stop: Optional[List[str]] = None, + run_manager: Optional[CallbackManagerForLLMRun] = None, + **kwargs: Any, + ) -> Iterator[GenerationChunk]: + """Call out to SambaNovaCloud complete endpoint. + + Args: + prompt: The prompt to pass into the model. + stop: Optional list of stop words to use when generating. + + Returns: + The string generated by the model. + """ + response = self._handle_request(prompt, stop, streaming=True) + for chunk in self._process_stream_response(response): + if run_manager: + run_manager.on_llm_new_token(chunk.text) + yield chunk diff --git a/libs/community/tests/integration_tests/llms/test_sambanova.py b/libs/community/tests/integration_tests/llms/test_sambanova.py index 5f082df52785c..345ae8bcf013a 100644 --- a/libs/community/tests/integration_tests/llms/test_sambanova.py +++ b/libs/community/tests/integration_tests/llms/test_sambanova.py @@ -1,13 +1,20 @@ -"""Test sambanova API wrapper. +"""Test sambanova API llm wrappers. -In order to run this test, you need to have a sambastudio base url, -project id, endpoint id, and api key. -You'll then need to set SAMBASTUDIO_BASE_URL, SAMBASTUDIO_BASE_URI -SAMBASTUDIO_PROJECT_ID, SAMBASTUDIO_ENDPOINT_ID, and SAMBASTUDIO_API_KEY -environment variables. +In order to run this test, you need to have a sambastudio url, and api key +and a sambanova cloud api key. +You'll then need to set SAMBASTUDIO_URL, and SAMBASTUDIO_API_KEY, +and SAMBANOVA_API_KEY environment variables. """ -from langchain_community.llms.sambanova import SambaStudio +from langchain_community.llms.sambanova import SambaNovaCloud, SambaStudio + + +def test_sambanova_cloud_call() -> None: + """Test simple non-streaming call to sambastudio.""" + llm = SambaNovaCloud() + output = llm.invoke("What is LangChain") + assert output + assert isinstance(output, str) def test_sambastudio_call() -> None: diff --git a/libs/community/tests/unit_tests/llms/test_imports.py b/libs/community/tests/unit_tests/llms/test_imports.py index df0fe68b59fd4..991996ec55802 100644 --- a/libs/community/tests/unit_tests/llms/test_imports.py +++ b/libs/community/tests/unit_tests/llms/test_imports.py @@ -78,6 +78,7 @@ "RWKV", "Replicate", "SagemakerEndpoint", + "SambaNovaCloud", "SambaStudio", "SelfHostedHuggingFaceLLM", "SelfHostedPipeline",