diff --git a/docs/examples.md b/docs/examples.md
index 9b349f315..70c9be43b 100644
--- a/docs/examples.md
+++ b/docs/examples.md
@@ -22,7 +22,6 @@ Build semantic/similarity/vector/neural search applications.
| [Build a QA database](https://github.com/neuml/txtai/blob/master/examples/34_Build_a_QA_database.ipynb) | Question matching with semantic search | [](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/34_Build_a_QA_database.ipynb) |
| [Semantic Graphs](https://github.com/neuml/txtai/blob/master/examples/38_Introducing_the_Semantic_Graph.ipynb) | Explore topics, data connectivity and run network analysis| [](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/38_Introducing_the_Semantic_Graph.ipynb) |
| [Topic Modeling with BM25](https://github.com/neuml/txtai/blob/master/examples/39_Classic_Topic_Modeling_with_BM25.ipynb) | Topic modeling backed by a BM25 index | [](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/39_Classic_Topic_Modeling_with_BM25.ipynb) |
-| [Getting started with LLM APIs](https://github.com/neuml/txtai/blob/master/examples/70_Getting_started_with_LLM_APIs.ipynb) | A code snippet that lets us call different LLM APIs using just one set of code. | [](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/70_Getting_started_with_LLM_APIs.ipynb) |
## LLM
@@ -43,6 +42,7 @@ Autonomous agents, retrieval augmented generation (RAG), chat with your data, pi
| [Speech to Speech RAG](https://github.com/neuml/txtai/blob/master/examples/65_Speech_to_Speech_RAG.ipynb) [▶️](https://www.youtube.com/watch?v=tH8QWwkVMKA) | Full cycle speech to speech workflow with RAG | [](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/65_Speech_to_Speech_RAG.ipynb) |
| [Analyzing Hugging Face Posts with Graphs and Agents](https://github.com/neuml/txtai/blob/master/examples/68_Analyzing_Hugging_Face_Posts_with_Graphs_and_Agents.ipynb) | Explore a rich dataset with Graph Analysis and Agents | [](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/68_Analyzing_Hugging_Face_Posts_with_Graphs_and_Agents.ipynb) |
| [Granting autonomy to agents](https://github.com/neuml/txtai/blob/master/examples/69_Granting_autonomy_to_agents.ipynb) | Agents that iteratively solve problems as they see fit | [](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/69_Granting_autonomy_to_agents.ipynb) |
+| [Getting started with LLM APIs](https://github.com/neuml/txtai/blob/master/examples/70_Getting_started_with_LLM_APIs.ipynb) | Generate embeddings and run LLMs with OpenAI, Claude, Gemini, Bedrock and more | [](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/70_Getting_started_with_LLM_APIs.ipynb) |
## Pipelines
diff --git a/docs/pipeline/text/llm.md b/docs/pipeline/text/llm.md
index 4da905e2c..7ec4667cd 100644
--- a/docs/pipeline/text/llm.md
+++ b/docs/pipeline/text/llm.md
@@ -121,6 +121,7 @@ See the links below for more detailed examples.
| [Generative Audio](https://github.com/neuml/txtai/blob/master/examples/66_Generative_Audio.ipynb) | Storytelling with generative audio workflows | [](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/66_Generative_Audio.ipynb) |
| [Analyzing Hugging Face Posts with Graphs and Agents](https://github.com/neuml/txtai/blob/master/examples/68_Analyzing_Hugging_Face_Posts_with_Graphs_and_Agents.ipynb) | Explore a rich dataset with Graph Analysis and Agents | [](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/68_Analyzing_Hugging_Face_Posts_with_Graphs_and_Agents.ipynb) |
| [Granting autonomy to agents](https://github.com/neuml/txtai/blob/master/examples/69_Granting_autonomy_to_agents.ipynb) | Agents that iteratively solve problems as they see fit | [](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/69_Granting_autonomy_to_agents.ipynb) |
+| [Getting started with LLM APIs](https://github.com/neuml/txtai/blob/master/examples/70_Getting_started_with_LLM_APIs.ipynb) | Generate embeddings and run LLMs with OpenAI, Claude, Gemini, Bedrock and more | [](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/70_Getting_started_with_LLM_APIs.ipynb) |
## Configuration-driven example
diff --git a/docs/pipeline/text/rag.md b/docs/pipeline/text/rag.md
index ed777070f..6a1c41cc9 100644
--- a/docs/pipeline/text/rag.md
+++ b/docs/pipeline/text/rag.md
@@ -62,6 +62,9 @@ See the links below for more detailed examples.
| [How RAG with txtai works](https://github.com/neuml/txtai/blob/master/examples/63_How_RAG_with_txtai_works.ipynb) | Create RAG processes, API services and Docker instances | [](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/63_How_RAG_with_txtai_works.ipynb) |
| [Speech to Speech RAG](https://github.com/neuml/txtai/blob/master/examples/65_Speech_to_Speech_RAG.ipynb) [▶️](https://www.youtube.com/watch?v=tH8QWwkVMKA) | Full cycle speech to speech workflow with RAG | [](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/65_Speech_to_Speech_RAG.ipynb) |
| [Generative Audio](https://github.com/neuml/txtai/blob/master/examples/66_Generative_Audio.ipynb) | Storytelling with generative audio workflows | [](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/66_Generative_Audio.ipynb) |
+| [Analyzing Hugging Face Posts with Graphs and Agents](https://github.com/neuml/txtai/blob/master/examples/68_Analyzing_Hugging_Face_Posts_with_Graphs_and_Agents.ipynb) | Explore a rich dataset with Graph Analysis and Agents | [](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/68_Analyzing_Hugging_Face_Posts_with_Graphs_and_Agents.ipynb) |
+| [Granting autonomy to agents](https://github.com/neuml/txtai/blob/master/examples/69_Granting_autonomy_to_agents.ipynb) | Agents that iteratively solve problems as they see fit | [](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/69_Granting_autonomy_to_agents.ipynb) |
+| [Getting started with LLM APIs](https://github.com/neuml/txtai/blob/master/examples/70_Getting_started_with_LLM_APIs.ipynb) | Generate embeddings and run LLMs with OpenAI, Claude, Gemini, Bedrock and more | [](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/70_Getting_started_with_LLM_APIs.ipynb) |
| [Extractive QA with txtai](https://github.com/neuml/txtai/blob/master/examples/05_Extractive_QA_with_txtai.ipynb) | Introduction to extractive question-answering with txtai | [](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/05_Extractive_QA_with_txtai.ipynb) |
| [Extractive QA with Elasticsearch](https://github.com/neuml/txtai/blob/master/examples/06_Extractive_QA_with_Elasticsearch.ipynb) | Run extractive question-answering queries with Elasticsearch | [](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/06_Extractive_QA_with_Elasticsearch.ipynb) |
| [Extractive QA to build structured data](https://github.com/neuml/txtai/blob/master/examples/20_Extractive_QA_to_build_structured_data.ipynb) | Build structured datasets using extractive question-answering | [](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/20_Extractive_QA_to_build_structured_data.ipynb) |
diff --git a/examples/70_Getting_started_with_LLM_APIs.ipynb b/examples/70_Getting_started_with_LLM_APIs.ipynb
index 466410f78..7c2d3d85e 100644
--- a/examples/70_Getting_started_with_LLM_APIs.ipynb
+++ b/examples/70_Getting_started_with_LLM_APIs.ipynb
@@ -1,61 +1,38 @@
{
- "nbformat": 4,
- "nbformat_minor": 0,
- "metadata": {
- "colab": {
- "provenance": [],
- "collapsed_sections": [
- "lhrk9y766Lw1",
- "wuCvBPCL-VQG",
- "pgQuDv4fZgHe",
- "0KsLGZFtrZVT",
- "I7a3d8EsrdBc",
- "xZg16h4KHpE5",
- "Cqh5j0a4JcHg",
- "oPq-sHZifHFZ"
- ],
- "toc_visible": true
- },
- "kernelspec": {
- "name": "python3",
- "display_name": "Python 3"
- },
- "language_info": {
- "name": "python"
- }
- },
"cells": [
{
"cell_type": "markdown",
+ "metadata": {
+ "id": "7btcrgth5z6f"
+ },
"source": [
"# Getting started with LLM APIs\n",
"\n",
- "This Notebook has a code snippet that lets us call different LLM APIs using just one set of code.\n",
+ "_Thank you to [Igor Ribeiro Lima](https://github.com/igorlima) for writing this notebook!_\n",
"\n",
- "While each LLM has its set of config, here's a cool perk of using [`txtai`](https://neuml.github.io/txtai/): plus [`litellm`](https://docs.litellm.ai/docs/) working behind the scenes. The big win with `txtai` is that it helps standardize inputs and outputs across several LLMs. **This means we can seamlessly call any LLM API**.\n",
+ "This notebook demonstrates how to call different LLM APIs using just one set of common functions.\n",
"\n",
- "In this Notebook, you can use the same code to call _**Gemini**_, _**VertexAI**_, _**Mistral**_, _**Cohere**_, _**AWS Bedrock**_, _**OpenAI**_, _**Claude by Anthropic**_, _**Groq**_, _and more_!\n",
+ "While each LLM has its own configuration, here's a cool perk of using [`txtai`](https://neuml.github.io/txtai/) plus [`litellm`](https://docs.litellm.ai/docs/) working behind the scenes. The big win with `txtai` is that it helps standardize inputs and outputs across several LLMs. **This means we can seamlessly call any LLM API**.\n",
"\n",
- "Isn't that amazing? One code for many different LLM APIs - **that's the beauty of these libraries**!"
- ],
- "metadata": {
- "id": "7btcrgth5z6f"
- }
+ "In this notebook, we use one set of common functions to call _**Gemini**_, _**VertexAI**_, _**Mistral**_, _**Cohere**_, _**AWS Bedrock**_, _**OpenAI**_, _**Claude by Anthropic**_, _**Groq**_, _and more_!\n",
+ "\n",
+ "Isn't that amazing? One method for many different LLM APIs - **that's the beauty of these libraries**!"
+ ]
},
{
"cell_type": "markdown",
+ "metadata": {
+ "id": "lhrk9y766Lw1"
+ },
"source": [
"## Install dependencies\n",
"\n",
- "This session should guide you through all the essential dependencies you'll need to run the Python script for various LLM APIs.\n",
+ "This session guides you through all the essential dependencies you'll need to run the Python script for various LLM APIs.\n",
"\n",
"You can choose the dependencies based on your specific needs. Each dependency is carefully commented on in the code, so you'll know exactly which API it supports.\n",
"\n",
"The core dependencies you'll typically need are: `txtai` and `txtai[pipeline]`."
- ],
- "metadata": {
- "id": "lhrk9y766Lw1"
- }
+ ]
},
{
"cell_type": "code",
@@ -86,30 +63,47 @@
},
{
"cell_type": "markdown",
+ "metadata": {
+ "id": "evHD8Kw09Aa0"
+ },
"source": [
"- _**A friendly reminder**: things can change unexpectedly in the ever-evolving world of coding. One day, your code works flawlessly; the next, it might throw a tantrum for no apparent reason. That's why it's essential to specify the dependencies' versions using the latest available versions when writing the code._\n",
- " - Even recognizing the importance of version control, I left each version one line above as a comment in the code. It serves as a guide to help you track dependencies and ensures the code runs smoothly across different environments, even if something goes awry.\n",
+ " - Even recognizing the importance of version control, each version has one line above as a comment in the code. It serves as a guide to help you track dependencies and ensures the code runs smoothly across different environments, even if something goes awry.\n",
" - While there is [a trade-off](https://github.com/neuml/txtai/pull/844#issuecomment-2564294186) in limiting code to a specific version, noting today's version as a comment should help you protect against potential issues with future updates."
- ],
- "metadata": {
- "id": "evHD8Kw09Aa0"
- }
+ ]
},
{
"cell_type": "markdown",
+ "metadata": {
+ "id": "wuCvBPCL-VQG"
+ },
"source": [
"## LLM API Configuration\n",
"\n",
"This session is like a special Python dictionary that holds all the cool configurations for our LLM AI model. It includes details like environment variables, the model's name, text embedding parameters, and other fancy settings.\n",
"\n",
"One important key here is `IS_ENABLED`. When it's set to `True`, it's like giving the model a green light to shine! But if you ever feel like taking a break or don't need it for a while, you can easily set this key to `False`, and the model will chill out."
- ],
- "metadata": {
- "id": "wuCvBPCL-VQG"
- }
+ ]
},
{
"cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "jyahNyOrO2yp",
+ "outputId": "8143c2b0-a6a9-429b-a66f-93f39e6885e4"
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "config set!\n"
+ ]
+ }
+ ],
"source": [
"import os, getpass\n",
"from txtai import LLM, Embeddings\n",
@@ -212,50 +206,38 @@
"# https://docs.litellm.ai/docs/debugging/local_debugging#set-verbose\n",
"litellm.set_verbose=False\n",
"\n",
- "def custom_setup ():\n",
- " def vertexai_custom_setup ():\n",
+ "def customsetup():\n",
+ " def vertexai():\n",
" # https://docs.litellm.ai/docs/embedding/supported_embedding#usage---embedding\n",
" # https://docs.litellm.ai/docs/providers/vertex\n",
" litellm.vertex_project = os.environ['GOOGLE_VERTEX_PROJECT']\n",
" litellm.vertex_location = os.environ['GOOGLE_VERTEX_LOCATION']\n",
- " LLM_MODEL_CONFIG['VERTEXAI']['ENV_VAR_SETUP'] = vertexai_custom_setup\n",
- "custom_setup()\n",
+ "\n",
+ " LLM_MODEL_CONFIG['VERTEXAI']['ENV_VAR_SETUP'] = vertexai()\n",
+ "\n",
+ "customsetup()\n",
"\n",
"LLM_MODELS = {k: v for k, v in LLM_MODEL_CONFIG.items() if v['IS_ENABLED']}\n",
"ENV_VARS = [v['ENV_VAR'] for k, v in LLM_MODELS.items()]\n",
"ENV_VARS = [x for xs in ENV_VARS for x in xs] # flatten array\n",
"\n",
"print(\"config set!\")"
- ],
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "id": "jyahNyOrO2yp",
- "outputId": "8143c2b0-a6a9-429b-a66f-93f39e6885e4"
- },
- "execution_count": null,
- "outputs": [
- {
- "output_type": "stream",
- "name": "stdout",
- "text": [
- "config set!\n"
- ]
- }
]
},
{
"cell_type": "markdown",
- "source": [
- "### Vertex AI"
- ],
"metadata": {
"id": "pgQuDv4fZgHe"
- }
+ },
+ "source": [
+ "### Vertex AI"
+ ]
},
{
"cell_type": "markdown",
+ "metadata": {
+ "id": "wG_2u_jKTyYg"
+ },
"source": [
"When working with VertexAI in a Jupyter notebook, an environment variable called `GOOGLE_APPLICATION_CREDENTIALS` points to a credentials file. This variable is like giving VertexAI a secret map to find the JSON file containing your service account key.\n",
"\n",
@@ -286,44 +268,44 @@
"├── lib32\n",
"└── lib64\n",
"```\n"
- ],
- "metadata": {
- "id": "wG_2u_jKTyYg"
- }
+ ]
},
{
"cell_type": "markdown",
- "source": [
- "### Groq"
- ],
"metadata": {
"id": "0KsLGZFtrZVT"
- }
+ },
+ "source": [
+ "### Groq"
+ ]
},
{
"cell_type": "markdown",
+ "metadata": {
+ "id": "-4RUn2dUrcbZ"
+ },
"source": [
"It's good to make the most of [free resources](https://github.com/BerriAI/litellm/issues/4922#issuecomment-2374234548) like this one! Of course, free options often come with some limitations, and this is no different. Currently, Groq doesn't offer an API for text embeddings.\n",
"\n",
"The Groq team highlights a pre-trained model that can be used locally. You can learn more about it on the [Groq Blog](https://groq.com/retrieval-augmented-generation-with-groq-api/), where they discuss the `jinaai/jina-embeddings-v2-base-en`, a model that can be hosted locally.\n",
"\n",
"This notebook primarily focuses on well-known online API models, so I've commented on the Groq line for text embedding. However, if you're curious, you can explore the model Groq recommends for local use by uncommenting it."
- ],
- "metadata": {
- "id": "-4RUn2dUrcbZ"
- }
+ ]
},
{
"cell_type": "markdown",
- "source": [
- "### Claude Anthropic"
- ],
"metadata": {
"id": "I7a3d8EsrdBc"
- }
+ },
+ "source": [
+ "### Claude Anthropic"
+ ]
},
{
"cell_type": "markdown",
+ "metadata": {
+ "id": "RHmrKCbZriz_"
+ },
"source": [
"While Claude Anthropic isn't a free resource, they're well-known for their top-notch LLM outputs. The only tiny hiccup for this Notebook context is that they don't offer an API for text embeddings.\n",
"\n",
@@ -332,42 +314,42 @@
"The Claude team highlights the Voyage AI embedding model on their documentation page. Voyage is an online resource, though the Claude Anthropic team does not host it.\n",
"\n",
"This notebook is about well-known online API models. So, I've commented on the Claude config line for text embedding. But if you're curious and want to explore the model Claude recommends, uncomment that line and explore away!"
- ],
- "metadata": {
- "id": "RHmrKCbZriz_"
- }
+ ]
},
{
"cell_type": "markdown",
- "source": [
- "### Voyage AI"
- ],
"metadata": {
"id": "IYpWxGBR2COD"
- }
+ },
+ "source": [
+ "### Voyage AI"
+ ]
},
{
"cell_type": "markdown",
- "source": [
- "Voyage AI provides only [top-notch embeddings](https://docs.voyageai.com/docs/introduction). While it doesn't offer [its LLM model](https://docs.voyageai.com/docs/embeddings), that doesn't diminish its offerings. The cool thing? Their embeddings are versatile and can be integrated into any model you choose. So, feel free to experiment and have fun with them! That's the beauty of this notebook - it's all about exploration and learning."
- ],
"metadata": {
"id": "j6vIcO0A2FjD"
- }
+ },
+ "source": [
+ "Voyage AI provides only [top-notch embeddings](https://docs.voyageai.com/docs/introduction). While it doesn't offer [its LLM model](https://docs.voyageai.com/docs/embeddings), that doesn't diminish its offerings. The cool thing? Their embeddings are versatile and can be integrated into any model you choose. So, feel free to experiment and have fun with them! That's the beauty of this notebook - it's all about exploration and learning."
+ ]
},
{
"cell_type": "markdown",
+ "metadata": {
+ "id": "xZg16h4KHpE5"
+ },
"source": [
"## Environment Variables\n",
"\n",
"This session has scripts to reset or set environment variables _(env vars)_. Using env vars is a great way to keep those sensitive API KEY values safe and sound, away from unwanted eyes."
- ],
- "metadata": {
- "id": "xZg16h4KHpE5"
- }
+ ]
},
{
"cell_type": "markdown",
+ "metadata": {
+ "id": "Cqh5j0a4JcHg"
+ },
"source": [
"### Script to Reset Environment Variables\n",
"\n",
@@ -376,32 +358,23 @@
"- **Typos Happen:** We're all human, and typos can sneak in!\n",
"- **Skipping Variables:** Sometimes we need to skip one or another, so we set the env var to empty.\n",
"- **Fresh Start:** After tweaking a script or setup, it's always good to rerun with a fresh environment.\n"
- ],
- "metadata": {
- "id": "Cqh5j0a4JcHg"
- }
+ ]
},
{
"cell_type": "code",
- "source": [
- "if 'MISTRAL_API_KEY' in os.environ:\n",
- " del os.environ['MISTRAL_API_KEY']"
- ],
+ "execution_count": null,
"metadata": {
"id": "VejHDLF1aKub"
},
- "execution_count": null,
- "outputs": []
+ "outputs": [],
+ "source": [
+ "if 'MISTRAL_API_KEY' in os.environ:\n",
+ " del os.environ['MISTRAL_API_KEY']"
+ ]
},
{
"cell_type": "code",
- "source": [
- "for ENV_VARS in [v['ENV_VAR'] for k, v in LLM_MODEL_CONFIG.items()]:\n",
- " for ENV_VAR in ENV_VARS:\n",
- " if ENV_VAR in os.environ:\n",
- " del os.environ[ENV_VAR]\n",
- " print(f'unset env var: {ENV_VAR}')\n"
- ],
+ "execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
@@ -409,11 +382,10 @@
"id": "vzrNvU5XJpIm",
"outputId": "05181b36-caa8-417d-fcd6-159e15118594"
},
- "execution_count": null,
"outputs": [
{
- "output_type": "stream",
"name": "stdout",
+ "output_type": "stream",
"text": [
"unset env var: GEMINI_API_KEY\n",
"unset env var: COHERE_API_KEY\n",
@@ -426,10 +398,20 @@
"unset env var: GOOGLE_VERTEX_LOCATION\n"
]
}
+ ],
+ "source": [
+ "for ENV_VARS in [v['ENV_VAR'] for k, v in LLM_MODEL_CONFIG.items()]:\n",
+ " for ENV_VAR in ENV_VARS:\n",
+ " if ENV_VAR in os.environ:\n",
+ " del os.environ[ENV_VAR]\n",
+ " print(f'unset env var: {ENV_VAR}')\n"
]
},
{
"cell_type": "markdown",
+ "metadata": {
+ "id": "GqZceA9qJIdq"
+ },
"source": [
"### Script to Set Environment Variables\n",
"\n",
@@ -438,25 +420,11 @@
"Once your favorites have been gathered, the script will prompt you to set all the necessary environment variables.\n",
"\n",
"The more LLM models you have enabled, the more prompts you'll see - but don't worry, each environment variable will prompt only once."
- ],
- "metadata": {
- "id": "GqZceA9qJIdq"
- }
+ ]
},
{
"cell_type": "code",
- "source": [
- "for ENV_VAR in ENV_VARS:\n",
- " os.environ[ENV_VAR] = getpass.getpass(f\"enter {ENV_VAR}: \") if not ENV_VAR in os.environ else os.environ[ENV_VAR]\n",
- "\n",
- "LLM_MODELS_WITH_CUSTOM_SETUP = {k: v for k, v in LLM_MODELS.items() if 'ENV_VAR_SETUP' in v}\n",
- "for LLM_MODEL, LLM_CONFIG in LLM_MODELS_WITH_CUSTOM_SETUP.items():\n",
- " print(f\"running custom env var setup for {LLM_MODEL}...\")\n",
- " LLM_CONFIG['ENV_VAR_SETUP']()\n",
- " print(f\"done setup for {LLM_MODEL}\")\n",
- "\n",
- "print(\"all env var set!\")"
- ],
+ "execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
@@ -464,11 +432,10 @@
"id": "zQmXuvbsHSwR",
"outputId": "07e94a99-817b-4fae-e649-c6e0337bb1d1"
},
- "execution_count": null,
"outputs": [
{
- "output_type": "stream",
"name": "stdout",
+ "output_type": "stream",
"text": [
"enter GEMINI_API_KEY: ··········\n",
"enter COHERE_API_KEY: ··········\n",
@@ -488,34 +455,49 @@
"all env var set!\n"
]
}
+ ],
+ "source": [
+ "for ENV_VAR in ENV_VARS:\n",
+ " os.environ[ENV_VAR] = getpass.getpass(f\"enter {ENV_VAR}: \") if not ENV_VAR in os.environ else os.environ[ENV_VAR]\n",
+ "\n",
+ "LLM_MODELS_WITH_CUSTOM_SETUP = {k: v for k, v in LLM_MODELS.items() if 'ENV_VAR_SETUP' in v}\n",
+ "for LLM_MODEL, LLM_CONFIG in LLM_MODELS_WITH_CUSTOM_SETUP.items():\n",
+ " print(f\"running custom env var setup for {LLM_MODEL}...\")\n",
+ " LLM_CONFIG['ENV_VAR_SETUP']()\n",
+ " print(f\"done setup for {LLM_MODEL}\")\n",
+ "\n",
+ "print(\"all env var set!\")"
]
},
{
"cell_type": "markdown",
+ "metadata": {
+ "id": "tnTSFJZcJW5p"
+ },
"source": [
"# Code Snippet\n",
"\n",
"You can run the code snippet after installing the necessary dependencies and setting up the required environment variables!\n",
"\n",
"\n"
- ],
- "metadata": {
- "id": "tnTSFJZcJW5p"
- }
+ ]
},
{
"cell_type": "markdown",
+ "metadata": {
+ "id": "oPq-sHZifHFZ"
+ },
"source": [
"## Introduction\n",
"\n",
"This code snippet is designed to achieve only two tasks: _(i)_ run an LLM Pipeline; _(ii)_ text Embed using the LLM Model."
- ],
- "metadata": {
- "id": "oPq-sHZifHFZ"
- }
+ ]
},
{
"cell_type": "markdown",
+ "metadata": {
+ "id": "gKd9GB4UfJTt"
+ },
"source": [
"The code snippet has two tasks:\n",
"1. **Running an LLM Pipeline**:\n",
@@ -552,97 +534,22 @@
"```\n",
"\n",
"- **friendly reminder**: _the library's author often [points out](https://github.com/neuml/txtai/pull/844#issuecomment-2563561232) that it's [not necessary](https://github.com/neuml/txtai/issues/843#issuecomment-2563244810) to explicitly pass the second argument `method='litellm'`. When you're learning something new, it's okay to avoid relying on shortcuts or \"magic\" until you're more comfortable. Once you understand the library better, you can start using these convenient features to your advantage. In this introduction, I'm intentionally including the second argument `method='litellm'` in the function. However, I'm choosing to leave it out in the Playground section_."
- ],
- "metadata": {
- "id": "gKd9GB4UfJTt"
- }
+ ]
},
{
"cell_type": "markdown",
+ "metadata": {
+ "id": "1I6o5f1ue_KX"
+ },
"source": [
"## Playground\n",
"\n",
"Once you've installed the necessary dependencies and configured the environment variables, you can play with and explore the code snippet. Enjoy your coding journey! 😊\n"
- ],
- "metadata": {
- "id": "1I6o5f1ue_KX"
- }
+ ]
},
{
"cell_type": "code",
- "source": [
- "# A text prompt to run through the LLM pipeline\n",
- "# https://neuml.github.io/txtai/pipeline/text/llm/\n",
- "LLM_PROMPT_INPUT = \"Where is one place you'd go in Washington, DC?\"\n",
- "# The embeddings dataset is versatile! It plays with lists, datasets, or even generators.\n",
- "# https://neuml.github.io/txtai/embeddings/\n",
- "EMBEDDING_DATA = [\n",
- " \"US tops 5 million confirmed virus cases\",\n",
- " \"Canada's last fully intact ice shelf has suddenly collapsed, forming a Manhattan-sized iceberg\",\n",
- " \"Beijing mobilises invasion craft along coast as Taiwan tensions escalate\",\n",
- " \"The National Park Service warns against sacrificing slower friends in a bear attack\",\n",
- " \"Maine man wins $1M from $25 lottery ticket\",\n",
- " \"Make huge profits without work, earn up to $100,000 a day\"\n",
- "]\n",
- "\n",
- "# https://neuml.github.io/txtai/pipeline/text/llm/\n",
- "def run_llm_api(prompt=\"\", llm_model_name=None):\n",
- " if llm_model_name is None:\n",
- " return\n",
- " else:\n",
- " # A quick note: you can skip specifying the `method` argument.\n",
- " # There's an autodetection logic designed to recognize it as a `litellm` model.\n",
- " # ```\n",
- " # llm = LLM(LLM_MODEL_NAME, method=\"litellm\")\n",
- " # ```\n",
- " llm = LLM(llm_model_name)\n",
- " # OR: print(llm(LLM_PROMPT_INPUT, defaultrole=\"user\"))\n",
- " print(llm([{\"role\": \"user\", \"content\": prompt}]))\n",
- "\n",
- "# https://neuml.github.io/txtai/embeddings/\n",
- "def run_text_embedding_api(embedding_data=None, text_embedding_path=None):\n",
- " if text_embedding_path is None:\n",
- " return\n",
- " else:\n",
- " embeddings = Embeddings(\n",
- " path=text_embedding_path,\n",
- " # a quick note: you can skip specifying the `method` argument - there is autodetection logic\n",
- " # method=\"litellm\"\n",
- " )\n",
- " # create an index for the list of text\n",
- " embeddings.index(embedding_data)\n",
- " print(\".\" * 50)\n",
- " print(\"%-20s %s\" % (\"Query\", \"Best Match\"))\n",
- " print(\".\" * 50)\n",
- " # run an embeddings search for each query\n",
- " for query in (\"feel good story\", \"climate change\",\n",
- " \"public health story\", \"war\", \"wildlife\", \"asia\",\n",
- " \"lucky\", \"dishonest junk\"):\n",
- " # extract uid of first result\n",
- " # search result format: (uid, score)\n",
- " uid = embeddings.search(query, 1)[0][0]\n",
- " # print text\n",
- " print(\"%-20s %s\" % (query, embedding_data[uid]))\n",
- "\n",
- "# Let's LOOP THROUGH each enabled LLM and embedding model.\n",
- "for LLM_MODEL, LLM_CONFIG in LLM_MODELS.items():\n",
- " LLM_MODEL_NAME = LLM_CONFIG['LLM_MODEL_NAME'] if 'LLM_MODEL_NAME' in LLM_CONFIG else None\n",
- " TEXT_EMBEDDING_PATH = LLM_CONFIG['TEXT_EMBEDDING_PATH'] if 'TEXT_EMBEDDING_PATH' in LLM_CONFIG else None\n",
- " print(\"-\" * 50)\n",
- " print(LLM_MODEL)\n",
- " # https://neuml.github.io/txtai/pipeline/text/llm/\n",
- " run_llm_api(\n",
- " prompt=LLM_PROMPT_INPUT,\n",
- " llm_model_name=LLM_MODEL_NAME\n",
- " )\n",
- " # https://neuml.github.io/txtai/embeddings/\n",
- " run_text_embedding_api(\n",
- " embedding_data=EMBEDDING_DATA,\n",
- " text_embedding_path=TEXT_EMBEDDING_PATH\n",
- " )\n",
- "\n",
- "print(\"-\" * 50)"
- ],
+ "execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
@@ -650,11 +557,10 @@
"id": "A3NetE3Sh0e0",
"outputId": "d9be990a-1f57-4abc-f2be-86f8e8ef7c6c"
},
- "execution_count": null,
"outputs": [
{
- "output_type": "stream",
"name": "stdout",
+ "output_type": "stream",
"text": [
"--------------------------------------------------\n",
"GEMINI\n",
@@ -786,7 +692,101 @@
"--------------------------------------------------\n"
]
}
+ ],
+ "source": [
+ "# A text prompt to run through the LLM pipeline\n",
+ "# https://neuml.github.io/txtai/pipeline/text/llm/\n",
+ "LLM_PROMPT_INPUT = \"Where is one place you'd go in Washington, DC?\"\n",
+ "\n",
+ "# The embeddings dataset is versatile! It plays with lists, datasets, or even generators.\n",
+ "# https://neuml.github.io/txtai/embeddings/\n",
+ "EMBEDDING_DATA = [\n",
+ " \"US tops 5 million confirmed virus cases\",\n",
+ " \"Canada's last fully intact ice shelf has suddenly collapsed, forming a Manhattan-sized iceberg\",\n",
+ " \"Beijing mobilises invasion craft along coast as Taiwan tensions escalate\",\n",
+ " \"The National Park Service warns against sacrificing slower friends in a bear attack\",\n",
+ " \"Maine man wins $1M from $25 lottery ticket\",\n",
+ " \"Make huge profits without work, earn up to $100,000 a day\"\n",
+ "]\n",
+ "\n",
+ "# https://neuml.github.io/txtai/pipeline/text/llm/\n",
+ "def runllm(prompt=\"\", path=None):\n",
+ " if path:\n",
+ " # A quick note: you can skip specifying the `method` argument.\n",
+ " # There's an autodetection logic designed to recognize it as a `litellm` model.\n",
+ " # \n",
+ " # llm = LLM(LLM_MODEL_NAME, method=\"litellm\")\n",
+ " # \n",
+ " llm = LLM(path)\n",
+ "\n",
+ " # OR: print(llm(LLM_PROMPT_INPUT, defaultrole=\"user\"))\n",
+ " print(llm([{\"role\": \"user\", \"content\": prompt}]))\n",
+ "\n",
+ "# https://neuml.github.io/txtai/embeddings/\n",
+ "def runembeddings(data=None, path=None):\n",
+ " if path:\n",
+ " embeddings = Embeddings(\n",
+ " path=path,\n",
+ " # a quick note: you can skip specifying the `method` argument - there is autodetection logic\n",
+ " # method=\"litellm\"\n",
+ " )\n",
+ " # create an index for the list of text\n",
+ " embeddings.index(data)\n",
+ "\n",
+ " print(\".\" * 50)\n",
+ " print(\"%-20s %s\" % (\"Query\", \"Best Match\"))\n",
+ " print(\".\" * 50)\n",
+ " \n",
+ " # run an embeddings search for each query\n",
+ " for query in (\"feel good story\", \"climate change\",\n",
+ " \"public health story\", \"war\", \"wildlife\", \"asia\",\n",
+ " \"lucky\", \"dishonest junk\"):\n",
+ " # extract uid of first result\n",
+ " # search result format: (uid, score)\n",
+ " uid = embeddings.search(query, 1)[0][0]\n",
+ " # print text\n",
+ " print(\"%-20s %s\" % (query, data[uid]))\n",
+ "\n",
+ "# Let's LOOP THROUGH each enabled LLM and embedding model.\n",
+ "for LLM_MODEL, LLM_CONFIG in LLM_MODELS.items():\n",
+ " LLM_MODEL_NAME = LLM_CONFIG['LLM_MODEL_NAME'] if 'LLM_MODEL_NAME' in LLM_CONFIG else None\n",
+ " TEXT_EMBEDDING_PATH = LLM_CONFIG['TEXT_EMBEDDING_PATH'] if 'TEXT_EMBEDDING_PATH' in LLM_CONFIG else None\n",
+ " print(\"-\" * 50)\n",
+ " print(LLM_MODEL)\n",
+ "\n",
+ " # https://neuml.github.io/txtai/pipeline/text/llm/\n",
+ " runllm(prompt=LLM_PROMPT_INPUT, path=LLM_MODEL_NAME)\n",
+ "\n",
+ " # https://neuml.github.io/txtai/embeddings/\n",
+ " runembeddings(data=EMBEDDING_DATA, path=TEXT_EMBEDDING_PATH)\n",
+ "\n",
+ "print(\"-\" * 50)"
]
}
- ]
+ ],
+ "metadata": {
+ "colab": {
+ "collapsed_sections": [
+ "lhrk9y766Lw1",
+ "wuCvBPCL-VQG",
+ "pgQuDv4fZgHe",
+ "0KsLGZFtrZVT",
+ "I7a3d8EsrdBc",
+ "xZg16h4KHpE5",
+ "Cqh5j0a4JcHg",
+ "oPq-sHZifHFZ"
+ ],
+ "provenance": [],
+ "toc_visible": true
+ },
+ "kernelspec": {
+ "display_name": "Python 3",
+ "name": "python3"
+ },
+ "language_info": {
+ "name": "python"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 0
}