examples/notebooks/langchain/RAG_with_sources_Langchain-vLLM-Milvus.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "2a331fd6",
   "metadata": {},
   "source": [
    "## RAG example with Langchain, Milvus, and vLLM\n",
    "\n",
    "Requirements:\n",
    "- A Milvus instance, either standalone or cluster.\n",
    "- Connection credentials to Milvus must be available as environment variables: MILVUS_USERNAME and MILVUS_PASSWORD.\n",
    "- A vLLM inference endpoint. In this example we use the OpenAI Compatible API."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e712b3e8-f406-4387-9188-3e2f20a6841f",
   "metadata": {},
   "source": [
    "### Needed packages and imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "d4a359bd-4f69-4e88-82c0-5763c26aa0af",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip available: \u001b[0m\u001b[31;49m22.2.2\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m24.0\u001b[0m\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!pip install -q einops==0.7.0 langchain==0.1.9 pymilvus==2.3.6 sentence-transformers==2.4.0 openai==1.13.3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "83e11d23-c0ad-4875-b67f-149fc8b14725",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import os\n",
    "from langchain.callbacks.base import BaseCallbackHandler\n",
    "from langchain.chains import RetrievalQA\n",
    "from langchain.embeddings.huggingface import HuggingFaceEmbeddings\n",
    "from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler\n",
    "from langchain_community.llms import VLLMOpenAI\n",
    "from langchain.prompts import PromptTemplate\n",
    "from langchain_community.vectorstores import Milvus"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9cd4537b",
   "metadata": {},
   "source": [
    "#### Bases parameters, Inference server and Milvus info"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "51baf1a6-4111-4b40-b43a-833438bdc222",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Replace values according to your Milvus deployment\n",
    "INFERENCE_SERVER_URL = \"http://vllm.llm-hosting.svc.cluster.local:8000/v1\"\n",
    "MODEL_NAME = \"mistralai/Mistral-7B-Instruct-v0.2\"\n",
    "MAX_TOKENS=1024\n",
    "TOP_P=0.95\n",
    "TEMPERATURE=0.01\n",
    "PRESENCE_PENALTY=1.03\n",
    "MILVUS_HOST = \"vectordb-milvus.milvus.svc.cluster.local\"\n",
    "MILVUS_PORT = 19530\n",
    "MILVUS_USERNAME = os.getenv('MILVUS_USERNAME')\n",
    "MILVUS_PASSWORD = os.getenv('MILVUS_PASSWORD')\n",
    "MILVUS_COLLECTION = \"red_hat_openshift_ai_self_managed_2_6\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fe4c1b1a",
   "metadata": {},
   "source": [
    "#### Initialize the connection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "bbb6a3e3-5ccd-441e-b80d-427555d9e9f6",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "You try to use a model that was created with version 2.4.0.dev0, however, your version is 2.4.0. This might cause unexpected behavior or errors. In that case, try to update to the latest version.\n",
      "\n",
      "\n",
      "\n",
      "<All keys matched successfully>\n"
     ]
    }
   ],
   "source": [
    "model_kwargs = {'trust_remote_code': True}\n",
    "embeddings = HuggingFaceEmbeddings(\n",
    "    model_name=\"nomic-ai/nomic-embed-text-v1\",\n",
    "    model_kwargs=model_kwargs,\n",
    "    show_progress=False\n",
    ")\n",
    "\n",
    "store = Milvus(\n",
    "    embedding_function=embeddings,\n",
    "    connection_args={\"host\": MILVUS_HOST, \"port\": MILVUS_PORT, \"user\": MILVUS_USERNAME, \"password\": MILVUS_PASSWORD},\n",
    "    collection_name=MILVUS_COLLECTION,\n",
    "    metadata_field=\"metadata\",\n",
    "    text_field=\"page_content\",\n",
    "    drop_old=False\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b72a3a2b",
   "metadata": {},
   "source": [
    "#### Initialize query chain"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "ed8fd396-0798-45c5-8969-6b6ede134c77",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "template=\"\"\"<s>[INST] <<SYS>>\n",
    "You are a helpful, respectful and honest assistant named HatBot answering questions.\n",
    "You will be given a question you need to answer, and a context to provide you with information. You must answer the question based as much as possible on this context.\n",
    "Always answer as helpfully as possible, while being safe. Your answers should not include any harmful, unethical, racist, sexist, toxic, dangerous, or illegal content. Please ensure that your responses are socially unbiased and positive in nature.\n",
    "\n",
    "If a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. If you don't know the answer to a question, please don't share false information.\n",
    "<</SYS>>\n",
    "\n",
    "Context: \n",
    "{context}\n",
    "\n",
    "Question: {question} [/INST]\n",
    "\"\"\"\n",
    "\n",
    "QA_CHAIN_PROMPT = PromptTemplate.from_template(template)\n",
    "\n",
    "llm =  VLLMOpenAI(\n",
    "    openai_api_key=\"EMPTY\",\n",
    "    openai_api_base=INFERENCE_SERVER_URL,\n",
    "    model_name=MODEL_NAME,\n",
    "    max_tokens=MAX_TOKENS,\n",
    "    top_p=TOP_P,\n",
    "    temperature=TEMPERATURE,\n",
    "    presence_penalty=PRESENCE_PENALTY,\n",
    "    streaming=True,\n",
    "    verbose=False,\n",
    "    callbacks=[StreamingStdOutCallbackHandler()]\n",
    ")\n",
    "\n",
    "qa_chain = RetrievalQA.from_chain_type(\n",
    "        llm,\n",
    "        retriever=store.as_retriever(\n",
    "            search_type=\"similarity\",\n",
    "            search_kwargs={\"k\": 4}\n",
    "            ),\n",
    "        chain_type_kwargs={\"prompt\": QA_CHAIN_PROMPT},\n",
    "        return_source_documents=True\n",
    "        )\n",
    "\n",
    "os.environ[\"TOKENIZERS_PARALLELISM\"] = \"false\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3a45ad23",
   "metadata": {},
   "source": [
    "#### Query example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "105d2fd1-f36c-409d-8e52-ec6d23a56ad1",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "To create a Data Science Project in Red Hat OpenShift AI, follow these steps:\n",
      "\n",
      "1. From the OpenShift AI dashboard, select \"Data Science Projects\" from the navigation menu. This page lists any existing projects that you have access to.\n",
      "2. If you don't have an active project that you'd like to use, click \"Create data science project\".\n",
      "3. Enter a display name and description for your project. Based on the display name, a resource name will be automatically generated, but you can change it if you'd like. The resource name must consist of lowercase alphanumeric characters, hyphens, and must start and end with an alphanumeric character.\n",
      "4. Click \"Create\" to create the project.\n",
      "\n",
      "Verification: You can now see the newly created project on the \"Data Science Projects\" page."
     ]
    }
   ],
   "source": [
    "question = \"How can I create a Data Science Project?\"\n",
    "result = qa_chain.invoke({\"query\": question})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "97d75d0c",
   "metadata": {},
   "source": [
    "#### Retrieve source"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "acda357e-558a-4879-8ad8-21f0567f2f2e",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "https://access.redhat.com/documentation/en-us/red_hat_openshift_ai_self-managed/2.6/html-single/getting_started_with_red_hat_openshift_ai_self-managed/index\n",
      "https://access.redhat.com/documentation/en-us/red_hat_openshift_ai_self-managed/2.6/html-single/openshift_ai_tutorial_-_fraud_detection_example/index\n",
      "https://access.redhat.com/documentation/en-us/red_hat_openshift_ai_self-managed/2.6/html-single/working_on_data_science_projects/index\n"
     ]
    }
   ],
   "source": [
    "def remove_duplicates(input_list):\n",
    "    unique_list = []\n",
    "    for item in input_list:\n",
    "        if item.metadata['source'] not in unique_list:\n",
    "            unique_list.append(item.metadata['source'])\n",
    "    return unique_list\n",
    "\n",
    "results = remove_duplicates(result['source_documents'])\n",
    "\n",
    "for s in results:\n",
    "    print(s)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.9",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.18"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}