04-Adding_Memory.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "01a8b5c0-87cb-4302-8e3c-dc809d0039fb",
   "metadata": {},
   "source": [
    "# Understanding Memory in LLMs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a2f73380-6395-4e9f-9c83-3f47a5d7e292",
   "metadata": {},
   "source": [
    "In the previous Notebook 03, we successfully explored how OpenAI models can enhance the results from Azure Cognitive Search. [Bing Chat](http://chat.bing.com/) is a search engine with a GPT-4 model that utilizes the content of search results to provide context and deliver accurate responses to queries.\n",
    "\n",
    "However, we have yet to discover how to engage in a conversation with the LLM. With Bing Chat, this is possible, as the LLM can understand and reference the previous responses.\n",
    "\n",
    "There is a common misconception that GPT models have memory. This is not true. While they possess knowledge, they do not retain information from previous questions asked to them.\n",
    "\n",
    "The aim of this Notebook is to demonstrate how we can \"provide memory\" to the LLM by utilizing prompts and context."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "733c782e-204c-47d0-8dae-c9df7091ab23",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import random\n",
    "from langchain.chat_models import AzureChatOpenAI\n",
    "from langchain.chains import LLMChain\n",
    "from langchain.prompts import PromptTemplate\n",
    "from langchain.chains import ConversationChain\n",
    "from langchain.chains.conversational_retrieval.prompts import CONDENSE_QUESTION_PROMPT\n",
    "from langchain.memory import ConversationBufferMemory, ConversationTokenBufferMemory\n",
    "from openai.error import OpenAIError\n",
    "from langchain.docstore.document import Document\n",
    "from langchain.memory import CosmosDBChatMessageHistory\n",
    "\n",
    "from IPython.display import Markdown, HTML, display  \n",
    "\n",
    "def printmd(string):\n",
    "    display(Markdown(string))\n",
    "\n",
    "#custom libraries that we will use later in the app\n",
    "from common.utils import (\n",
    "    get_search_results,\n",
    "    order_search_results,\n",
    "    model_tokens_limit,\n",
    "    num_tokens_from_docs,\n",
    "    embed_docs,\n",
    "    search_docs,\n",
    "    get_answer,\n",
    ")\n",
    "\n",
    "from common.prompts import COMBINE_QUESTION_PROMPT, COMBINE_PROMPT, COMBINE_CHAT_PROMPT\n",
    "\n",
    "from dotenv import load_dotenv\n",
    "load_dotenv(\"credentials.env\")\n",
    "\n",
    "import logging\n",
    "\n",
    "# Get the root logger\n",
    "logger = logging.getLogger()\n",
    "# Set the logging level to a higher level to ignore INFO messages\n",
    "logger.setLevel(logging.WARNING)\n",
    "\n",
    "DATASOURCE_SAS_TOKEN = \"?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2023-11-29T01:50:59Z&st=2023-05-10T16:50:59Z&spr=https&sig=ZT7MLy%2BnlvAxUKKj5v0RwoWObXaab3gO4ec2%2Fci6iL0%3D\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "6bc63c55-a57d-49a7-b6c7-0f18bca8199e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set the ENV variables that Langchain needs to connect to Azure OpenAI\n",
    "os.environ[\"OPENAI_API_BASE\"] = os.environ[\"AZURE_OPENAI_ENDPOINT\"]\n",
    "os.environ[\"OPENAI_API_KEY\"] = os.environ[\"AZURE_OPENAI_API_KEY\"]\n",
    "os.environ[\"OPENAI_API_VERSION\"] = os.environ[\"AZURE_OPENAI_API_VERSION\"]\n",
    "os.environ[\"OPENAI_API_TYPE\"] = \"azure\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3dc72b22-11c2-4df0-91b8-033d01829663",
   "metadata": {},
   "source": [
    "### Let's start with the basics\n",
    "Let's use a very simple example to see if the GPT model of Azure OpenAI have memory. We again will be using langchain to simplify our code "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "3eef5dc9-8b80-4085-980c-865fa41fa1f6",
   "metadata": {},
   "outputs": [],
   "source": [
    "QUESTION = \"Tell me some use cases for reinforcement learning?\"\n",
    "FOLLOW_UP_QUESTION = \"Can you summarize your last response?\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "a00181d5-bd76-4ce4-a256-75ac5b58c60f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define model\n",
    "MODEL = \"gpt-35-turbo\"\n",
    "# Create an OpenAI instance\n",
    "llm = AzureChatOpenAI(deployment_name=MODEL, temperature=0.5, max_tokens=1000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "9502d0f1-fddf-40d1-95d2-a1461dcc498a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# We create a very simple prompt template, just the question as is:\n",
    "prompt = PromptTemplate(\n",
    "    input_variables=[\"question\"],\n",
    "    template=\"{question}\",\n",
    ")\n",
    "\n",
    "chain = LLMChain(llm=llm, prompt=prompt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "c5c9903e-15c7-4e05-87a1-58e5a7917ba2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "Reinforcement learning can be used in a variety of applications, including:\n",
       "\n",
       "1. Game playing: Reinforcement learning algorithms have been used to train agents to play games such as chess, Go, and poker.\n",
       "\n",
       "2. Robotics: Reinforcement learning can be used to train robots to perform tasks such as grasping objects, walking, and navigating in complex environments.\n",
       "\n",
       "3. Autonomous driving: Reinforcement learning can be used to train self-driving cars to make decisions based on real-world data.\n",
       "\n",
       "4. Personalized recommendation systems: Reinforcement learning can be used to learn user preferences and recommend products or services that are tailored to their individual needs.\n",
       "\n",
       "5. Energy management: Reinforcement learning can be used to optimize energy consumption in buildings and other systems.\n",
       "\n",
       "6. Healthcare: Reinforcement learning can be used to develop personalized treatment plans for patients based on their individual health data.\n",
       "\n",
       "7. Advertising: Reinforcement learning can be used to optimize ad placement and targeting to maximize revenue.\n",
       "\n",
       "8. Finance: Reinforcement learning can be used to develop trading algorithms that learn to make profitable trades based on market data."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Let's see what the GPT model responds\n",
    "response = chain.run(QUESTION)\n",
    "printmd(response)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "99acaf3c-ce68-4b87-b24a-6065b15ff9a8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"I'm sorry, as an AI language model, I cannot summarize my last response without knowing which response you are referring to. Please provide me with more context.\""
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Now let's ask a follow up question\n",
    "chain.run(FOLLOW_UP_QUESTION)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a3e1c143-c95f-4566-a8b4-af8c42f08dd2",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "tags": []
   },
   "source": [
    "As you can see, it doesn't remember what it just responded. This proof that the LLM does NOT have memory and that we need to give the memory as a a conversation history as part of the prompt, like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "0946ce71-6285-432e-b011-9c2dc1ba7b8a",
   "metadata": {},
   "outputs": [],
   "source": [
    "hist_prompt = PromptTemplate(\n",
    "    input_variables=[\"history\", \"question\"],\n",
    "    template=\"\"\"\n",
    "                {history}\n",
    "                Human: {question}\n",
    "                AI:\n",
    "            \"\"\"\n",
    "    )\n",
    "chain = LLMChain(llm=llm, prompt=hist_prompt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "6d088e51-e5eb-4143-b87d-b2be429eb864",
   "metadata": {},
   "outputs": [],
   "source": [
    "Conversation_history = \"\"\"\n",
    "Human: {question}\n",
    "AI: {response}\n",
    "\"\"\".format(question=QUESTION, response=response)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "d99e34ad-5539-44dd-b080-3ad05efd2f01",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Reinforcement learning can be used in various applications such as game playing, robotics, autonomous driving, personalized recommendation systems, energy management, healthcare, advertising, and finance. It involves training algorithms to make decisions based on real-world data and optimize outcomes.'"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "chain.run({\"history\":Conversation_history, \"question\": FOLLOW_UP_QUESTION})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "045e5af6-55d6-4353-b3f6-3275c95db00a",
   "metadata": {},
   "source": [
    "**Bingo!**, so we now know how to create a chatbot using LLMs, we just need to keep the state/history of the conversation and pass it as context every time"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eafd1694-0077-4aa8-bd01-e9f763ce36a3",
   "metadata": {},
   "source": [
    "## Now that we understand the concept of memory via adding history as a context, let's go back to our GPT Smart Search engine"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d8b27c45-7fbb-40da-a2e3-61e66a8e49b0",
   "metadata": {},
   "source": [
    "In order to not duplicate code, we have put many of the code used in Notebook 3 into functions. These functions are in the `common/utils.py` and `common/prompts.py` files This way we can use these functios in the app that we will build later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "ef9f459b-e8b8-40b9-a94d-80c079968594",
   "metadata": {},
   "outputs": [],
   "source": [
    "index1_name = \"cogsrch-index-files\"\n",
    "index2_name = \"cogsrch-index-csv\"\n",
    "indexes = [index1_name, index2_name]\n",
    "\n",
    "agg_search_results = get_search_results(QUESTION, indexes)\n",
    "ordered_results = order_search_results(agg_search_results, reranker_threshold=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "9b2a3595-c3b7-4376-b9c5-0db7a42b3ee4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Custom token limit for gpt-35-turbo-16k : 14500\n",
      "Combined docs tokens count: 112561\n"
     ]
    }
   ],
   "source": [
    "docs = []\n",
    "for key,value in ordered_results.items():\n",
    "    for page in value[\"chunks\"]:\n",
    "        location = value[\"location\"] if value[\"location\"] is not None else \"\"\n",
    "        docs.append(Document(page_content=page, metadata={\"source\": location+DATASOURCE_SAS_TOKEN}))\n",
    "\n",
    "# Calculate number of tokens of our docs\n",
    "tokens_limit = model_tokens_limit(MODEL)\n",
    "\n",
    "if(len(docs)>0):\n",
    "    num_tokens = num_tokens_from_docs(docs)\n",
    "    print(\"Custom token limit for\", MODEL, \":\", tokens_limit)\n",
    "    print(\"Combined docs tokens count:\",num_tokens)\n",
    "        \n",
    "else:\n",
    "    print(\"NO RESULTS FROM AZURE SEARCH\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "c26d7540-feb8-4581-849e-003f4bf2a601",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Token count after similarity search: 2169\n",
      "Chain Type selected: stuff\n",
      "CPU times: user 728 ms, sys: 27.2 ms, total: 755 ms\n",
      "Wall time: 9.67 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "if num_tokens > tokens_limit:\n",
    "    index = embed_docs(docs)\n",
    "    top_docs = search_docs(index,QUESTION,k=2)\n",
    "    \n",
    "    # Now we need to recalculate the tokens count of the top results from similarity vector search\n",
    "    # in order to select the chain type: stuff or map_reduce\n",
    "    \n",
    "    num_tokens = num_tokens_from_docs(top_docs)   \n",
    "    print(\"Token count after similarity search:\", num_tokens)\n",
    "    chain_type = \"map_reduce\" if num_tokens > tokens_limit else \"stuff\"\n",
    "    \n",
    "else:\n",
    "    # if total tokens is less than our limit, we don't need to vectorize and do similarity search\n",
    "    top_docs = docs\n",
    "    chain_type = \"stuff\"\n",
    "    \n",
    "print(\"Chain Type selected:\", chain_type)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "3ce6efa9-2b8f-4810-904d-5986b4ae0372",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "Reinforcement learning has been used as a practical computational tool for constructing autonomous systems that improve themselves with experience in various applications, such as robotics, industrial manufacturing, and computer game playing <sup><a href=\"https://demodatasetsp.blob.core.windows.net/arxivcs/9605/9605103v1.pdf?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2023-11-29T01:50:59Z&st=2023-05-10T16:50:59Z&spr=https&sig=ZT7MLy%2BnlvAxUKKj5v0RwoWObXaab3gO4ec2%2Fci6iL0%3D\" target=\"_blank\">[1]</a></sup>. Some use cases include game playing, where reinforcement learning algorithms can be adapted to work for a very general class of games, such as backgammon, which has been used as a test case for reinforcement learning, and robotics, where shaping, local reinforcement signals, imitation, problem decomposition, and reflexes can be used to bias the learning process <sup><a href=\"https://demodatasetsp.blob.core.windows.net/arxivcs/9605/9605103v1.pdf?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2023-11-29T01:50:59Z&st=2023-05-10T16:50:59Z&spr=https&sig=ZT7MLy%2BnlvAxUKKj5v0RwoWObXaab3gO4ec2%2Fci6iL0%3D\" target=\"_blank\">[1]</a></sup>."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Get the answer\n",
    "response = get_answer(llm=llm, docs=top_docs, query=QUESTION, language=\"English\", chain_type=chain_type)\n",
    "printmd(response['output_text'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "27501f1b-7db0-4ee3-9cb1-e609254ffa3d",
   "metadata": {},
   "source": [
    "And if we ask the follow up question:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "5cf5b323-3b9c-479b-8502-acfc4f7915dd",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "The article discusses the challenges of scaling reinforcement learning techniques to larger problems and suggests incorporating bias, such as shaping, local reinforcement signals, imitation, problem decomposition, and reflexes, to aid learning. It also provides examples of practical applications of reinforcement learning, including game playing, and how they can inform research questions. No specific summary of the previous response was provided. <sup><a href=\"https://demodatasetsp.blob.core.windows.net/arxivcs/9605/9605103v1.pdf?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2023-11-29T01:50:59Z&st=2023-05-10T16:50:59Z&spr=https&sig=ZT7MLy%2BnlvAxUKKj5v0RwoWObXaab3gO4ec2%2Fci6iL0%3D\" target=\"_blank\">[1]</a></sup>."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "response = get_answer(llm=llm, docs=top_docs,  query=FOLLOW_UP_QUESTION, language=\"English\", chain_type=chain_type)\n",
    "printmd(response['output_text'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "035fa6e6-226c-400f-a504-30255385f43b",
   "metadata": {},
   "source": [
    "You might get a different response from above, but it doesn't matter what response you get, it will be based on the context given, not on previous answers.\n",
    "\n",
    "Until now we just have the same as the prior Notebook 03: results from Azure Search enhanced by OpenAI model, with no memory\n",
    "\n",
    "**Now let's add memory to it:**\n",
    "\n",
    "Reference: https://python.langchain.com/docs/modules/memory/how_to/adding_memory_chain_multiple_inputs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "d98b876e-d264-48ae-b5ed-9801d6a9152b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "Reinforcement learning has been applied in various practical computational tools for constructing autonomous systems that improve themselves with experience, including robotics, industrial manufacturing, and computer game playing. In game playing, reinforcement-learning algorithms can be adapted to work for a very general class of games, and many researchers have used reinforcement learning in these environments. One example is TD-Gammon, which applied the temporal difference algorithm to backgammon, using a neural network as a function approximator for the value function. TD-Gammon won none of the backgammon tournaments but came sufficiently close that it is now considered one of the best few players in the world<sup><a href=\"https://demodatasetsp.blob.core.windows.net/arxivcs/9605/9605103v1.pdf?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2023-11-29T01:50:59Z&st=2023-05-10T16:50:59Z&spr=https&sig=ZT7MLy%2BnlvAxUKKj5v0RwoWObXaab3gO4ec2%2Fci6iL0%3D\" target=\"_blank\">[1]</a></sup>. Other use cases for reinforcement learning include industrial automation and control, robotics, and combinatorial search problems<sup><a href=\"https://demodatasetsp.blob.core.windows.net/arxivcs/9605/9605103v1.pdf?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2023-11-29T01:50:59Z&st=2023-05-10T16:50:59Z&spr=https&sig=ZT7MLy%2BnlvAxUKKj5v0RwoWObXaab3gO4ec2%2Fci6iL0%3D\" target=\"_blank\">[1]</a></sup>."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# memory object, which is neccessary to track the inputs/outputs and hold a conversation.\n",
    "memory = ConversationBufferMemory(memory_key=\"chat_history\",input_key=\"question\")\n",
    "\n",
    "response = get_answer(llm=llm, docs=top_docs, query=QUESTION, language=\"English\", chain_type=chain_type, \n",
    "                        memory=memory)\n",
    "printmd(response['output_text'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "bf28927b-d9ee-4412-bb07-13e055e832a7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "I previously provided information on various practical computational tools that have applied reinforcement learning, including robotics, industrial manufacturing, and computer game playing, among others. One example of reinforcement learning applied in game playing is TD-Gammon, which applied the temporal difference algorithm to backgammon, using a neural network as a function approximator for the value function. Other use cases for reinforcement learning include industrial automation and control, robotics, and combinatorial search problems<sup><a href=\"https://demodatasetsp.blob.core.windows.net/arxivcs/9605/9605103v1.pdf?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2023-11-29T01:50:59Z&st=2023-05-10T16:50:59Z&spr=https&sig=ZT7MLy%2BnlvAxUKKj5v0RwoWObXaab3gO4ec2%2Fci6iL0%3D\" target=\"_blank\">[1]</a></sup>. Would you like me to provide more information on any specific use case?"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Now we add a follow up question:\n",
    "response = get_answer(llm=llm, docs=top_docs, query=FOLLOW_UP_QUESTION, language=\"English\", chain_type=chain_type, \n",
    "                      memory=memory)\n",
    "printmd(response['output_text'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "3830b0b8-0ca2-4d0a-9747-f6273368002b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "I'm sorry, I don't understand what you are asking for. Can you please clarify?"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Another follow up query\n",
    "response = get_answer(llm=llm, docs=top_docs, query=\"Thank you\", language=\"English\", chain_type=chain_type,  \n",
    "                      memory=memory)\n",
    "printmd(response['output_text'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "111e732b-3c8c-4df3-8fcb-c3d01e7bec74",
   "metadata": {},
   "source": [
    "You might get a different answer on the above cell, and it is ok, this bot is not yet well configured to answer any question that is not related to its knowledge base, including salutations.\n",
    "\n",
    "Let's check our memory to see that it's keeping the conversation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "1279692c-7eb0-4300-8a66-c7025f02c318",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Human: Tell me some use cases for reinforcement learning?\\nAI: Reinforcement learning has been applied in various practical computational tools for constructing autonomous systems that improve themselves with experience, including robotics, industrial manufacturing, and computer game playing. In game playing, reinforcement-learning algorithms can be adapted to work for a very general class of games, and many researchers have used reinforcement learning in these environments. One example is TD-Gammon, which applied the temporal difference algorithm to backgammon, using a neural network as a function approximator for the value function. TD-Gammon won none of the backgammon tournaments but came sufficiently close that it is now considered one of the best few players in the world<sup><a href=\"https://demodatasetsp.blob.core.windows.net/arxivcs/9605/9605103v1.pdf?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2023-11-29T01:50:59Z&st=2023-05-10T16:50:59Z&spr=https&sig=ZT7MLy%2BnlvAxUKKj5v0RwoWObXaab3gO4ec2%2Fci6iL0%3D\" target=\"_blank\">[1]</a></sup>. Other use cases for reinforcement learning include industrial automation and control, robotics, and combinatorial search problems<sup><a href=\"https://demodatasetsp.blob.core.windows.net/arxivcs/9605/9605103v1.pdf?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2023-11-29T01:50:59Z&st=2023-05-10T16:50:59Z&spr=https&sig=ZT7MLy%2BnlvAxUKKj5v0RwoWObXaab3gO4ec2%2Fci6iL0%3D\" target=\"_blank\">[1]</a></sup>.\\nHuman: Can you summarize your last response?\\nAI: I previously provided information on various practical computational tools that have applied reinforcement learning, including robotics, industrial manufacturing, and computer game playing, among others. One example of reinforcement learning applied in game playing is TD-Gammon, which applied the temporal difference algorithm to backgammon, using a neural network as a function approximator for the value function. Other use cases for reinforcement learning include industrial automation and control, robotics, and combinatorial search problems<sup><a href=\"https://demodatasetsp.blob.core.windows.net/arxivcs/9605/9605103v1.pdf?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2023-11-29T01:50:59Z&st=2023-05-10T16:50:59Z&spr=https&sig=ZT7MLy%2BnlvAxUKKj5v0RwoWObXaab3gO4ec2%2Fci6iL0%3D\" target=\"_blank\">[1]</a></sup>. Would you like me to provide more information on any specific use case?\\nHuman: Thank you\\nAI: I\\'m sorry, I don\\'t understand what you are asking for. Can you please clarify?'"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "memory.buffer"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "87405173",
   "metadata": {},
   "source": [
    "## Using CosmosDB as persistent memory\n",
    "\n",
    "In previous cell we have added local RAM memory to our chatbot. However, it is not persistent, it gets deleted once the app user's session is terminated. It is necessary then to use a Database for persistent storage of each of the bot user conversations, not only for Analytics and Auditing, but also if we wisg to provide recommendations. \n",
    "\n",
    "Here we will store the conversation history into CosmosDB for future auditing purpose.\n",
    "We will use a class in LangChain use CosmosDBChatMessageHistory, see [HERE](https://python.langchain.com/en/latest/_modules/langchain/memory/chat_message_histories/cosmos_db.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "c7131daa",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create CosmosDB instance from langchain cosmos class.\n",
    "cosmos = CosmosDBChatMessageHistory(\n",
    "    cosmos_endpoint=os.environ['AZURE_COSMOSDB_ENDPOINT'],\n",
    "    cosmos_database=os.environ['AZURE_COSMOSDB_NAME'],\n",
    "    cosmos_container=os.environ['AZURE_COSMOSDB_CONTAINER_NAME'],\n",
    "    connection_string=os.environ['AZURE_COMOSDB_CONNECTION_STRING'],\n",
    "    session_id=\"Agent-Test-Session\" + str(random.randint(1, 1000)),\n",
    "    user_id=\"Agent-Test-User\" + str(random.randint(1, 1000))\n",
    "    )\n",
    "\n",
    "# prepare the cosmosdb instance\n",
    "cosmos.prepare_cosmos()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "d87cc7c6-5ef1-4492-b133-9f63a392e223",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create or Memory Object\n",
    "memory = ConversationBufferMemory(memory_key=\"chat_history\",input_key=\"question\",chat_memory=cosmos)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "27ceb47a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "Reinforcement learning has been used in a variety of practical computational applications, including robotics, industrial manufacturing, and computer game playing. In game playing, reinforcement learning algorithms can be adapted to work for a very general class of games, and many researchers have used reinforcement learning in these environments. One application, TD-Gammon, used a backpropagation-based three-layer neural network as a function approximator for the value function, and was able to learn to play backgammon at a professional level <sup><a href=\"https://demodatasetsp.blob.core.windows.net/arxivcs/9605/9605103v1.pdf?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2023-11-29T01:50:59Z&st=2023-05-10T16:50:59Z&spr=https&sig=ZT7MLy%2BnlvAxUKKj5v0RwoWObXaab3gO4ec2%2Fci6iL0%3D\">[1]</a></sup>. Additionally, reinforcement learning techniques can be used to solve complex problems by incorporating bias that will give leverage to the learning process. Bias can come in a variety of forms, including shaping, local reinforcement signals, imitation, problem decomposition, and reflexes <sup><a href=\"https://demodatasetsp.blob.core.windows.net/arxivcs/9605/9605103v1.pdf?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2023-11-29T01:50:59Z&st=2023-05-10T16:50:59Z&spr=https&sig=ZT7MLy%2BnlvAxUKKj5v0RwoWObXaab3gO4ec2%2Fci6iL0%3D\">[1]</a></sup>."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Testing using our Question\n",
    "response = get_answer(llm=llm, docs=top_docs, query=QUESTION, language=\"English\", chain_type=chain_type, \n",
    "                        memory=memory)\n",
    "printmd(response['output_text'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "1a5ff826",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'I provided a summary of practical applications of reinforcement learning, including game playing, robotics, and industrial manufacturing<sup><a href=\"https://demodatasetsp.blob.core.windows.net/arxivcs/9605/9605103v1.pdf?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2023-11-29T01:50:59Z&st=2023-05-10T16:50:59Z&spr=https&sig=ZT7MLy%2BnlvAxUKKj5v0RwoWObXaab3gO4ec2%2Fci6iL0%3D\">[1]</a></sup>. Anything else I can help you with?'"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Now we add a follow up question:\n",
    "response = get_answer(llm=llm, docs=top_docs, query=FOLLOW_UP_QUESTION, language=\"English\", chain_type=chain_type, \n",
    "                      memory=memory)\n",
    "printmd(response['output_text'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "be1620fa",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "There are several practical applications of reinforcement learning, including robotics, industrial manufacturing, and computer game playing <sup><a href=\"https://demodatasetsp.blob.core.windows.net/arxivcs/9605/9605103v1.pdf?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2023-11-29T01:50:59Z&st=2023-05-10T16:50:59Z&spr=https&sig=ZT7MLy%2BnlvAxUKKj5v0RwoWObXaab3gO4ec2%2Fci6iL0%3D\">[1]</a></sup>. Reinforcement learning algorithms have been adapted to work for a very general class of games, and many researchers have used reinforcement learning in game playing environments. One application, TD-Gammon, used a backpropagation-based three-layer neural network as a function approximator for the value function, and was able to learn to play backgammon at a professional level <sup><a href=\"https://demodatasetsp.blob.core.windows.net/arxivcs/9605/9605103v1.pdf?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2023-11-29T01:50:59Z&st=2023-05-10T16:50:59Z&spr=https&sig=ZT7MLy%2BnlvAxUKKj5v0RwoWObXaab3gO4ec2%2Fci6iL0%3D\">[1]</a></sup>. Additionally, reinforcement learning techniques can be used to solve complex problems by incorporating bias that will give leverage to the learning process. Bias can come in a variety of forms, including shaping, local reinforcement signals, imitation, problem decomposition, and reflexes <sup><a href=\"https://demodatasetsp.blob.core.windows.net/arxivcs/9605/9605103v1.pdf?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2023-11-29T01:50:59Z&st=2023-05-10T16:50:59Z&spr=https&sig=ZT7MLy%2BnlvAxUKKj5v0RwoWObXaab3gO4ec2%2Fci6iL0%3D\">[1]</a></sup>."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Another follow up query\n",
    "response = get_answer(llm=llm, docs=top_docs, query=\"Thank you\", language=\"English\", chain_type=chain_type,  \n",
    "                      memory=memory)\n",
    "printmd(response['output_text'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cdc5ac98",
   "metadata": {},
   "source": [
    "Let's check our Azure CosmosDB to see the whole conversation\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "e1d7688a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[HumanMessage(content='Tell me some use cases for reinforcement learning?', additional_kwargs={}, example=False),\n",
       " AIMessage(content='Reinforcement learning has been used in a variety of practical computational applications, including robotics, industrial manufacturing, and computer game playing. In game playing, reinforcement learning algorithms can be adapted to work for a very general class of games, and many researchers have used reinforcement learning in these environments. One application, TD-Gammon, used a backpropagation-based three-layer neural network as a function approximator for the value function, and was able to learn to play backgammon at a professional level <sup><a href=\"https://demodatasetsp.blob.core.windows.net/arxivcs/9605/9605103v1.pdf?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2023-11-29T01:50:59Z&st=2023-05-10T16:50:59Z&spr=https&sig=ZT7MLy%2BnlvAxUKKj5v0RwoWObXaab3gO4ec2%2Fci6iL0%3D\">[1]</a></sup>. Additionally, reinforcement learning techniques can be used to solve complex problems by incorporating bias that will give leverage to the learning process. Bias can come in a variety of forms, including shaping, local reinforcement signals, imitation, problem decomposition, and reflexes <sup><a href=\"https://demodatasetsp.blob.core.windows.net/arxivcs/9605/9605103v1.pdf?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2023-11-29T01:50:59Z&st=2023-05-10T16:50:59Z&spr=https&sig=ZT7MLy%2BnlvAxUKKj5v0RwoWObXaab3gO4ec2%2Fci6iL0%3D\">[1]</a></sup>.', additional_kwargs={}, example=False),\n",
       " HumanMessage(content='Thank you', additional_kwargs={}, example=False),\n",
       " AIMessage(content='There are several practical applications of reinforcement learning, including robotics, industrial manufacturing, and computer game playing <sup><a href=\"https://demodatasetsp.blob.core.windows.net/arxivcs/9605/9605103v1.pdf?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2023-11-29T01:50:59Z&st=2023-05-10T16:50:59Z&spr=https&sig=ZT7MLy%2BnlvAxUKKj5v0RwoWObXaab3gO4ec2%2Fci6iL0%3D\">[1]</a></sup>. Reinforcement learning algorithms have been adapted to work for a very general class of games, and many researchers have used reinforcement learning in game playing environments. One application, TD-Gammon, used a backpropagation-based three-layer neural network as a function approximator for the value function, and was able to learn to play backgammon at a professional level <sup><a href=\"https://demodatasetsp.blob.core.windows.net/arxivcs/9605/9605103v1.pdf?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2023-11-29T01:50:59Z&st=2023-05-10T16:50:59Z&spr=https&sig=ZT7MLy%2BnlvAxUKKj5v0RwoWObXaab3gO4ec2%2Fci6iL0%3D\">[1]</a></sup>. Additionally, reinforcement learning techniques can be used to solve complex problems by incorporating bias that will give leverage to the learning process. Bias can come in a variety of forms, including shaping, local reinforcement signals, imitation, problem decomposition, and reflexes <sup><a href=\"https://demodatasetsp.blob.core.windows.net/arxivcs/9605/9605103v1.pdf?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2023-11-29T01:50:59Z&st=2023-05-10T16:50:59Z&spr=https&sig=ZT7MLy%2BnlvAxUKKj5v0RwoWObXaab3gO4ec2%2Fci6iL0%3D\">[1]</a></sup>.', additional_kwargs={}, example=False)]"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#load message from cosmosdb\n",
    "cosmos.load_messages()\n",
    "cosmos.messages"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f5e30694-ae2a-47bb-a5c7-db51ecdbba1e",
   "metadata": {},
   "source": [
    "![CosmosDB Memory](./images/cosmos-chathistory.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6789cada-23a3-451a-a91a-0906ceb0bd14",
   "metadata": {},
   "source": [
    "# Summary\n",
    "##### Adding memory to our application allows the user to have a conversation, however this feature is not something that comes with the LLM, but instead, memory is something that we must provide to the LLM in form of context of the question.\n",
    "\n",
    "We added persitent memory using CosmosDB.\n",
    "\n",
    "We also can notice that the current chain that we are using is smart, but not that much. Although we have given memory to it, it searches for similar docs everytime, it struggles to respond to prompts like: Hello, Thank you, Bye, What's your name, What's the weather and any other task that is not search in the knowledge base.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c629ebf4-aced-45b7-a6a2-315810d37d48",
   "metadata": {},
   "source": [
    "# NEXT\n",
    "We know now how to do a Smart Search Engine that can power a chatbot!! great!\n",
    "\n",
    "But, does this solve all the possible scenarios that a virtual assistant will require?  **What about if the answer to the Smart Search Engine is not related to text, but instead requires to look into tabular data?** The next notebook 05 explains and solves the tabular problem and the concept of Agents"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6d4d9da4-3918-4da6-b235-a3320f0dcb12",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.10 - SDK v2",
   "language": "python",
   "name": "python310-sdkv2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}