tutorials/mixture-of-agents/mixture_of_agents.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Mixture of Agents Powered by Groq using Langchain LCEL\n",
    "\n",
    "Mixture of Agents (MoA) is an advanced approach in the field of Generative AI and Large Language Models (LLMs) that combines multiple AI models to produce more robust and comprehensive responses. This implementation showcases an agentic workflow, where multiple AI agents collaborate to solve complex tasks, leading to more nuanced and reliable outputs than single-model approaches.\n",
    "\n",
    "This notebook demonstrates the implementation of a Mixture of Agents (MoA) architecture using Langchain and Groq. The MoA approach combines multiple open source models to produce responses that are on par or better than SOTA proprietary models like GPT4.\n",
    "\n",
    "This tutorial will walk you through how to:\n",
    "\n",
    "1. Set up the environment and dependencies.\n",
    "2. Create helper functions.\n",
    "3. Configure and build the Mixture of Agents pipeline.\n",
    "4. Chat with the Agent.\n",
    "\n",
    "![Mixture of Agents diagram](moa_diagram.svg)\n",
    "\n",
    "You can create a developer account for free at https://console.groq.com/ and generate a free API key to follow this tutorial!\n",
    "\n",
    "This implementation is based on the research paper:\n",
    "\n",
    "```\n",
    "@article{wang2024mixture,\n",
    "  title={Mixture-of-Agents Enhances Large Language Model Capabilities},\n",
    "  author={Wang, Junlin and Wang, Jue and Athiwaratkun, Ben and Zhang, Ce and Zou, James},\n",
    "  journal={arXiv preprint arXiv:2406.04692},\n",
    "  year={2024}\n",
    "}\n",
    "```\n",
    "The main difference between the implementation by the authors of the paper and this notebook is the addition of configurating system prompts of the agents within the layer.\n",
    "We acknowledge the authors for their contributions to the field and encourage readers to refer to the original paper for a deeper understanding of the Mixture-of-Agents concept."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install langchain -q\n",
    "!pip install langchain_groq -q\n",
    "!pip install langchain_community -q"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import nest_asyncio\n",
    "nest_asyncio.apply()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Set up the Environment and Dependencies\n",
    "\n",
    "To use [Groq](https://groq.com), you need to make sure that `GROQ_API_KEY` is specified as an environment variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "import os\n",
    "\n",
    "os.environ[\"GROQ_API_KEY\"] = \"gsk...\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Create helper functions\n",
    "\n",
    "To help us configure our agentic workflow pipeline, we will need some helper functions:\n",
    "- `create_agent` : This function takes in a system prompt and returns a Langchain Runnable that we can chain together using LCEL\n",
    "- `concat_response` : This function takes in a dictionary of inputs, which within the pipeline will be to concenate and format the responses given by the layer agent and returns a string with the formatted response"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from typing import Dict, Optional, Generator\n",
    "from textwrap import dedent\n",
    "\n",
    "from langchain_groq import ChatGroq\n",
    "from langchain.memory import ConversationBufferMemory\n",
    "from langchain_core.runnables import RunnablePassthrough, RunnableLambda, RunnableSerializable\n",
    "from langchain_core.output_parsers import StrOutputParser\n",
    "from langchain.prompts import ChatPromptTemplate, MessagesPlaceholder\n",
    "\n",
    "\n",
    "# Helper method to create an LCEL chain\n",
    "def create_agent(\n",
    "    system_prompt: str = \"You are a helpful assistant.\\n{helper_response}\",\n",
    "    model_name: str = \"llama3-8b-8192\",\n",
    "    **llm_kwargs\n",
    ") -> RunnableSerializable[Dict, str]:\n",
    "    \"\"\"Create a simple Langchain LCEL chain agent based on a system prompt\"\"\"\n",
    "\n",
    "    prompt = ChatPromptTemplate.from_messages([\n",
    "        (\"system\", system_prompt),\n",
    "        MessagesPlaceholder(variable_name=\"messages\", optional=True),\n",
    "        (\"human\", \"{input}\")\n",
    "    ])\n",
    "\n",
    "    assert 'helper_response' in prompt.input_variables, \"{helper_response} prompt variable not found in prompt. Please add it\" # To make sure we can add layer agent outputs into the prompt\n",
    "    llm = ChatGroq(model=model_name, **llm_kwargs)\n",
    "    \n",
    "    chain = prompt | llm | StrOutputParser()\n",
    "    return chain\n",
    "\n",
    "def concat_response(\n",
    "    inputs: Dict[str, str],\n",
    "    reference_system_prompt: Optional[str] = None\n",
    ") -> str:\n",
    "    \"\"\"Concatenate and format layer agent responses\"\"\"\n",
    "\n",
    "    REFERENCE_SYSTEM_PROMPT = dedent(\"\"\"\\\n",
    "    You have been provided with a set of responses from various open-source models to the latest user query. \n",
    "    Your task is to synthesize these responses into a single, high-quality response. \n",
    "    It is crucial to critically evaluate the information provided in these responses, recognizing that some of it may be biased or incorrect. \n",
    "    Your response should not simply replicate the given answers but should offer a refined, accurate, and comprehensive reply to the instruction. \n",
    "    Ensure your response is well-structured, coherent, and adheres to the highest standards of accuracy and reliability.\n",
    "    Responses from models:\n",
    "    {responses}\n",
    "    \"\"\")\n",
    "    reference_system_prompt = reference_system_prompt or REFERENCE_SYSTEM_PROMPT\n",
    "\n",
    "    assert \"{responses}\" in reference_system_prompt, \"{responses} prompt variable not found in prompt. Please add it\"\n",
    "    responses = \"\"\n",
    "    res_list = []\n",
    "    for i, out in enumerate(inputs.values()):\n",
    "        responses += f\"{i}. {out}\\n\"\n",
    "        res_list.append(out)\n",
    "\n",
    "    formatted_prompt = reference_system_prompt.format(responses=responses)\n",
    "    return formatted_prompt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Configure and build the Mixture of Agents pipeline.\n",
    "\n",
    "Let's configure and build out the whole workflow!\n",
    "\n",
    "Here is a breakdown of the different components:\n",
    "- `CHAT_MEMORY` : This is used to store and retrieve the chat history of the workflow.\n",
    "- `CYCLES` : Number of times the input and helper responses are passed through to the `LAYER_AGENT`\n",
    "- `LAYER_AGENT` : Each agent within this layer agent runs in parallel, and the responses are concatenated using the `concat_response` helper function.\n",
    "- `MAIN_AGENT` : The final agent that responds to the user's query based on the layer agents"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Hyperparameters of agent\n",
    "# Re run this if you want to delete chats\n",
    "CHAT_MEMORY = ConversationBufferMemory(\n",
    "    memory_key=\"messages\",\n",
    "    return_messages=True\n",
    ")\n",
    "CYCLES = 3\n",
    "LAYER_AGENT = ( # Each layer agent in this dictionary runs in parallel\n",
    "    {\n",
    "        'layer_agent_1': RunnablePassthrough() | create_agent(\n",
    "            system_prompt=\"You are an expert planner agent. Break down and plan out how you can answer the user's question {helper_response}\",\n",
    "            model_name='llama3-8b-8192'\n",
    "        ),\n",
    "        'layer_agent_2': RunnablePassthrough() | create_agent(\n",
    "            system_prompt=\"Respond with a thought and then your response to the question. {helper_response}\",\n",
    "            model_name='mixtral-8x7b-32768'\n",
    "        ),\n",
    "        'layer_agent_3': RunnablePassthrough() | create_agent(\n",
    "            system_prompt=\"Think through your response step by step. {helper_response}\",\n",
    "            model_name='gemma2-9b-it'\n",
    "        ),\n",
    "        # Add/Remove agents as needed...\n",
    "    }\n",
    "    |\n",
    "    RunnableLambda(concat_response) # Format layer agent outputs\n",
    ")\n",
    "\n",
    "MAIN_AGENT = create_agent(\n",
    "    system_prompt=\"You are a helpful assistant named Bob.\\n{helper_response}\",\n",
    "    model_name=\"llama3-70b-8192\",\n",
    "    temperature=0.1,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Build the mixture of agents pipeline and create the chat function to ask questions to the agents\n",
    "\n",
    "The `chat_stream` function takes in a query and passes it through the mixture of agents workflow.\n",
    "The query is:\n",
    "1. Passed through the LAYER_AGENT, which in parallel, generates responses from each of the layer agents and conctenates it using the `concat_response` function.\n",
    "2. If `CYCLES` is more than 1, it passes through again through the LAYER_AGENT, this time with the previous concatenated responses and the user's query. This repeats `CYCLES` times.\n",
    "3. The final layer concatenated response and the user's query is passed to the `MAIN_AGENT`, which then stream the final response as and when done. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def chat_stream(query: str) -> Generator[str, None, None]:\n",
    "    \"\"\"Run Mixture of Agents LCEL pipeline\"\"\"\n",
    "\n",
    "    llm_inp = {\n",
    "    'input': query,\n",
    "    'messages': CHAT_MEMORY.load_memory_variables({})['messages'],\n",
    "    'helper_response': \"\"\n",
    "    }\n",
    "    for _ in range(CYCLES):\n",
    "        llm_inp = {\n",
    "            'input': query,\n",
    "            'messages': CHAT_MEMORY.load_memory_variables({})['messages'],\n",
    "            'helper_response': LAYER_AGENT.invoke(llm_inp)\n",
    "        }\n",
    "\n",
    "    response = \"\"\n",
    "    for chunk in MAIN_AGENT.stream(llm_inp):\n",
    "        yield chunk\n",
    "        response += chunk\n",
    "    \n",
    "    # Save response to memory\n",
    "    CHAT_MEMORY.save_context({'input': query}, {'output': response})\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. (Optional) Add observability to view your Agent's execution\n",
    "\n",
    "We can optionally use [Arize Phoenix](https://github.com/Arize-ai/phoenix) to trace and evaluate the execution of our agent. Phoenix can be run locally or in the cloud. We'll use a local instance to keep things simple."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: arize-phoenix in /opt/anaconda3/envs/envName=phoenix/lib/python3.10/site-packages (4.35.1)\n",
      "Requirement already satisfied: openinference-instrumentation-langchain in /opt/anaconda3/envs/envName=phoenix/lib/python3.10/site-packages (0.1.27)\n",
      "Requirement already satisfied: aioitertools in /opt/anaconda3/envs/envName=phoenix/lib/python3.10/site-packages (from arize-phoenix) (0.12.0)\n",
      "Requirement already satisfied: aiosqlite in /opt/anaconda3/envs/envName=phoenix/lib/python3.10/site-packages (from arize-phoenix) (0.20.0)\n",
      "Requirement already satisfied: alembic<2,>=1.3.0 in /opt/anaconda3/envs/envName=phoenix/lib/python3.10/site-packages (from arize-phoenix) (1.13.2)\n",
      "Requirement already satisfied: arize-phoenix-evals>=0.13.1 in /opt/anaconda3/envs/envName=phoenix/lib/python3.10/site-packages (from arize-phoenix) (0.15.1)\n",
      "Requirement already satisfied: arize-phoenix-otel>=0.4.1 in /opt/anaconda3/envs/envName=phoenix/lib/python3.10/site-packages (from arize-phoenix) (0.4.1)\n",
      "Requirement already satisfied: cachetools in /opt/anaconda3/envs/envName=phoenix/lib/python3.10/site-packages (from arize-phoenix) (5.5.0)\n",
      "Requirement already satisfied: fastapi in /opt/anaconda3/envs/envName=phoenix/lib/python3.10/site-packages (from arize-phoenix) (0.111.1)\n",
      "Requirement already satisfied: grpcio in /opt/anaconda3/envs/envName=phoenix/lib/python3.10/site-packages (from arize-phoenix) (1.66.1)\n",
      "Requirement already satisfied: hdbscan>=0.8.33 in /opt/anaconda3/envs/envName=phoenix/lib/python3.10/site-packages (from arize-phoenix) (0.8.38.post1)\n",
      "Requirement already satisfied: httpx in /opt/anaconda3/envs/envName=phoenix/lib/python3.10/site-packages (from arize-phoenix) (0.27.2)\n",
      "Requirement already satisfied: jinja2 in /opt/anaconda3/envs/envName=phoenix/lib/python3.10/site-packages (from arize-phoenix) (3.1.4)\n",
      "Requirement already satisfied: numpy<2 in /opt/anaconda3/envs/envName=phoenix/lib/python3.10/site-packages (from arize-phoenix) (1.26.4)\n",
      "Requirement already satisfied: openinference-instrumentation-llama-index>=2.2.1 in /opt/anaconda3/envs/envName=phoenix/lib/python3.10/site-packages (from arize-phoenix) (3.0.2)\n",
      "Requirement already satisfied: openinference-instrumentation-openai>=0.1.11 in /opt/anaconda3/envs/envName=phoenix/lib/python3.10/site-packages (from arize-phoenix) (0.1.14)\n",
      "Requirement already satisfied: openinference-instrumentation>=0.1.12 in /opt/anaconda3/envs/envName=phoenix/lib/python3.10/site-packages (from arize-phoenix) (0.1.18)\n",
      "Requirement already satisfied: openinference-semantic-conventions>=0.1.9 in /opt/anaconda3/envs/envName=phoenix/lib/python3.10/site-packages (from arize-phoenix) (0.1.10)\n",
      "Requirement already satisfied: opentelemetry-exporter-otlp in /opt/anaconda3/envs/envName=phoenix/lib/python3.10/site-packages (from arize-phoenix) (1.27.0)\n",
      "Requirement already satisfied: opentelemetry-proto>=1.12.0 in /opt/anaconda3/envs/envName=phoenix/lib/python3.10/site-packages (from arize-phoenix) (1.27.0)\n",
      "Requirement already satisfied: opentelemetry-sdk in /opt/anaconda3/envs/envName=phoenix/lib/python3.10/site-packages (from arize-phoenix) (1.27.0)\n",
      "Requirement already satisfied: opentelemetry-semantic-conventions in /opt/anaconda3/envs/envName=phoenix/lib/python3.10/site-packages (from arize-phoenix) (0.48b0)\n",
      "Requirement already satisfied: pandas>=1.0 in /opt/anaconda3/envs/envName=phoenix/lib/python3.10/site-packages (from arize-phoenix) (2.2.2)\n",
      "Requirement already satisfied: protobuf<6.0,>=3.20 in /opt/anaconda3/envs/envName=phoenix/lib/python3.10/site-packages (from arize-phoenix) (4.25.4)\n",
      "Requirement already satisfied: psutil in /opt/anaconda3/envs/envName=phoenix/lib/python3.10/site-packages (from arize-phoenix) (5.9.0)\n",
      "Requirement already satisfied: pyarrow in /opt/anaconda3/envs/envName=phoenix/lib/python3.10/site-packages (from arize-phoenix) (17.0.0)\n",
      "Requirement already satisfied: pydantic!=2.0.*,<3,>=1.0 in /opt/anaconda3/envs/envName=phoenix/lib/python3.10/site-packages (from arize-phoenix) (2.8.2)\n",
      "Requirement already satisfied: pyjwt in /opt/anaconda3/envs/envName=phoenix/lib/python3.10/site-packages (from arize-phoenix) (2.9.0)\n",
      "Requirement already satisfied: python-multipart in /opt/anaconda3/envs/envName=phoenix/lib/python3.10/site-packages (from arize-phoenix) (0.0.9)\n",
      "Requirement already satisfied: scikit-learn in /opt/anaconda3/envs/envName=phoenix/lib/python3.10/site-packages (from arize-phoenix) (1.2.2)\n",
      "Requirement already satisfied: scipy in /opt/anaconda3/envs/envName=phoenix/lib/python3.10/site-packages (from arize-phoenix) (1.14.1)\n",
      "Requirement already satisfied: sqlalchemy<3,>=2.0.4 in /opt/anaconda3/envs/envName=phoenix/lib/python3.10/site-packages (from sqlalchemy[asyncio]<3,>=2.0.4->arize-phoenix) (2.0.34)\n",
      "Requirement already satisfied: sqlean-py>=3.45.1 in /opt/anaconda3/envs/envName=phoenix/lib/python3.10/site-packages (from arize-phoenix) (3.45.1)\n",
      "Requirement already satisfied: starlette in /opt/anaconda3/envs/envName=phoenix/lib/python3.10/site-packages (from arize-phoenix) (0.37.2)\n",
      "Requirement already satisfied: strawberry-graphql==0.236.0 in /opt/anaconda3/envs/envName=phoenix/lib/python3.10/site-packages (from arize-phoenix) (0.236.0)\n",
      "Requirement already satisfied: tqdm in /opt/anaconda3/envs/envName=phoenix/lib/python3.10/site-packages (from arize-phoenix) (4.66.5)\n",
      "Requirement already satisfied: typing-extensions>=4.5 in /opt/anaconda3/envs/envName=phoenix/lib/python3.10/site-packages (from arize-phoenix) (4.12.2)\n",
      "Requirement already satisfied: umap-learn in /opt/anaconda3/envs/envName=phoenix/lib/python3.10/site-packages (from arize-phoenix) (0.5.6)\n",
      "Requirement already satisfied: uvicorn in /opt/anaconda3/envs/envName=phoenix/lib/python3.10/site-packages (from arize-phoenix) (0.30.6)\n",
      "Requirement already satisfied: wrapt in /opt/anaconda3/envs/envName=phoenix/lib/python3.10/site-packages (from arize-phoenix) (1.16.0)\n",
      "Requirement already satisfied: graphql-core<3.3.0,>=3.2.0 in /opt/anaconda3/envs/envName=phoenix/lib/python3.10/site-packages (from strawberry-graphql==0.236.0->arize-phoenix) (3.2.4)\n",
      "Requirement already satisfied: python-dateutil<3.0.0,>=2.7.0 in /opt/anaconda3/envs/envName=phoenix/lib/python3.10/site-packages (from strawberry-graphql==0.236.0->arize-phoenix) (2.9.0.post0)\n",
      "Requirement already satisfied: opentelemetry-api in /opt/anaconda3/envs/envName=phoenix/lib/python3.10/site-packages (from openinference-instrumentation-langchain) (1.27.0)\n",
      "Requirement already satisfied: opentelemetry-instrumentation in /opt/anaconda3/envs/envName=phoenix/lib/python3.10/site-packages (from openinference-instrumentation-langchain) (0.48b0)\n",
      "Requirement already satisfied: Mako in /opt/anaconda3/envs/envName=phoenix/lib/python3.10/site-packages (from alembic<2,>=1.3.0->arize-phoenix) (1.3.5)\n",
      "Requirement already satisfied: joblib>=1.0 in /opt/anaconda3/envs/envName=phoenix/lib/python3.10/site-packages (from hdbscan>=0.8.33->arize-phoenix) (1.4.2)\n",
      "Requirement already satisfied: pytz>=2020.1 in /opt/anaconda3/envs/envName=phoenix/lib/python3.10/site-packages (from pandas>=1.0->arize-phoenix) (2024.1)\n",
      "Requirement already satisfied: tzdata>=2022.7 in /opt/anaconda3/envs/envName=phoenix/lib/python3.10/site-packages (from pandas>=1.0->arize-phoenix) (2024.1)\n",
      "Requirement already satisfied: annotated-types>=0.4.0 in /opt/anaconda3/envs/envName=phoenix/lib/python3.10/site-packages (from pydantic!=2.0.*,<3,>=1.0->arize-phoenix) (0.7.0)\n",
      "Requirement already satisfied: pydantic-core==2.20.1 in /opt/anaconda3/envs/envName=phoenix/lib/python3.10/site-packages (from pydantic!=2.0.*,<3,>=1.0->arize-phoenix) (2.20.1)\n",
      "Requirement already satisfied: threadpoolctl>=2.0.0 in /opt/anaconda3/envs/envName=phoenix/lib/python3.10/site-packages (from scikit-learn->arize-phoenix) (3.5.0)\n",
      "Requirement already satisfied: greenlet!=0.4.17 in /opt/anaconda3/envs/envName=phoenix/lib/python3.10/site-packages (from sqlalchemy[asyncio]<3,>=2.0.4->arize-phoenix) (3.0.3)\n",
      "Requirement already satisfied: fastapi-cli>=0.0.2 in /opt/anaconda3/envs/envName=phoenix/lib/python3.10/site-packages (from fastapi->arize-phoenix) (0.0.5)\n",
      "Requirement already satisfied: email_validator>=2.0.0 in /opt/anaconda3/envs/envName=phoenix/lib/python3.10/site-packages (from fastapi->arize-phoenix) (2.2.0)\n",
      "Requirement already satisfied: anyio in /opt/anaconda3/envs/envName=phoenix/lib/python3.10/site-packages (from httpx->arize-phoenix) (4.4.0)\n",
      "Requirement already satisfied: certifi in /opt/anaconda3/envs/envName=phoenix/lib/python3.10/site-packages (from httpx->arize-phoenix) (2024.7.4)\n",
      "Requirement already satisfied: httpcore==1.* in /opt/anaconda3/envs/envName=phoenix/lib/python3.10/site-packages (from httpx->arize-phoenix) (1.0.5)\n",
      "Requirement already satisfied: idna in /opt/anaconda3/envs/envName=phoenix/lib/python3.10/site-packages (from httpx->arize-phoenix) (3.8)\n",
      "Requirement already satisfied: sniffio in /opt/anaconda3/envs/envName=phoenix/lib/python3.10/site-packages (from httpx->arize-phoenix) (1.3.1)\n",
      "Requirement already satisfied: h11<0.15,>=0.13 in /opt/anaconda3/envs/envName=phoenix/lib/python3.10/site-packages (from httpcore==1.*->httpx->arize-phoenix) (0.14.0)\n",
      "Requirement already satisfied: MarkupSafe>=2.0 in /opt/anaconda3/envs/envName=phoenix/lib/python3.10/site-packages (from jinja2->arize-phoenix) (2.1.5)\n",
      "Requirement already satisfied: click>=7.0 in /opt/anaconda3/envs/envName=phoenix/lib/python3.10/site-packages (from uvicorn->arize-phoenix) (8.1.7)\n",
      "Requirement already satisfied: deprecated>=1.2.6 in /opt/anaconda3/envs/envName=phoenix/lib/python3.10/site-packages (from opentelemetry-api->openinference-instrumentation-langchain) (1.2.14)\n",
      "Requirement already satisfied: importlib-metadata<=8.4.0,>=6.0 in /opt/anaconda3/envs/envName=phoenix/lib/python3.10/site-packages (from opentelemetry-api->openinference-instrumentation-langchain) (8.4.0)\n",
      "Requirement already satisfied: opentelemetry-exporter-otlp-proto-grpc==1.27.0 in /opt/anaconda3/envs/envName=phoenix/lib/python3.10/site-packages (from opentelemetry-exporter-otlp->arize-phoenix) (1.27.0)\n",
      "Requirement already satisfied: opentelemetry-exporter-otlp-proto-http==1.27.0 in /opt/anaconda3/envs/envName=phoenix/lib/python3.10/site-packages (from opentelemetry-exporter-otlp->arize-phoenix) (1.27.0)\n",
      "Requirement already satisfied: googleapis-common-protos~=1.52 in /opt/anaconda3/envs/envName=phoenix/lib/python3.10/site-packages (from opentelemetry-exporter-otlp-proto-grpc==1.27.0->opentelemetry-exporter-otlp->arize-phoenix) (1.65.0)\n",
      "Requirement already satisfied: opentelemetry-exporter-otlp-proto-common==1.27.0 in /opt/anaconda3/envs/envName=phoenix/lib/python3.10/site-packages (from opentelemetry-exporter-otlp-proto-grpc==1.27.0->opentelemetry-exporter-otlp->arize-phoenix) (1.27.0)\n",
      "Requirement already satisfied: requests~=2.7 in /opt/anaconda3/envs/envName=phoenix/lib/python3.10/site-packages (from opentelemetry-exporter-otlp-proto-http==1.27.0->opentelemetry-exporter-otlp->arize-phoenix) (2.32.3)\n",
      "Requirement already satisfied: setuptools>=16.0 in /opt/anaconda3/envs/envName=phoenix/lib/python3.10/site-packages (from opentelemetry-instrumentation->openinference-instrumentation-langchain) (72.1.0)\n",
      "Requirement already satisfied: numba>=0.51.2 in /opt/anaconda3/envs/envName=phoenix/lib/python3.10/site-packages (from umap-learn->arize-phoenix) (0.60.0)\n",
      "Requirement already satisfied: pynndescent>=0.5 in /opt/anaconda3/envs/envName=phoenix/lib/python3.10/site-packages (from umap-learn->arize-phoenix) (0.5.13)\n",
      "Requirement already satisfied: exceptiongroup>=1.0.2 in /opt/anaconda3/envs/envName=phoenix/lib/python3.10/site-packages (from anyio->httpx->arize-phoenix) (1.2.2)\n",
      "Requirement already satisfied: dnspython>=2.0.0 in /opt/anaconda3/envs/envName=phoenix/lib/python3.10/site-packages (from email_validator>=2.0.0->fastapi->arize-phoenix) (2.6.1)\n",
      "Requirement already satisfied: typer>=0.12.3 in /opt/anaconda3/envs/envName=phoenix/lib/python3.10/site-packages (from fastapi-cli>=0.0.2->fastapi->arize-phoenix) (0.12.5)\n",
      "Requirement already satisfied: zipp>=0.5 in /opt/anaconda3/envs/envName=phoenix/lib/python3.10/site-packages (from importlib-metadata<=8.4.0,>=6.0->opentelemetry-api->openinference-instrumentation-langchain) (3.20.1)\n",
      "Requirement already satisfied: llvmlite<0.44,>=0.43.0dev0 in /opt/anaconda3/envs/envName=phoenix/lib/python3.10/site-packages (from numba>=0.51.2->umap-learn->arize-phoenix) (0.43.0)\n",
      "Requirement already satisfied: six>=1.5 in /opt/anaconda3/envs/envName=phoenix/lib/python3.10/site-packages (from python-dateutil<3.0.0,>=2.7.0->strawberry-graphql==0.236.0->arize-phoenix) (1.16.0)\n",
      "Requirement already satisfied: httptools>=0.5.0 in /opt/anaconda3/envs/envName=phoenix/lib/python3.10/site-packages (from uvicorn[standard]>=0.12.0->fastapi->arize-phoenix) (0.6.1)\n",
      "Requirement already satisfied: python-dotenv>=0.13 in /opt/anaconda3/envs/envName=phoenix/lib/python3.10/site-packages (from uvicorn[standard]>=0.12.0->fastapi->arize-phoenix) (1.0.1)\n",
      "Requirement already satisfied: pyyaml>=5.1 in /opt/anaconda3/envs/envName=phoenix/lib/python3.10/site-packages (from uvicorn[standard]>=0.12.0->fastapi->arize-phoenix) (6.0.2)\n",
      "Requirement already satisfied: uvloop!=0.15.0,!=0.15.1,>=0.14.0 in /opt/anaconda3/envs/envName=phoenix/lib/python3.10/site-packages (from uvicorn[standard]>=0.12.0->fastapi->arize-phoenix) (0.20.0)\n",
      "Requirement already satisfied: watchfiles>=0.13 in /opt/anaconda3/envs/envName=phoenix/lib/python3.10/site-packages (from uvicorn[standard]>=0.12.0->fastapi->arize-phoenix) (0.24.0)\n",
      "Requirement already satisfied: websockets>=10.4 in /opt/anaconda3/envs/envName=phoenix/lib/python3.10/site-packages (from uvicorn[standard]>=0.12.0->fastapi->arize-phoenix) (12.0)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in /opt/anaconda3/envs/envName=phoenix/lib/python3.10/site-packages (from requests~=2.7->opentelemetry-exporter-otlp-proto-http==1.27.0->opentelemetry-exporter-otlp->arize-phoenix) (3.3.2)\n",
      "Requirement already satisfied: urllib3<3,>=1.21.1 in /opt/anaconda3/envs/envName=phoenix/lib/python3.10/site-packages (from requests~=2.7->opentelemetry-exporter-otlp-proto-http==1.27.0->opentelemetry-exporter-otlp->arize-phoenix) (2.2.2)\n",
      "Requirement already satisfied: shellingham>=1.3.0 in /opt/anaconda3/envs/envName=phoenix/lib/python3.10/site-packages (from typer>=0.12.3->fastapi-cli>=0.0.2->fastapi->arize-phoenix) (1.5.4)\n",
      "Requirement already satisfied: rich>=10.11.0 in /opt/anaconda3/envs/envName=phoenix/lib/python3.10/site-packages (from typer>=0.12.3->fastapi-cli>=0.0.2->fastapi->arize-phoenix) (13.8.0)\n",
      "Requirement already satisfied: markdown-it-py>=2.2.0 in /opt/anaconda3/envs/envName=phoenix/lib/python3.10/site-packages (from rich>=10.11.0->typer>=0.12.3->fastapi-cli>=0.0.2->fastapi->arize-phoenix) (3.0.0)\n",
      "Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /opt/anaconda3/envs/envName=phoenix/lib/python3.10/site-packages (from rich>=10.11.0->typer>=0.12.3->fastapi-cli>=0.0.2->fastapi->arize-phoenix) (2.18.0)\n",
      "Requirement already satisfied: mdurl~=0.1 in /opt/anaconda3/envs/envName=phoenix/lib/python3.10/site-packages (from markdown-it-py>=2.2.0->rich>=10.11.0->typer>=0.12.3->fastapi-cli>=0.0.2->fastapi->arize-phoenix) (0.1.2)\n"
     ]
    }
   ],
   "source": [
    "! pip install arize-phoenix openinference-instrumentation-langchain"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/anaconda3/envs/envName=phoenix/lib/python3.10/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
      "  from .autonotebook import tqdm as notebook_tqdm\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "🌍 To view the Phoenix app in your browser, visit http://localhost:6006/\n",
      "📖 For more information on how to use Phoenix, check out https://docs.arize.com/phoenix\n",
      "🔭 OpenTelemetry Tracing Details 🔭\n",
      "|  Phoenix Project: default\n",
      "|  Span Processor: SimpleSpanProcessor\n",
      "|  Collector Endpoint: localhost:4317\n",
      "|  Transport: gRPC\n",
      "|  Transport Headers: {'user-agent': '****'}\n",
      "|  \n",
      "|  Using a default SpanProcessor. `add_span_processor` will overwrite this default.\n",
      "|  \n",
      "|  `register` has set this TracerProvider as the global OpenTelemetry default.\n",
      "|  To disable this behavior, call `register` with `set_global_tracer_provider=False`.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from openinference.instrumentation.langchain import LangChainInstrumentor\n",
    "import phoenix as px\n",
    "from phoenix.otel import register\n",
    "\n",
    "session = px.launch_app()\n",
    "tracer_provider = register()\n",
    "LangChainInstrumentor().instrument(tracer_provider=tracer_provider)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "📺 Opening a view to the Phoenix app. The app is running at http://localhost:6006/\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"100%\"\n",
       "            height=\"1000\"\n",
       "            src=\"http://localhost:6006/\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "            \n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x3415c9870>"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "session.view()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Chat with the Agent\n",
    "\n",
    "Let's chat with our mixture of agents!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "User: tell me about opentelemetry\n",
      "AI: OpenTelemetry is an open-source observability framework that provides a vendor-neutral, language-agnostic way to collect, transmit, and analyze telemetry data from applications. Its primary goal is to standardize telemetry instrumentation, making it easier to collect and analyze data from various sources and platforms.\n",
      "\n",
      "**Key Components:**\n",
      "\n",
      "1. **APIs**: OpenTelemetry provides a set of APIs for various programming languages (e.g., Java, .NET, Python, Go) to instrument applications and generate telemetry data.\n",
      "2. **Collectors**: Collectors are responsible for collecting telemetry data from applications and sending it to a backend storage system, such as a database or an analytics platform.\n",
      "3. **Processors**: Processors can be used to transform, filter, or enrich telemetry data before it's sent to the collector or storage system.\n",
      "4. **SDKs (Software Development Kits)**: SDKs are language-specific implementations of the OpenTelemetry API, handling the details of collecting and exporting telemetry data.\n",
      "5. **Exporters**: Exporters send the collected telemetry data to your chosen observability backend.\n",
      "6. **Propagators**: Propagators ensure that trace context is carried across different services and components.\n",
      "\n",
      "**Benefits:**\n",
      "\n",
      "1. **Vendor-Neutrality**: OpenTelemetry is not tied to a specific vendor or platform, making it a great choice for organizations with diverse technology stacks.\n",
      "2. **Standardization**: By using OpenTelemetry, developers can focus on instrumentation and data collection, rather than worrying about vendor-specific APIs or protocols.\n",
      "3. **Improved Observability**: OpenTelemetry enables organizations to collect and analyze telemetry data more effectively, providing valuable insights for monitoring, debugging, and optimizing their applications.\n",
      "4. **Reduced Vendor Lock-in**: OpenTelemetry allows developers to choose the best tools for their needs without being tied to a single platform.\n",
      "\n",
      "**Use Cases:**\n",
      "\n",
      "1. **Application Performance Monitoring (APM)**: OpenTelemetry can be used to collect data on application performance, latency, and errors, helping organizations identify bottlenecks and optimize their applications.\n",
      "2. **Log Collection and Analysis**: OpenTelemetry can be used to collect and analyze log data, providing insights into application behavior, errors, and security incidents.\n",
      "3. **Cloud Cost Optimization**: By collecting telemetry data on cloud resource usage, organizations can optimize their cloud costs and improve resource allocation.\n",
      "\n",
      "**Implementation:**\n",
      "\n",
      "1. **Instrument Your Applications**: Use OpenTelemetry APIs to instrument your applications, generating telemetry data that can be collected and analyzed.\n",
      "2. **Configure Collectors**: Set up collectors to collect telemetry data from your applications and send it to a backend storage system or analytics platform.\n",
      "3. **Integrate with Analytics Platforms**: Integrate OpenTelemetry with your preferred analytics platform (e.g., Prometheus, Grafana, New Relic) to visualize and analyze your telemetry data.\n",
      "\n",
      "**Challenges and Limitations:**\n",
      "\n",
      "1. **Instrumentation Complexity**: Instrumenting applications can be complex, especially for large-scale or legacy systems.\n",
      "2. **Data Volume and Complexity**: Collecting and analyzing large amounts of telemetry data can be challenging, requiring significant resources and expertise.\n",
      "3. **Security and Compliance**: OpenTelemetry may require additional security measures to ensure the integrity and confidentiality of sensitive data.\n",
      "\n",
      "**Conclusion:**\n",
      "\n",
      "OpenTelemetry is a powerful observability framework that can help organizations improve their application monitoring, debugging, and optimization capabilities. By understanding its components, benefits, use cases, implementation, and challenges, you can effectively leverage OpenTelemetry to drive better business outcomes.\n",
      "\n",
      "OpenTelemetry is backed by major players in the industry, including Google, Microsoft, and the Cloud Native Computing Foundation (CNCF), ensuring its growth and long-term support. Using OpenTelemetry, developers can gain deep insights into their applications' performance, resource utilization, and potential issues, leading to more reliable and efficient systems.\n",
      "User: \n",
      "AI: "
     ]
    }
   ],
   "source": [
    "# Chat with Agent\n",
    "while True:\n",
    "    inp = input(\"\\nAsk a question: \")\n",
    "    print(f\"\\nUser: {inp}\")\n",
    "    if inp.lower() == \"quit\":\n",
    "        print(\"\\nStopped by User\\n\")\n",
    "        break\n",
    "    stream = chat_stream(inp)\n",
    "    print(f\"AI: \", end=\"\")\n",
    "    for chunk in stream:\n",
    "        print(chunk, end=\"\", flush=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "In this notebook we demonstrated how to build a fairly complex agentic workflow using Groq's fast AI inference.\n",
    "MoA and other agentic workflows offer significant advantages in working with Large Language Models (LLMs). By enabling LLMs to \"think,\" refine their responses, and break down complex tasks, these approaches enhance accuracy and problem-solving capabilities. Moreover, they present a cost-effective solution by allowing the use of smaller, open-source models in combination, even when multiple LLM calls are required. This notebook serves as an introduction to agentic workflows and in production, should be adapted to your use case and evaluated thoroughly.\n",
    "\n",
    "For a more Object Oriented approach, streamlit demo app and an easier way to configure the workflow please checkout [this repo](https://github.com/skapadia3214/groq-moa)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}