Exploring Structured Planning Agent-Need Help

[mraguth]

I am exploring the concept of a structured planning agent (as shown in link below). To initiate the agent, it appears that I need to create a tool (for example, uber_tool) and pass that tool to the worker as demonstrated in the code. However, I am currently using a query engine similar to what is shown in the code below. Is it possible to convert my query engine into the required tool? By doing so, I would be able to initiate a structured planner. Your help is appreciated.

https://github.com/run-llama/llama_index/blob/main/docs/docs/examples/agent/structured_planner.ipynb

Suggested code in structured Planning agent py:

lyft_tool = QueryEngineTool.from_defaults(
    lyft_index.as_query_engine(),
    name="lyft_2021",
    description="Useful for asking questions about Lyft's 2021 10-K filling.",
)

uber_tool = QueryEngineTool.from_defaults(
    uber_index.as_query_engine(),
    name="uber_2021",
    description="Useful for asking questions about Uber's 2021 10-K filling.",
)
from llama_index.core.agent import (
    StructuredPlannerAgent,
    FunctionCallingAgentWorker,
    ReActAgentWorker,
)

# create the function calling worker for reasoning
worker = FunctionCallingAgentWorker.from_tools(
    [lyft_tool, uber_tool], verbose=True
)

# wrap the worker in the top-level planner
agent = StructuredPlannerAgent(
    worker, tools=[lyft_tool, uber_tool], verbose=True
)

My query engine that need to be ported into above tool:

query_engine = RAGStringQueryEngine(
    retriever=recursive_retriever,
    response_synthesizer=response_synthesizer,
    pandas_prompt=pandas_prompt,
    pandas_output_parser=pandas_output_parser,
    llm=llm,
    qa_prompt=qa_prompt,
    synthesize_response=True,
    response_synthesis_prompt=response_synthesis_prompt,
    callback_manager=Settings.callback_manager,  # Important for Langfuse.
)

[dosubot[bot]]

Yes, it is possible to convert your existing query engine into the required tool to initiate a structured planner using LlamaIndex. Here is how you can achieve this:

1. Convert Your Query Engine to a Tool:
   Use the QueryEngineTool.from_defaults method to create a tool from your existing RAGStringQueryEngine.

   from llama_index.core.tools import QueryEngineTool
   
   # Assuming your RAGStringQueryEngine is already initialized and indexed
   query_engine = RAGStringQueryEngine(
       retriever=recursive_retriever,
       response_synthesizer=response_synthesizer,
       pandas_prompt=pandas_prompt,
       pandas_output_parser=pandas_output_parser,
       llm=llm,
       qa_prompt=qa_prompt,
       synthesize_response=True,
       response_synthesis_prompt=response_synthesis_prompt,
       callback_manager=Settings.callback_manager,
   )
   
   # Create a QueryEngineTool instance
   query_engine_tool = QueryEngineTool.from_defaults(
       query_engine=query_engine,
       name="rag_string_query_engine",
       description="Provides information using RAGStringQueryEngine."
   )

2. Initialize the Structured Planner Agent:
   Use the created tool to initialize the FunctionCallingAgentWorker and StructuredPlannerAgent.

   from llama_index.core.agent import (
       StructuredPlannerAgent,
       FunctionCallingAgentWorker,
   )
   
   # Create the function calling worker for reasoning
   worker = FunctionCallingAgentWorker.from_tools(
       [query_engine_tool], verbose=True
   )
   
   # Wrap the worker in the top-level planner
   agent = StructuredPlannerAgent(
       worker, tools=[query_engine_tool], verbose=True
   )

By following these steps, you can convert your existing RAGStringQueryEngine into a tool and use it to initiate a structured planner ^[1].

To continue talking to Dosu, mention @dosu.

[PRABADHARSHAN]

Description:

I'm trying to create an agent using FunctionCallingAgentWorker and StructuredPlannerAgent in LlamaIndex to handle SQL and vector search queries. However, I encounter a TypeError: too many positional arguments when calling the agent.chat() method.

I suspect that this error arises from the way the tools are passed to the FunctionCallingAgentWorker. Here’s the setup I’m working with:

Code:

# Define the tools for querying
query_engine_tools = [
    QueryEngineTool(
        query_engine=sql_query_engine,
        metadata=ToolMetadata(
           name="SQLQueryTool",
           description=(
            "This tool is ideal for handling queries where the user specifies the exact name or identifier of a document. It translates natural language queries into SQL queries to search a table with columns such as file_name, text, and url. Use this tool when the request involves retrieving documents based on their name or specific identifier."
        ),
        ),
    ),
    QueryEngineTool(
        query_engine=retriever_query_engine,
        metadata=ToolMetadata(
          name="vectorsearch",
          description=(
        "This tool excels at addressing queries related to the content or subject matter of documents. It uses vector embeddings to understand the semantic meaning and context of documents. Use this tool when the request involves finding documents based on their content or topic, such as discussions about 'selling pens'."
    ),
        ),
    ),
]

from llama_index.core.agent import (
    StructuredPlannerAgent,
    FunctionCallingAgentWorker,
    ReActAgentWorker,
)

# Create the function calling worker for reasoning
worker = FunctionCallingAgentWorker.from_tools(
    [query_engine_tools], verbose=True
)

# Wrap the worker in the top-level planner
agent = StructuredPlannerAgent(
    worker, tools=[query_engine_tools], verbose=True
)

# Execute a chat query
response = agent.chat(message="give me document which talks about selling a pen")