A smarter parser for tools

[krypterro]

Although ChatGPT generally works for LLM tool choices, I get poor results from other LLM. My Vicuna LLM likes to reply with "Use Search" for example, which fails the simple regex in output_parser, so I am applying this ugly fix for my app that is using AgentType.CONVERSATIONAL_REACT_DESCRIPTION:
`

def parse(self, text: str) -> Union[AgentAction, AgentFinish]:

    if f"{self.ai_prefix}:" in text:
        return AgentFinish(
            {"output": text.split(f"{self.ai_prefix}:")[-1].strip()}, text
        )
    regex = r"Action: (.*?)[\n]*Action Input: (.*)"
    match = re.search(regex, text)
    if not match:
        raise OutputParserException(f"Could not parse LLM output: `{text}`")
    action = match.group(1)
    
    # monkey patch
    txt = action.lower()
    if txt.find("search") >= 0:
        action = "Search"
    elif txt.find("calculator") >= 0:
        action = "Calculator"
        
    action_input = match.group(2)
    return AgentAction(action.strip(), action_input.strip(" ").strip('"'), text)

`

Wouldn't it be more elegant to pass in a list of available tools, and just check to see if the LLM mentions any of them in the reply, and then choose the one mentioned?

This would allow for extra text from the LLM.

[hinthornw]

Can also lead to false positives since they frequently mention multiple tools, sometimes not as a means to invoke

[krypterro]

Noted. It looks like the best option is to train the model to play nice with LangChain. Now, if I could find a dataset of successful LangChain integration that would be great, but I can't. So I'm left with generating my own by using ChatGPT. This will take awhile.