Using embeddings to reduce costs of the openapi agent

[raphox]

@agola11 int his part of code https://github.com/hwchase17/langchain/blob/910da8518f561199d6f3edf0503e4117fdfa9ed6/langchain/agents/agent_toolkits/openapi/toolkit.py#L49-L53 you are loading and putting, in the prompt, the entire JSON API code.

The cost of that is expensive using OpenAI as LLM. What do you think about using embeddings to filter just the endpoints related to the user's question?

[raphox]

I am interested in helping with the project but I knew about it recently. I started to study LLM a few weeks ago.
@agola11, are you working on something like that? Do you want my help? Do you have code references that could help me apply my new approach?

[agola11]

That's actually not the case. The JSON agent is not putting the entire json/open api spec into the llm prompt. That would be expensive and slow, as you pointed out. Instead, the agent is able to iteratively sift through the OpenAPI spec and find the relevant information by retrieving the keys, then extracting the right values over and over again.

[raphox]

The agent approach tries to be generic, without any previous information. We could use the chain approach https://python.langchain.com/en/latest/modules/chains/examples/openapi.html too.

I didn't work to make another code to decrease the costs because any approach I thought was costly. But, I thought to use a vector database to store all endpoint documentation, and before requesting the API URL, we could insert the related endpoint docs in the prompt beginning. Similar to here https://python.langchain.com/en/latest/modules/chains/index_examples/question_answering.html

[davidzshi]

@raphox did you end up going with the vector database approach?

[davidzshi]

Update: I built my own version of this using Chroma and the performance even with the out of the box embedding model is quite good.

[warlockdn]

any chance you'll open source the code ? or may be the methodology used ?

[davidzshi]

@warlockdn https://github.com/operatorlabs/nft-demo - you can check out the methodology here. it's a bit clunky, but in essence I took the openapi spec, broke it down into the different endpoints, described them using gpt-3.5 turbo, generated and stored the embedding in chroma, then created a retrieval mechanism that ranked the relevance of the query with the generated endpoint description.

when the agent selects the tool, it get the endpoint related info and generates a curl request. in this case, I'm running custom requests against the reservoir.tools api.