Chat as a main multi-agent interface

[Max-Levitskiy]

@daveshap mentioned about the communication between agents problem. In the project we're with my team doing, we're going with chat interface where agents should be able to mention each other and people. The agent should think about other models, prompts, agents, chats and people in the same way as about "tools" and desides, if it makes sense to solve some issue using "tools".
I'm planning to do some POC about it in my project with using latest OpenAI update with JSON response (to form tasks list) and multi-function calls to @mention different "tools". Probably, after this POC I'll have a better picture of how it will work and share the result.
Until then, I'm oppened to hear any thoughts/criticism/suggestions about such an approach.
From this framework perspective, I think it's possible to do the same - create different chats to make agents talk with each other via mentioning.
I think it should be something like json/function call return from the OpenAI api, which looks like this:

{
  "mentions": [
    {"type": "agent", "id": "very-funny-agent", "content":{"type": "text", "text": "tell me a joke"}},
    {"type": "model", "id": "dall-e", "content":{"type": "text", "text": "generate a funny image"}}
  ]
}

Then agent is waiting for responses from them and continue to work.

[daveshap]

I like this. It seems imminently reasonable and doable.

[SwiftCloudOps]

Json is a clear win IMHO due to human readability + easy formatting.

For PoC, Discord is excellent, with potential for access controls to rooms. Discord does support read-only room-access, for each layer down from SOB.

In the original ACE framework it was a clean linear hierarchy up/down, so Pinecone probably adds complexity without UI payoff of discord - just my 02.

Note we're mixing lifespan context: Strategy might change monthly, base telemetry by the second.

(RogerV)

[mfilter]

Maybe the idea of using EPCIS 2.0 fits in here? #30 (reply in thread)

[RafaDallaCosta]

I know that gpt4 can output json, but can it handle json input as well as it does with natural language? I think the json output is essential, but I believe we will need an engine to translate the json into natural language and forward it to the agents mentioned in the metadata.

[Max-Levitskiy]

It can understand the JSON input, but you need JSON as a router to concrete model/agent/person. Type and ID are routing parameters. Content is a part, which contains the data, which you pass to the model as parameters.
Different models have different parameters, so you need to fit it in the code, it won't be automatically.
It requires writing a transformation code that will "transform" the JSON to a function call.

[RafaDallaCosta]

"It requires writing a transformation code that will "transform" the JSON to a function call." - this is what I mean by the engine needed.
I understand that we must use JSON output.

[AlexandruMuresanM]

I don't know much about programing in python and automation, I have some base knowledge from my highschool days, but can you use Discord chat to supervise the conversation between bots?

[SwiftCloudOps]

Yes - that's an advantage of discord is human readability; json is readable even by a layperson without meaningful code knowledge.

[trungtin]

That looks like the right direction @Max-Levitskiy . Have you done the POC yet? I would love to take a look. Is it open source?

In my app, I have added the ability for users to mention an assistant and run the thread with it instead of the default assistant. But I would love to see how to effectively enable assistant-to-assistant communication. Do you need a master to orchestrate the assistant calls, or can they call each other directly?

My app is deployed at https://agentround.com/ and is open source (on my profile).
If someone wants to try it:

• Add your API key (locally stored only)
• Create a new thread with a defined assistant (created on OpenAPI playground)
• Chat with it just like ChatGPT
• Tag another assistant (@assistantname) to ask for different perspectives

[Max-Levitskiy]

I didn't run an assistant API poc right now. But we already have a multi model context in our chat. The solution is not open source, it's a part of my startup. I can give you a read mode to take a look, but it's not something clean where you can easily understand everything. :) in general, it's a chat based on Google firebase. We're storing messages and sending context to other models and giving more context control to the user.
The idea with assistant for us is in the splitting the chat to "archived" part which will be uploaded as files and "on demand" which will be included in the thread. I'm not sure it will work efficiently. I'm not sure how long the thread can be. And really interested in the token usage, if it will be more efficient.

[Max-Levitskiy]

Btw, if you wanna check out the interface, you can find it here: SymbiotAI.com

[jjshab]

I was just writing along these lines: " Would something like "open interpreter" https://github.com/KillianLucas/open-interpreter help us with this?

I’m wondering if once agents have access to a machine, couldn’t they more easily and naturally communicate via a simple chat app like Discord and then use the machine access to perform various other actions and/or functions?

Instead of making the network work in a typical Python or other codebase or via an API system, change the playing field completely. This changes the rules fundamentally and opens up options.

I’m sure this has probably already been discussed and shot down for a thousand reasons, but I couldn’t find any discussion history. Even if moving the agents into the code interpreter world isn’t the solution, changing the agents' fundamental context can be done in many different ways.

So rather than try and build from scratch or try to force the agents into a network in a hostile environment for this, so to speak, let’s find another environment that allows easy communication/networking for these agents without too many other tradeoffs.

[OWigginsHay]

@Max-Levitskiy I like this idea and I like that you tied it to the method used in tool creation which was my work. If you are interested in this idea still, I think the idea of calling agents in a similar manner is very doable. I made post her ages ago: #123 where I describe my vision for agents with extensions. I think what you are describing would be a natural follow on from tool_making and be a good candidate for the second extension on a unit.