Document structured output for for AssistantAgent in its API doc, and note strict=True is required for FunctionTool

[chengyu-liu-cs]

What feature would you like to be added?

Current AssistantAgent does not support response_format for structured output.
extra_create_args={ "response_format": "Pydantic model", }
It would be perfect if respose_format can be supported by both on_messages and on_messages_stream.

Why is this needed?

It is a good feature supported by many models to return structured output. It would simplify integration with other application in cases where predefined structures are expected from AssistantAgent.

It seems it is supported by OpenAIChatCompletionClient

[ekzhu]

You can pass the client with structured output into the agent, which will then generate structured output: https://microsoft.github.io/autogen/stable/user-guide/agentchat-user-guide/tutorial/agents.html#structured-output

Make sure your FunctionTool is set to strict=True when using this.

[chengyu-liu-cs]

@ekzhu , It would be different right if the response_format is configured at client and create levels. If it sets in client level, all the create will generate the structured outputs. While configuring at create level should provide much better flexibility.

For example, if a user says "hello", then the output should be just "hello" instead of enforcing structured outputs.

[chengyu-liu-cs]

@ekzhu ,
I was testing the example in the link you provided. Instead of testing the example as it is, I was testing with model_client_stream=True,. There was an error below. Is this something AutoGen will work on ? Is it possible to stream in this scenario, and is there workaround ?
It seems from OpenAI side this is possible

"""
autogen/lib/python3.10/site-packages/openai/resources/chat/completions.py", line 1786, in validate_response_format raise TypeError( TypeError: You tried to pass a BaseModelclass tochat.completions.create(); You must use beta.chat.completions.parse() instead
"""

[ekzhu]

This is indeed a bug in the model client. Can you create a separate issue for this?

Open AI's client uses a different api for structure output. I think the create_stream method has not been updated to use it.

[ekzhu]

@ekzhu , It would be different right if the response_format is configured at client and create levels. If it sets in client level, all the create will generate the structured outputs. While configuring at create level should provide much better flexibility.

For example, if a user says "hello", then the output should be just "hello" instead of enforcing structured outputs.

How does one know when to use structured outputs and when not to? The model cannot decide this. When you pass in a base model in the response format in create method, it will still enforce the structure output. So it becomes the same thing as passing response format in the client which gets passed into the agent.

Perhaps some dynamic approach to detect user intent and then decide whether to use structured output is possible. However, the AssistantAgent is already quite complex and if we add more complex behaviour it will become confusing.

[chengyu-liu-cs]

Perhaps some dynamic approach to detect user intent and then decide whether to use structured output is possible. However, the AssistantAgent is already quite complex and if we add more complex behaviour it will become confusing.

Exactly. First, detect intents and then dynamically decide if tools and response_format should be attached in model calls.

Any tips or example implementations if this would not be included in AssistantAgent ?

[chengyu-liu-cs]

This is indeed a bug in the model client. Can you create a separate issue for this?

Open AI's client uses a different api for structure output. I think the create_stream method has not been updated to use it.

Yes. I will create a new issue for this.

[ekzhu]

Any tips or example implementations if this would not be included in AssistantAgent ?

I think this can be a great idea for a custom agent that is meant to be a multi-purpose agent. Or, perhaps a selector group chat with dynamic routing based on user's intent.