Function calling

[chadbailey59]

No description provided.

[chadbailey59]

So, function calling is... weird.

Without function calling, a chatbot pipeline is pretty straightforward:

• The user says something
• Pipecat appends whatever the user said to a messages list and send that list to the LLM
• The LLM generates an assistant response
• Pipecat generates TTS from that assistant response and plays that audio through the transport

With function calling, the flow is different:

• The user says something
• Pipecat appends whatever the user said to a messages list (along with some possible "tools" it may choose to use) and sends that list to the LLM
• The LLM generates an assistant response that may be text, or it may be a "tool call", i.e. the LLM decides to use one of the available "tools"
  • If it's text:
    • Pipecat generates TTS from that assistant response and plays that audio through the transport
  • if it's a tool call:
    • Pipecat needs to append the assistant message with the tool call, including its params, to the message list
    • Pipecat then needs to call the requested function with the provided params (e.g. "check_weather" with params {location: 'san francisco'}
    • Pipecat then needs to append the results from that function call to the messages list in a weird format
    • Pipecat then needs to re-prompt the LLM with the new messages list to generate an answer to the user's question
    • Finally, Pipecat generates TTS from that second assistant response and plays that audio through the transport

Right now, that entire second branch is implemented in the 15-function-calling example as a FunctionCaller class that pushes a context frame back up the pipeline for the re-prompting. We should probably be handling all this inside the framework itself, but that starts to touch on how much context management we should be doing on behalf of the user.

[aconchillo]

All the comments are removed? And we should use WakeCheckFilter instead.

[aconchillo]

I believe this can be added internally when we push LLMFunctionCallFrame. There's nothing the user should do in here.

[chadbailey59]

I'm hesitant to do this on behalf of the user, because as a general rule we don't mess with the context in the framework. In the patient-intake example, I'm kind of misusing function calling and not actually inserting function call results into the context, for example.

[aconchillo]

I see. Well, in this case it's not the result, it's just the function call itself. Maybe for this one maybe we could add an argument to the LLM service, something like include_function_calling_in_context or something like that (and it defaults to True). Asking the user to handle all this is too much probably?

[chadbailey59]

@aconchillo I think I've addressed the concerns and the function calling code is ready to merge. But I'm still concerned I may have inadvertently undone some of your changes through various merges and rebases.

[aconchillo]

Remove LLMFunctionStartFrame and CallFrame

[chadbailey59]

OK, I think I've gotten the new function calling approach where it needs to be. Let's get this one merged, and I can remove the function call frame types in a follow-up PR. @aconchillo

[aconchillo]

should all these functions return None?

[aconchillo]

They do implicitly, but maybe better to be more explicit about it, if so.