Examples in the ActionPlanner

[lukasz-appstream]

Hey folks,

I've been pondering over the idea of rewriting examples in the ActionPlanner class (

semantic-kernel/dotnet/src/Extensions/Planning.ActionPlanner/ActionPlanner.cs

Line 160 in b31e36b

// TODO: generate string programmatically

) to generate them from code. However, I couldn't help but wonder, why go through the trouble? Initially, it appears more practical to keep them within the prompt.

Suppose we reach a stage where we have an abundance of examples and need to select the most suitable one for a specific goal. In that case, wouldn't it be better to explore goal embeddings and leverage similarity search as an effective extension? As of now, it seems unnecessary to invest effort in generating examples through code when we have one good example and one edge case.

If I've overlooked something crucial, I'm open to being proven wrong and would appreciate your insights.

Cheers,
Lukasz

[dluc]

It's a very good point which has come up several times. When using few-shots learning, it would be nice having an option to manage the prompt with a configurable list - still optional, ie one can still use a plain text prompt. Here's some relevant features I proposed a while back:

• create a "memory" of good and bad examples (aka "procedural memory"); allow to search examples relevant to the goal and inject those in the prompt. Include bad examples with a reason.
• allow developers and users (depending on the UI) to upvote/downvote plans. When a plan is upvoted, save it as a good example. When downvoted ask why and save it as a bad example.
• when calling any planner (there should be many planners) allow to pass a custom prompt
• when calling any planner allow to pass a custom list of examples
• leverage the "procedural memory" as a central repository of functions and plugins; allow to save plans as new functions. This would allow to train a system to achieve new goals, and build on this capability to achieve more complex goals.
• allow planners to tell which plugins/functions are missing in order to achieve a goal, allowing the user to train the system. E.g. "to do X I need a plugin to send emails, I don't know how to send emails, please teach me".

You might have noticed the new models launched in past few months, e.g. chat models, support for functions. In the past we also had Codex, fine tuned for coding (planning is a form of coding). I think over time we'll see these capabilities (planning and learning) exposed as specific models/APIs, potentially with runtime state and pluggable private memory. If you like, you could build all of this within ChatGPT and Copilot Chat leveraging plugins and RAG/semantic memory, with some work.

[lukasz-appstream]

That's an interesting perspective! I have also come across the new function calling capabilities, although I haven't had a chance to try them out yet. It seems like a built-in ActionPlanner functionality, and having it fine-tuned and accurate is definitely a plus. However, it does raise questions about planning next steps of Semantic Kernel development, doesn't it? 😉

I completely agree with you about the potential for expansion in different directions with planners. I was playing with a tough of making it like a "recurrent" planner, where each step builds upon the knowledge from the previous step, so that we decide "locally" what should happen next, but never had time to dive in and test it ;)

You think private memory will be part of the model API? I know it's a must for multiple use cases to have reliable and accurate memory which LLM doesn't have, but it seems like a job for an orchestrator to connect model to any DB that a specific use case demand?