Program optimization features

[joshmouch]

I'd like to discuss the possibility of adding self-training and evaluation capabilities to Semantic Kernel, similar to what DSPy offers. Specifically:

Is there any consideration on the roadmap for features that would allow:

1. Automated prompt optimization through testing and evaluation

   • Self-training capabilities for plugins and semantic functions
   • Systematic evaluation frameworks for prompt templates
   • Built-in support for collecting and formatting chat histories for fine-tuning
   • Pre-built plugins for common prompting patterns like:
     • Chain of Thought (CoT) reasoning
     • Few-shot learning
     • Self-consistency checking
     • Tree of Thoughts
     • These could standardize best practices while remaining customizable

2. I've noticed the documentation and recent features (like Processors and Agents) have moved toward a code-first approach, moving away from text-based configuration and prompts. While I understand the benefits of type safety and IDE support, having configurable/optimizable prompts could offer significant advantages for:

   • A/B testing different prompt strategies
   • Automated prompt refinement based on performance metrics
   • More flexible deployment and updates without code changes

3. Additionally, I see an opportunity for cost optimization through fine-tuning support:

   • Making it easier to capture chat histories from plugins using premium models (like GPT-4o or GPT-4o-1)
   • Providing built-in formatting tools to prepare this data for fine-tuning cheaper models (like GPT-4o-mini)
   • Creating a seamless workflow to transition from expensive model exploration to cost-effective deployment

This could help teams start with powerful models for exploration and initial development, then efficiently transition to more cost-effective fine-tuned models for production use.
For context, I'm looking at frameworks like DSPy which treat prompts as trainable parameters rather than static strings. Would Semantic Kernel consider a hybrid approach that maintains the code-first benefits while adding these optimization capabilities?
Happy to provide more specific examples or use cases if helpful.

Also happy to contribute experimental PRs if this aligns with the Semantic Kernel vision.

[sophialagerkranspandey]

This is a great suggestion, thanks for the feedback. Currently we have evaluations for Python (https://learn.microsoft.com/en-us/semantic-kernel/overview/).