Auto calling Kernel Function not working

[KishanJ98]

Based on the examples, I have a YAML prompt with the following settings:

I have a class called ChatPlugin with the following method:

I can see that in the Kernel there is a ChatPlugin with a function called 'GenerateImage'.
In the YAML prompt template I have the instruction: "If asked to generate an image, use the GenerateImage method to generate an image url and return the outcome of the method as your answer."
I'm using the following to execute the function:

However, the model is not calling this method when the user message intent is to generate an image. How to make this work?

[moonbox3]

Adding @SergeyMenshykh for visibility.

[SergeyMenshykh]

Hi @KishanJ98, I can think of a few reasons the function is not called:

1. The chat completion service is not registered as the default, but function call behavior is enabled for the "default" service in your YAML prompt template config. If that is the case, then either register the chat completion service as the default by not supplying any argument for the serviceId parameter in the Add{Azure}OpenChatCompletion method call, or update the YAML prompt config to match the service id with which the chat completion service is registered.
2. The YAML prompt formatting is incorrect, even though it does not appear that way in the screenshot. YAML is sensitive to extra tabs and whitespaces.
3. Your instructions don't include a prompt. The AI model responded to me with, "Sure, could you please provide me with the details or theme of the image you wish to generate?" and did not call the function when I sent the provided instructions. However, adding the prompt resolved the issue - the function was called, and the model returned the image url.

If the function still isn't called after you've checked the reasons above, try running this code snippet and let me know if it works:

[Fact]
public async Task RunPromptTemplateConfigWithAutoFunctionChoiceBehaviorUsingStreamingAsync()
{
    // Create kernel
    IKernelBuilder builder = Kernel.CreateBuilder();

    builder.AddOpenAIChatCompletion(TestConfiguration.OpenAI.ChatModelId, TestConfiguration.OpenAI.ApiKey);

    Kernel kernel = builder.Build();

    kernel.ImportPluginFromType<ChatPlugin>();

    string promptTemplateConfig = """
        template_format: semantic-kernel
        template: If asked to generate an image, use the GenerateImage method to generate an image url and return the outcome of the method as your answer. {{$prompt}}
        execution_settings:
          default:
            function_choice_behavior:
              type: auto
              functions:
                - ChatPlugin.GenerateImage
        """;

    KernelFunction promptFunction = kernel.CreateFunctionFromPromptYaml(promptTemplateConfig);

    KernelArguments arguments = new()
    {
        { "prompt", "Generate an image of a cat." }
    };

    StringBuilder stringBuilder = new();

    await foreach (StreamingChatMessageContent update in kernel.InvokeStreamingAsync(promptFunction, arguments))
    {
        if(!string.IsNullOrEmpty(update.Content))
        {
            stringBuilder.Append(update.Content);
        }
    }

    Console.Write(stringBuilder.ToString());
}

private sealed class ChatPlugin
{
    [KernelFunction, Description("Generate an image based on the prompt.")]
    public async Task<string?> GenerateImageAsync([Description("The user's message or user intent")] string message)
    {
        return "https://fake-image-url/";
    }
}

The output:

[KishanJ98]

Hi Sergey,

I have this as code right now:
`public class ImagePlugin
{
private readonly Kernel _kernel;
private readonly ILogger _logger;

public ImagePlugin(Kernel kernel, ILogger logger)
{
    _kernel = kernel;
    _logger = logger;
}

/// <summary>
/// Method that generates an image using Dall-e based on the given prompt.
/// </summary>
/// <param name="message">The prompt to generate an image for.</param>
[KernelFunction, Description("Generates an image based on the prompt.")]
public async Task<string?> GenerateImageAsync(
    [Description("The user's message or user intent")] string message)
{
    var dallE = _kernel.GetRequiredService<ITextToImageService>();
    try
    {
        var image = await dallE.GenerateImageAsync(message, 1024, 1024);
        return image;
    }
    catch (Exception e)
    {
        _logger.LogError($"Image generation failed: {e.Message}");
        return null;
    }
}

}`

Execution:

  var imagePlugin = new ImagePlugin(Kernel, Logger);
        Kernel.ImportPluginFromObject(imagePlugin, "ImagePlugin");
        string promptTemplateConfig = """
                                      template_format: semantic-kernel
                                      template: If asked to generate an image, use the GenerateImage method to generate an image url and return the outcome of the method as your answer. {{$message}}
                                      execution_settings:
                                        default:
                                          function_choice_behavior:
                                            type: auto
                                            functions:
                                              - ImagePlugin.GenerateImage
                                      """;

        KernelFunction promptFunction = Kernel.CreateFunctionFromPromptYaml(promptTemplateConfig);
        var result = await Kernel.InvokeAsync(promptFunction, chatKernelArguments, cancellationToken);

However this is not calling the function, $message contains the prompt, but the function is not getting called.

[KishanJ98]

@SergeyMenshykh Can you look at my reply please?