How do I record the cost of each LLM call in SK

[unsung189]

Hello, everyone.

Recently, SK is planning to build the company's ai application, the general idea is to form a service with multiple plug-ins, and the terminal can access different services according to needs. There are two issues involved

1: The plug-in does not have a annotation to mark it, nor does it implement a fixed interface, which makes it difficult to dynamically configure multiple plug-ins into an application. I wish it had a [SKPlugin] to tag and describe what it does. Also, the plugin constructors are not uniform, which also causes difficulties when I want to create them on demand while the program is running.

2: After each call to the LLM, the content returned can record the model of each visit and the cost.

[RogerBarreto]

It was not very clear for me the first ASK, as we have [SKFunction] attributes and those functions have the SkillName which actually is the actual plugin, so eventually if you have your function reference you are able to track which was executed from a specific Skill/Plugin.

For the second item currently you can do that using the ModelResult property from the SKContext.

If you are using Azure or OpenAI packages there's a special extension method that converts the ModelResult object in the actual detailed result where you can get info like usage and model executed.

Check here for reference: Example43_GetModelResult.cs

The result will be something on this structure for OpenAI requests.

[
    {
            "Id": "cmpl-7S6nc9YnYNNv3YD0UWhcVhU8ZXmpe",
            "Created": 1686933676,
            "Model": "text-davinci-003",
            "Choices": [
              {
                "Text": "Excuse: I was busy trying to break the world record for the longest game of hide and seek.",
                "Index": 0,
                "Logprobs": null,
                "FinishReason": "stop"
              },
              {
                "Text": "Excuse: I was busy trying to break the world record for the longest game of hide and seek.",
                "Index": 1,
                "Logprobs": null,
                "FinishReason": "stop"
              },
              {
                "Text": "Excuse: I was busy trying to break the world record for the longest game of hide and seek.",
                "Index": 2,
                "Logprobs": null,
                "FinishReason": "stop"
              }
            ],
            "Usage": {
              "CompletionTokens": 63,
              "PromptTokens": 90,
              "TotalTokens": 153
            }
    },
    ...
]

[unsung189]

First of all, thank you very much for your answer. The second question is very clear.

The first problem you can look at is my example code: I want to configure a set of plugins for an app, and when the app runs, it needs to instantiate it according to the configured plugin. The following code is very complex and unstable in getting the plugin collection and instantiating it

`public async Task<ActionResult> InvokeAsync([FromServices] IKernel kernel, Guid appId, [FromBody] AskDto ask)
{
// Input check
var app = await _appRepository.FindAsync(x => x.Id == appId);
Check.NotNull(app, nameof(app));

    var appPlugins = await _appPluginRepository.GetListAsync(x => x.AppId == appId);
    Check.NotNull(appPlugins, nameof(appPlugins));

    var pluginIds = appPlugins.Select(plugin => plugin.PluginId).ToList();
    var plugins = await _pluginRepository.GetListAsync(x => pluginIds.Contains(x.Id));
    Check.NotNull(plugins, nameof(plugins));

    // Initializes the parameter variable
    var context = kernel.CreateNewContext();
    foreach (var input in ask.Variables)
    {
        context.Variables.Set(input.Key, input.Value);
    }
    // initialize plug-ins
    var loadedSkills = await _pluginDomainService.GetPluginTypes();
    loadedSkills.Where(x => plugins.Any(plugin => plugin.Namespace == x.Namespace && plugin.Name == x.Name)).ToList().ForEach(x =>
    {
        var skillInstance = _pluginDomainService.CreateInstance(x, new object[] { kernel, _serviceProvider });
        if (skillInstance != null)
        {
            kernel.ImportSkill(skillInstance, x.Name);
        }
    });
    var plan = app.PlannerXml.ToPlanFromXml(ask.Input, context);
    var result = plan.InvokeAsync(context);

    return new ActionResult<AskResultDto>(new AskResultDto());
}`