ASP.Net Core SemanticTextMemory - How to use plugin

[hemantkathuria]

I have registered the Memory store / Plugin as below in ASP.Net core application

I looked at the below sample where with Kernel.Invoke kernel functions are being used.

Sample

  builder.Services.AddTransient((serviceProvider) => {
    Kernel kernel = new Kernel(serviceProvider);
    IDatabase database = connectionMultiplexer.GetDatabase();
    // If unspecified, the default vector size is 1536.
    int vectorSize = 1536;
    // Initialize a memory store using the redis database
    IMemoryStore memoryStore = new RedisMemoryStore(database, vectorSize);
    ITextEmbeddingGenerationService embeddingService = kernel.GetRequiredService<ITextEmbeddingGenerationService>();
    // Initialize a SemanticTextMemory using the memory store and embedding generation service.
    SemanticTextMemory textMemory = new(memoryStore, embeddingService);
    // Initialize a TextMemoryPlugin using the text memory.
    TextMemoryPlugin memoryPlugin = new TextMemoryPlugin(textMemory);
    kernel.Plugins.AddFromObject(memoryPlugin,"memory");
    return kernel;
 });

I have couple of questions.

1. With IChatCompletionService SementicKernel (SK) should automatically invoke Memory.Search/Recall or I need to call search first and add the results in chat history? With the syntax below Memory.Recall/Search is never invoked? I feel, if I have to explicitly call SemanticTextMemory.Search, then what is the benefit of it adding as plugin. Please let me know if SK can automatically call recall / search on memory store. With SK SDK I am completely lost on when to use Kernel.Invoke and When to use ChatCompletionAPI, as it is very confusing

   OpenAIPromptExecutionSettings openAIPromptExecutionSettings = new()
   {
   ToolCallBehavior = ToolCallBehavior.AutoInvokeKernelFunctions
   };
   _result = await _chatCompletionService.GetChatMessageContentAsync(
   _chatHistory, executionSettings: openAIPromptExecutionSettings, _kernel);

2. With ASP.Net Core what is the right way to keep chat history? Shall we store the history for each user in a cache like redis? I don't see any good samples around to tell where to keep chat history in case of ASP.Net applicaitions.

[markwallace-microsoft]

Hi @hemantkathuria

1. When using auto invocation of functions the model will call the function if it supports function calling and if it need to call the function to respond the the current ask.
2. We don't have guidance on how to store chat history. @matthewbolanos should we create something in our backlog for this?

[hemantkathuria]

@markwallace-microsoft - I am using [gpt-4o] on Azure Open AI but my memory plugin is never called. Also, all the examples are available with Kernel.invoke like here and here also there is an explicit call to recall. I have not found many examples with chat completion API.

Also, this link is giving me the impression that memory plugin will not be invoked automatically Could you please cross check and confirm one more time please?

Also, it will be great if you share guidance on chat history ?

[hk4git]

Yes, please let us know, how TextMemoryPlugin can be used with chat complition auto invoke functionality.

[markwallace-microsoft]

Hi @hemantkathuria
I experimented and I can see the problem you are facing. The methods exposed by TextMemoryPlugin aren't useful for function calling and were intended to be used when you need to inject context into a prompt.
I've created a sample showing how to solve this problem in the short term, see: https://github.com/microsoft/semantic-kernel/pull/8968/files.
We are in the process of updating our memory abstractions and search capabilities and supporting function calling is one of the use cases we are targeting. We expect to release the new functionality in the next few weeks and we'll present this at the weekly community call. The design document for the new functionality is available here: https://github.com/microsoft/semantic-kernel/blob/21f8a278e55c23b34e96c9de3ab06e8564dca703/docs/decisions/00NN-text-search.md