Telemetry and observability

[douglasware]

I'm curious what the philosophy around telemetry/observability is on the dotnet side. I am surprised at the general lack of it considering the logger is passed around so much. I personally consider this to be very important in a system doing orchestrations. Looking at the App Insights for the fully deployed Chat Copilot sample, I am struck by how little one can tell about it as it runs.

Is this on purpose or would contributions to add more tracing be welcome?

[glahaye]

@douglasware We actually observed the same thing today on the Chat Copilot. I believe it might have been instantiated without the configuration to make it work properly. I'll verify this today and get back to you shortly.

[glahaye]

@douglasware By default, only warnings and above are sent to App Insights.

If you want to see more logs, you can adjust the level sent to App Insights:

  // Logging configuration
  "Logging": {
    "LogLevel": {
      "Default": "Information",
      "CopilotChat.WebApi": "Information",
      "Microsoft.SemanticKernel": "Information",
      "Microsoft.AspNetCore.Hosting": "Information",
      "Microsoft.Hosting.Lifetime": "Information"
    },
    "ApplicationInsights": {
      "LogLevel": {
        // Configure log levels specific to Application Insights to better
        // control the volume of logs being sent
        "Default": "Information"
      }
    }
  }

[douglasware]

Thanks @glahaye!
This is the totality of the logging in Connectors.AI.OpenAI.

I think it would be good to be able to see the requests, responses, tokenization, etc.

I'd be happy to do a small PR or two to add more tracing if it is welcome.

[glahaye]

@douglasware If you can make logging more useful, then, yes, by all means submit PR's!

[douglasware]

In case anyone else reads this. WRT Chat Copilot, I am using these settings:

  // Logging configuration
  "Logging": {
    "LogLevel": {
      "Default": "Warning",
      "CopilotChat.WebApi": "Trace",
      "Microsoft.SemanticKernel": "Trace",
      "Microsoft.AspNetCore.Hosting": "Warning",
      "Microsoft.Hosting.Lifetime": "Error"
    },
    "ApplicationInsights": {
      "LogLevel": {
        // Configure log levels specific to Application Insights to better
        // control the volume of logs being sent
        "Default": "Information"
      }
    }
  }

The above will give you a decent picture of what is going on at the prompt/request/response level as there is good tracing in the prompt formatter code. It doesn't show you everything but it helps with the questions asked here #1239 and here #2355.

If you haven't set up AppInsights you can add:

builder.Logging.AddConsole();

to program.cs in the Chat Copilot Web API project to see the output in the local console at runtime.

I still plan to add some tracing in SK and submit a PR but in the meantime, I hope this helps.

[evchaki]

Check out the following blogs on how to use telemetry with SK - https://devblogs.microsoft.com/semantic-kernel/track-your-token-usage-and-costs-with-semantic-kernel/ and https://devblogs.microsoft.com/semantic-kernel/unlock-the-power-of-telemetry-in-semantic-kernel-sdk/