Correlated trace with Google.Cloud.Logging.Console

[charring]

I read the threads where this was discussed and an issue was added

#9948

And see that's now in the docs
https://cloud.google.com/dotnet/docs/reference/Google.Cloud.Logging.Console/latest#log-trace-correlation

My traces aren't correlated, and I assume that I have to do something in my logging code to have that happen. But I don't know what.

This page [https://cloud.google.com/logging/docs/view/correlate-logs] says

"However, you can group, or correlate your log entries by using the trace field in the LogEntry object.
When you correlate log entries, you can view a parent log entry with multiple child log entries in the Logs Explorer."

The docs says
"When writing a server application, the current .NET Activity needs to be initialized with the trace context included in the current request. ASP.NET Core does this for you automatically based on HTTP headers"
But doesn't elaborate. Must I have something in the http request header to drive the trace correlation?

[amanda-tarafa]

On the bullet point above the one you are citing it says:

Google Cloud runtimes export some traces automatically. (For example, Cloud Run will export traces for received requests by default, but you would need to export traces for any additional information.)

This implies, and it's explicitly stated in the linked documentation, that: "The standard W3C trace context propagation header traceparent is automatically populated for Cloud Run requests."

This all means that if you are running ASP.NET Core in CloudRun, in principle the only thing you need to is configure the logger by specifying your Google Cloud Project Id as shown in the documentation you linked (as follows)

using Google.Cloud.Logging.Console;
...
var builder = WebApplication.CreateBuilder(args);
builder.Logging.AddGoogleCloudConsole(options => options.TraceGoogleCloudProjectId = "google-project-id");

Because:
a) CloudRun will include the trace context information on the request headers.
b) CloudRun is exporting this trace context to Google Cloud Trace.
c) ASP.NET Core will extract this trace context information from the header and initialize the current Acitivity.

Did you configured the console logger as shown and still your log entries were not correlated?

[charring]

As I had configured the logger as you have shown, I figured that the issue must be with my use of the Logs Explorer. And I figured it out after re-reading https://cloud.google.com/logging/docs/view/correlate-logs

As it says, you have to choose the right log. I choose "run.googleapis.com/requests" and got a correlated traces.

The logs explorer is a powerful tool and I'll have to invest some time in learning it.

[charring]

I've come back to working on my Cloud Run app, and now I can't get correlated logs.
I went back to the doc that I had used: https://cloud.google.com/logging/docs/view/correlate-logs
But the log explorer has been updated while this doc has not been updated. Can someone either update that doc and/or convey what are the correct selections in the new log explorer one needs to make?

[amanda-tarafa]

Can you be more specific about what differences you are seeing on the documentation vs. the log explorer. After a quick overview, what I read and see on https://cloud.google.com/logging/docs/view/correlate-logs matches what I see on the log explorer.

But also, it's best if you use the Send Feedback button at the top right of the page, that's the fasted way to make certain that the issues you find get to the right team sooner.

[charring]

From that doc:

• In the Resource drop-down menu, select GAE Application.
• In the Log Name drop-down menu, select request_log.

I see neither GAE Application nor request_log

I tried submitting feedback. I get a wait spinner for 30 sec and then the message "we could not send your feedback - retry". So I'm pasting here what I wrote. I'm not including the screenshot

---

Previously I followed this doc to setup correlated logs: https://cloud.google.com/logging/docs/view/correlate-logs
But that doc no longer matches what I see here. I see no "GAE Application" in step 2, nor "request_log" in step 3.

In the new Log Explorer, there is a "Correlate by" dropdown. If I select "requests" then the log does get collapsed
to just show requests. But the stdout messages are not nested within those log entries.

I assume that a) the doc hasn't been updated or b) I'm on the wrong page or c) I'm not using it correctly.

---

Note: I removed the screenshot and then the send feedback worked. I sent an additional feedback explaining that it wouldn't work with an attached screenshot.

[amanda-tarafa]

As far as I know the dropdown shows elements for which there is information to display. For instance I can see "GAE Application" in my logs explorer for a GCP project where I have App Engine services running, but I cannot see it in a project where I don't have App Engine services running. Note that the documentation says:

For example, to display your App Engine request log entry and application log entries, do the following:
...
2. In the Resource drop-down menu, select GAE Application.

If your application was running say, in Cloud Run, you would have to select something different than GAE Application.

A couple of checks for you to try:

• Are you 100% certain that the application that is supposed to be generating logs is actually up and running? And that is has executed code paths that should log? Is it running on App Engine or some other Google Runtime?
• Is there any application specific message that you include in your logs that you can search for? Say, your application logs something like "Request received, starting processing...". If you search for that, do you find it? You can then use the log_name field on that entry to infer what you need to select on the filters.

I'm afraid we won't be able to help you much more with Logs Explorer usability as this is a tool that we don't maintain or have significant knowledge of. If you need further assistance here, you can follow Google Operations Suite support channels. So I'll be closing this issue now.

Of course, if it turns out that your application is correctly setup but it's not logging or correlating logs to traces, then feel free to reopen and we can look into it.

[charring]

Under "correlate by" dropdown, I have selected "Cloud Run requests"

Are you 100% certain that the application that is supposed to be generating logs is actually up and running? And that is has executed code paths that should log? Is it running on App Engine or some other Google Runtime?

Yes. The stdout messages are present if I don't correlate

Is there any application specific message that you include in your logs that you can search for?

If it's not in correlate mode, I can search and find expected messages. If I correlate, no messages are found.

feel free to reopen

I don't see how to reopen an issue

[amanda-tarafa]

OK, if you can see logs when not correlating but can't find them when correlating there might be an issue with either correlation setup or a bug in how we correlate. I'm reopening the issue so we can investigate.

A few things for you to try and/or answer:

• Before, you were deploying your application on App Engine and you were able to see correlated logs.

• Now, you have started to deploy your application on Cloud Run and you are not able to see any logs when correlating, but logs are found when correlation is not configured.

• Is the application that you are deploying on Cloud Run the same as the one you deployed in App Engine? Has something changed there configuration wise?

• Can you deploy the application in App Engine again and check if you see correlated logs?

• Can you check the Using Distributed Tracing in Cloud Run documentation to confirm that everything in your setup is fine. I'd expecte it is as Cloud Run traces automatically. But do notice the section on Trace Sampling Rate and make certain you are receiving enough requests to actually get traces.

• Also please review the documentation on Viewing Traces and try to find the traces for your application and check if these have logs associated to them? Try to then find those logs in the Logs Explorer.

I'm reopening this issue while you follow the steps above and once you have an anwer to each of these points, we'll be able to look further.

[charring]

Hi,

Your questions led me to look more carefully at my notes, and to see that it was a different version of the app that had correlation working.

The difference, for anyone who comes upon similar behavior, was that some logging wasn't using the Google Cloud Console log provider. Those logged messages would be displayed when logs are not correlated but would not be present in the expanded request when correlation was enabled. And that makes sense.

There is change in the Log Explorer that I think should be reflected in the documentation. The documentation steps are:

1. In the Resource drop-down menu, select GAE Application.
2. In the Log Name drop-down menu, select request_log.
3. In the Query results pane, to expand a log entry, click "> Expand"

But correlation does not occur unless you also select "requests" in the "Correlate By" control.

Thanks again for your help!

[amanda-tarafa]

For the documentation changes request, please follow the Send Feedback button you used earlier. That will guarentee the request reaching the right team sooner.

Otherwise, I'm glad to see everything is working as expected. I'll close this issue now as it seems everything is sorted.