Log metrics for Services don't show unless "log.level" emitted

[roshan-elastic]

This isn't technically a bug but I'd like to discuss if we can avoid a situation where when a service only emits logs without log.level then neither the log rate or log error % will show:

New Services Inventory

Logs-only Service View

We require log.level vs any logs in the logs-* index to filter out a lot of noise from APM likely to relating failed transactions (to avoid duplication) and general unhelpful noise.

Tasks

Preview Give feedback

1. Get docs created #189498
   needs-team +1
   
2. Provide copy for designs #189499
   needs-team +1
   
3. Provide short-links to point to from UI #189500
   needs-team +1
   
4. https://github.com/elastic/observability-design/issues/302
5. [APM][ECO] Logs callout when logs.level is not available #189731
   Team:obs-ux-infra_services enhancement +2
   

Related

• [Request]: Update 'service.name' docs with additional guidance observability-docs#4102
• Update short-links to point at docs #189501

🎨 Design

Service View message
Figma

Copy:

No log metrics have been detected against this service
Please ensure you are surfacing log.levelin your logs to display log metrics. Learn more

Service Inventory log rate and log error % columns

Copy:

Want to see more?
In order to see log metrics against this service, please declare log.level in your logs.

Learn more

[elasticmachine]

Pinging @elastic/obs-ux-infra_services-team (Team:obs-ux-infra_services)

[roshan-elastic]

Why do we filter out anything without log.level?

Looking into the underlying data, I think I found why we restricted this to only count when log.level is set. The APM services emit a lot of noise to the logs-* indices which get filtered out if we filter for log.level : *:

Sample logs for an APM service

If I look at all services as a whole, you can see it's broken down into these categories:

Digging around these messages, they generally seem to be APM-specific relating to transactions that I don't think are helpful to the user (and perhaps captured in the 'failed transactions' metrics anyway?):

What should we do?

Assuming I'm right about the above, I think we should keep this filter and follow up on @cauemarcondes' suggestion to add the (i) N/A state for the log rate/log error rate to explain to the user how to get these populated:

Sample message

(and update the docs too)

We can then explain to the user why log metrics aren't there so they can take action and we can also wait for feedback to see how this plays out with customers.

Be keen to hear thoughts when this gets picked up.

[roshan-elastic]

Decision

cc @kkurstak @kpatticha @cauemarcondes

We spoke on the design call and agreed:

1. We'll stick with the requirement for log.level for the log metrics
2. We'll stick the N/A box in the log columns as per the above comment
3. On the service view page, we'll add a permanent warning to explain why the log metrics aren't populated (with links to docs)
4. On the service view page, we'll ensure the 'Add APM' CTA is dismissible

Actions

• @kkurstak - To make designs and share here
• @roshan-elastic to provide copy
• @roshan-elastic to get docs created

[cauemarcondes]

I don't think we need to mention the service.name on the N/A copy. If the user is seeing that service means the service.name was already provided.

[roshan-elastic]

I don't think we need to mention the service.name on the N/A copy. If the user is seeing that service means the service.name was already provided.

I hear you. Let me think about the copy and I'll post it once the designs are shared.

[kkurstak]

FYI -
Created this issue for the design work: https://github.com/elastic/observability-design/issues/302
Design is ready, final copy to be provided by @roshan-elastic

[roshan-elastic]

FYI copy in issue description for when engineering pick up

[roshan-elastic]

Removing issues which don't block this work

[kpatticha]

closing this one since the PR has been merged into main

#189644