-
Notifications
You must be signed in to change notification settings - Fork 8.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[APM] Fix missing error.culprit #205242
[APM] Fix missing error.culprit #205242
Conversation
/ci |
Pinging @elastic/obs-ux-infra_services-team (Team:obs-ux-infra_services) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
Starting backport for target branches: 8.16, 8.17, 8.x |
fixes [elastic#205240](elastic#205240) ## Summary Fixes the issue where `error.culprit` is always missing. <img width="800" alt="image" src="https://github.com/user-attachments/assets/9dad6797-dd40-4206-b508-daf8d69d653d" /> <img width="800" alt="image" src="https://github.com/user-attachments/assets/6a0ac25a-bc4b-4a26-b126-86eecbe7e769" /> ### How to test - Run `simple_trace` synthtrace scenario - Run the request below <details> <summary>POST logs-apm.error-default/_doc</summary> ``` POST logs-apm.error-default/_doc { "container": { "id": "80cc20ecab1b57865d7aca0f66ab75a0c6ec2c51f7c53da04a8540215369c359" }, "kubernetes": { "node": { "name": "ip-192-168-55-136.ec2.internal" }, "pod": { "uid": "9bbfe3dc-dace-4638-b8f5-aa6ccf435cbe", "name": "otel-demo-green-3-cartservice-5d5ccc86dd-gcxtr" }, "namespace": "otel-demo-green-3" }, "parent": { "id": "f39639f56a540e0e" }, "agent": { "name": "opentelemetry/dotnet", "version": "1.6.0" }, "source": { "port": 35136, "ip": "::ffff:192.168.50.132" }, "error": { "exception": [ { "handled": true, "message": "Exception (plain, .NET) without culprit", "type": "Grpc.Core.RpcException" } ], "culprit": "culprit msg", // remove to test the N/A scenario "id": "60674e9b23c8b06d83889e63ffe82c1c", //change the id on every run "stack_trace": "Grpc.Core.RpcException: Status(StatusCode=\"FailedPrecondition\", Detail=\"Can't access cart storage. System.ApplicationException: Wasn't able to connect to redis\n at cartservice.cartstore.RedisCartStore.EnsureRedisConnected() in /usr/src/app/src/cartstore/RedisCartStore.cs:line 79\n at cartservice.cartstore.RedisCartStore.EmptyCartAsync(String userId) in /usr/src/app/src/cartstore/RedisCartStore.cs:line 157\")\n at cartservice.cartstore.RedisCartStore.EmptyCartAsync(String userId) in /usr/src/app/src/cartstore/RedisCartStore.cs:line 166\n at cartservice.services.CartService.EmptyCart(EmptyCartRequest request, ServerCallContext context) in /usr/src/app/src/services/CartService.cs:line 65", "grouping_key": "6082c58afd7fa6838ebc27eef6e04829" }, "message": "Exception (plain, .NET)", "url": { "path": "/oteldemo.CartService/EmptyCart", "original": "http://otel-demo-green-3-cartservice:8080/oteldemo.CartService/EmptyCart", "scheme": "http", "port": 8080, "domain": "otel-demo-green-3-cartservice", "full": "http://otel-demo-green-3-cartservice:8080/oteldemo.CartService/EmptyCart" }, "labels": { "k8s_pod_ip": "192.168.62.174", "k8s_pod_start_time": "2024-12-17T02:55:54Z", "service_namespace": "opentelemetry-demo", "k8s_deployment_name": "otel-demo-green-3-cartservice" }, "observer": { "hostname": "450fbf7ab701", "type": "apm-server", "version": "8.13.2" }, "trace": { "id": "699f6d1039fc873191a1c983790db3a2" }, "@timestamp": "2024-12-30T10:30:00.000Z", "data_stream": { "namespace": "default", "type": "logs", "dataset": "apm.error" }, "service": { "node": { "name": "80cc20ecab1b57865d7aca0f66ab75a0c6ec2c51f7c53da04a8540215369c359" }, "framework": { "name": "Microsoft.AspNetCore" }, "name": "synth-node-0", "language": { "name": "dotnet" } }, "host": { "hostname": "ip-192-168-55-136.ec2.internal", "name": "ip-192-168-55-136.ec2.internal" }, "client": { "port": 35136, "ip": "::ffff:192.168.50.132" }, "http": { "request": { "method": "POST" }, "response": { "status_code": 200 }, "version": "2" }, "event": { "agent_id_status": "missing", "ingested": "2024-12-18T18:04:22Z" }, "transaction": { "id": "f39639f56a540e0e", "type": "request", "sampled": true }, "user_agent": { "original": "grpc-go/1.59.0", "name": "Other", "device": { "name": "Other" } }, "span": { "id": "f39639f56a540e0e" }, "timestamp": { "us": 1704892605838000 } } ``` </details> (cherry picked from commit 12608c3)
fixes [elastic#205240](elastic#205240) ## Summary Fixes the issue where `error.culprit` is always missing. <img width="800" alt="image" src="https://github.com/user-attachments/assets/9dad6797-dd40-4206-b508-daf8d69d653d" /> <img width="800" alt="image" src="https://github.com/user-attachments/assets/6a0ac25a-bc4b-4a26-b126-86eecbe7e769" /> ### How to test - Run `simple_trace` synthtrace scenario - Run the request below <details> <summary>POST logs-apm.error-default/_doc</summary> ``` POST logs-apm.error-default/_doc { "container": { "id": "80cc20ecab1b57865d7aca0f66ab75a0c6ec2c51f7c53da04a8540215369c359" }, "kubernetes": { "node": { "name": "ip-192-168-55-136.ec2.internal" }, "pod": { "uid": "9bbfe3dc-dace-4638-b8f5-aa6ccf435cbe", "name": "otel-demo-green-3-cartservice-5d5ccc86dd-gcxtr" }, "namespace": "otel-demo-green-3" }, "parent": { "id": "f39639f56a540e0e" }, "agent": { "name": "opentelemetry/dotnet", "version": "1.6.0" }, "source": { "port": 35136, "ip": "::ffff:192.168.50.132" }, "error": { "exception": [ { "handled": true, "message": "Exception (plain, .NET) without culprit", "type": "Grpc.Core.RpcException" } ], "culprit": "culprit msg", // remove to test the N/A scenario "id": "60674e9b23c8b06d83889e63ffe82c1c", //change the id on every run "stack_trace": "Grpc.Core.RpcException: Status(StatusCode=\"FailedPrecondition\", Detail=\"Can't access cart storage. System.ApplicationException: Wasn't able to connect to redis\n at cartservice.cartstore.RedisCartStore.EnsureRedisConnected() in /usr/src/app/src/cartstore/RedisCartStore.cs:line 79\n at cartservice.cartstore.RedisCartStore.EmptyCartAsync(String userId) in /usr/src/app/src/cartstore/RedisCartStore.cs:line 157\")\n at cartservice.cartstore.RedisCartStore.EmptyCartAsync(String userId) in /usr/src/app/src/cartstore/RedisCartStore.cs:line 166\n at cartservice.services.CartService.EmptyCart(EmptyCartRequest request, ServerCallContext context) in /usr/src/app/src/services/CartService.cs:line 65", "grouping_key": "6082c58afd7fa6838ebc27eef6e04829" }, "message": "Exception (plain, .NET)", "url": { "path": "/oteldemo.CartService/EmptyCart", "original": "http://otel-demo-green-3-cartservice:8080/oteldemo.CartService/EmptyCart", "scheme": "http", "port": 8080, "domain": "otel-demo-green-3-cartservice", "full": "http://otel-demo-green-3-cartservice:8080/oteldemo.CartService/EmptyCart" }, "labels": { "k8s_pod_ip": "192.168.62.174", "k8s_pod_start_time": "2024-12-17T02:55:54Z", "service_namespace": "opentelemetry-demo", "k8s_deployment_name": "otel-demo-green-3-cartservice" }, "observer": { "hostname": "450fbf7ab701", "type": "apm-server", "version": "8.13.2" }, "trace": { "id": "699f6d1039fc873191a1c983790db3a2" }, "@timestamp": "2024-12-30T10:30:00.000Z", "data_stream": { "namespace": "default", "type": "logs", "dataset": "apm.error" }, "service": { "node": { "name": "80cc20ecab1b57865d7aca0f66ab75a0c6ec2c51f7c53da04a8540215369c359" }, "framework": { "name": "Microsoft.AspNetCore" }, "name": "synth-node-0", "language": { "name": "dotnet" } }, "host": { "hostname": "ip-192-168-55-136.ec2.internal", "name": "ip-192-168-55-136.ec2.internal" }, "client": { "port": 35136, "ip": "::ffff:192.168.50.132" }, "http": { "request": { "method": "POST" }, "response": { "status_code": 200 }, "version": "2" }, "event": { "agent_id_status": "missing", "ingested": "2024-12-18T18:04:22Z" }, "transaction": { "id": "f39639f56a540e0e", "type": "request", "sampled": true }, "user_agent": { "original": "grpc-go/1.59.0", "name": "Other", "device": { "name": "Other" } }, "span": { "id": "f39639f56a540e0e" }, "timestamp": { "us": 1704892605838000 } } ``` </details> (cherry picked from commit 12608c3)
💔 Some backports could not be created
Note: Successful backport PRs will be merged automatically after passing CI. Manual backportTo create the backport manually run:
Questions ?Please refer to the Backport tool documentation |
fixes [elastic#205240](elastic#205240) ## Summary Fixes the issue where `error.culprit` is always missing. <img width="800" alt="image" src="https://github.com/user-attachments/assets/9dad6797-dd40-4206-b508-daf8d69d653d" /> <img width="800" alt="image" src="https://github.com/user-attachments/assets/6a0ac25a-bc4b-4a26-b126-86eecbe7e769" /> ### How to test - Run `simple_trace` synthtrace scenario - Run the request below <details> <summary>POST logs-apm.error-default/_doc</summary> ``` POST logs-apm.error-default/_doc { "container": { "id": "80cc20ecab1b57865d7aca0f66ab75a0c6ec2c51f7c53da04a8540215369c359" }, "kubernetes": { "node": { "name": "ip-192-168-55-136.ec2.internal" }, "pod": { "uid": "9bbfe3dc-dace-4638-b8f5-aa6ccf435cbe", "name": "otel-demo-green-3-cartservice-5d5ccc86dd-gcxtr" }, "namespace": "otel-demo-green-3" }, "parent": { "id": "f39639f56a540e0e" }, "agent": { "name": "opentelemetry/dotnet", "version": "1.6.0" }, "source": { "port": 35136, "ip": "::ffff:192.168.50.132" }, "error": { "exception": [ { "handled": true, "message": "Exception (plain, .NET) without culprit", "type": "Grpc.Core.RpcException" } ], "culprit": "culprit msg", // remove to test the N/A scenario "id": "60674e9b23c8b06d83889e63ffe82c1c", //change the id on every run "stack_trace": "Grpc.Core.RpcException: Status(StatusCode=\"FailedPrecondition\", Detail=\"Can't access cart storage. System.ApplicationException: Wasn't able to connect to redis\n at cartservice.cartstore.RedisCartStore.EnsureRedisConnected() in /usr/src/app/src/cartstore/RedisCartStore.cs:line 79\n at cartservice.cartstore.RedisCartStore.EmptyCartAsync(String userId) in /usr/src/app/src/cartstore/RedisCartStore.cs:line 157\")\n at cartservice.cartstore.RedisCartStore.EmptyCartAsync(String userId) in /usr/src/app/src/cartstore/RedisCartStore.cs:line 166\n at cartservice.services.CartService.EmptyCart(EmptyCartRequest request, ServerCallContext context) in /usr/src/app/src/services/CartService.cs:line 65", "grouping_key": "6082c58afd7fa6838ebc27eef6e04829" }, "message": "Exception (plain, .NET)", "url": { "path": "/oteldemo.CartService/EmptyCart", "original": "http://otel-demo-green-3-cartservice:8080/oteldemo.CartService/EmptyCart", "scheme": "http", "port": 8080, "domain": "otel-demo-green-3-cartservice", "full": "http://otel-demo-green-3-cartservice:8080/oteldemo.CartService/EmptyCart" }, "labels": { "k8s_pod_ip": "192.168.62.174", "k8s_pod_start_time": "2024-12-17T02:55:54Z", "service_namespace": "opentelemetry-demo", "k8s_deployment_name": "otel-demo-green-3-cartservice" }, "observer": { "hostname": "450fbf7ab701", "type": "apm-server", "version": "8.13.2" }, "trace": { "id": "699f6d1039fc873191a1c983790db3a2" }, "@timestamp": "2024-12-30T10:30:00.000Z", "data_stream": { "namespace": "default", "type": "logs", "dataset": "apm.error" }, "service": { "node": { "name": "80cc20ecab1b57865d7aca0f66ab75a0c6ec2c51f7c53da04a8540215369c359" }, "framework": { "name": "Microsoft.AspNetCore" }, "name": "synth-node-0", "language": { "name": "dotnet" } }, "host": { "hostname": "ip-192-168-55-136.ec2.internal", "name": "ip-192-168-55-136.ec2.internal" }, "client": { "port": 35136, "ip": "::ffff:192.168.50.132" }, "http": { "request": { "method": "POST" }, "response": { "status_code": 200 }, "version": "2" }, "event": { "agent_id_status": "missing", "ingested": "2024-12-18T18:04:22Z" }, "transaction": { "id": "f39639f56a540e0e", "type": "request", "sampled": true }, "user_agent": { "original": "grpc-go/1.59.0", "name": "Other", "device": { "name": "Other" } }, "span": { "id": "f39639f56a540e0e" }, "timestamp": { "us": 1704892605838000 } } ``` </details> (cherry picked from commit 12608c3) # Conflicts: # x-pack/test/api_integration/deployment_agnostic/apis/observability/apm/mobile/errors/group_id_samples.spec.ts
💚 All backports created successfully
Note: Successful backport PRs will be merged automatically after passing CI. Questions ?Please refer to the Backport tool documentation |
# Backport This will backport the following commits from `main` to `8.17`: - [[APM Fix missing error.culprit (#205242)](#205242) <!--- Backport version: 9.4.3 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Carlos Crespo","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-12-30T12:50:48Z","message":"[APM Fix missing error.culprit (#205242)\n\nfixes [#205240](https://github.com/elastic/kibana/issues/205240)\r\n\r\n## Summary\r\n\r\nFixes the issue where `error.culprit` is always missing.\r\n\r\n<img width=\"800\" alt=\"image\"\r\nsrc=\"https://github.com/user-attachments/assets/9dad6797-dd40-4206-b508-daf8d69d653d\"\r\n/>\r\n\r\n<img width=\"800\" alt=\"image\"\r\nsrc=\"https://github.com/user-attachments/assets/6a0ac25a-bc4b-4a26-b126-86eecbe7e769\"\r\n/>\r\n\r\n\r\n\r\n### How to test \r\n\r\n- Run `simple_trace` synthtrace scenario\r\n- Run the request below\r\n<details>\r\n<summary>POST logs-apm.error-default/_doc</summary>\r\n\r\n```\r\nPOST logs-apm.error-default/_doc\r\n{\r\n\r\n \"container\": {\r\n \"id\": \"80cc20ecab1b57865d7aca0f66ab75a0c6ec2c51f7c53da04a8540215369c359\"\r\n },\r\n \"kubernetes\": {\r\n \"node\": {\r\n \"name\": \"ip-192-168-55-136.ec2.internal\"\r\n },\r\n \"pod\": {\r\n \"uid\": \"9bbfe3dc-dace-4638-b8f5-aa6ccf435cbe\",\r\n \"name\": \"otel-demo-green-3-cartservice-5d5ccc86dd-gcxtr\"\r\n },\r\n \"namespace\": \"otel-demo-green-3\"\r\n },\r\n \"parent\": {\r\n \"id\": \"f39639f56a540e0e\"\r\n },\r\n \"agent\": {\r\n \"name\": \"opentelemetry/dotnet\",\r\n \"version\": \"1.6.0\"\r\n },\r\n \"source\": {\r\n \"port\": 35136,\r\n \"ip\": \"::ffff:192.168.50.132\"\r\n },\r\n \"error\": {\r\n \"exception\": [\r\n {\r\n \"handled\": true,\r\n \"message\": \"Exception (plain, .NET) without culprit\",\r\n \"type\": \"Grpc.Core.RpcException\"\r\n }\r\n ],\r\n \"culprit\": \"culprit msg\", // remove to test the N/A scenario\r\n \"id\": \"60674e9b23c8b06d83889e63ffe82c1c\", //change the id on every run\r\n \"stack_trace\": \"Grpc.Core.RpcException: Status(StatusCode=\\\"FailedPrecondition\\\", Detail=\\\"Can't access cart storage. System.ApplicationException: Wasn't able to connect to redis\\n at cartservice.cartstore.RedisCartStore.EnsureRedisConnected() in /usr/src/app/src/cartstore/RedisCartStore.cs:line 79\\n at cartservice.cartstore.RedisCartStore.EmptyCartAsync(String userId) in /usr/src/app/src/cartstore/RedisCartStore.cs:line 157\\\")\\n at cartservice.cartstore.RedisCartStore.EmptyCartAsync(String userId) in /usr/src/app/src/cartstore/RedisCartStore.cs:line 166\\n at cartservice.services.CartService.EmptyCart(EmptyCartRequest request, ServerCallContext context) in /usr/src/app/src/services/CartService.cs:line 65\",\r\n \"grouping_key\": \"6082c58afd7fa6838ebc27eef6e04829\"\r\n },\r\n \"message\": \"Exception (plain, .NET)\",\r\n \"url\": {\r\n \"path\": \"/oteldemo.CartService/EmptyCart\",\r\n \"original\": \"http://otel-demo-green-3-cartservice:8080/oteldemo.CartService/EmptyCart\",\r\n \"scheme\": \"http\",\r\n \"port\": 8080,\r\n \"domain\": \"otel-demo-green-3-cartservice\",\r\n \"full\": \"http://otel-demo-green-3-cartservice:8080/oteldemo.CartService/EmptyCart\"\r\n },\r\n \"labels\": {\r\n \"k8s_pod_ip\": \"192.168.62.174\",\r\n \"k8s_pod_start_time\": \"2024-12-17T02:55:54Z\",\r\n \"service_namespace\": \"opentelemetry-demo\",\r\n \"k8s_deployment_name\": \"otel-demo-green-3-cartservice\"\r\n },\r\n \"observer\": {\r\n \"hostname\": \"450fbf7ab701\",\r\n \"type\": \"apm-server\",\r\n \"version\": \"8.13.2\"\r\n },\r\n \"trace\": {\r\n \"id\": \"699f6d1039fc873191a1c983790db3a2\"\r\n },\r\n \"@timestamp\": \"2024-12-30T10:30:00.000Z\",\r\n \"data_stream\": {\r\n \"namespace\": \"default\",\r\n \"type\": \"logs\",\r\n \"dataset\": \"apm.error\"\r\n },\r\n \"service\": {\r\n \"node\": {\r\n \"name\": \"80cc20ecab1b57865d7aca0f66ab75a0c6ec2c51f7c53da04a8540215369c359\"\r\n },\r\n \"framework\": {\r\n \"name\": \"Microsoft.AspNetCore\"\r\n },\r\n \"name\": \"synth-node-0\",\r\n \"language\": {\r\n \"name\": \"dotnet\"\r\n }\r\n },\r\n \"host\": {\r\n \"hostname\": \"ip-192-168-55-136.ec2.internal\",\r\n \"name\": \"ip-192-168-55-136.ec2.internal\"\r\n },\r\n \"client\": {\r\n \"port\": 35136,\r\n \"ip\": \"::ffff:192.168.50.132\"\r\n },\r\n \"http\": {\r\n \"request\": {\r\n \"method\": \"POST\"\r\n },\r\n \"response\": {\r\n \"status_code\": 200\r\n },\r\n \"version\": \"2\"\r\n },\r\n \"event\": {\r\n \"agent_id_status\": \"missing\",\r\n \"ingested\": \"2024-12-18T18:04:22Z\"\r\n },\r\n \"transaction\": {\r\n \"id\": \"f39639f56a540e0e\",\r\n \"type\": \"request\",\r\n \"sampled\": true\r\n },\r\n \"user_agent\": {\r\n \"original\": \"grpc-go/1.59.0\",\r\n \"name\": \"Other\",\r\n \"device\": {\r\n \"name\": \"Other\"\r\n }\r\n },\r\n \"span\": {\r\n \"id\": \"f39639f56a540e0e\"\r\n },\r\n \"timestamp\": {\r\n \"us\": 1704892605838000\r\n }\r\n }\r\n```\r\n</details>","sha":"12608c3659a2aaea702c1d341bd0fab30f968056","branchLabelMapping":{"^v9.0.0$":"main","^v8.18.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","v9.0.0","backport:prev-major","Team:obs-ux-infra_services"],"title":"[APM Fix missing error.culprit","number":205242,"url":"https://github.com/elastic/kibana/pull/205242","mergeCommit":{"message":"[APM Fix missing error.culprit (#205242)\n\nfixes [#205240](https://github.com/elastic/kibana/issues/205240)\r\n\r\n## Summary\r\n\r\nFixes the issue where `error.culprit` is always missing.\r\n\r\n<img width=\"800\" alt=\"image\"\r\nsrc=\"https://github.com/user-attachments/assets/9dad6797-dd40-4206-b508-daf8d69d653d\"\r\n/>\r\n\r\n<img width=\"800\" alt=\"image\"\r\nsrc=\"https://github.com/user-attachments/assets/6a0ac25a-bc4b-4a26-b126-86eecbe7e769\"\r\n/>\r\n\r\n\r\n\r\n### How to test \r\n\r\n- Run `simple_trace` synthtrace scenario\r\n- Run the request below\r\n<details>\r\n<summary>POST logs-apm.error-default/_doc</summary>\r\n\r\n```\r\nPOST logs-apm.error-default/_doc\r\n{\r\n\r\n \"container\": {\r\n \"id\": \"80cc20ecab1b57865d7aca0f66ab75a0c6ec2c51f7c53da04a8540215369c359\"\r\n },\r\n \"kubernetes\": {\r\n \"node\": {\r\n \"name\": \"ip-192-168-55-136.ec2.internal\"\r\n },\r\n \"pod\": {\r\n \"uid\": \"9bbfe3dc-dace-4638-b8f5-aa6ccf435cbe\",\r\n \"name\": \"otel-demo-green-3-cartservice-5d5ccc86dd-gcxtr\"\r\n },\r\n \"namespace\": \"otel-demo-green-3\"\r\n },\r\n \"parent\": {\r\n \"id\": \"f39639f56a540e0e\"\r\n },\r\n \"agent\": {\r\n \"name\": \"opentelemetry/dotnet\",\r\n \"version\": \"1.6.0\"\r\n },\r\n \"source\": {\r\n \"port\": 35136,\r\n \"ip\": \"::ffff:192.168.50.132\"\r\n },\r\n \"error\": {\r\n \"exception\": [\r\n {\r\n \"handled\": true,\r\n \"message\": \"Exception (plain, .NET) without culprit\",\r\n \"type\": \"Grpc.Core.RpcException\"\r\n }\r\n ],\r\n \"culprit\": \"culprit msg\", // remove to test the N/A scenario\r\n \"id\": \"60674e9b23c8b06d83889e63ffe82c1c\", //change the id on every run\r\n \"stack_trace\": \"Grpc.Core.RpcException: Status(StatusCode=\\\"FailedPrecondition\\\", Detail=\\\"Can't access cart storage. System.ApplicationException: Wasn't able to connect to redis\\n at cartservice.cartstore.RedisCartStore.EnsureRedisConnected() in /usr/src/app/src/cartstore/RedisCartStore.cs:line 79\\n at cartservice.cartstore.RedisCartStore.EmptyCartAsync(String userId) in /usr/src/app/src/cartstore/RedisCartStore.cs:line 157\\\")\\n at cartservice.cartstore.RedisCartStore.EmptyCartAsync(String userId) in /usr/src/app/src/cartstore/RedisCartStore.cs:line 166\\n at cartservice.services.CartService.EmptyCart(EmptyCartRequest request, ServerCallContext context) in /usr/src/app/src/services/CartService.cs:line 65\",\r\n \"grouping_key\": \"6082c58afd7fa6838ebc27eef6e04829\"\r\n },\r\n \"message\": \"Exception (plain, .NET)\",\r\n \"url\": {\r\n \"path\": \"/oteldemo.CartService/EmptyCart\",\r\n \"original\": \"http://otel-demo-green-3-cartservice:8080/oteldemo.CartService/EmptyCart\",\r\n \"scheme\": \"http\",\r\n \"port\": 8080,\r\n \"domain\": \"otel-demo-green-3-cartservice\",\r\n \"full\": \"http://otel-demo-green-3-cartservice:8080/oteldemo.CartService/EmptyCart\"\r\n },\r\n \"labels\": {\r\n \"k8s_pod_ip\": \"192.168.62.174\",\r\n \"k8s_pod_start_time\": \"2024-12-17T02:55:54Z\",\r\n \"service_namespace\": \"opentelemetry-demo\",\r\n \"k8s_deployment_name\": \"otel-demo-green-3-cartservice\"\r\n },\r\n \"observer\": {\r\n \"hostname\": \"450fbf7ab701\",\r\n \"type\": \"apm-server\",\r\n \"version\": \"8.13.2\"\r\n },\r\n \"trace\": {\r\n \"id\": \"699f6d1039fc873191a1c983790db3a2\"\r\n },\r\n \"@timestamp\": \"2024-12-30T10:30:00.000Z\",\r\n \"data_stream\": {\r\n \"namespace\": \"default\",\r\n \"type\": \"logs\",\r\n \"dataset\": \"apm.error\"\r\n },\r\n \"service\": {\r\n \"node\": {\r\n \"name\": \"80cc20ecab1b57865d7aca0f66ab75a0c6ec2c51f7c53da04a8540215369c359\"\r\n },\r\n \"framework\": {\r\n \"name\": \"Microsoft.AspNetCore\"\r\n },\r\n \"name\": \"synth-node-0\",\r\n \"language\": {\r\n \"name\": \"dotnet\"\r\n }\r\n },\r\n \"host\": {\r\n \"hostname\": \"ip-192-168-55-136.ec2.internal\",\r\n \"name\": \"ip-192-168-55-136.ec2.internal\"\r\n },\r\n \"client\": {\r\n \"port\": 35136,\r\n \"ip\": \"::ffff:192.168.50.132\"\r\n },\r\n \"http\": {\r\n \"request\": {\r\n \"method\": \"POST\"\r\n },\r\n \"response\": {\r\n \"status_code\": 200\r\n },\r\n \"version\": \"2\"\r\n },\r\n \"event\": {\r\n \"agent_id_status\": \"missing\",\r\n \"ingested\": \"2024-12-18T18:04:22Z\"\r\n },\r\n \"transaction\": {\r\n \"id\": \"f39639f56a540e0e\",\r\n \"type\": \"request\",\r\n \"sampled\": true\r\n },\r\n \"user_agent\": {\r\n \"original\": \"grpc-go/1.59.0\",\r\n \"name\": \"Other\",\r\n \"device\": {\r\n \"name\": \"Other\"\r\n }\r\n },\r\n \"span\": {\r\n \"id\": \"f39639f56a540e0e\"\r\n },\r\n \"timestamp\": {\r\n \"us\": 1704892605838000\r\n }\r\n }\r\n```\r\n</details>","sha":"12608c3659a2aaea702c1d341bd0fab30f968056"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/205242","number":205242,"mergeCommit":{"message":"[APM Fix missing error.culprit (#205242)\n\nfixes [#205240](https://github.com/elastic/kibana/issues/205240)\r\n\r\n## Summary\r\n\r\nFixes the issue where `error.culprit` is always missing.\r\n\r\n<img width=\"800\" alt=\"image\"\r\nsrc=\"https://github.com/user-attachments/assets/9dad6797-dd40-4206-b508-daf8d69d653d\"\r\n/>\r\n\r\n<img width=\"800\" alt=\"image\"\r\nsrc=\"https://github.com/user-attachments/assets/6a0ac25a-bc4b-4a26-b126-86eecbe7e769\"\r\n/>\r\n\r\n\r\n\r\n### How to test \r\n\r\n- Run `simple_trace` synthtrace scenario\r\n- Run the request below\r\n<details>\r\n<summary>POST logs-apm.error-default/_doc</summary>\r\n\r\n```\r\nPOST logs-apm.error-default/_doc\r\n{\r\n\r\n \"container\": {\r\n \"id\": \"80cc20ecab1b57865d7aca0f66ab75a0c6ec2c51f7c53da04a8540215369c359\"\r\n },\r\n \"kubernetes\": {\r\n \"node\": {\r\n \"name\": \"ip-192-168-55-136.ec2.internal\"\r\n },\r\n \"pod\": {\r\n \"uid\": \"9bbfe3dc-dace-4638-b8f5-aa6ccf435cbe\",\r\n \"name\": \"otel-demo-green-3-cartservice-5d5ccc86dd-gcxtr\"\r\n },\r\n \"namespace\": \"otel-demo-green-3\"\r\n },\r\n \"parent\": {\r\n \"id\": \"f39639f56a540e0e\"\r\n },\r\n \"agent\": {\r\n \"name\": \"opentelemetry/dotnet\",\r\n \"version\": \"1.6.0\"\r\n },\r\n \"source\": {\r\n \"port\": 35136,\r\n \"ip\": \"::ffff:192.168.50.132\"\r\n },\r\n \"error\": {\r\n \"exception\": [\r\n {\r\n \"handled\": true,\r\n \"message\": \"Exception (plain, .NET) without culprit\",\r\n \"type\": \"Grpc.Core.RpcException\"\r\n }\r\n ],\r\n \"culprit\": \"culprit msg\", // remove to test the N/A scenario\r\n \"id\": \"60674e9b23c8b06d83889e63ffe82c1c\", //change the id on every run\r\n \"stack_trace\": \"Grpc.Core.RpcException: Status(StatusCode=\\\"FailedPrecondition\\\", Detail=\\\"Can't access cart storage. System.ApplicationException: Wasn't able to connect to redis\\n at cartservice.cartstore.RedisCartStore.EnsureRedisConnected() in /usr/src/app/src/cartstore/RedisCartStore.cs:line 79\\n at cartservice.cartstore.RedisCartStore.EmptyCartAsync(String userId) in /usr/src/app/src/cartstore/RedisCartStore.cs:line 157\\\")\\n at cartservice.cartstore.RedisCartStore.EmptyCartAsync(String userId) in /usr/src/app/src/cartstore/RedisCartStore.cs:line 166\\n at cartservice.services.CartService.EmptyCart(EmptyCartRequest request, ServerCallContext context) in /usr/src/app/src/services/CartService.cs:line 65\",\r\n \"grouping_key\": \"6082c58afd7fa6838ebc27eef6e04829\"\r\n },\r\n \"message\": \"Exception (plain, .NET)\",\r\n \"url\": {\r\n \"path\": \"/oteldemo.CartService/EmptyCart\",\r\n \"original\": \"http://otel-demo-green-3-cartservice:8080/oteldemo.CartService/EmptyCart\",\r\n \"scheme\": \"http\",\r\n \"port\": 8080,\r\n \"domain\": \"otel-demo-green-3-cartservice\",\r\n \"full\": \"http://otel-demo-green-3-cartservice:8080/oteldemo.CartService/EmptyCart\"\r\n },\r\n \"labels\": {\r\n \"k8s_pod_ip\": \"192.168.62.174\",\r\n \"k8s_pod_start_time\": \"2024-12-17T02:55:54Z\",\r\n \"service_namespace\": \"opentelemetry-demo\",\r\n \"k8s_deployment_name\": \"otel-demo-green-3-cartservice\"\r\n },\r\n \"observer\": {\r\n \"hostname\": \"450fbf7ab701\",\r\n \"type\": \"apm-server\",\r\n \"version\": \"8.13.2\"\r\n },\r\n \"trace\": {\r\n \"id\": \"699f6d1039fc873191a1c983790db3a2\"\r\n },\r\n \"@timestamp\": \"2024-12-30T10:30:00.000Z\",\r\n \"data_stream\": {\r\n \"namespace\": \"default\",\r\n \"type\": \"logs\",\r\n \"dataset\": \"apm.error\"\r\n },\r\n \"service\": {\r\n \"node\": {\r\n \"name\": \"80cc20ecab1b57865d7aca0f66ab75a0c6ec2c51f7c53da04a8540215369c359\"\r\n },\r\n \"framework\": {\r\n \"name\": \"Microsoft.AspNetCore\"\r\n },\r\n \"name\": \"synth-node-0\",\r\n \"language\": {\r\n \"name\": \"dotnet\"\r\n }\r\n },\r\n \"host\": {\r\n \"hostname\": \"ip-192-168-55-136.ec2.internal\",\r\n \"name\": \"ip-192-168-55-136.ec2.internal\"\r\n },\r\n \"client\": {\r\n \"port\": 35136,\r\n \"ip\": \"::ffff:192.168.50.132\"\r\n },\r\n \"http\": {\r\n \"request\": {\r\n \"method\": \"POST\"\r\n },\r\n \"response\": {\r\n \"status_code\": 200\r\n },\r\n \"version\": \"2\"\r\n },\r\n \"event\": {\r\n \"agent_id_status\": \"missing\",\r\n \"ingested\": \"2024-12-18T18:04:22Z\"\r\n },\r\n \"transaction\": {\r\n \"id\": \"f39639f56a540e0e\",\r\n \"type\": \"request\",\r\n \"sampled\": true\r\n },\r\n \"user_agent\": {\r\n \"original\": \"grpc-go/1.59.0\",\r\n \"name\": \"Other\",\r\n \"device\": {\r\n \"name\": \"Other\"\r\n }\r\n },\r\n \"span\": {\r\n \"id\": \"f39639f56a540e0e\"\r\n },\r\n \"timestamp\": {\r\n \"us\": 1704892605838000\r\n }\r\n }\r\n```\r\n</details>","sha":"12608c3659a2aaea702c1d341bd0fab30f968056"}}]}] BACKPORT--> Co-authored-by: Carlos Crespo <[email protected]>
# Backport This will backport the following commits from `main` to `8.x`: - [[APM Fix missing error.culprit (#205242)](#205242) <!--- Backport version: 9.4.3 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Carlos Crespo","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-12-30T12:50:48Z","message":"[APM Fix missing error.culprit (#205242)\n\nfixes [#205240](https://github.com/elastic/kibana/issues/205240)\r\n\r\n## Summary\r\n\r\nFixes the issue where `error.culprit` is always missing.\r\n\r\n<img width=\"800\" alt=\"image\"\r\nsrc=\"https://github.com/user-attachments/assets/9dad6797-dd40-4206-b508-daf8d69d653d\"\r\n/>\r\n\r\n<img width=\"800\" alt=\"image\"\r\nsrc=\"https://github.com/user-attachments/assets/6a0ac25a-bc4b-4a26-b126-86eecbe7e769\"\r\n/>\r\n\r\n\r\n\r\n### How to test \r\n\r\n- Run `simple_trace` synthtrace scenario\r\n- Run the request below\r\n<details>\r\n<summary>POST logs-apm.error-default/_doc</summary>\r\n\r\n```\r\nPOST logs-apm.error-default/_doc\r\n{\r\n\r\n \"container\": {\r\n \"id\": \"80cc20ecab1b57865d7aca0f66ab75a0c6ec2c51f7c53da04a8540215369c359\"\r\n },\r\n \"kubernetes\": {\r\n \"node\": {\r\n \"name\": \"ip-192-168-55-136.ec2.internal\"\r\n },\r\n \"pod\": {\r\n \"uid\": \"9bbfe3dc-dace-4638-b8f5-aa6ccf435cbe\",\r\n \"name\": \"otel-demo-green-3-cartservice-5d5ccc86dd-gcxtr\"\r\n },\r\n \"namespace\": \"otel-demo-green-3\"\r\n },\r\n \"parent\": {\r\n \"id\": \"f39639f56a540e0e\"\r\n },\r\n \"agent\": {\r\n \"name\": \"opentelemetry/dotnet\",\r\n \"version\": \"1.6.0\"\r\n },\r\n \"source\": {\r\n \"port\": 35136,\r\n \"ip\": \"::ffff:192.168.50.132\"\r\n },\r\n \"error\": {\r\n \"exception\": [\r\n {\r\n \"handled\": true,\r\n \"message\": \"Exception (plain, .NET) without culprit\",\r\n \"type\": \"Grpc.Core.RpcException\"\r\n }\r\n ],\r\n \"culprit\": \"culprit msg\", // remove to test the N/A scenario\r\n \"id\": \"60674e9b23c8b06d83889e63ffe82c1c\", //change the id on every run\r\n \"stack_trace\": \"Grpc.Core.RpcException: Status(StatusCode=\\\"FailedPrecondition\\\", Detail=\\\"Can't access cart storage. System.ApplicationException: Wasn't able to connect to redis\\n at cartservice.cartstore.RedisCartStore.EnsureRedisConnected() in /usr/src/app/src/cartstore/RedisCartStore.cs:line 79\\n at cartservice.cartstore.RedisCartStore.EmptyCartAsync(String userId) in /usr/src/app/src/cartstore/RedisCartStore.cs:line 157\\\")\\n at cartservice.cartstore.RedisCartStore.EmptyCartAsync(String userId) in /usr/src/app/src/cartstore/RedisCartStore.cs:line 166\\n at cartservice.services.CartService.EmptyCart(EmptyCartRequest request, ServerCallContext context) in /usr/src/app/src/services/CartService.cs:line 65\",\r\n \"grouping_key\": \"6082c58afd7fa6838ebc27eef6e04829\"\r\n },\r\n \"message\": \"Exception (plain, .NET)\",\r\n \"url\": {\r\n \"path\": \"/oteldemo.CartService/EmptyCart\",\r\n \"original\": \"http://otel-demo-green-3-cartservice:8080/oteldemo.CartService/EmptyCart\",\r\n \"scheme\": \"http\",\r\n \"port\": 8080,\r\n \"domain\": \"otel-demo-green-3-cartservice\",\r\n \"full\": \"http://otel-demo-green-3-cartservice:8080/oteldemo.CartService/EmptyCart\"\r\n },\r\n \"labels\": {\r\n \"k8s_pod_ip\": \"192.168.62.174\",\r\n \"k8s_pod_start_time\": \"2024-12-17T02:55:54Z\",\r\n \"service_namespace\": \"opentelemetry-demo\",\r\n \"k8s_deployment_name\": \"otel-demo-green-3-cartservice\"\r\n },\r\n \"observer\": {\r\n \"hostname\": \"450fbf7ab701\",\r\n \"type\": \"apm-server\",\r\n \"version\": \"8.13.2\"\r\n },\r\n \"trace\": {\r\n \"id\": \"699f6d1039fc873191a1c983790db3a2\"\r\n },\r\n \"@timestamp\": \"2024-12-30T10:30:00.000Z\",\r\n \"data_stream\": {\r\n \"namespace\": \"default\",\r\n \"type\": \"logs\",\r\n \"dataset\": \"apm.error\"\r\n },\r\n \"service\": {\r\n \"node\": {\r\n \"name\": \"80cc20ecab1b57865d7aca0f66ab75a0c6ec2c51f7c53da04a8540215369c359\"\r\n },\r\n \"framework\": {\r\n \"name\": \"Microsoft.AspNetCore\"\r\n },\r\n \"name\": \"synth-node-0\",\r\n \"language\": {\r\n \"name\": \"dotnet\"\r\n }\r\n },\r\n \"host\": {\r\n \"hostname\": \"ip-192-168-55-136.ec2.internal\",\r\n \"name\": \"ip-192-168-55-136.ec2.internal\"\r\n },\r\n \"client\": {\r\n \"port\": 35136,\r\n \"ip\": \"::ffff:192.168.50.132\"\r\n },\r\n \"http\": {\r\n \"request\": {\r\n \"method\": \"POST\"\r\n },\r\n \"response\": {\r\n \"status_code\": 200\r\n },\r\n \"version\": \"2\"\r\n },\r\n \"event\": {\r\n \"agent_id_status\": \"missing\",\r\n \"ingested\": \"2024-12-18T18:04:22Z\"\r\n },\r\n \"transaction\": {\r\n \"id\": \"f39639f56a540e0e\",\r\n \"type\": \"request\",\r\n \"sampled\": true\r\n },\r\n \"user_agent\": {\r\n \"original\": \"grpc-go/1.59.0\",\r\n \"name\": \"Other\",\r\n \"device\": {\r\n \"name\": \"Other\"\r\n }\r\n },\r\n \"span\": {\r\n \"id\": \"f39639f56a540e0e\"\r\n },\r\n \"timestamp\": {\r\n \"us\": 1704892605838000\r\n }\r\n }\r\n```\r\n</details>","sha":"12608c3659a2aaea702c1d341bd0fab30f968056","branchLabelMapping":{"^v9.0.0$":"main","^v8.18.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","v9.0.0","backport:prev-major","Team:obs-ux-infra_services"],"title":"[APM Fix missing error.culprit","number":205242,"url":"https://github.com/elastic/kibana/pull/205242","mergeCommit":{"message":"[APM Fix missing error.culprit (#205242)\n\nfixes [#205240](https://github.com/elastic/kibana/issues/205240)\r\n\r\n## Summary\r\n\r\nFixes the issue where `error.culprit` is always missing.\r\n\r\n<img width=\"800\" alt=\"image\"\r\nsrc=\"https://github.com/user-attachments/assets/9dad6797-dd40-4206-b508-daf8d69d653d\"\r\n/>\r\n\r\n<img width=\"800\" alt=\"image\"\r\nsrc=\"https://github.com/user-attachments/assets/6a0ac25a-bc4b-4a26-b126-86eecbe7e769\"\r\n/>\r\n\r\n\r\n\r\n### How to test \r\n\r\n- Run `simple_trace` synthtrace scenario\r\n- Run the request below\r\n<details>\r\n<summary>POST logs-apm.error-default/_doc</summary>\r\n\r\n```\r\nPOST logs-apm.error-default/_doc\r\n{\r\n\r\n \"container\": {\r\n \"id\": \"80cc20ecab1b57865d7aca0f66ab75a0c6ec2c51f7c53da04a8540215369c359\"\r\n },\r\n \"kubernetes\": {\r\n \"node\": {\r\n \"name\": \"ip-192-168-55-136.ec2.internal\"\r\n },\r\n \"pod\": {\r\n \"uid\": \"9bbfe3dc-dace-4638-b8f5-aa6ccf435cbe\",\r\n \"name\": \"otel-demo-green-3-cartservice-5d5ccc86dd-gcxtr\"\r\n },\r\n \"namespace\": \"otel-demo-green-3\"\r\n },\r\n \"parent\": {\r\n \"id\": \"f39639f56a540e0e\"\r\n },\r\n \"agent\": {\r\n \"name\": \"opentelemetry/dotnet\",\r\n \"version\": \"1.6.0\"\r\n },\r\n \"source\": {\r\n \"port\": 35136,\r\n \"ip\": \"::ffff:192.168.50.132\"\r\n },\r\n \"error\": {\r\n \"exception\": [\r\n {\r\n \"handled\": true,\r\n \"message\": \"Exception (plain, .NET) without culprit\",\r\n \"type\": \"Grpc.Core.RpcException\"\r\n }\r\n ],\r\n \"culprit\": \"culprit msg\", // remove to test the N/A scenario\r\n \"id\": \"60674e9b23c8b06d83889e63ffe82c1c\", //change the id on every run\r\n \"stack_trace\": \"Grpc.Core.RpcException: Status(StatusCode=\\\"FailedPrecondition\\\", Detail=\\\"Can't access cart storage. System.ApplicationException: Wasn't able to connect to redis\\n at cartservice.cartstore.RedisCartStore.EnsureRedisConnected() in /usr/src/app/src/cartstore/RedisCartStore.cs:line 79\\n at cartservice.cartstore.RedisCartStore.EmptyCartAsync(String userId) in /usr/src/app/src/cartstore/RedisCartStore.cs:line 157\\\")\\n at cartservice.cartstore.RedisCartStore.EmptyCartAsync(String userId) in /usr/src/app/src/cartstore/RedisCartStore.cs:line 166\\n at cartservice.services.CartService.EmptyCart(EmptyCartRequest request, ServerCallContext context) in /usr/src/app/src/services/CartService.cs:line 65\",\r\n \"grouping_key\": \"6082c58afd7fa6838ebc27eef6e04829\"\r\n },\r\n \"message\": \"Exception (plain, .NET)\",\r\n \"url\": {\r\n \"path\": \"/oteldemo.CartService/EmptyCart\",\r\n \"original\": \"http://otel-demo-green-3-cartservice:8080/oteldemo.CartService/EmptyCart\",\r\n \"scheme\": \"http\",\r\n \"port\": 8080,\r\n \"domain\": \"otel-demo-green-3-cartservice\",\r\n \"full\": \"http://otel-demo-green-3-cartservice:8080/oteldemo.CartService/EmptyCart\"\r\n },\r\n \"labels\": {\r\n \"k8s_pod_ip\": \"192.168.62.174\",\r\n \"k8s_pod_start_time\": \"2024-12-17T02:55:54Z\",\r\n \"service_namespace\": \"opentelemetry-demo\",\r\n \"k8s_deployment_name\": \"otel-demo-green-3-cartservice\"\r\n },\r\n \"observer\": {\r\n \"hostname\": \"450fbf7ab701\",\r\n \"type\": \"apm-server\",\r\n \"version\": \"8.13.2\"\r\n },\r\n \"trace\": {\r\n \"id\": \"699f6d1039fc873191a1c983790db3a2\"\r\n },\r\n \"@timestamp\": \"2024-12-30T10:30:00.000Z\",\r\n \"data_stream\": {\r\n \"namespace\": \"default\",\r\n \"type\": \"logs\",\r\n \"dataset\": \"apm.error\"\r\n },\r\n \"service\": {\r\n \"node\": {\r\n \"name\": \"80cc20ecab1b57865d7aca0f66ab75a0c6ec2c51f7c53da04a8540215369c359\"\r\n },\r\n \"framework\": {\r\n \"name\": \"Microsoft.AspNetCore\"\r\n },\r\n \"name\": \"synth-node-0\",\r\n \"language\": {\r\n \"name\": \"dotnet\"\r\n }\r\n },\r\n \"host\": {\r\n \"hostname\": \"ip-192-168-55-136.ec2.internal\",\r\n \"name\": \"ip-192-168-55-136.ec2.internal\"\r\n },\r\n \"client\": {\r\n \"port\": 35136,\r\n \"ip\": \"::ffff:192.168.50.132\"\r\n },\r\n \"http\": {\r\n \"request\": {\r\n \"method\": \"POST\"\r\n },\r\n \"response\": {\r\n \"status_code\": 200\r\n },\r\n \"version\": \"2\"\r\n },\r\n \"event\": {\r\n \"agent_id_status\": \"missing\",\r\n \"ingested\": \"2024-12-18T18:04:22Z\"\r\n },\r\n \"transaction\": {\r\n \"id\": \"f39639f56a540e0e\",\r\n \"type\": \"request\",\r\n \"sampled\": true\r\n },\r\n \"user_agent\": {\r\n \"original\": \"grpc-go/1.59.0\",\r\n \"name\": \"Other\",\r\n \"device\": {\r\n \"name\": \"Other\"\r\n }\r\n },\r\n \"span\": {\r\n \"id\": \"f39639f56a540e0e\"\r\n },\r\n \"timestamp\": {\r\n \"us\": 1704892605838000\r\n }\r\n }\r\n```\r\n</details>","sha":"12608c3659a2aaea702c1d341bd0fab30f968056"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/205242","number":205242,"mergeCommit":{"message":"[APM Fix missing error.culprit (#205242)\n\nfixes [#205240](https://github.com/elastic/kibana/issues/205240)\r\n\r\n## Summary\r\n\r\nFixes the issue where `error.culprit` is always missing.\r\n\r\n<img width=\"800\" alt=\"image\"\r\nsrc=\"https://github.com/user-attachments/assets/9dad6797-dd40-4206-b508-daf8d69d653d\"\r\n/>\r\n\r\n<img width=\"800\" alt=\"image\"\r\nsrc=\"https://github.com/user-attachments/assets/6a0ac25a-bc4b-4a26-b126-86eecbe7e769\"\r\n/>\r\n\r\n\r\n\r\n### How to test \r\n\r\n- Run `simple_trace` synthtrace scenario\r\n- Run the request below\r\n<details>\r\n<summary>POST logs-apm.error-default/_doc</summary>\r\n\r\n```\r\nPOST logs-apm.error-default/_doc\r\n{\r\n\r\n \"container\": {\r\n \"id\": \"80cc20ecab1b57865d7aca0f66ab75a0c6ec2c51f7c53da04a8540215369c359\"\r\n },\r\n \"kubernetes\": {\r\n \"node\": {\r\n \"name\": \"ip-192-168-55-136.ec2.internal\"\r\n },\r\n \"pod\": {\r\n \"uid\": \"9bbfe3dc-dace-4638-b8f5-aa6ccf435cbe\",\r\n \"name\": \"otel-demo-green-3-cartservice-5d5ccc86dd-gcxtr\"\r\n },\r\n \"namespace\": \"otel-demo-green-3\"\r\n },\r\n \"parent\": {\r\n \"id\": \"f39639f56a540e0e\"\r\n },\r\n \"agent\": {\r\n \"name\": \"opentelemetry/dotnet\",\r\n \"version\": \"1.6.0\"\r\n },\r\n \"source\": {\r\n \"port\": 35136,\r\n \"ip\": \"::ffff:192.168.50.132\"\r\n },\r\n \"error\": {\r\n \"exception\": [\r\n {\r\n \"handled\": true,\r\n \"message\": \"Exception (plain, .NET) without culprit\",\r\n \"type\": \"Grpc.Core.RpcException\"\r\n }\r\n ],\r\n \"culprit\": \"culprit msg\", // remove to test the N/A scenario\r\n \"id\": \"60674e9b23c8b06d83889e63ffe82c1c\", //change the id on every run\r\n \"stack_trace\": \"Grpc.Core.RpcException: Status(StatusCode=\\\"FailedPrecondition\\\", Detail=\\\"Can't access cart storage. System.ApplicationException: Wasn't able to connect to redis\\n at cartservice.cartstore.RedisCartStore.EnsureRedisConnected() in /usr/src/app/src/cartstore/RedisCartStore.cs:line 79\\n at cartservice.cartstore.RedisCartStore.EmptyCartAsync(String userId) in /usr/src/app/src/cartstore/RedisCartStore.cs:line 157\\\")\\n at cartservice.cartstore.RedisCartStore.EmptyCartAsync(String userId) in /usr/src/app/src/cartstore/RedisCartStore.cs:line 166\\n at cartservice.services.CartService.EmptyCart(EmptyCartRequest request, ServerCallContext context) in /usr/src/app/src/services/CartService.cs:line 65\",\r\n \"grouping_key\": \"6082c58afd7fa6838ebc27eef6e04829\"\r\n },\r\n \"message\": \"Exception (plain, .NET)\",\r\n \"url\": {\r\n \"path\": \"/oteldemo.CartService/EmptyCart\",\r\n \"original\": \"http://otel-demo-green-3-cartservice:8080/oteldemo.CartService/EmptyCart\",\r\n \"scheme\": \"http\",\r\n \"port\": 8080,\r\n \"domain\": \"otel-demo-green-3-cartservice\",\r\n \"full\": \"http://otel-demo-green-3-cartservice:8080/oteldemo.CartService/EmptyCart\"\r\n },\r\n \"labels\": {\r\n \"k8s_pod_ip\": \"192.168.62.174\",\r\n \"k8s_pod_start_time\": \"2024-12-17T02:55:54Z\",\r\n \"service_namespace\": \"opentelemetry-demo\",\r\n \"k8s_deployment_name\": \"otel-demo-green-3-cartservice\"\r\n },\r\n \"observer\": {\r\n \"hostname\": \"450fbf7ab701\",\r\n \"type\": \"apm-server\",\r\n \"version\": \"8.13.2\"\r\n },\r\n \"trace\": {\r\n \"id\": \"699f6d1039fc873191a1c983790db3a2\"\r\n },\r\n \"@timestamp\": \"2024-12-30T10:30:00.000Z\",\r\n \"data_stream\": {\r\n \"namespace\": \"default\",\r\n \"type\": \"logs\",\r\n \"dataset\": \"apm.error\"\r\n },\r\n \"service\": {\r\n \"node\": {\r\n \"name\": \"80cc20ecab1b57865d7aca0f66ab75a0c6ec2c51f7c53da04a8540215369c359\"\r\n },\r\n \"framework\": {\r\n \"name\": \"Microsoft.AspNetCore\"\r\n },\r\n \"name\": \"synth-node-0\",\r\n \"language\": {\r\n \"name\": \"dotnet\"\r\n }\r\n },\r\n \"host\": {\r\n \"hostname\": \"ip-192-168-55-136.ec2.internal\",\r\n \"name\": \"ip-192-168-55-136.ec2.internal\"\r\n },\r\n \"client\": {\r\n \"port\": 35136,\r\n \"ip\": \"::ffff:192.168.50.132\"\r\n },\r\n \"http\": {\r\n \"request\": {\r\n \"method\": \"POST\"\r\n },\r\n \"response\": {\r\n \"status_code\": 200\r\n },\r\n \"version\": \"2\"\r\n },\r\n \"event\": {\r\n \"agent_id_status\": \"missing\",\r\n \"ingested\": \"2024-12-18T18:04:22Z\"\r\n },\r\n \"transaction\": {\r\n \"id\": \"f39639f56a540e0e\",\r\n \"type\": \"request\",\r\n \"sampled\": true\r\n },\r\n \"user_agent\": {\r\n \"original\": \"grpc-go/1.59.0\",\r\n \"name\": \"Other\",\r\n \"device\": {\r\n \"name\": \"Other\"\r\n }\r\n },\r\n \"span\": {\r\n \"id\": \"f39639f56a540e0e\"\r\n },\r\n \"timestamp\": {\r\n \"us\": 1704892605838000\r\n }\r\n }\r\n```\r\n</details>","sha":"12608c3659a2aaea702c1d341bd0fab30f968056"}}]}] BACKPORT--> Co-authored-by: Carlos Crespo <[email protected]>
# Backport This will backport the following commits from `main` to `8.16`: - [[APM Fix missing error.culprit (#205242)](#205242) <!--- Backport version: 8.9.8 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Carlos Crespo","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-12-30T12:50:48Z","message":"[APM Fix missing error.culprit (#205242)\n\nfixes [#205240](https://github.com/elastic/kibana/issues/205240)\r\n\r\n## Summary\r\n\r\nFixes the issue where `error.culprit` is always missing.\r\n\r\n<img width=\"800\" alt=\"image\"\r\nsrc=\"https://github.com/user-attachments/assets/9dad6797-dd40-4206-b508-daf8d69d653d\"\r\n/>\r\n\r\n<img width=\"800\" alt=\"image\"\r\nsrc=\"https://github.com/user-attachments/assets/6a0ac25a-bc4b-4a26-b126-86eecbe7e769\"\r\n/>\r\n\r\n\r\n\r\n### How to test \r\n\r\n- Run `simple_trace` synthtrace scenario\r\n- Run the request below\r\n<details>\r\n<summary>POST logs-apm.error-default/_doc</summary>\r\n\r\n```\r\nPOST logs-apm.error-default/_doc\r\n{\r\n\r\n \"container\": {\r\n \"id\": \"80cc20ecab1b57865d7aca0f66ab75a0c6ec2c51f7c53da04a8540215369c359\"\r\n },\r\n \"kubernetes\": {\r\n \"node\": {\r\n \"name\": \"ip-192-168-55-136.ec2.internal\"\r\n },\r\n \"pod\": {\r\n \"uid\": \"9bbfe3dc-dace-4638-b8f5-aa6ccf435cbe\",\r\n \"name\": \"otel-demo-green-3-cartservice-5d5ccc86dd-gcxtr\"\r\n },\r\n \"namespace\": \"otel-demo-green-3\"\r\n },\r\n \"parent\": {\r\n \"id\": \"f39639f56a540e0e\"\r\n },\r\n \"agent\": {\r\n \"name\": \"opentelemetry/dotnet\",\r\n \"version\": \"1.6.0\"\r\n },\r\n \"source\": {\r\n \"port\": 35136,\r\n \"ip\": \"::ffff:192.168.50.132\"\r\n },\r\n \"error\": {\r\n \"exception\": [\r\n {\r\n \"handled\": true,\r\n \"message\": \"Exception (plain, .NET) without culprit\",\r\n \"type\": \"Grpc.Core.RpcException\"\r\n }\r\n ],\r\n \"culprit\": \"culprit msg\", // remove to test the N/A scenario\r\n \"id\": \"60674e9b23c8b06d83889e63ffe82c1c\", //change the id on every run\r\n \"stack_trace\": \"Grpc.Core.RpcException: Status(StatusCode=\\\"FailedPrecondition\\\", Detail=\\\"Can't access cart storage. System.ApplicationException: Wasn't able to connect to redis\\n at cartservice.cartstore.RedisCartStore.EnsureRedisConnected() in /usr/src/app/src/cartstore/RedisCartStore.cs:line 79\\n at cartservice.cartstore.RedisCartStore.EmptyCartAsync(String userId) in /usr/src/app/src/cartstore/RedisCartStore.cs:line 157\\\")\\n at cartservice.cartstore.RedisCartStore.EmptyCartAsync(String userId) in /usr/src/app/src/cartstore/RedisCartStore.cs:line 166\\n at cartservice.services.CartService.EmptyCart(EmptyCartRequest request, ServerCallContext context) in /usr/src/app/src/services/CartService.cs:line 65\",\r\n \"grouping_key\": \"6082c58afd7fa6838ebc27eef6e04829\"\r\n },\r\n \"message\": \"Exception (plain, .NET)\",\r\n \"url\": {\r\n \"path\": \"/oteldemo.CartService/EmptyCart\",\r\n \"original\": \"http://otel-demo-green-3-cartservice:8080/oteldemo.CartService/EmptyCart\",\r\n \"scheme\": \"http\",\r\n \"port\": 8080,\r\n \"domain\": \"otel-demo-green-3-cartservice\",\r\n \"full\": \"http://otel-demo-green-3-cartservice:8080/oteldemo.CartService/EmptyCart\"\r\n },\r\n \"labels\": {\r\n \"k8s_pod_ip\": \"192.168.62.174\",\r\n \"k8s_pod_start_time\": \"2024-12-17T02:55:54Z\",\r\n \"service_namespace\": \"opentelemetry-demo\",\r\n \"k8s_deployment_name\": \"otel-demo-green-3-cartservice\"\r\n },\r\n \"observer\": {\r\n \"hostname\": \"450fbf7ab701\",\r\n \"type\": \"apm-server\",\r\n \"version\": \"8.13.2\"\r\n },\r\n \"trace\": {\r\n \"id\": \"699f6d1039fc873191a1c983790db3a2\"\r\n },\r\n \"@timestamp\": \"2024-12-30T10:30:00.000Z\",\r\n \"data_stream\": {\r\n \"namespace\": \"default\",\r\n \"type\": \"logs\",\r\n \"dataset\": \"apm.error\"\r\n },\r\n \"service\": {\r\n \"node\": {\r\n \"name\": \"80cc20ecab1b57865d7aca0f66ab75a0c6ec2c51f7c53da04a8540215369c359\"\r\n },\r\n \"framework\": {\r\n \"name\": \"Microsoft.AspNetCore\"\r\n },\r\n \"name\": \"synth-node-0\",\r\n \"language\": {\r\n \"name\": \"dotnet\"\r\n }\r\n },\r\n \"host\": {\r\n \"hostname\": \"ip-192-168-55-136.ec2.internal\",\r\n \"name\": \"ip-192-168-55-136.ec2.internal\"\r\n },\r\n \"client\": {\r\n \"port\": 35136,\r\n \"ip\": \"::ffff:192.168.50.132\"\r\n },\r\n \"http\": {\r\n \"request\": {\r\n \"method\": \"POST\"\r\n },\r\n \"response\": {\r\n \"status_code\": 200\r\n },\r\n \"version\": \"2\"\r\n },\r\n \"event\": {\r\n \"agent_id_status\": \"missing\",\r\n \"ingested\": \"2024-12-18T18:04:22Z\"\r\n },\r\n \"transaction\": {\r\n \"id\": \"f39639f56a540e0e\",\r\n \"type\": \"request\",\r\n \"sampled\": true\r\n },\r\n \"user_agent\": {\r\n \"original\": \"grpc-go/1.59.0\",\r\n \"name\": \"Other\",\r\n \"device\": {\r\n \"name\": \"Other\"\r\n }\r\n },\r\n \"span\": {\r\n \"id\": \"f39639f56a540e0e\"\r\n },\r\n \"timestamp\": {\r\n \"us\": 1704892605838000\r\n }\r\n }\r\n```\r\n</details>","sha":"12608c3659a2aaea702c1d341bd0fab30f968056","branchLabelMapping":{"^v9.0.0$":"main","^v8.18.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","v9.0.0","backport:prev-major","Team:obs-ux-infra_services"],"number":205242,"url":"https://github.com/elastic/kibana/pull/205242","mergeCommit":{"message":"[APM Fix missing error.culprit (#205242)\n\nfixes [#205240](https://github.com/elastic/kibana/issues/205240)\r\n\r\n## Summary\r\n\r\nFixes the issue where `error.culprit` is always missing.\r\n\r\n<img width=\"800\" alt=\"image\"\r\nsrc=\"https://github.com/user-attachments/assets/9dad6797-dd40-4206-b508-daf8d69d653d\"\r\n/>\r\n\r\n<img width=\"800\" alt=\"image\"\r\nsrc=\"https://github.com/user-attachments/assets/6a0ac25a-bc4b-4a26-b126-86eecbe7e769\"\r\n/>\r\n\r\n\r\n\r\n### How to test \r\n\r\n- Run `simple_trace` synthtrace scenario\r\n- Run the request below\r\n<details>\r\n<summary>POST logs-apm.error-default/_doc</summary>\r\n\r\n```\r\nPOST logs-apm.error-default/_doc\r\n{\r\n\r\n \"container\": {\r\n \"id\": \"80cc20ecab1b57865d7aca0f66ab75a0c6ec2c51f7c53da04a8540215369c359\"\r\n },\r\n \"kubernetes\": {\r\n \"node\": {\r\n \"name\": \"ip-192-168-55-136.ec2.internal\"\r\n },\r\n \"pod\": {\r\n \"uid\": \"9bbfe3dc-dace-4638-b8f5-aa6ccf435cbe\",\r\n \"name\": \"otel-demo-green-3-cartservice-5d5ccc86dd-gcxtr\"\r\n },\r\n \"namespace\": \"otel-demo-green-3\"\r\n },\r\n \"parent\": {\r\n \"id\": \"f39639f56a540e0e\"\r\n },\r\n \"agent\": {\r\n \"name\": \"opentelemetry/dotnet\",\r\n \"version\": \"1.6.0\"\r\n },\r\n \"source\": {\r\n \"port\": 35136,\r\n \"ip\": \"::ffff:192.168.50.132\"\r\n },\r\n \"error\": {\r\n \"exception\": [\r\n {\r\n \"handled\": true,\r\n \"message\": \"Exception (plain, .NET) without culprit\",\r\n \"type\": \"Grpc.Core.RpcException\"\r\n }\r\n ],\r\n \"culprit\": \"culprit msg\", // remove to test the N/A scenario\r\n \"id\": \"60674e9b23c8b06d83889e63ffe82c1c\", //change the id on every run\r\n \"stack_trace\": \"Grpc.Core.RpcException: Status(StatusCode=\\\"FailedPrecondition\\\", Detail=\\\"Can't access cart storage. System.ApplicationException: Wasn't able to connect to redis\\n at cartservice.cartstore.RedisCartStore.EnsureRedisConnected() in /usr/src/app/src/cartstore/RedisCartStore.cs:line 79\\n at cartservice.cartstore.RedisCartStore.EmptyCartAsync(String userId) in /usr/src/app/src/cartstore/RedisCartStore.cs:line 157\\\")\\n at cartservice.cartstore.RedisCartStore.EmptyCartAsync(String userId) in /usr/src/app/src/cartstore/RedisCartStore.cs:line 166\\n at cartservice.services.CartService.EmptyCart(EmptyCartRequest request, ServerCallContext context) in /usr/src/app/src/services/CartService.cs:line 65\",\r\n \"grouping_key\": \"6082c58afd7fa6838ebc27eef6e04829\"\r\n },\r\n \"message\": \"Exception (plain, .NET)\",\r\n \"url\": {\r\n \"path\": \"/oteldemo.CartService/EmptyCart\",\r\n \"original\": \"http://otel-demo-green-3-cartservice:8080/oteldemo.CartService/EmptyCart\",\r\n \"scheme\": \"http\",\r\n \"port\": 8080,\r\n \"domain\": \"otel-demo-green-3-cartservice\",\r\n \"full\": \"http://otel-demo-green-3-cartservice:8080/oteldemo.CartService/EmptyCart\"\r\n },\r\n \"labels\": {\r\n \"k8s_pod_ip\": \"192.168.62.174\",\r\n \"k8s_pod_start_time\": \"2024-12-17T02:55:54Z\",\r\n \"service_namespace\": \"opentelemetry-demo\",\r\n \"k8s_deployment_name\": \"otel-demo-green-3-cartservice\"\r\n },\r\n \"observer\": {\r\n \"hostname\": \"450fbf7ab701\",\r\n \"type\": \"apm-server\",\r\n \"version\": \"8.13.2\"\r\n },\r\n \"trace\": {\r\n \"id\": \"699f6d1039fc873191a1c983790db3a2\"\r\n },\r\n \"@timestamp\": \"2024-12-30T10:30:00.000Z\",\r\n \"data_stream\": {\r\n \"namespace\": \"default\",\r\n \"type\": \"logs\",\r\n \"dataset\": \"apm.error\"\r\n },\r\n \"service\": {\r\n \"node\": {\r\n \"name\": \"80cc20ecab1b57865d7aca0f66ab75a0c6ec2c51f7c53da04a8540215369c359\"\r\n },\r\n \"framework\": {\r\n \"name\": \"Microsoft.AspNetCore\"\r\n },\r\n \"name\": \"synth-node-0\",\r\n \"language\": {\r\n \"name\": \"dotnet\"\r\n }\r\n },\r\n \"host\": {\r\n \"hostname\": \"ip-192-168-55-136.ec2.internal\",\r\n \"name\": \"ip-192-168-55-136.ec2.internal\"\r\n },\r\n \"client\": {\r\n \"port\": 35136,\r\n \"ip\": \"::ffff:192.168.50.132\"\r\n },\r\n \"http\": {\r\n \"request\": {\r\n \"method\": \"POST\"\r\n },\r\n \"response\": {\r\n \"status_code\": 200\r\n },\r\n \"version\": \"2\"\r\n },\r\n \"event\": {\r\n \"agent_id_status\": \"missing\",\r\n \"ingested\": \"2024-12-18T18:04:22Z\"\r\n },\r\n \"transaction\": {\r\n \"id\": \"f39639f56a540e0e\",\r\n \"type\": \"request\",\r\n \"sampled\": true\r\n },\r\n \"user_agent\": {\r\n \"original\": \"grpc-go/1.59.0\",\r\n \"name\": \"Other\",\r\n \"device\": {\r\n \"name\": \"Other\"\r\n }\r\n },\r\n \"span\": {\r\n \"id\": \"f39639f56a540e0e\"\r\n },\r\n \"timestamp\": {\r\n \"us\": 1704892605838000\r\n }\r\n }\r\n```\r\n</details>","sha":"12608c3659a2aaea702c1d341bd0fab30f968056"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","labelRegex":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/205242","number":205242,"mergeCommit":{"message":"[APM Fix missing error.culprit (#205242)\n\nfixes [#205240](https://github.com/elastic/kibana/issues/205240)\r\n\r\n## Summary\r\n\r\nFixes the issue where `error.culprit` is always missing.\r\n\r\n<img width=\"800\" alt=\"image\"\r\nsrc=\"https://github.com/user-attachments/assets/9dad6797-dd40-4206-b508-daf8d69d653d\"\r\n/>\r\n\r\n<img width=\"800\" alt=\"image\"\r\nsrc=\"https://github.com/user-attachments/assets/6a0ac25a-bc4b-4a26-b126-86eecbe7e769\"\r\n/>\r\n\r\n\r\n\r\n### How to test \r\n\r\n- Run `simple_trace` synthtrace scenario\r\n- Run the request below\r\n<details>\r\n<summary>POST logs-apm.error-default/_doc</summary>\r\n\r\n```\r\nPOST logs-apm.error-default/_doc\r\n{\r\n\r\n \"container\": {\r\n \"id\": \"80cc20ecab1b57865d7aca0f66ab75a0c6ec2c51f7c53da04a8540215369c359\"\r\n },\r\n \"kubernetes\": {\r\n \"node\": {\r\n \"name\": \"ip-192-168-55-136.ec2.internal\"\r\n },\r\n \"pod\": {\r\n \"uid\": \"9bbfe3dc-dace-4638-b8f5-aa6ccf435cbe\",\r\n \"name\": \"otel-demo-green-3-cartservice-5d5ccc86dd-gcxtr\"\r\n },\r\n \"namespace\": \"otel-demo-green-3\"\r\n },\r\n \"parent\": {\r\n \"id\": \"f39639f56a540e0e\"\r\n },\r\n \"agent\": {\r\n \"name\": \"opentelemetry/dotnet\",\r\n \"version\": \"1.6.0\"\r\n },\r\n \"source\": {\r\n \"port\": 35136,\r\n \"ip\": \"::ffff:192.168.50.132\"\r\n },\r\n \"error\": {\r\n \"exception\": [\r\n {\r\n \"handled\": true,\r\n \"message\": \"Exception (plain, .NET) without culprit\",\r\n \"type\": \"Grpc.Core.RpcException\"\r\n }\r\n ],\r\n \"culprit\": \"culprit msg\", // remove to test the N/A scenario\r\n \"id\": \"60674e9b23c8b06d83889e63ffe82c1c\", //change the id on every run\r\n \"stack_trace\": \"Grpc.Core.RpcException: Status(StatusCode=\\\"FailedPrecondition\\\", Detail=\\\"Can't access cart storage. System.ApplicationException: Wasn't able to connect to redis\\n at cartservice.cartstore.RedisCartStore.EnsureRedisConnected() in /usr/src/app/src/cartstore/RedisCartStore.cs:line 79\\n at cartservice.cartstore.RedisCartStore.EmptyCartAsync(String userId) in /usr/src/app/src/cartstore/RedisCartStore.cs:line 157\\\")\\n at cartservice.cartstore.RedisCartStore.EmptyCartAsync(String userId) in /usr/src/app/src/cartstore/RedisCartStore.cs:line 166\\n at cartservice.services.CartService.EmptyCart(EmptyCartRequest request, ServerCallContext context) in /usr/src/app/src/services/CartService.cs:line 65\",\r\n \"grouping_key\": \"6082c58afd7fa6838ebc27eef6e04829\"\r\n },\r\n \"message\": \"Exception (plain, .NET)\",\r\n \"url\": {\r\n \"path\": \"/oteldemo.CartService/EmptyCart\",\r\n \"original\": \"http://otel-demo-green-3-cartservice:8080/oteldemo.CartService/EmptyCart\",\r\n \"scheme\": \"http\",\r\n \"port\": 8080,\r\n \"domain\": \"otel-demo-green-3-cartservice\",\r\n \"full\": \"http://otel-demo-green-3-cartservice:8080/oteldemo.CartService/EmptyCart\"\r\n },\r\n \"labels\": {\r\n \"k8s_pod_ip\": \"192.168.62.174\",\r\n \"k8s_pod_start_time\": \"2024-12-17T02:55:54Z\",\r\n \"service_namespace\": \"opentelemetry-demo\",\r\n \"k8s_deployment_name\": \"otel-demo-green-3-cartservice\"\r\n },\r\n \"observer\": {\r\n \"hostname\": \"450fbf7ab701\",\r\n \"type\": \"apm-server\",\r\n \"version\": \"8.13.2\"\r\n },\r\n \"trace\": {\r\n \"id\": \"699f6d1039fc873191a1c983790db3a2\"\r\n },\r\n \"@timestamp\": \"2024-12-30T10:30:00.000Z\",\r\n \"data_stream\": {\r\n \"namespace\": \"default\",\r\n \"type\": \"logs\",\r\n \"dataset\": \"apm.error\"\r\n },\r\n \"service\": {\r\n \"node\": {\r\n \"name\": \"80cc20ecab1b57865d7aca0f66ab75a0c6ec2c51f7c53da04a8540215369c359\"\r\n },\r\n \"framework\": {\r\n \"name\": \"Microsoft.AspNetCore\"\r\n },\r\n \"name\": \"synth-node-0\",\r\n \"language\": {\r\n \"name\": \"dotnet\"\r\n }\r\n },\r\n \"host\": {\r\n \"hostname\": \"ip-192-168-55-136.ec2.internal\",\r\n \"name\": \"ip-192-168-55-136.ec2.internal\"\r\n },\r\n \"client\": {\r\n \"port\": 35136,\r\n \"ip\": \"::ffff:192.168.50.132\"\r\n },\r\n \"http\": {\r\n \"request\": {\r\n \"method\": \"POST\"\r\n },\r\n \"response\": {\r\n \"status_code\": 200\r\n },\r\n \"version\": \"2\"\r\n },\r\n \"event\": {\r\n \"agent_id_status\": \"missing\",\r\n \"ingested\": \"2024-12-18T18:04:22Z\"\r\n },\r\n \"transaction\": {\r\n \"id\": \"f39639f56a540e0e\",\r\n \"type\": \"request\",\r\n \"sampled\": true\r\n },\r\n \"user_agent\": {\r\n \"original\": \"grpc-go/1.59.0\",\r\n \"name\": \"Other\",\r\n \"device\": {\r\n \"name\": \"Other\"\r\n }\r\n },\r\n \"span\": {\r\n \"id\": \"f39639f56a540e0e\"\r\n },\r\n \"timestamp\": {\r\n \"us\": 1704892605838000\r\n }\r\n }\r\n```\r\n</details>","sha":"12608c3659a2aaea702c1d341bd0fab30f968056"}},{"url":"https://github.com/elastic/kibana/pull/205254","number":205254,"branch":"8.17","state":"OPEN"},{"url":"https://github.com/elastic/kibana/pull/205255","number":205255,"branch":"8.x","state":"OPEN"}]}] BACKPORT-->
fixes [elastic#205240](elastic#205240) ## Summary Fixes the issue where `error.culprit` is always missing. <img width="800" alt="image" src="https://github.com/user-attachments/assets/9dad6797-dd40-4206-b508-daf8d69d653d" /> <img width="800" alt="image" src="https://github.com/user-attachments/assets/6a0ac25a-bc4b-4a26-b126-86eecbe7e769" /> ### How to test - Run `simple_trace` synthtrace scenario - Run the request below <details> <summary>POST logs-apm.error-default/_doc</summary> ``` POST logs-apm.error-default/_doc { "container": { "id": "80cc20ecab1b57865d7aca0f66ab75a0c6ec2c51f7c53da04a8540215369c359" }, "kubernetes": { "node": { "name": "ip-192-168-55-136.ec2.internal" }, "pod": { "uid": "9bbfe3dc-dace-4638-b8f5-aa6ccf435cbe", "name": "otel-demo-green-3-cartservice-5d5ccc86dd-gcxtr" }, "namespace": "otel-demo-green-3" }, "parent": { "id": "f39639f56a540e0e" }, "agent": { "name": "opentelemetry/dotnet", "version": "1.6.0" }, "source": { "port": 35136, "ip": "::ffff:192.168.50.132" }, "error": { "exception": [ { "handled": true, "message": "Exception (plain, .NET) without culprit", "type": "Grpc.Core.RpcException" } ], "culprit": "culprit msg", // remove to test the N/A scenario "id": "60674e9b23c8b06d83889e63ffe82c1c", //change the id on every run "stack_trace": "Grpc.Core.RpcException: Status(StatusCode=\"FailedPrecondition\", Detail=\"Can't access cart storage. System.ApplicationException: Wasn't able to connect to redis\n at cartservice.cartstore.RedisCartStore.EnsureRedisConnected() in /usr/src/app/src/cartstore/RedisCartStore.cs:line 79\n at cartservice.cartstore.RedisCartStore.EmptyCartAsync(String userId) in /usr/src/app/src/cartstore/RedisCartStore.cs:line 157\")\n at cartservice.cartstore.RedisCartStore.EmptyCartAsync(String userId) in /usr/src/app/src/cartstore/RedisCartStore.cs:line 166\n at cartservice.services.CartService.EmptyCart(EmptyCartRequest request, ServerCallContext context) in /usr/src/app/src/services/CartService.cs:line 65", "grouping_key": "6082c58afd7fa6838ebc27eef6e04829" }, "message": "Exception (plain, .NET)", "url": { "path": "/oteldemo.CartService/EmptyCart", "original": "http://otel-demo-green-3-cartservice:8080/oteldemo.CartService/EmptyCart", "scheme": "http", "port": 8080, "domain": "otel-demo-green-3-cartservice", "full": "http://otel-demo-green-3-cartservice:8080/oteldemo.CartService/EmptyCart" }, "labels": { "k8s_pod_ip": "192.168.62.174", "k8s_pod_start_time": "2024-12-17T02:55:54Z", "service_namespace": "opentelemetry-demo", "k8s_deployment_name": "otel-demo-green-3-cartservice" }, "observer": { "hostname": "450fbf7ab701", "type": "apm-server", "version": "8.13.2" }, "trace": { "id": "699f6d1039fc873191a1c983790db3a2" }, "@timestamp": "2024-12-30T10:30:00.000Z", "data_stream": { "namespace": "default", "type": "logs", "dataset": "apm.error" }, "service": { "node": { "name": "80cc20ecab1b57865d7aca0f66ab75a0c6ec2c51f7c53da04a8540215369c359" }, "framework": { "name": "Microsoft.AspNetCore" }, "name": "synth-node-0", "language": { "name": "dotnet" } }, "host": { "hostname": "ip-192-168-55-136.ec2.internal", "name": "ip-192-168-55-136.ec2.internal" }, "client": { "port": 35136, "ip": "::ffff:192.168.50.132" }, "http": { "request": { "method": "POST" }, "response": { "status_code": 200 }, "version": "2" }, "event": { "agent_id_status": "missing", "ingested": "2024-12-18T18:04:22Z" }, "transaction": { "id": "f39639f56a540e0e", "type": "request", "sampled": true }, "user_agent": { "original": "grpc-go/1.59.0", "name": "Other", "device": { "name": "Other" } }, "span": { "id": "f39639f56a540e0e" }, "timestamp": { "us": 1704892605838000 } } ``` </details>
fixes [elastic#205240](elastic#205240) ## Summary Fixes the issue where `error.culprit` is always missing. <img width="800" alt="image" src="https://github.com/user-attachments/assets/9dad6797-dd40-4206-b508-daf8d69d653d" /> <img width="800" alt="image" src="https://github.com/user-attachments/assets/6a0ac25a-bc4b-4a26-b126-86eecbe7e769" /> ### How to test - Run `simple_trace` synthtrace scenario - Run the request below <details> <summary>POST logs-apm.error-default/_doc</summary> ``` POST logs-apm.error-default/_doc { "container": { "id": "80cc20ecab1b57865d7aca0f66ab75a0c6ec2c51f7c53da04a8540215369c359" }, "kubernetes": { "node": { "name": "ip-192-168-55-136.ec2.internal" }, "pod": { "uid": "9bbfe3dc-dace-4638-b8f5-aa6ccf435cbe", "name": "otel-demo-green-3-cartservice-5d5ccc86dd-gcxtr" }, "namespace": "otel-demo-green-3" }, "parent": { "id": "f39639f56a540e0e" }, "agent": { "name": "opentelemetry/dotnet", "version": "1.6.0" }, "source": { "port": 35136, "ip": "::ffff:192.168.50.132" }, "error": { "exception": [ { "handled": true, "message": "Exception (plain, .NET) without culprit", "type": "Grpc.Core.RpcException" } ], "culprit": "culprit msg", // remove to test the N/A scenario "id": "60674e9b23c8b06d83889e63ffe82c1c", //change the id on every run "stack_trace": "Grpc.Core.RpcException: Status(StatusCode=\"FailedPrecondition\", Detail=\"Can't access cart storage. System.ApplicationException: Wasn't able to connect to redis\n at cartservice.cartstore.RedisCartStore.EnsureRedisConnected() in /usr/src/app/src/cartstore/RedisCartStore.cs:line 79\n at cartservice.cartstore.RedisCartStore.EmptyCartAsync(String userId) in /usr/src/app/src/cartstore/RedisCartStore.cs:line 157\")\n at cartservice.cartstore.RedisCartStore.EmptyCartAsync(String userId) in /usr/src/app/src/cartstore/RedisCartStore.cs:line 166\n at cartservice.services.CartService.EmptyCart(EmptyCartRequest request, ServerCallContext context) in /usr/src/app/src/services/CartService.cs:line 65", "grouping_key": "6082c58afd7fa6838ebc27eef6e04829" }, "message": "Exception (plain, .NET)", "url": { "path": "/oteldemo.CartService/EmptyCart", "original": "http://otel-demo-green-3-cartservice:8080/oteldemo.CartService/EmptyCart", "scheme": "http", "port": 8080, "domain": "otel-demo-green-3-cartservice", "full": "http://otel-demo-green-3-cartservice:8080/oteldemo.CartService/EmptyCart" }, "labels": { "k8s_pod_ip": "192.168.62.174", "k8s_pod_start_time": "2024-12-17T02:55:54Z", "service_namespace": "opentelemetry-demo", "k8s_deployment_name": "otel-demo-green-3-cartservice" }, "observer": { "hostname": "450fbf7ab701", "type": "apm-server", "version": "8.13.2" }, "trace": { "id": "699f6d1039fc873191a1c983790db3a2" }, "@timestamp": "2024-12-30T10:30:00.000Z", "data_stream": { "namespace": "default", "type": "logs", "dataset": "apm.error" }, "service": { "node": { "name": "80cc20ecab1b57865d7aca0f66ab75a0c6ec2c51f7c53da04a8540215369c359" }, "framework": { "name": "Microsoft.AspNetCore" }, "name": "synth-node-0", "language": { "name": "dotnet" } }, "host": { "hostname": "ip-192-168-55-136.ec2.internal", "name": "ip-192-168-55-136.ec2.internal" }, "client": { "port": 35136, "ip": "::ffff:192.168.50.132" }, "http": { "request": { "method": "POST" }, "response": { "status_code": 200 }, "version": "2" }, "event": { "agent_id_status": "missing", "ingested": "2024-12-18T18:04:22Z" }, "transaction": { "id": "f39639f56a540e0e", "type": "request", "sampled": true }, "user_agent": { "original": "grpc-go/1.59.0", "name": "Other", "device": { "name": "Other" } }, "span": { "id": "f39639f56a540e0e" }, "timestamp": { "us": 1704892605838000 } } ``` </details>
fixes [elastic#205240](elastic#205240) ## Summary Fixes the issue where `error.culprit` is always missing. <img width="800" alt="image" src="https://github.com/user-attachments/assets/9dad6797-dd40-4206-b508-daf8d69d653d" /> <img width="800" alt="image" src="https://github.com/user-attachments/assets/6a0ac25a-bc4b-4a26-b126-86eecbe7e769" /> ### How to test - Run `simple_trace` synthtrace scenario - Run the request below <details> <summary>POST logs-apm.error-default/_doc</summary> ``` POST logs-apm.error-default/_doc { "container": { "id": "80cc20ecab1b57865d7aca0f66ab75a0c6ec2c51f7c53da04a8540215369c359" }, "kubernetes": { "node": { "name": "ip-192-168-55-136.ec2.internal" }, "pod": { "uid": "9bbfe3dc-dace-4638-b8f5-aa6ccf435cbe", "name": "otel-demo-green-3-cartservice-5d5ccc86dd-gcxtr" }, "namespace": "otel-demo-green-3" }, "parent": { "id": "f39639f56a540e0e" }, "agent": { "name": "opentelemetry/dotnet", "version": "1.6.0" }, "source": { "port": 35136, "ip": "::ffff:192.168.50.132" }, "error": { "exception": [ { "handled": true, "message": "Exception (plain, .NET) without culprit", "type": "Grpc.Core.RpcException" } ], "culprit": "culprit msg", // remove to test the N/A scenario "id": "60674e9b23c8b06d83889e63ffe82c1c", //change the id on every run "stack_trace": "Grpc.Core.RpcException: Status(StatusCode=\"FailedPrecondition\", Detail=\"Can't access cart storage. System.ApplicationException: Wasn't able to connect to redis\n at cartservice.cartstore.RedisCartStore.EnsureRedisConnected() in /usr/src/app/src/cartstore/RedisCartStore.cs:line 79\n at cartservice.cartstore.RedisCartStore.EmptyCartAsync(String userId) in /usr/src/app/src/cartstore/RedisCartStore.cs:line 157\")\n at cartservice.cartstore.RedisCartStore.EmptyCartAsync(String userId) in /usr/src/app/src/cartstore/RedisCartStore.cs:line 166\n at cartservice.services.CartService.EmptyCart(EmptyCartRequest request, ServerCallContext context) in /usr/src/app/src/services/CartService.cs:line 65", "grouping_key": "6082c58afd7fa6838ebc27eef6e04829" }, "message": "Exception (plain, .NET)", "url": { "path": "/oteldemo.CartService/EmptyCart", "original": "http://otel-demo-green-3-cartservice:8080/oteldemo.CartService/EmptyCart", "scheme": "http", "port": 8080, "domain": "otel-demo-green-3-cartservice", "full": "http://otel-demo-green-3-cartservice:8080/oteldemo.CartService/EmptyCart" }, "labels": { "k8s_pod_ip": "192.168.62.174", "k8s_pod_start_time": "2024-12-17T02:55:54Z", "service_namespace": "opentelemetry-demo", "k8s_deployment_name": "otel-demo-green-3-cartservice" }, "observer": { "hostname": "450fbf7ab701", "type": "apm-server", "version": "8.13.2" }, "trace": { "id": "699f6d1039fc873191a1c983790db3a2" }, "@timestamp": "2024-12-30T10:30:00.000Z", "data_stream": { "namespace": "default", "type": "logs", "dataset": "apm.error" }, "service": { "node": { "name": "80cc20ecab1b57865d7aca0f66ab75a0c6ec2c51f7c53da04a8540215369c359" }, "framework": { "name": "Microsoft.AspNetCore" }, "name": "synth-node-0", "language": { "name": "dotnet" } }, "host": { "hostname": "ip-192-168-55-136.ec2.internal", "name": "ip-192-168-55-136.ec2.internal" }, "client": { "port": 35136, "ip": "::ffff:192.168.50.132" }, "http": { "request": { "method": "POST" }, "response": { "status_code": 200 }, "version": "2" }, "event": { "agent_id_status": "missing", "ingested": "2024-12-18T18:04:22Z" }, "transaction": { "id": "f39639f56a540e0e", "type": "request", "sampled": true }, "user_agent": { "original": "grpc-go/1.59.0", "name": "Other", "device": { "name": "Other" } }, "span": { "id": "f39639f56a540e0e" }, "timestamp": { "us": 1704892605838000 } } ``` </details>
fixes #205240
Summary
Fixes the issue where
error.culprit
is always missing.How to test
simple_trace
synthtrace scenarioPOST logs-apm.error-default/_doc