[EEM] Add built in definitions for core Kubernetes entities #196916
Merged
+802
−4
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
cauemarcondes
added a commit
that referenced
this pull request
Oct 24, 2024
closes #196155 Blocked by #196916 (K8s entities alias patterns don't exist yet.) ``` node scripts/synthtrace many_entities.ts --clean --live node scripts/synthtrace k8s_entities.ts --clean --live ``` https://github.com/user-attachments/assets/5861ebc7-8386-4a4b-a68b-50adc5244d43
klacabane
reviewed
Oct 25, 2024
| export { BUILT_IN_ID_PREFIX } from './constants'; | ||
|
|
||
| export const builtInDefinitions: EntityDefinition[] = [ | ||
| builtInServicesFromEcsEntityDefinition, | ||
| builtInHostsFromEcsEntityDefinition, | ||
| builtInContainersFromEcsEntityDefinition, | ||
| ...Object.values(kubernetes), |
There was a problem hiding this comment.
probably a good time to make the enablement call async with all these new definitions. The enablement endpoint will also install every definitions in parallel, could that be problematic with the current amount of definitions ? we can also consider throttling the installations on our side otherwise
|
Hey @crespocarlos @miltonhultgren,
Let's not have a separate definition for Kubernetes containers. We can use the existing one (which is keyed on cc @mlunadia If the K8s containers observed in OTel aren't already mapped to ECS - we can ignore them completely in this issue.
Let's just do it for ECS for now and if we can open a separate issue if we need one for OTel (there is no definition for OTel schema k8s services right now.
I don't think so? If we have a separate definition per schema then each definition per schema shouldn't care about to other? |
This was referenced Oct 29, 2024
cauemarcondes
added a commit
to cauemarcondes/kibana
that referenced
this pull request
Nov 1, 2024
closes elastic#196155 Blocked by elastic#196916 (K8s entities alias patterns don't exist yet.) ``` node scripts/synthtrace many_entities.ts --clean --live node scripts/synthtrace k8s_entities.ts --clean --live ``` https://github.com/user-attachments/assets/5861ebc7-8386-4a4b-a68b-50adc5244d43 (cherry picked from commit c6f4178) # Conflicts: # x-pack/plugins/observability_solution/inventory/public/components/search_bar/index.tsx
cauemarcondes
added a commit
to cauemarcondes/kibana
that referenced
this pull request
Nov 1, 2024
closes elastic#196155 Blocked by elastic#196916 (K8s entities alias patterns don't exist yet.) ``` node scripts/synthtrace many_entities.ts --clean --live node scripts/synthtrace k8s_entities.ts --clean --live ``` https://github.com/user-attachments/assets/5861ebc7-8386-4a4b-a68b-50adc5244d43 (cherry picked from commit c6f4178) # Conflicts: # x-pack/plugins/observability_solution/inventory/public/components/search_bar/index.tsx
cauemarcondes
added a commit
that referenced
this pull request
Nov 1, 2024
# Backport This will backport the following commits from `main` to `8.x`: - [Inventory] List k8s entities in the grid (#197292) (c6f4178) <!--- Backport version: 8.9.8 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Cauê Marcondes","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-10-24T13:35:19Z","message":"[Inventory] List k8s entities in the grid (#197292)\n\ncloses https://github.com/elastic/kibana/issues/196155\r\nBlocked by #196916 (K8s entities\r\nalias patterns don't exist yet.)\r\n\r\n```\r\nnode scripts/synthtrace many_entities.ts --clean --live\r\nnode scripts/synthtrace k8s_entities.ts --clean --live\r\n```\r\n\r\n\r\nhttps://github.com/user-attachments/assets/5861ebc7-8386-4a4b-a68b-50adc5244d43","sha":"c6f41783bf853f2828787cc4c48a96511fec2eec"},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[]}] BACKPORT-->
crespocarlos
added a commit
that referenced
this pull request
Nov 4, 2024
closes [#196142](#196142) ## Summary Links kubernetes ECS entities to their corresponding dashboards > [!IMPORTANT] > ECS `replicaset` doesn't have a dedicated dashboard. `container` will be handled in a separate ticket > Semconv won't link to any dashboard/page <img width="800" alt="image" src="https://github.com/user-attachments/assets/711dbd28-f0ef-4af0-a658-afe7f1595697">  ### How to test - While #196916 is not merged, change `ENTITIES_LATEST_ALIAS` constant to `'.entities.v1.latest*'` - Start a local kibana and es instances - Run ` node scripts/synthtrace k8s_entities.ts --live --clean ` - Run `PUT kbn:/internal/entities/managed/enablement` on the devtools - Install the kubernetes integration package to have the dashboards installed. - Navigate to `Inventory` and click through the k8s entities --------- Co-authored-by: kibanamachine <[email protected]> Co-authored-by: Elastic Machine <[email protected]>
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Nov 4, 2024
) closes [elastic#196142](elastic#196142) ## Summary Links kubernetes ECS entities to their corresponding dashboards > [!IMPORTANT] > ECS `replicaset` doesn't have a dedicated dashboard. `container` will be handled in a separate ticket > Semconv won't link to any dashboard/page <img width="800" alt="image" src="https://github.com/user-attachments/assets/711dbd28-f0ef-4af0-a658-afe7f1595697">  ### How to test - While elastic#196916 is not merged, change `ENTITIES_LATEST_ALIAS` constant to `'.entities.v1.latest*'` - Start a local kibana and es instances - Run ` node scripts/synthtrace k8s_entities.ts --live --clean ` - Run `PUT kbn:/internal/entities/managed/enablement` on the devtools - Install the kubernetes integration package to have the dashboards installed. - Navigate to `Inventory` and click through the k8s entities --------- Co-authored-by: kibanamachine <[email protected]> Co-authored-by: Elastic Machine <[email protected]> (cherry picked from commit 8145cb7)
kibanamachine
added a commit
that referenced
this pull request
Nov 4, 2024
…) (#198815) # Backport This will backport the following commits from `main` to `8.x`: - [[ECO][Inventory] Redirect ECS k8s entities to dashboards (#197222)](#197222) <!--- Backport version: 9.4.3 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Carlos Crespo","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-11-04T16:01:05Z","message":"[ECO][Inventory] Redirect ECS k8s entities to dashboards (#197222)\n\ncloses [#196142](https://github.com/elastic/kibana/issues/196142)\r\n\r\n## Summary\r\n\r\nLinks kubernetes ECS entities to their corresponding dashboards\r\n\r\n> [!IMPORTANT]\r\n> ECS `replicaset` doesn't have a dedicated dashboard. `container` will\r\nbe handled in a separate ticket\r\n> Semconv won't link to any dashboard/page\r\n\r\n<img width=\"800\" alt=\"image\"\r\nsrc=\"https://github.com/user-attachments/assets/711dbd28-f0ef-4af0-a658-afe7f1595697\">\r\n\r\n\r\n\r\n\r\n\r\n### How to test\r\n- While #196916 is not merged,\r\nchange `ENTITIES_LATEST_ALIAS` constant to `'.entities.v1.latest*'`\r\n- Start a local kibana and es instances \r\n- Run ` node scripts/synthtrace k8s_entities.ts --live --clean `\r\n- Run `PUT kbn:/internal/entities/managed/enablement` on the devtools\r\n- Install the kubernetes integration package to have the dashboards\r\ninstalled.\r\n- Navigate to `Inventory` and click through the k8s entities\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine <[email protected]>\r\nCo-authored-by: Elastic Machine <[email protected]>","sha":"8145cb7c6f483c3a8aa561b492fa098e3ce52027","branchLabelMapping":{"^v9.0.0$":"main","^v8.17.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","v9.0.0","backport:prev-minor","ci:project-deploy-observability","Team:obs-ux-infra_services"],"title":"[ECO][Inventory] Redirect ECS k8s entities to dashboards","number":197222,"url":"https://github.com/elastic/kibana/pull/197222","mergeCommit":{"message":"[ECO][Inventory] Redirect ECS k8s entities to dashboards (#197222)\n\ncloses [#196142](https://github.com/elastic/kibana/issues/196142)\r\n\r\n## Summary\r\n\r\nLinks kubernetes ECS entities to their corresponding dashboards\r\n\r\n> [!IMPORTANT]\r\n> ECS `replicaset` doesn't have a dedicated dashboard. `container` will\r\nbe handled in a separate ticket\r\n> Semconv won't link to any dashboard/page\r\n\r\n<img width=\"800\" alt=\"image\"\r\nsrc=\"https://github.com/user-attachments/assets/711dbd28-f0ef-4af0-a658-afe7f1595697\">\r\n\r\n\r\n\r\n\r\n\r\n### How to test\r\n- While #196916 is not merged,\r\nchange `ENTITIES_LATEST_ALIAS` constant to `'.entities.v1.latest*'`\r\n- Start a local kibana and es instances \r\n- Run ` node scripts/synthtrace k8s_entities.ts --live --clean `\r\n- Run `PUT kbn:/internal/entities/managed/enablement` on the devtools\r\n- Install the kubernetes integration package to have the dashboards\r\ninstalled.\r\n- Navigate to `Inventory` and click through the k8s entities\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine <[email protected]>\r\nCo-authored-by: Elastic Machine <[email protected]>","sha":"8145cb7c6f483c3a8aa561b492fa098e3ce52027"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/197222","number":197222,"mergeCommit":{"message":"[ECO][Inventory] Redirect ECS k8s entities to dashboards (#197222)\n\ncloses [#196142](https://github.com/elastic/kibana/issues/196142)\r\n\r\n## Summary\r\n\r\nLinks kubernetes ECS entities to their corresponding dashboards\r\n\r\n> [!IMPORTANT]\r\n> ECS `replicaset` doesn't have a dedicated dashboard. `container` will\r\nbe handled in a separate ticket\r\n> Semconv won't link to any dashboard/page\r\n\r\n<img width=\"800\" alt=\"image\"\r\nsrc=\"https://github.com/user-attachments/assets/711dbd28-f0ef-4af0-a658-afe7f1595697\">\r\n\r\n\r\n\r\n\r\n\r\n### How to test\r\n- While #196916 is not merged,\r\nchange `ENTITIES_LATEST_ALIAS` constant to `'.entities.v1.latest*'`\r\n- Start a local kibana and es instances \r\n- Run ` node scripts/synthtrace k8s_entities.ts --live --clean `\r\n- Run `PUT kbn:/internal/entities/managed/enablement` on the devtools\r\n- Install the kubernetes integration package to have the dashboards\r\ninstalled.\r\n- Navigate to `Inventory` and click through the k8s entities\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine <[email protected]>\r\nCo-authored-by: Elastic Machine <[email protected]>","sha":"8145cb7c6f483c3a8aa561b492fa098e3ce52027"}}]}] BACKPORT--> Co-authored-by: Carlos Crespo <[email protected]>
simianhacker
added
backport:skip
|
@elasticmachine merge upstream |
💚 Build Succeeded
Metrics [docs]
History
|
klacabane
approved these changes
Nov 19, 2024
This was referenced Nov 21, 2024
This was referenced Nov 25, 2024
💚 All backports created successfully
Note: Successful backport PRs will be merged automatically after passing CI. Questions ?Please refer to the Backport tool documentation |
simianhacker
pushed a commit
to simianhacker/kibana
that referenced
this pull request
Nov 25, 2024
…196916) ## 🍒 Summary This PR adds the OTEL and ECS entity definition for Kubernetes. This covers the following datasets: - Cluster - Service (ECS Only) - Pod - ReplicaSet - Deployment - Statefulset - DaemonSet - Job - CronJob - Node This PR does not include Container per @roshan-elastic ### ✅ TODO - [X] Use correct index pattern for SemConv data (`metrics-k8sclusterreceiver.otel-default`, `metrics-kubeletstatsreceiver.otel-default`) Use global IDs instead of local IDs - [X] Add minimal list of labels to track beyond what was already added (wildcards are not supported, example `container.image.name` for containers to allow to find all "redis" containers) - [ ] Test with ECS data, SemConv data and mixed data (to check if we get duplicates, with the container definition for example). ### 🐴 Follow up EEM features elastic/elastic-entity-model#170 (Add dedicated aggregation for display name and use that instead to provide a better label than the global ID) elastic/elastic-entity-model#193 (Add entity type display label to allow UI to not hard code a user friendly label) --------- Co-authored-by: Chris Cowan <[email protected]> Co-authored-by: Elastic Machine <[email protected]> (cherry picked from commit 080d0ff)
simianhacker
added a commit
that referenced
this pull request
Nov 26, 2024
…96916) (#201660) # Backport This will backport the following commits from `main` to `8.x`: - [[EEM] Add built in definitions for core Kubernetes entities (#196916)](#196916) <!--- Backport version: 8.9.8 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Milton Hultgren","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-11-19T21:23:52Z","message":"[EEM] Add built in definitions for core Kubernetes entities (#196916)\n\n## 🍒 Summary\r\n\r\nThis PR adds the OTEL and ECS entity definition for Kubernetes. This\r\ncovers the following datasets:\r\n- Cluster\r\n- Service (ECS Only)\r\n- Pod\r\n- ReplicaSet\r\n- Deployment\r\n- Statefulset\r\n- DaemonSet\r\n- Job\r\n- CronJob\r\n- Node\r\n\r\nThis PR does not include Container per @roshan-elastic \r\n\r\n### ✅ TODO\r\n- [X] Use correct index pattern for SemConv data\r\n(`metrics-k8sclusterreceiver.otel-default`,\r\n`metrics-kubeletstatsreceiver.otel-default`)\r\nUse global IDs instead of local IDs\r\n- [X] Add minimal list of labels to track beyond what was already added\r\n(wildcards are not supported, example `container.image.name` for\r\ncontainers to allow to find all \"redis\" containers)\r\n- [ ] Test with ECS data, SemConv data and mixed data (to check if we\r\nget duplicates, with the container definition for example).\r\n\r\n### 🐴 Follow up EEM features \r\nhttps://github.com/elastic/elastic-entity-model/issues/170 (Add\r\ndedicated aggregation for display name and use that instead to provide a\r\nbetter label than the global ID)\r\nhttps://github.com/elastic/elastic-entity-model/issues/193 (Add entity\r\ntype display label to allow UI to not hard code a user friendly label)\r\n\r\n---------\r\n\r\nCo-authored-by: Chris Cowan <[email protected]>\r\nCo-authored-by: Elastic Machine <[email protected]>","sha":"080d0ff97f00bf564dedfd8fd37cdac0370e1349","branchLabelMapping":{"^v9.0.0$":"main","^v8.17.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["backport:skip","v9.0.0","release_note:feature","v8.17.0"],"number":196916,"url":"https://github.com/elastic/kibana/pull/196916","mergeCommit":{"message":"[EEM] Add built in definitions for core Kubernetes entities (#196916)\n\n## 🍒 Summary\r\n\r\nThis PR adds the OTEL and ECS entity definition for Kubernetes. This\r\ncovers the following datasets:\r\n- Cluster\r\n- Service (ECS Only)\r\n- Pod\r\n- ReplicaSet\r\n- Deployment\r\n- Statefulset\r\n- DaemonSet\r\n- Job\r\n- CronJob\r\n- Node\r\n\r\nThis PR does not include Container per @roshan-elastic \r\n\r\n### ✅ TODO\r\n- [X] Use correct index pattern for SemConv data\r\n(`metrics-k8sclusterreceiver.otel-default`,\r\n`metrics-kubeletstatsreceiver.otel-default`)\r\nUse global IDs instead of local IDs\r\n- [X] Add minimal list of labels to track beyond what was already added\r\n(wildcards are not supported, example `container.image.name` for\r\ncontainers to allow to find all \"redis\" containers)\r\n- [ ] Test with ECS data, SemConv data and mixed data (to check if we\r\nget duplicates, with the container definition for example).\r\n\r\n### 🐴 Follow up EEM features \r\nhttps://github.com/elastic/elastic-entity-model/issues/170 (Add\r\ndedicated aggregation for display name and use that instead to provide a\r\nbetter label than the global ID)\r\nhttps://github.com/elastic/elastic-entity-model/issues/193 (Add entity\r\ntype display label to allow UI to not hard code a user friendly label)\r\n\r\n---------\r\n\r\nCo-authored-by: Chris Cowan <[email protected]>\r\nCo-authored-by: Elastic Machine <[email protected]>","sha":"080d0ff97f00bf564dedfd8fd37cdac0370e1349"}},"sourceBranch":"main","suggestedTargetBranches":["8.17"],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","labelRegex":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/196916","number":196916,"mergeCommit":{"message":"[EEM] Add built in definitions for core Kubernetes entities (#196916)\n\n## 🍒 Summary\r\n\r\nThis PR adds the OTEL and ECS entity definition for Kubernetes. This\r\ncovers the following datasets:\r\n- Cluster\r\n- Service (ECS Only)\r\n- Pod\r\n- ReplicaSet\r\n- Deployment\r\n- Statefulset\r\n- DaemonSet\r\n- Job\r\n- CronJob\r\n- Node\r\n\r\nThis PR does not include Container per @roshan-elastic \r\n\r\n### ✅ TODO\r\n- [X] Use correct index pattern for SemConv data\r\n(`metrics-k8sclusterreceiver.otel-default`,\r\n`metrics-kubeletstatsreceiver.otel-default`)\r\nUse global IDs instead of local IDs\r\n- [X] Add minimal list of labels to track beyond what was already added\r\n(wildcards are not supported, example `container.image.name` for\r\ncontainers to allow to find all \"redis\" containers)\r\n- [ ] Test with ECS data, SemConv data and mixed data (to check if we\r\nget duplicates, with the container definition for example).\r\n\r\n### 🐴 Follow up EEM features \r\nhttps://github.com/elastic/elastic-entity-model/issues/170 (Add\r\ndedicated aggregation for display name and use that instead to provide a\r\nbetter label than the global ID)\r\nhttps://github.com/elastic/elastic-entity-model/issues/193 (Add entity\r\ntype display label to allow UI to not hard code a user friendly label)\r\n\r\n---------\r\n\r\nCo-authored-by: Chris Cowan <[email protected]>\r\nCo-authored-by: Elastic Machine <[email protected]>","sha":"080d0ff97f00bf564dedfd8fd37cdac0370e1349"}},{"branch":"8.x","label":"v8.17.0","labelRegex":"^v8.17.0$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT--> Co-authored-by: Milton Hultgren <[email protected]>
simianhacker
added a commit
that referenced
this pull request
Nov 26, 2024
…196916) (#201661) # Backport This will backport the following commits from `main` to `8.17`: - [[EEM] Add built in definitions for core Kubernetes entities (#196916)](#196916) <!--- Backport version: 8.9.8 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Milton Hultgren","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-11-19T21:23:52Z","message":"[EEM] Add built in definitions for core Kubernetes entities (#196916)\n\n## 🍒 Summary\r\n\r\nThis PR adds the OTEL and ECS entity definition for Kubernetes. This\r\ncovers the following datasets:\r\n- Cluster\r\n- Service (ECS Only)\r\n- Pod\r\n- ReplicaSet\r\n- Deployment\r\n- Statefulset\r\n- DaemonSet\r\n- Job\r\n- CronJob\r\n- Node\r\n\r\nThis PR does not include Container per @roshan-elastic \r\n\r\n### ✅ TODO\r\n- [X] Use correct index pattern for SemConv data\r\n(`metrics-k8sclusterreceiver.otel-default`,\r\n`metrics-kubeletstatsreceiver.otel-default`)\r\nUse global IDs instead of local IDs\r\n- [X] Add minimal list of labels to track beyond what was already added\r\n(wildcards are not supported, example `container.image.name` for\r\ncontainers to allow to find all \"redis\" containers)\r\n- [ ] Test with ECS data, SemConv data and mixed data (to check if we\r\nget duplicates, with the container definition for example).\r\n\r\n### 🐴 Follow up EEM features \r\nhttps://github.com/elastic/elastic-entity-model/issues/170 (Add\r\ndedicated aggregation for display name and use that instead to provide a\r\nbetter label than the global ID)\r\nhttps://github.com/elastic/elastic-entity-model/issues/193 (Add entity\r\ntype display label to allow UI to not hard code a user friendly label)\r\n\r\n---------\r\n\r\nCo-authored-by: Chris Cowan <[email protected]>\r\nCo-authored-by: Elastic Machine <[email protected]>","sha":"080d0ff97f00bf564dedfd8fd37cdac0370e1349","branchLabelMapping":{"^v9.0.0$":"main","^v8.17.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["backport:skip","v9.0.0","release_note:feature","v8.17.0"],"number":196916,"url":"https://github.com/elastic/kibana/pull/196916","mergeCommit":{"message":"[EEM] Add built in definitions for core Kubernetes entities (#196916)\n\n## 🍒 Summary\r\n\r\nThis PR adds the OTEL and ECS entity definition for Kubernetes. This\r\ncovers the following datasets:\r\n- Cluster\r\n- Service (ECS Only)\r\n- Pod\r\n- ReplicaSet\r\n- Deployment\r\n- Statefulset\r\n- DaemonSet\r\n- Job\r\n- CronJob\r\n- Node\r\n\r\nThis PR does not include Container per @roshan-elastic \r\n\r\n### ✅ TODO\r\n- [X] Use correct index pattern for SemConv data\r\n(`metrics-k8sclusterreceiver.otel-default`,\r\n`metrics-kubeletstatsreceiver.otel-default`)\r\nUse global IDs instead of local IDs\r\n- [X] Add minimal list of labels to track beyond what was already added\r\n(wildcards are not supported, example `container.image.name` for\r\ncontainers to allow to find all \"redis\" containers)\r\n- [ ] Test with ECS data, SemConv data and mixed data (to check if we\r\nget duplicates, with the container definition for example).\r\n\r\n### 🐴 Follow up EEM features \r\nhttps://github.com/elastic/elastic-entity-model/issues/170 (Add\r\ndedicated aggregation for display name and use that instead to provide a\r\nbetter label than the global ID)\r\nhttps://github.com/elastic/elastic-entity-model/issues/193 (Add entity\r\ntype display label to allow UI to not hard code a user friendly label)\r\n\r\n---------\r\n\r\nCo-authored-by: Chris Cowan <[email protected]>\r\nCo-authored-by: Elastic Machine <[email protected]>","sha":"080d0ff97f00bf564dedfd8fd37cdac0370e1349"}},"sourceBranch":"main","suggestedTargetBranches":["8.17"],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","labelRegex":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/196916","number":196916,"mergeCommit":{"message":"[EEM] Add built in definitions for core Kubernetes entities (#196916)\n\n## 🍒 Summary\r\n\r\nThis PR adds the OTEL and ECS entity definition for Kubernetes. This\r\ncovers the following datasets:\r\n- Cluster\r\n- Service (ECS Only)\r\n- Pod\r\n- ReplicaSet\r\n- Deployment\r\n- Statefulset\r\n- DaemonSet\r\n- Job\r\n- CronJob\r\n- Node\r\n\r\nThis PR does not include Container per @roshan-elastic \r\n\r\n### ✅ TODO\r\n- [X] Use correct index pattern for SemConv data\r\n(`metrics-k8sclusterreceiver.otel-default`,\r\n`metrics-kubeletstatsreceiver.otel-default`)\r\nUse global IDs instead of local IDs\r\n- [X] Add minimal list of labels to track beyond what was already added\r\n(wildcards are not supported, example `container.image.name` for\r\ncontainers to allow to find all \"redis\" containers)\r\n- [ ] Test with ECS data, SemConv data and mixed data (to check if we\r\nget duplicates, with the container definition for example).\r\n\r\n### 🐴 Follow up EEM features \r\nhttps://github.com/elastic/elastic-entity-model/issues/170 (Add\r\ndedicated aggregation for display name and use that instead to provide a\r\nbetter label than the global ID)\r\nhttps://github.com/elastic/elastic-entity-model/issues/193 (Add entity\r\ntype display label to allow UI to not hard code a user friendly label)\r\n\r\n---------\r\n\r\nCo-authored-by: Chris Cowan <[email protected]>\r\nCo-authored-by: Elastic Machine <[email protected]>","sha":"080d0ff97f00bf564dedfd8fd37cdac0370e1349"}},{"branch":"8.x","label":"v8.17.0","labelRegex":"^v8.17.0$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT--> Co-authored-by: Milton Hultgren <[email protected]>
crespocarlos
added a commit
that referenced
this pull request
Nov 26, 2024
closes [#201226](#201226) ## Summary This PR makes the final adjustments on k8s entities after the #196916 was merged. I had to fix most of the ECS entities because they had their `entityId` defined with `uid` field. Most of the ECS K8s entities don't have this field and should use the `name` field instead (see [metricsets](https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-module-kubernetes.html#_metricsets_41)) ~I also had to fix the transforms to include an aggregation for the `displayNameTemplate` field, when it doesn't match with the `entity.Id`~ ### Real data Otel The screenshots below are from a tests running the `opentemeletry-demo` sending otel data <img width="710" alt="image" src="https://github.com/user-attachments/assets/d223858d-3b99-4bb0-b69e-3b70112a2c17"> <img width="710" alt="image" src="https://github.com/user-attachments/assets/1ef27dcd-682c-4681-b92b-be9b4f2b32b8"> ### Real data ECS The screenshots below are from a tests running the `opentemeletry-demo` with elastic agent installed with Kubernetes integration <img width="710" alt="image" src="https://github.com/user-attachments/assets/e2fb6cbd-60a0-4995-bc94-8ccbc8911db1"> <img width="710" alt="image" src="https://github.com/user-attachments/assets/a3d9dba3-fee7-42af-972b-34a151b52f2b"> ### Additional test with synthtrace <img width="709" alt="image" src="https://github.com/user-attachments/assets/bf56da1e-0bbb-40a0-9f44-06b9f86427a8"> ### Fix ECS k8s service entity was missing the link to its corresponding dashboard <img width="709" alt="image" src="https://github.com/user-attachments/assets/488891a6-1c61-4001-b604-208478e1c798"> <img width="709" alt="image" src="https://github.com/user-attachments/assets/0b30c56b-db52-4d82-a7e7-100dae91a35e"> ### How to test - Start local kibana and es instances - run ` node scripts/synthtrace k8s_entities.ts --clean --live` - Navigate to Inventory and enable EEM --------- Co-authored-by: Elastic Machine <[email protected]>
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Nov 26, 2024
closes [elastic#201226](elastic#201226) ## Summary This PR makes the final adjustments on k8s entities after the elastic#196916 was merged. I had to fix most of the ECS entities because they had their `entityId` defined with `uid` field. Most of the ECS K8s entities don't have this field and should use the `name` field instead (see [metricsets](https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-module-kubernetes.html#_metricsets_41)) ~I also had to fix the transforms to include an aggregation for the `displayNameTemplate` field, when it doesn't match with the `entity.Id`~ ### Real data Otel The screenshots below are from a tests running the `opentemeletry-demo` sending otel data <img width="710" alt="image" src="https://github.com/user-attachments/assets/d223858d-3b99-4bb0-b69e-3b70112a2c17"> <img width="710" alt="image" src="https://github.com/user-attachments/assets/1ef27dcd-682c-4681-b92b-be9b4f2b32b8"> ### Real data ECS The screenshots below are from a tests running the `opentemeletry-demo` with elastic agent installed with Kubernetes integration <img width="710" alt="image" src="https://github.com/user-attachments/assets/e2fb6cbd-60a0-4995-bc94-8ccbc8911db1"> <img width="710" alt="image" src="https://github.com/user-attachments/assets/a3d9dba3-fee7-42af-972b-34a151b52f2b"> ### Additional test with synthtrace <img width="709" alt="image" src="https://github.com/user-attachments/assets/bf56da1e-0bbb-40a0-9f44-06b9f86427a8"> ### Fix ECS k8s service entity was missing the link to its corresponding dashboard <img width="709" alt="image" src="https://github.com/user-attachments/assets/488891a6-1c61-4001-b604-208478e1c798"> <img width="709" alt="image" src="https://github.com/user-attachments/assets/0b30c56b-db52-4d82-a7e7-100dae91a35e"> ### How to test - Start local kibana and es instances - run ` node scripts/synthtrace k8s_entities.ts --clean --live` - Navigate to Inventory and enable EEM --------- Co-authored-by: Elastic Machine <[email protected]> (cherry picked from commit aead7b9)
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Nov 26, 2024
closes [elastic#201226](elastic#201226) ## Summary This PR makes the final adjustments on k8s entities after the elastic#196916 was merged. I had to fix most of the ECS entities because they had their `entityId` defined with `uid` field. Most of the ECS K8s entities don't have this field and should use the `name` field instead (see [metricsets](https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-module-kubernetes.html#_metricsets_41)) ~I also had to fix the transforms to include an aggregation for the `displayNameTemplate` field, when it doesn't match with the `entity.Id`~ ### Real data Otel The screenshots below are from a tests running the `opentemeletry-demo` sending otel data <img width="710" alt="image" src="https://github.com/user-attachments/assets/d223858d-3b99-4bb0-b69e-3b70112a2c17"> <img width="710" alt="image" src="https://github.com/user-attachments/assets/1ef27dcd-682c-4681-b92b-be9b4f2b32b8"> ### Real data ECS The screenshots below are from a tests running the `opentemeletry-demo` with elastic agent installed with Kubernetes integration <img width="710" alt="image" src="https://github.com/user-attachments/assets/e2fb6cbd-60a0-4995-bc94-8ccbc8911db1"> <img width="710" alt="image" src="https://github.com/user-attachments/assets/a3d9dba3-fee7-42af-972b-34a151b52f2b"> ### Additional test with synthtrace <img width="709" alt="image" src="https://github.com/user-attachments/assets/bf56da1e-0bbb-40a0-9f44-06b9f86427a8"> ### Fix ECS k8s service entity was missing the link to its corresponding dashboard <img width="709" alt="image" src="https://github.com/user-attachments/assets/488891a6-1c61-4001-b604-208478e1c798"> <img width="709" alt="image" src="https://github.com/user-attachments/assets/0b30c56b-db52-4d82-a7e7-100dae91a35e"> ### How to test - Start local kibana and es instances - run ` node scripts/synthtrace k8s_entities.ts --clean --live` - Navigate to Inventory and enable EEM --------- Co-authored-by: Elastic Machine <[email protected]> (cherry picked from commit aead7b9)
kibanamachine
added a commit
that referenced
this pull request
Nov 26, 2024
# Backport This will backport the following commits from `main` to `8.17`: - [[Inventory] Inventory k8s entities fixes (#201260)](#201260) <!--- Backport version: 9.4.3 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Carlos Crespo","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-11-26T08:29:23Z","message":"[Inventory] Inventory k8s entities fixes (#201260)\n\ncloses [#201226](https://github.com/elastic/kibana/issues/201226)\r\n\r\n## Summary\r\n\r\nThis PR makes the final adjustments on k8s entities after the\r\nhttps://github.com//pull/196916 was merged.\r\n\r\nI had to fix most of the ECS entities because they had their `entityId`\r\ndefined with `uid` field. Most of the ECS K8s entities don't have this\r\nfield and should use the `name` field instead (see\r\n[metricsets](https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-module-kubernetes.html#_metricsets_41))\r\n\r\n~I also had to fix the transforms to include an aggregation for the\r\n`displayNameTemplate` field, when it doesn't match with the `entity.Id`~\r\n\r\n### Real data Otel\r\n\r\nThe screenshots below are from a tests running the `opentemeletry-demo`\r\nsending otel data\r\n\r\n<img width=\"710\" alt=\"image\"\r\nsrc=\"https://github.com/user-attachments/assets/d223858d-3b99-4bb0-b69e-3b70112a2c17\">\r\n\r\n<img width=\"710\" alt=\"image\"\r\nsrc=\"https://github.com/user-attachments/assets/1ef27dcd-682c-4681-b92b-be9b4f2b32b8\">\r\n\r\n### Real data ECS\r\n\r\nThe screenshots below are from a tests running the `opentemeletry-demo`\r\nwith elastic agent installed with Kubernetes integration\r\n\r\n<img width=\"710\" alt=\"image\"\r\nsrc=\"https://github.com/user-attachments/assets/e2fb6cbd-60a0-4995-bc94-8ccbc8911db1\">\r\n<img width=\"710\" alt=\"image\"\r\nsrc=\"https://github.com/user-attachments/assets/a3d9dba3-fee7-42af-972b-34a151b52f2b\">\r\n\r\n\r\n### Additional test with synthtrace\r\n\r\n<img width=\"709\" alt=\"image\"\r\nsrc=\"https://github.com/user-attachments/assets/bf56da1e-0bbb-40a0-9f44-06b9f86427a8\">\r\n\r\n### Fix\r\n\r\nECS k8s service entity was missing the link to its corresponding\r\ndashboard\r\n\r\n<img width=\"709\" alt=\"image\"\r\nsrc=\"https://github.com/user-attachments/assets/488891a6-1c61-4001-b604-208478e1c798\">\r\n\r\n<img width=\"709\" alt=\"image\"\r\nsrc=\"https://github.com/user-attachments/assets/0b30c56b-db52-4d82-a7e7-100dae91a35e\">\r\n\r\n\r\n### How to test\r\n\r\n- Start local kibana and es instances\r\n- run ` node scripts/synthtrace k8s_entities.ts --clean --live`\r\n- Navigate to Inventory and enable EEM\r\n\r\n---------\r\n\r\nCo-authored-by: Elastic Machine <[email protected]>","sha":"aead7b9acd5de0e5668c5f860eda473071a5a42d","branchLabelMapping":{"^v9.0.0$":"main","^v8.18.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","v9.0.0","backport:prev-minor","ci:project-deploy-observability","Team:obs-ux-infra_services","Team:obs-entities","backport:version","v8.17.0"],"title":"[Inventory] Inventory k8s entities fixes","number":201260,"url":"https://github.com/elastic/kibana/pull/201260","mergeCommit":{"message":"[Inventory] Inventory k8s entities fixes (#201260)\n\ncloses [#201226](https://github.com/elastic/kibana/issues/201226)\r\n\r\n## Summary\r\n\r\nThis PR makes the final adjustments on k8s entities after the\r\nhttps://github.com//pull/196916 was merged.\r\n\r\nI had to fix most of the ECS entities because they had their `entityId`\r\ndefined with `uid` field. Most of the ECS K8s entities don't have this\r\nfield and should use the `name` field instead (see\r\n[metricsets](https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-module-kubernetes.html#_metricsets_41))\r\n\r\n~I also had to fix the transforms to include an aggregation for the\r\n`displayNameTemplate` field, when it doesn't match with the `entity.Id`~\r\n\r\n### Real data Otel\r\n\r\nThe screenshots below are from a tests running the `opentemeletry-demo`\r\nsending otel data\r\n\r\n<img width=\"710\" alt=\"image\"\r\nsrc=\"https://github.com/user-attachments/assets/d223858d-3b99-4bb0-b69e-3b70112a2c17\">\r\n\r\n<img width=\"710\" alt=\"image\"\r\nsrc=\"https://github.com/user-attachments/assets/1ef27dcd-682c-4681-b92b-be9b4f2b32b8\">\r\n\r\n### Real data ECS\r\n\r\nThe screenshots below are from a tests running the `opentemeletry-demo`\r\nwith elastic agent installed with Kubernetes integration\r\n\r\n<img width=\"710\" alt=\"image\"\r\nsrc=\"https://github.com/user-attachments/assets/e2fb6cbd-60a0-4995-bc94-8ccbc8911db1\">\r\n<img width=\"710\" alt=\"image\"\r\nsrc=\"https://github.com/user-attachments/assets/a3d9dba3-fee7-42af-972b-34a151b52f2b\">\r\n\r\n\r\n### Additional test with synthtrace\r\n\r\n<img width=\"709\" alt=\"image\"\r\nsrc=\"https://github.com/user-attachments/assets/bf56da1e-0bbb-40a0-9f44-06b9f86427a8\">\r\n\r\n### Fix\r\n\r\nECS k8s service entity was missing the link to its corresponding\r\ndashboard\r\n\r\n<img width=\"709\" alt=\"image\"\r\nsrc=\"https://github.com/user-attachments/assets/488891a6-1c61-4001-b604-208478e1c798\">\r\n\r\n<img width=\"709\" alt=\"image\"\r\nsrc=\"https://github.com/user-attachments/assets/0b30c56b-db52-4d82-a7e7-100dae91a35e\">\r\n\r\n\r\n### How to test\r\n\r\n- Start local kibana and es instances\r\n- run ` node scripts/synthtrace k8s_entities.ts --clean --live`\r\n- Navigate to Inventory and enable EEM\r\n\r\n---------\r\n\r\nCo-authored-by: Elastic Machine <[email protected]>","sha":"aead7b9acd5de0e5668c5f860eda473071a5a42d"}},"sourceBranch":"main","suggestedTargetBranches":["8.17"],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/201260","number":201260,"mergeCommit":{"message":"[Inventory] Inventory k8s entities fixes (#201260)\n\ncloses [#201226](https://github.com/elastic/kibana/issues/201226)\r\n\r\n## Summary\r\n\r\nThis PR makes the final adjustments on k8s entities after the\r\nhttps://github.com//pull/196916 was merged.\r\n\r\nI had to fix most of the ECS entities because they had their `entityId`\r\ndefined with `uid` field. Most of the ECS K8s entities don't have this\r\nfield and should use the `name` field instead (see\r\n[metricsets](https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-module-kubernetes.html#_metricsets_41))\r\n\r\n~I also had to fix the transforms to include an aggregation for the\r\n`displayNameTemplate` field, when it doesn't match with the `entity.Id`~\r\n\r\n### Real data Otel\r\n\r\nThe screenshots below are from a tests running the `opentemeletry-demo`\r\nsending otel data\r\n\r\n<img width=\"710\" alt=\"image\"\r\nsrc=\"https://github.com/user-attachments/assets/d223858d-3b99-4bb0-b69e-3b70112a2c17\">\r\n\r\n<img width=\"710\" alt=\"image\"\r\nsrc=\"https://github.com/user-attachments/assets/1ef27dcd-682c-4681-b92b-be9b4f2b32b8\">\r\n\r\n### Real data ECS\r\n\r\nThe screenshots below are from a tests running the `opentemeletry-demo`\r\nwith elastic agent installed with Kubernetes integration\r\n\r\n<img width=\"710\" alt=\"image\"\r\nsrc=\"https://github.com/user-attachments/assets/e2fb6cbd-60a0-4995-bc94-8ccbc8911db1\">\r\n<img width=\"710\" alt=\"image\"\r\nsrc=\"https://github.com/user-attachments/assets/a3d9dba3-fee7-42af-972b-34a151b52f2b\">\r\n\r\n\r\n### Additional test with synthtrace\r\n\r\n<img width=\"709\" alt=\"image\"\r\nsrc=\"https://github.com/user-attachments/assets/bf56da1e-0bbb-40a0-9f44-06b9f86427a8\">\r\n\r\n### Fix\r\n\r\nECS k8s service entity was missing the link to its corresponding\r\ndashboard\r\n\r\n<img width=\"709\" alt=\"image\"\r\nsrc=\"https://github.com/user-attachments/assets/488891a6-1c61-4001-b604-208478e1c798\">\r\n\r\n<img width=\"709\" alt=\"image\"\r\nsrc=\"https://github.com/user-attachments/assets/0b30c56b-db52-4d82-a7e7-100dae91a35e\">\r\n\r\n\r\n### How to test\r\n\r\n- Start local kibana and es instances\r\n- run ` node scripts/synthtrace k8s_entities.ts --clean --live`\r\n- Navigate to Inventory and enable EEM\r\n\r\n---------\r\n\r\nCo-authored-by: Elastic Machine <[email protected]>","sha":"aead7b9acd5de0e5668c5f860eda473071a5a42d"}},{"branch":"8.17","label":"v8.17.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT--> Co-authored-by: Carlos Crespo <[email protected]>
kibanamachine
added a commit
that referenced
this pull request
Nov 26, 2024
# Backport This will backport the following commits from `main` to `8.x`: - [[Inventory] Inventory k8s entities fixes (#201260)](#201260) <!--- Backport version: 9.4.3 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Carlos Crespo","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-11-26T08:29:23Z","message":"[Inventory] Inventory k8s entities fixes (#201260)\n\ncloses [#201226](https://github.com/elastic/kibana/issues/201226)\r\n\r\n## Summary\r\n\r\nThis PR makes the final adjustments on k8s entities after the\r\nhttps://github.com//pull/196916 was merged.\r\n\r\nI had to fix most of the ECS entities because they had their `entityId`\r\ndefined with `uid` field. Most of the ECS K8s entities don't have this\r\nfield and should use the `name` field instead (see\r\n[metricsets](https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-module-kubernetes.html#_metricsets_41))\r\n\r\n~I also had to fix the transforms to include an aggregation for the\r\n`displayNameTemplate` field, when it doesn't match with the `entity.Id`~\r\n\r\n### Real data Otel\r\n\r\nThe screenshots below are from a tests running the `opentemeletry-demo`\r\nsending otel data\r\n\r\n<img width=\"710\" alt=\"image\"\r\nsrc=\"https://github.com/user-attachments/assets/d223858d-3b99-4bb0-b69e-3b70112a2c17\">\r\n\r\n<img width=\"710\" alt=\"image\"\r\nsrc=\"https://github.com/user-attachments/assets/1ef27dcd-682c-4681-b92b-be9b4f2b32b8\">\r\n\r\n### Real data ECS\r\n\r\nThe screenshots below are from a tests running the `opentemeletry-demo`\r\nwith elastic agent installed with Kubernetes integration\r\n\r\n<img width=\"710\" alt=\"image\"\r\nsrc=\"https://github.com/user-attachments/assets/e2fb6cbd-60a0-4995-bc94-8ccbc8911db1\">\r\n<img width=\"710\" alt=\"image\"\r\nsrc=\"https://github.com/user-attachments/assets/a3d9dba3-fee7-42af-972b-34a151b52f2b\">\r\n\r\n\r\n### Additional test with synthtrace\r\n\r\n<img width=\"709\" alt=\"image\"\r\nsrc=\"https://github.com/user-attachments/assets/bf56da1e-0bbb-40a0-9f44-06b9f86427a8\">\r\n\r\n### Fix\r\n\r\nECS k8s service entity was missing the link to its corresponding\r\ndashboard\r\n\r\n<img width=\"709\" alt=\"image\"\r\nsrc=\"https://github.com/user-attachments/assets/488891a6-1c61-4001-b604-208478e1c798\">\r\n\r\n<img width=\"709\" alt=\"image\"\r\nsrc=\"https://github.com/user-attachments/assets/0b30c56b-db52-4d82-a7e7-100dae91a35e\">\r\n\r\n\r\n### How to test\r\n\r\n- Start local kibana and es instances\r\n- run ` node scripts/synthtrace k8s_entities.ts --clean --live`\r\n- Navigate to Inventory and enable EEM\r\n\r\n---------\r\n\r\nCo-authored-by: Elastic Machine <[email protected]>","sha":"aead7b9acd5de0e5668c5f860eda473071a5a42d","branchLabelMapping":{"^v9.0.0$":"main","^v8.18.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","v9.0.0","backport:prev-minor","ci:project-deploy-observability","Team:obs-ux-infra_services","Team:obs-entities","backport:version","v8.17.0"],"title":"[Inventory] Inventory k8s entities fixes","number":201260,"url":"https://github.com/elastic/kibana/pull/201260","mergeCommit":{"message":"[Inventory] Inventory k8s entities fixes (#201260)\n\ncloses [#201226](https://github.com/elastic/kibana/issues/201226)\r\n\r\n## Summary\r\n\r\nThis PR makes the final adjustments on k8s entities after the\r\nhttps://github.com//pull/196916 was merged.\r\n\r\nI had to fix most of the ECS entities because they had their `entityId`\r\ndefined with `uid` field. Most of the ECS K8s entities don't have this\r\nfield and should use the `name` field instead (see\r\n[metricsets](https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-module-kubernetes.html#_metricsets_41))\r\n\r\n~I also had to fix the transforms to include an aggregation for the\r\n`displayNameTemplate` field, when it doesn't match with the `entity.Id`~\r\n\r\n### Real data Otel\r\n\r\nThe screenshots below are from a tests running the `opentemeletry-demo`\r\nsending otel data\r\n\r\n<img width=\"710\" alt=\"image\"\r\nsrc=\"https://github.com/user-attachments/assets/d223858d-3b99-4bb0-b69e-3b70112a2c17\">\r\n\r\n<img width=\"710\" alt=\"image\"\r\nsrc=\"https://github.com/user-attachments/assets/1ef27dcd-682c-4681-b92b-be9b4f2b32b8\">\r\n\r\n### Real data ECS\r\n\r\nThe screenshots below are from a tests running the `opentemeletry-demo`\r\nwith elastic agent installed with Kubernetes integration\r\n\r\n<img width=\"710\" alt=\"image\"\r\nsrc=\"https://github.com/user-attachments/assets/e2fb6cbd-60a0-4995-bc94-8ccbc8911db1\">\r\n<img width=\"710\" alt=\"image\"\r\nsrc=\"https://github.com/user-attachments/assets/a3d9dba3-fee7-42af-972b-34a151b52f2b\">\r\n\r\n\r\n### Additional test with synthtrace\r\n\r\n<img width=\"709\" alt=\"image\"\r\nsrc=\"https://github.com/user-attachments/assets/bf56da1e-0bbb-40a0-9f44-06b9f86427a8\">\r\n\r\n### Fix\r\n\r\nECS k8s service entity was missing the link to its corresponding\r\ndashboard\r\n\r\n<img width=\"709\" alt=\"image\"\r\nsrc=\"https://github.com/user-attachments/assets/488891a6-1c61-4001-b604-208478e1c798\">\r\n\r\n<img width=\"709\" alt=\"image\"\r\nsrc=\"https://github.com/user-attachments/assets/0b30c56b-db52-4d82-a7e7-100dae91a35e\">\r\n\r\n\r\n### How to test\r\n\r\n- Start local kibana and es instances\r\n- run ` node scripts/synthtrace k8s_entities.ts --clean --live`\r\n- Navigate to Inventory and enable EEM\r\n\r\n---------\r\n\r\nCo-authored-by: Elastic Machine <[email protected]>","sha":"aead7b9acd5de0e5668c5f860eda473071a5a42d"}},"sourceBranch":"main","suggestedTargetBranches":["8.17"],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/201260","number":201260,"mergeCommit":{"message":"[Inventory] Inventory k8s entities fixes (#201260)\n\ncloses [#201226](https://github.com/elastic/kibana/issues/201226)\r\n\r\n## Summary\r\n\r\nThis PR makes the final adjustments on k8s entities after the\r\nhttps://github.com//pull/196916 was merged.\r\n\r\nI had to fix most of the ECS entities because they had their `entityId`\r\ndefined with `uid` field. Most of the ECS K8s entities don't have this\r\nfield and should use the `name` field instead (see\r\n[metricsets](https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-module-kubernetes.html#_metricsets_41))\r\n\r\n~I also had to fix the transforms to include an aggregation for the\r\n`displayNameTemplate` field, when it doesn't match with the `entity.Id`~\r\n\r\n### Real data Otel\r\n\r\nThe screenshots below are from a tests running the `opentemeletry-demo`\r\nsending otel data\r\n\r\n<img width=\"710\" alt=\"image\"\r\nsrc=\"https://github.com/user-attachments/assets/d223858d-3b99-4bb0-b69e-3b70112a2c17\">\r\n\r\n<img width=\"710\" alt=\"image\"\r\nsrc=\"https://github.com/user-attachments/assets/1ef27dcd-682c-4681-b92b-be9b4f2b32b8\">\r\n\r\n### Real data ECS\r\n\r\nThe screenshots below are from a tests running the `opentemeletry-demo`\r\nwith elastic agent installed with Kubernetes integration\r\n\r\n<img width=\"710\" alt=\"image\"\r\nsrc=\"https://github.com/user-attachments/assets/e2fb6cbd-60a0-4995-bc94-8ccbc8911db1\">\r\n<img width=\"710\" alt=\"image\"\r\nsrc=\"https://github.com/user-attachments/assets/a3d9dba3-fee7-42af-972b-34a151b52f2b\">\r\n\r\n\r\n### Additional test with synthtrace\r\n\r\n<img width=\"709\" alt=\"image\"\r\nsrc=\"https://github.com/user-attachments/assets/bf56da1e-0bbb-40a0-9f44-06b9f86427a8\">\r\n\r\n### Fix\r\n\r\nECS k8s service entity was missing the link to its corresponding\r\ndashboard\r\n\r\n<img width=\"709\" alt=\"image\"\r\nsrc=\"https://github.com/user-attachments/assets/488891a6-1c61-4001-b604-208478e1c798\">\r\n\r\n<img width=\"709\" alt=\"image\"\r\nsrc=\"https://github.com/user-attachments/assets/0b30c56b-db52-4d82-a7e7-100dae91a35e\">\r\n\r\n\r\n### How to test\r\n\r\n- Start local kibana and es instances\r\n- run ` node scripts/synthtrace k8s_entities.ts --clean --live`\r\n- Navigate to Inventory and enable EEM\r\n\r\n---------\r\n\r\nCo-authored-by: Elastic Machine <[email protected]>","sha":"aead7b9acd5de0e5668c5f860eda473071a5a42d"}},{"branch":"8.17","label":"v8.17.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT--> Co-authored-by: Carlos Crespo <[email protected]>
paulinashakirova
pushed a commit
to paulinashakirova/kibana
that referenced
this pull request
Nov 26, 2024
…196916) ## 🍒 Summary This PR adds the OTEL and ECS entity definition for Kubernetes. This covers the following datasets: - Cluster - Service (ECS Only) - Pod - ReplicaSet - Deployment - Statefulset - DaemonSet - Job - CronJob - Node This PR does not include Container per @roshan-elastic ### ✅ TODO - [X] Use correct index pattern for SemConv data (`metrics-k8sclusterreceiver.otel-default`, `metrics-kubeletstatsreceiver.otel-default`) Use global IDs instead of local IDs - [X] Add minimal list of labels to track beyond what was already added (wildcards are not supported, example `container.image.name` for containers to allow to find all "redis" containers) - [ ] Test with ECS data, SemConv data and mixed data (to check if we get duplicates, with the container definition for example). ### 🐴 Follow up EEM features elastic/elastic-entity-model#170 (Add dedicated aggregation for display name and use that instead to provide a better label than the global ID) elastic/elastic-entity-model#193 (Add entity type display label to allow UI to not hard code a user friendly label) --------- Co-authored-by: Chris Cowan <[email protected]> Co-authored-by: Elastic Machine <[email protected]>
paulinashakirova
pushed a commit
to paulinashakirova/kibana
that referenced
this pull request
Nov 26, 2024
closes [elastic#201226](elastic#201226) ## Summary This PR makes the final adjustments on k8s entities after the elastic#196916 was merged. I had to fix most of the ECS entities because they had their `entityId` defined with `uid` field. Most of the ECS K8s entities don't have this field and should use the `name` field instead (see [metricsets](https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-module-kubernetes.html#_metricsets_41)) ~I also had to fix the transforms to include an aggregation for the `displayNameTemplate` field, when it doesn't match with the `entity.Id`~ ### Real data Otel The screenshots below are from a tests running the `opentemeletry-demo` sending otel data <img width="710" alt="image" src="https://github.com/user-attachments/assets/d223858d-3b99-4bb0-b69e-3b70112a2c17"> <img width="710" alt="image" src="https://github.com/user-attachments/assets/1ef27dcd-682c-4681-b92b-be9b4f2b32b8"> ### Real data ECS The screenshots below are from a tests running the `opentemeletry-demo` with elastic agent installed with Kubernetes integration <img width="710" alt="image" src="https://github.com/user-attachments/assets/e2fb6cbd-60a0-4995-bc94-8ccbc8911db1"> <img width="710" alt="image" src="https://github.com/user-attachments/assets/a3d9dba3-fee7-42af-972b-34a151b52f2b"> ### Additional test with synthtrace <img width="709" alt="image" src="https://github.com/user-attachments/assets/bf56da1e-0bbb-40a0-9f44-06b9f86427a8"> ### Fix ECS k8s service entity was missing the link to its corresponding dashboard <img width="709" alt="image" src="https://github.com/user-attachments/assets/488891a6-1c61-4001-b604-208478e1c798"> <img width="709" alt="image" src="https://github.com/user-attachments/assets/0b30c56b-db52-4d82-a7e7-100dae91a35e"> ### How to test - Start local kibana and es instances - run ` node scripts/synthtrace k8s_entities.ts --clean --live` - Navigate to Inventory and enable EEM --------- Co-authored-by: Elastic Machine <[email protected]>
CAWilson94
pushed a commit
to CAWilson94/kibana
that referenced
this pull request
Dec 12, 2024
…196916) ## 🍒 Summary This PR adds the OTEL and ECS entity definition for Kubernetes. This covers the following datasets: - Cluster - Service (ECS Only) - Pod - ReplicaSet - Deployment - Statefulset - DaemonSet - Job - CronJob - Node This PR does not include Container per @roshan-elastic ### ✅ TODO - [X] Use correct index pattern for SemConv data (`metrics-k8sclusterreceiver.otel-default`, `metrics-kubeletstatsreceiver.otel-default`) Use global IDs instead of local IDs - [X] Add minimal list of labels to track beyond what was already added (wildcards are not supported, example `container.image.name` for containers to allow to find all "redis" containers) - [ ] Test with ECS data, SemConv data and mixed data (to check if we get duplicates, with the container definition for example). ### 🐴 Follow up EEM features elastic/elastic-entity-model#170 (Add dedicated aggregation for display name and use that instead to provide a better label than the global ID) elastic/elastic-entity-model#193 (Add entity type display label to allow UI to not hard code a user friendly label) --------- Co-authored-by: Chris Cowan <[email protected]> Co-authored-by: Elastic Machine <[email protected]>
CAWilson94
pushed a commit
to CAWilson94/kibana
that referenced
this pull request
Dec 12, 2024
closes [elastic#201226](elastic#201226) ## Summary This PR makes the final adjustments on k8s entities after the elastic#196916 was merged. I had to fix most of the ECS entities because they had their `entityId` defined with `uid` field. Most of the ECS K8s entities don't have this field and should use the `name` field instead (see [metricsets](https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-module-kubernetes.html#_metricsets_41)) ~I also had to fix the transforms to include an aggregation for the `displayNameTemplate` field, when it doesn't match with the `entity.Id`~ ### Real data Otel The screenshots below are from a tests running the `opentemeletry-demo` sending otel data <img width="710" alt="image" src="https://github.com/user-attachments/assets/d223858d-3b99-4bb0-b69e-3b70112a2c17"> <img width="710" alt="image" src="https://github.com/user-attachments/assets/1ef27dcd-682c-4681-b92b-be9b4f2b32b8"> ### Real data ECS The screenshots below are from a tests running the `opentemeletry-demo` with elastic agent installed with Kubernetes integration <img width="710" alt="image" src="https://github.com/user-attachments/assets/e2fb6cbd-60a0-4995-bc94-8ccbc8911db1"> <img width="710" alt="image" src="https://github.com/user-attachments/assets/a3d9dba3-fee7-42af-972b-34a151b52f2b"> ### Additional test with synthtrace <img width="709" alt="image" src="https://github.com/user-attachments/assets/bf56da1e-0bbb-40a0-9f44-06b9f86427a8"> ### Fix ECS k8s service entity was missing the link to its corresponding dashboard <img width="709" alt="image" src="https://github.com/user-attachments/assets/488891a6-1c61-4001-b604-208478e1c798"> <img width="709" alt="image" src="https://github.com/user-attachments/assets/0b30c56b-db52-4d82-a7e7-100dae91a35e"> ### How to test - Start local kibana and es instances - run ` node scripts/synthtrace k8s_entities.ts --clean --live` - Navigate to Inventory and enable EEM --------- Co-authored-by: Elastic Machine <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
🍒 Summary
This PR adds the OTEL and ECS entity definition for Kubernetes. This covers the following datasets:
This PR does not include Container per @roshan-elastic
✅ TODO
metrics-k8sclusterreceiver.otel-default,metrics-kubeletstatsreceiver.otel-default)Use global IDs instead of local IDs
container.image.namefor containers to allow to find all "redis" containers)🐴 Follow up EEM features
https://github.com/elastic/elastic-entity-model/issues/170 (Add dedicated aggregation for display name and use that instead to provide a better label than the global ID)
https://github.com/elastic/elastic-entity-model/issues/193 (Add entity type display label to allow UI to not hard code a user friendly label)