diff --git a/packages/vsphere/_dev/build/docs/README.md b/packages/vsphere/_dev/build/docs/README.md index 60df973abe6..209917a7c1b 100644 --- a/packages/vsphere/_dev/build/docs/README.md +++ b/packages/vsphere/_dev/build/docs/README.md @@ -16,16 +16,17 @@ The vSphere integration collects logs and metrics. Logs help you keep a record of events that happen on your machine. The `Log` data stream collected by vSphere as integration is `log`. -Metrics give you insight into the statistics of the vSphere. The `Metric` data stream collected by the vSphere integration are `cluster`, `datastore`, `host`, `resourcepool` and `virtualmachine` so that the user can monitor and troubleshoot the performance of the vSphere instance. +Metrics give you insight into the statistics of the vSphere. The `Metric` data stream collected by the vSphere integration are `cluster`, `datastore`, `datastorecluster`, `host`, `resourcepool` and `virtualmachine` so that the user can monitor and troubleshoot the performance of the vSphere instance. -Data streams: -- `log`: This data stream collects logs generated by VMware vSphere using a syslog daemon. -- `cluster`: This data stream collects metrics from VMware vSphere, such as lists of datastores, hosts and networks associated with cluster. -- `datastore`: This data stream collects datastore metrics from VMware vSphere, including performance statistics such as capacity, usage, read/write operations, latency, and throughput. -- `host`: This data stream collects host metrics from VMware vSphere, including performance statistics such as CPU usage, memory usage, disk I/O, and network activity. -- `resourcepool`: This data stream collects metrics from VMware vSphere, such as CPU and memory usage, CPU and memory reservation, and CPU and memory limit. -- `virtualmachine`: This data stream collects virtual machine metrics from VMware vSphere, including performance statistics such as status, uptime, CPU usage, memory usage, and network activity. +Data Streams: +- **`log`**: This data stream collects logs generated by VMware vSphere using a syslog daemon. +- **`cluster`**: This data stream collects metrics from VMware vSphere, such as lists of datastores, hosts and networks associated with cluster. +- **`datastore`**: This data stream gathers datastore metrics from VMware vSphere, including performance statistics such as capacity, usage, read/write operations, latency, and throughput. +- **`datastorecluster`**: This data stream gathers metrics for datastore clusters from VMware vSphere, including statistics like cluster capacity and available free space. Additionally, it provides information about the individual datastores that comprise the cluster. +- **`host`**: This data stream collects host metrics from VMware vSphere, including performance statistics such as CPU usage, memory usage, disk I/O, and network activity. +- **`resourcepool`**: This data stream collects metrics from VMware vSphere, such as CPU and memory usage, CPU and memory reservation, and CPU and memory limit. +- **`virtualmachine`**: This data stream gathers virtual machine metrics from VMware vSphere, including performance statistics such as status, uptime, CPU usage, memory usage, and network activity. Note: - Users can monitor and see the log inside the ingested documents for vSphere in the `logs-*` index pattern from `Discover`, and for metrics, the index pattern is `metrics-*`. @@ -95,6 +96,18 @@ Please refer to the following [document](https://www.elastic.co/guide/en/ecs/cur {{fields "datastore"}} +### Datastore Cluster + +Datastore clusters in vSphere group multiple datastores for optimized management and automated load balancing, ensuring efficient storage utilization and simplified administration. + +{{event "datastorecluster"}} + +**ECS Field Reference** + +Please refer to the following [document](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html) for detailed information on ECS fields. + +{{fields "datastorecluster"}} + ### Host This is the `host` data stream. This data stream collects host metrics from VMware vSphere, including performance statistics such as CPU usage, memory usage, disk I/O, and network activity. diff --git a/packages/vsphere/_dev/deploy/docker/Dockerfile b/packages/vsphere/_dev/deploy/docker/Dockerfile index ce3f58eb072..1663a3654dd 100644 --- a/packages/vsphere/_dev/deploy/docker/Dockerfile +++ b/packages/vsphere/_dev/deploy/docker/Dockerfile @@ -6,4 +6,4 @@ RUN apk add --no-cache curl git RUN go install github.com/vmware/govmomi/vcsim@v0.30.4 HEALTHCHECK --interval=1s --retries=60 --timeout=10s CMD curl http://localhost:8989/ -CMD vcsim -l :8989 +CMD vcsim -pod 1 -l :8989 diff --git a/packages/vsphere/_dev/deploy/docker/docker-compose.yml b/packages/vsphere/_dev/deploy/docker/docker-compose.yml index 4dce57d63a1..e7efe65d980 100644 --- a/packages/vsphere/_dev/deploy/docker/docker-compose.yml +++ b/packages/vsphere/_dev/deploy/docker/docker-compose.yml @@ -1,7 +1,6 @@ version: '2.3' services: vsphere-metrics: - image: docker.elastic.co/integrations-ci/beats-vsphere:${VSPHERE_GOLANG_VERSION:-1.17}-1 build: context: . args: diff --git a/packages/vsphere/changelog.yml b/packages/vsphere/changelog.yml index c5f7768751b..9ecf8da5121 100644 --- a/packages/vsphere/changelog.yml +++ b/packages/vsphere/changelog.yml @@ -1,6 +1,9 @@ # newer versions go on top - version: "1.15.0-next" changes: + - description: Add new datastorecluster datastream. + type: enhancement + link: https://github.com/elastic/integrations/pull/11089 - description: Add additional metrics for virtualmachine datastream type: enhancement link: https://github.com/elastic/integrations/pull/10942 diff --git a/packages/vsphere/data_stream/datastorecluster/_dev/test/pipeline/test-datastorecluster-metrics.json b/packages/vsphere/data_stream/datastorecluster/_dev/test/pipeline/test-datastorecluster-metrics.json new file mode 100644 index 00000000000..5fed9137da8 --- /dev/null +++ b/packages/vsphere/data_stream/datastorecluster/_dev/test/pipeline/test-datastorecluster-metrics.json @@ -0,0 +1,23 @@ +{ + "events": [ + { + "vsphere": { + "datastorecluster": { + "name": "datastore_cluster1", + "capacity": { + "bytes": 8795019280384 + }, + "free_space": { + "bytes": 8788836876288 + }, + "datastore": { + "count": 1, + "names": [ + "LocalDS_0" + ] + } + } + } + } + ] +} \ No newline at end of file diff --git a/packages/vsphere/data_stream/datastorecluster/_dev/test/pipeline/test-datastorecluster-metrics.json-expected.json b/packages/vsphere/data_stream/datastorecluster/_dev/test/pipeline/test-datastorecluster-metrics.json-expected.json new file mode 100644 index 00000000000..499d8cfe3c7 --- /dev/null +++ b/packages/vsphere/data_stream/datastorecluster/_dev/test/pipeline/test-datastorecluster-metrics.json-expected.json @@ -0,0 +1,26 @@ +{ + "expected": [ + { + "ecs": { + "version": "8.11.0" + }, + "vsphere": { + "datastorecluster": { + "capacity": { + "bytes": 8795019280384 + }, + "datastore": { + "count": 1, + "names": [ + "LocalDS_0" + ] + }, + "free_space": { + "bytes": 8788836876288 + }, + "name": "datastore_cluster1" + } + } + } + ] +} \ No newline at end of file diff --git a/packages/vsphere/data_stream/datastorecluster/_dev/test/system/test-default-config.yml b/packages/vsphere/data_stream/datastorecluster/_dev/test/system/test-default-config.yml new file mode 100644 index 00000000000..04abd36de05 --- /dev/null +++ b/packages/vsphere/data_stream/datastorecluster/_dev/test/system/test-default-config.yml @@ -0,0 +1,9 @@ +service: vsphere-metrics +vars: + hosts: + - "{{Hostname}}:8989/sdk" + period: 10s + username: "user" + password: "password" + # If insecure is true, don't verify the server's certificate chain + insecure: true diff --git a/packages/vsphere/data_stream/datastorecluster/agent/stream/stream.yml.hbs b/packages/vsphere/data_stream/datastorecluster/agent/stream/stream.yml.hbs new file mode 100644 index 00000000000..6532df6ddfa --- /dev/null +++ b/packages/vsphere/data_stream/datastorecluster/agent/stream/stream.yml.hbs @@ -0,0 +1,20 @@ +metricsets: ["datastorecluster"] +hosts: +{{#each hosts}} + - {{this}} +{{/each}} +period: {{period}} +username: {{username}} +password: {{password}} +# If insecure is true, don't verify the server's certificate chain +insecure: {{insecure}} + +tags: +{{#each tags as |tag|}} + - {{tag}} +{{/each}} + +{{#if processors}} +processors: +{{processors}} +{{/if}} diff --git a/packages/vsphere/data_stream/datastorecluster/elasticsearch/ingest_pipeline/default.yml b/packages/vsphere/data_stream/datastorecluster/elasticsearch/ingest_pipeline/default.yml new file mode 100644 index 00000000000..ebb730935f6 --- /dev/null +++ b/packages/vsphere/data_stream/datastorecluster/elasticsearch/ingest_pipeline/default.yml @@ -0,0 +1,10 @@ +--- +description: Pipeline for datastorecluster +processors: +- set: + field: ecs.version + value: '8.11.0' +on_failure: +- set: + field: error.message + value: '{{ _ingest.on_failure_message }}' diff --git a/packages/vsphere/data_stream/datastorecluster/fields/agent.yml b/packages/vsphere/data_stream/datastorecluster/fields/agent.yml new file mode 100644 index 00000000000..bc8fba8464d --- /dev/null +++ b/packages/vsphere/data_stream/datastorecluster/fields/agent.yml @@ -0,0 +1,20 @@ +- name: host + title: Host + group: 2 + description: 'A host is defined as a general computing instance. ECS host.* fields should be populated with details about the host on which the event happened, or from which the measurement was taken. Host types include hardware, virtual machines, Docker containers, and Kubernetes nodes.' + type: group + fields: + - name: name + level: core + type: keyword + dimension: true + ignore_above: 1024 + description: 'Name of the host. It can contain what `hostname` returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use.' +- name: agent + title: Agent + type: group + fields: + - name: id + type: keyword + ignore_above: 1024 + dimension: true diff --git a/packages/vsphere/data_stream/datastorecluster/fields/base-fields.yml b/packages/vsphere/data_stream/datastorecluster/fields/base-fields.yml new file mode 100644 index 00000000000..0e364f6fc43 --- /dev/null +++ b/packages/vsphere/data_stream/datastorecluster/fields/base-fields.yml @@ -0,0 +1,20 @@ +- name: data_stream.type + type: constant_keyword + description: Data stream type. +- name: data_stream.dataset + type: constant_keyword + description: Data stream dataset. +- name: data_stream.namespace + type: constant_keyword + description: Data stream namespace. +- name: '@timestamp' + type: date + description: Event timestamp. +- name: event.module + type: constant_keyword + description: Event module + value: vsphere +- name: event.dataset + type: constant_keyword + description: Event dataset + value: vsphere.datastorecluster diff --git a/packages/vsphere/data_stream/datastorecluster/fields/ecs.yml b/packages/vsphere/data_stream/datastorecluster/fields/ecs.yml new file mode 100644 index 00000000000..cee8d299187 --- /dev/null +++ b/packages/vsphere/data_stream/datastorecluster/fields/ecs.yml @@ -0,0 +1,21 @@ +- name: service.address + external: ecs + dimension: true +- external: ecs + name: cloud.instance.id + dimension: true +- external: ecs + name: cloud.provider + dimension: true +- external: ecs + name: container.id + dimension: true +- external: ecs + name: cloud.account.id + dimension: true +- external: ecs + name: cloud.region + dimension: true +- external: ecs + name: cloud.availability_zone + dimension: true diff --git a/packages/vsphere/data_stream/datastorecluster/fields/fields.yml b/packages/vsphere/data_stream/datastorecluster/fields/fields.yml new file mode 100644 index 00000000000..dd77cd0a1b6 --- /dev/null +++ b/packages/vsphere/data_stream/datastorecluster/fields/fields.yml @@ -0,0 +1,57 @@ +- name: vsphere.datastorecluster + type: group + description: > + Datastore Cluster + fields: + - name: capacity.bytes + type: long + metric_type: gauge + unit: byte + description: > + Total capacity of this storage pod, in bytes. + format: bytes + - name: datastore + type: group + fields: + - name: names + type: keyword + description: List of all the Datastore names associated with the Datastore Cluster. + - name: count + type: long + metric_type: gauge + description: Number of datastores in the Datastore Cluster. + - name: free_space.bytes + type: long + metric_type: gauge + unit: byte + description: > + Total free space on this storage pod, in bytes. + format: bytes + - name: name + type: keyword + # Reason to add as a dimension field: unique identifier of the host + dimension: true + description: The Datastore Cluster name. + - name: triggered_alarms + type: group + description: > + List of all the triggered alarms. + fields: + - name: status + type: keyword + description: Status of the alarm. + - name: triggered_time + type: date + description: Time when the alarm was triggered. + - name: description + type: keyword + description: Description of the alarm. + - name: entity_name + type: keyword + description: Name of the entity associated with the alarm. + - name: name + type: keyword + description: Name of the alarm. + - name: id + type: keyword + description: Unique identifier for the alarm. diff --git a/packages/vsphere/data_stream/datastorecluster/manifest.yml b/packages/vsphere/data_stream/datastorecluster/manifest.yml new file mode 100644 index 00000000000..09737039678 --- /dev/null +++ b/packages/vsphere/data_stream/datastorecluster/manifest.yml @@ -0,0 +1,29 @@ +title: "vSphere DatastoreCluster metrics" +type: metrics +streams: + - input: vsphere/metrics + title: vSphere Datastore Cluster metrics + description: vSphere Datastore Cluster metrics + vars: + - name: period + type: text + title: Period + description: The period for fetching metrics, e.g., every 1s, 1m, 1h. + default: 20s + - name: tags + type: text + title: Tags + multi: true + required: true + show_user: false + default: + - vsphere-datastorecluster + - name: processors + type: yaml + title: Processors + multi: false + required: false + show_user: false + description: Processors are used to reduce the number of fields in the exported event or to enhance the event with metadata. This executes in the agent before the events are shipped. See [Processors](https://www.elastic.co/guide/en/fleet/current/elastic-agent-processor-configuration.html) for details. +elasticsearch: + index_mode: "time_series" diff --git a/packages/vsphere/data_stream/datastorecluster/sample_event.json b/packages/vsphere/data_stream/datastorecluster/sample_event.json new file mode 100644 index 00000000000..f524e63af28 --- /dev/null +++ b/packages/vsphere/data_stream/datastorecluster/sample_event.json @@ -0,0 +1,79 @@ +{ + "@timestamp": "2024-09-22T05:28:46.315Z", + "agent": { + "ephemeral_id": "8b7e9ea4-0517-4e98-a795-b6fe529f4a2f", + "id": "7737279e-51e9-4d90-a0d0-2c12dc4446bf", + "name": "elastic-agent-23128", + "type": "metricbeat", + "version": "8.15.2" + }, + "data_stream": { + "dataset": "vsphere.datastorecluster", + "namespace": "65218", + "type": "metrics" + }, + "ecs": { + "version": "8.11.0" + }, + "elastic_agent": { + "id": "7737279e-51e9-4d90-a0d0-2c12dc4446bf", + "snapshot": true, + "version": "8.15.2" + }, + "event": { + "agent_id_status": "verified", + "dataset": "vsphere.datastorecluster", + "duration": 10772332, + "ingested": "2024-09-22T05:28:49Z", + "module": "vsphere" + }, + "host": { + "architecture": "x86_64", + "containerized": true, + "hostname": "elastic-agent-23128", + "id": "57723763cd1b4ff48e54a505de4ebe6c", + "ip": [ + "192.168.240.2", + "192.168.255.5" + ], + "mac": [ + "02-42-C0-A8-F0-02", + "02-42-C0-A8-FF-05" + ], + "name": "elastic-agent-23128", + "os": { + "codename": "focal", + "family": "debian", + "kernel": "3.10.0-1160.118.1.el7.x86_64", + "name": "Ubuntu", + "platform": "ubuntu", + "type": "linux", + "version": "20.04.6 LTS (Focal Fossa)" + } + }, + "metricset": { + "name": "datastorecluster", + "period": 20000 + }, + "service": { + "address": "https://svc-vsphere-metrics:8989/sdk", + "type": "vsphere" + }, + "tags": [ + "vsphere-datastorecluster" + ], + "vsphere": { + "datastorecluster": { + "capacity": { + "bytes": 0 + }, + "datastore": { + "count": 0 + }, + "free_space": { + "bytes": 0 + }, + "name": "DC0_POD0" + } + } +} \ No newline at end of file diff --git a/packages/vsphere/docs/README.md b/packages/vsphere/docs/README.md index bd4385d6508..da04610e592 100644 --- a/packages/vsphere/docs/README.md +++ b/packages/vsphere/docs/README.md @@ -16,16 +16,17 @@ The vSphere integration collects logs and metrics. Logs help you keep a record of events that happen on your machine. The `Log` data stream collected by vSphere as integration is `log`. -Metrics give you insight into the statistics of the vSphere. The `Metric` data stream collected by the vSphere integration are `cluster`, `datastore`, `host`, `resourcepool` and `virtualmachine` so that the user can monitor and troubleshoot the performance of the vSphere instance. +Metrics give you insight into the statistics of the vSphere. The `Metric` data stream collected by the vSphere integration are `cluster`, `datastore`, `datastorecluster`, `host`, `resourcepool` and `virtualmachine` so that the user can monitor and troubleshoot the performance of the vSphere instance. -Data streams: -- `log`: This data stream collects logs generated by VMware vSphere using a syslog daemon. -- `cluster`: This data stream collects metrics from VMware vSphere, such as lists of datastores, hosts and networks associated with cluster. -- `datastore`: This data stream collects datastore metrics from VMware vSphere, including performance statistics such as capacity, usage, read/write operations, latency, and throughput. -- `host`: This data stream collects host metrics from VMware vSphere, including performance statistics such as CPU usage, memory usage, disk I/O, and network activity. -- `resourcepool`: This data stream collects metrics from VMware vSphere, such as CPU and memory usage, CPU and memory reservation, and CPU and memory limit. -- `virtualmachine`: This data stream collects virtual machine metrics from VMware vSphere, including performance statistics such as status, uptime, CPU usage, memory usage, and network activity. +Data Streams: +- **`log`**: This data stream collects logs generated by VMware vSphere using a syslog daemon. +- **`cluster`**: This data stream collects metrics from VMware vSphere, such as lists of datastores, hosts and networks associated with cluster. +- **`datastore`**: This data stream gathers datastore metrics from VMware vSphere, including performance statistics such as capacity, usage, read/write operations, latency, and throughput. +- **`datastorecluster`**: This data stream gathers metrics for datastore clusters from VMware vSphere, including statistics like cluster capacity and available free space. Additionally, it provides information about the individual datastores that comprise the cluster. +- **`host`**: This data stream collects host metrics from VMware vSphere, including performance statistics such as CPU usage, memory usage, disk I/O, and network activity. +- **`resourcepool`**: This data stream collects metrics from VMware vSphere, such as CPU and memory usage, CPU and memory reservation, and CPU and memory limit. +- **`virtualmachine`**: This data stream gathers virtual machine metrics from VMware vSphere, including performance statistics such as status, uptime, CPU usage, memory usage, and network activity. Note: - Users can monitor and see the log inside the ingested documents for vSphere in the `logs-*` index pattern from `Discover`, and for metrics, the index pattern is `metrics-*`. @@ -473,6 +474,130 @@ Please refer to the following [document](https://www.elastic.co/guide/en/ecs/cur | vsphere.datastore.write.bytes | Rate of writing data to the datastore. | long | byte | gauge | +### Datastore Cluster + +Datastore clusters in vSphere group multiple datastores for optimized management and automated load balancing, ensuring efficient storage utilization and simplified administration. + +An example event for `datastorecluster` looks as following: + +```json +{ + "@timestamp": "2024-09-22T05:28:46.315Z", + "agent": { + "ephemeral_id": "8b7e9ea4-0517-4e98-a795-b6fe529f4a2f", + "id": "7737279e-51e9-4d90-a0d0-2c12dc4446bf", + "name": "elastic-agent-23128", + "type": "metricbeat", + "version": "8.15.2" + }, + "data_stream": { + "dataset": "vsphere.datastorecluster", + "namespace": "65218", + "type": "metrics" + }, + "ecs": { + "version": "8.11.0" + }, + "elastic_agent": { + "id": "7737279e-51e9-4d90-a0d0-2c12dc4446bf", + "snapshot": true, + "version": "8.15.2" + }, + "event": { + "agent_id_status": "verified", + "dataset": "vsphere.datastorecluster", + "duration": 10772332, + "ingested": "2024-09-22T05:28:49Z", + "module": "vsphere" + }, + "host": { + "architecture": "x86_64", + "containerized": true, + "hostname": "elastic-agent-23128", + "id": "57723763cd1b4ff48e54a505de4ebe6c", + "ip": [ + "192.168.240.2", + "192.168.255.5" + ], + "mac": [ + "02-42-C0-A8-F0-02", + "02-42-C0-A8-FF-05" + ], + "name": "elastic-agent-23128", + "os": { + "codename": "focal", + "family": "debian", + "kernel": "3.10.0-1160.118.1.el7.x86_64", + "name": "Ubuntu", + "platform": "ubuntu", + "type": "linux", + "version": "20.04.6 LTS (Focal Fossa)" + } + }, + "metricset": { + "name": "datastorecluster", + "period": 20000 + }, + "service": { + "address": "https://svc-vsphere-metrics:8989/sdk", + "type": "vsphere" + }, + "tags": [ + "vsphere-datastorecluster" + ], + "vsphere": { + "datastorecluster": { + "capacity": { + "bytes": 0 + }, + "datastore": { + "count": 0 + }, + "free_space": { + "bytes": 0 + }, + "name": "DC0_POD0" + } + } +} +``` + +**ECS Field Reference** + +Please refer to the following [document](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html) for detailed information on ECS fields. + +**Exported fields** + +| Field | Description | Type | Unit | Metric Type | +|---|---|---|---|---| +| @timestamp | Event timestamp. | date | | | +| agent.id | | keyword | | | +| cloud.account.id | The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier. | keyword | | | +| cloud.availability_zone | Availability zone in which this host, resource, or service is located. | keyword | | | +| cloud.instance.id | Instance ID of the host machine. | keyword | | | +| cloud.provider | Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean. | keyword | | | +| cloud.region | Region in which this host, resource, or service is located. | keyword | | | +| container.id | Unique container id. | keyword | | | +| data_stream.dataset | Data stream dataset. | constant_keyword | | | +| data_stream.namespace | Data stream namespace. | constant_keyword | | | +| data_stream.type | Data stream type. | constant_keyword | | | +| event.dataset | Event dataset | constant_keyword | | | +| event.module | Event module | constant_keyword | | | +| host.name | Name of the host. It can contain what `hostname` returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use. | keyword | | | +| service.address | Address where data about this service was collected from. This should be a URI, network address (ipv4:port or [ipv6]:port) or a resource path (sockets). | keyword | | | +| vsphere.datastorecluster.capacity.bytes | Total capacity of this storage pod, in bytes. | long | byte | gauge | +| vsphere.datastorecluster.datastore.count | Number of datastores in the Datastore Cluster. | long | | gauge | +| vsphere.datastorecluster.datastore.names | List of all the Datastore names associated with the Datastore Cluster. | keyword | | | +| vsphere.datastorecluster.free_space.bytes | Total free space on this storage pod, in bytes. | long | byte | gauge | +| vsphere.datastorecluster.name | The Datastore Cluster name. | keyword | | | +| vsphere.datastorecluster.triggered_alarms.description | Description of the alarm. | keyword | | | +| vsphere.datastorecluster.triggered_alarms.entity_name | Name of the entity associated with the alarm. | keyword | | | +| vsphere.datastorecluster.triggered_alarms.id | Unique identifier for the alarm. | keyword | | | +| vsphere.datastorecluster.triggered_alarms.name | Name of the alarm. | keyword | | | +| vsphere.datastorecluster.triggered_alarms.status | Status of the alarm. | keyword | | | +| vsphere.datastorecluster.triggered_alarms.triggered_time | Time when the alarm was triggered. | date | | | + + ### Host This is the `host` data stream. This data stream collects host metrics from VMware vSphere, including performance statistics such as CPU usage, memory usage, disk I/O, and network activity.