[HA Agent] Add datadog.agent.ha_agent.integration_runs #33233

AlexandreYang · 2025-01-22T10:50:18Z

What does this PR do?

[HA Agent] Add datadog.agent.ha_agent.integration_runs

Motivation

datadog.agent.ha_agent.integration_runs for HA Agent feature OOTB dashboard, see example below.

It helps know which agent is running which integrations over time.

Describe how you validated your changes

Possible Drawbacks / Trade-offs

Additional Notes

agent-platform-auto-pr · 2025-01-22T11:16:30Z

Uncompressed package size comparison

Comparison with ancestor afb45257bea753d2e534ec9584bc1a3afe3fdee5

Diff per package

package	diff	status	size	ancestor	threshold
datadog-iot-agent-x86_64-rpm	0.01MB	⚠️	93.91MB	93.90MB	0.50MB
datadog-iot-agent-x86_64-suse	0.01MB	⚠️	93.91MB	93.90MB	0.50MB
datadog-iot-agent-amd64-deb	0.01MB	⚠️	93.84MB	93.84MB	0.50MB
datadog-iot-agent-arm64-deb	0.01MB	⚠️	89.89MB	89.89MB	0.50MB
datadog-iot-agent-aarch64-rpm	0.00MB	✅	89.96MB	89.96MB	0.50MB
datadog-agent-amd64-deb	0.00MB	✅	877.19MB	877.18MB	0.50MB
datadog-agent-x86_64-rpm	0.00MB	✅	886.93MB	886.92MB	0.50MB
datadog-agent-x86_64-suse	0.00MB	✅	886.93MB	886.92MB	0.50MB
datadog-agent-arm64-deb	0.00MB	✅	864.97MB	864.97MB	0.50MB
datadog-agent-aarch64-rpm	0.00MB	✅	874.69MB	874.69MB	0.50MB
datadog-heroku-agent-amd64-deb	0.00MB	✅	456.45MB	456.45MB	0.50MB
datadog-dogstatsd-amd64-deb	0.00MB	✅	59.02MB	59.02MB	0.50MB
datadog-dogstatsd-x86_64-rpm	0.00MB	✅	59.10MB	59.10MB	0.50MB
datadog-dogstatsd-x86_64-suse	0.00MB	✅	59.10MB	59.10MB	0.50MB
datadog-dogstatsd-arm64-deb	0.00MB	✅	56.51MB	56.51MB	0.50MB

Decision

⚠️ Warning

agent-platform-auto-pr · 2025-01-22T11:16:44Z

Test changes on VM

Use this command from test-infra-definitions to manually test this PR changes on a VM:

inv aws.create-vm --pipeline-id=54653946 --os-family=ubuntu

Note: This applies to commit b5001c3

cit-pr-commenter · 2025-01-22T11:41:38Z

Regression Detector

Regression Detector Results

Metrics dashboard
Target profiles
Run ID: d419b0e5-5edd-4ab8-bd47-38b43ec68b99

Baseline: afb4525
Comparison: b5001c3
Diff

Optimization Goals: ✅ No significant changes detected

Fine details of change detection per experiment

perf	experiment	goal	Δ mean %	Δ mean % CI	trials	links
➖	quality_gate_idle	memory utilization	+0.47	[+0.43, +0.52]	1	Logs bounds checks dashboard
➖	file_to_blackhole_1000ms_latency_linear_load	egress throughput	+0.17	[-0.30, +0.64]	1	Logs
➖	file_to_blackhole_500ms_latency	egress throughput	+0.08	[-0.70, +0.86]	1	Logs
➖	file_to_blackhole_1000ms_latency	egress throughput	+0.04	[-0.74, +0.83]	1	Logs
➖	file_to_blackhole_0ms_latency_http1	egress throughput	+0.03	[-0.85, +0.90]	1	Logs
➖	file_to_blackhole_0ms_latency_http2	egress throughput	+0.02	[-0.82, +0.87]	1	Logs
➖	uds_dogstatsd_to_api	ingress throughput	+0.00	[-0.28, +0.28]	1	Logs
➖	quality_gate_idle_all_features	memory utilization	+0.00	[-0.06, +0.07]	1	Logs bounds checks dashboard
➖	tcp_dd_logs_filter_exclude	ingress throughput	+0.00	[-0.02, +0.02]	1	Logs
➖	file_to_blackhole_300ms_latency	egress throughput	-0.00	[-0.63, +0.63]	1	Logs
➖	file_to_blackhole_0ms_latency	egress throughput	-0.01	[-0.90, +0.88]	1	Logs
➖	tcp_syslog_to_blackhole	ingress throughput	-0.06	[-0.13, +0.01]	1	Logs
➖	file_to_blackhole_100ms_latency	egress throughput	-0.06	[-0.74, +0.62]	1	Logs
➖	file_tree	memory utilization	-0.30	[-0.37, -0.23]	1	Logs
➖	quality_gate_logs	% cpu utilization	-1.25	[-4.26, +1.76]	1	Logs
➖	uds_dogstatsd_to_api_cpu	% cpu utilization	-1.45	[-2.31, -0.58]	1	Logs

Bounds Checks: ✅ Passed

perf	experiment	bounds_check_name	replicates_passed	links
✅	file_to_blackhole_0ms_latency	lost_bytes	10/10
✅	file_to_blackhole_0ms_latency	memory_usage	10/10
✅	file_to_blackhole_0ms_latency_http1	lost_bytes	10/10
✅	file_to_blackhole_0ms_latency_http1	memory_usage	10/10
✅	file_to_blackhole_0ms_latency_http2	lost_bytes	10/10
✅	file_to_blackhole_0ms_latency_http2	memory_usage	10/10
✅	file_to_blackhole_1000ms_latency	memory_usage	10/10
✅	file_to_blackhole_1000ms_latency_linear_load	memory_usage	10/10
✅	file_to_blackhole_100ms_latency	lost_bytes	10/10
✅	file_to_blackhole_100ms_latency	memory_usage	10/10
✅	file_to_blackhole_300ms_latency	lost_bytes	10/10
✅	file_to_blackhole_300ms_latency	memory_usage	10/10
✅	file_to_blackhole_500ms_latency	lost_bytes	10/10
✅	file_to_blackhole_500ms_latency	memory_usage	10/10
✅	quality_gate_idle	intake_connections	10/10	bounds checks dashboard
✅	quality_gate_idle	memory_usage	10/10	bounds checks dashboard
✅	quality_gate_idle_all_features	intake_connections	10/10	bounds checks dashboard
✅	quality_gate_idle_all_features	memory_usage	10/10	bounds checks dashboard
✅	quality_gate_logs	intake_connections	10/10
✅	quality_gate_logs	lost_bytes	10/10
✅	quality_gate_logs	memory_usage	10/10

Explanation

Confidence level: 90.00%
Effect size tolerance: |Δ mean %| ≥ 5.00%

Performance changes are noted in the perf column of each table:

✅ = significantly better comparison variant performance
❌ = significantly worse comparison variant performance
➖ = no significant change in performance

A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".

For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:

Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.
Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.
Its configuration does not mark it "erratic".

CI Pass/Fail Decision

✅ Passed. All Quality Gates passed.

quality_gate_idle_all_features, bounds check intake_connections: 10/10 replicas passed. Gate passed.
quality_gate_idle_all_features, bounds check memory_usage: 10/10 replicas passed. Gate passed.
quality_gate_logs, bounds check memory_usage: 10/10 replicas passed. Gate passed.
quality_gate_logs, bounds check intake_connections: 10/10 replicas passed. Gate passed.
quality_gate_logs, bounds check lost_bytes: 10/10 replicas passed. Gate passed.
quality_gate_idle, bounds check memory_usage: 10/10 replicas passed. Gate passed.
quality_gate_idle, bounds check intake_connections: 10/10 replicas passed. Gate passed.

This reverts commit 228fc9b.

AlexandreYang · 2025-01-23T08:50:35Z

pkg/collector/worker/worker.go

+			senderShouldCommit = true
+		}
+		if senderShouldCommit {


Note for reviewers: Using senderShouldCommit bool helps avoid sender.Commit() twice around line 208 and 164.

…313-integration-metric

AlexandreYang · 2025-02-01T20:47:07Z

/merge

dd-devflow · 2025-02-01T20:47:18Z

Devflow running: `/merge`

View all feedbacks in Devflow UI.

2025-02-01 20:47:17 UTC ℹ️ MergeQueue: waiting for PR to be ready

This merge request is not mergeable yet, because of pending checks/missing approvals. It will be added to the queue as soon as checks pass and/or get approvals.
Note: if you pushed new commits since the last approval, you may need additional approval.
You can remove it from the waiting list with /remove command.

2025-02-01 20:50:24 UTC ⚠️ MergeQueue: This merge request was unqueued

[email protected] unqueued this merge request

AlexandreYang · 2025-02-01T20:50:19Z

/remove

dd-devflow · 2025-02-01T20:50:23Z

Devflow running: `/remove`

View all feedbacks in Devflow UI.

2025-02-01 20:50:23 UTC ℹ️ Devflow: /remove

AlexandreYang · 2025-02-01T21:05:28Z

/remove

dd-devflow · 2025-02-01T21:05:32Z

Devflow running: `/remove`

View all feedbacks in Devflow UI.

2025-02-01 21:05:31 UTC ❌ Devflow: /remove

This merge request is not in the queue and can't be unqueued

To get help about command usage, write /remove --help

If you need support, contact us on Slack #devflow with those details!

hush-hush

Added a quick question but look good overall for agent-configuration.

hush-hush · 2025-02-03T13:52:43Z

pkg/collector/check/stats/stats.go

@@ -249,6 +257,9 @@ func (cs *Stats) Add(t time.Duration, err error, warnings []error, metricStats S
 		cs.TotalEventPlatformEvents[k] = cs.TotalEventPlatformEvents[k] + v
 		cs.EventPlatformEvents[k] = v
 	}
+	if haagent != nil && haagent.Enabled() && haagent.IsHaIntegration(cs.CheckName) {


Do you think we could include the haagent.Enabled() check in IsHaIntegration to simplify usage ?

Hum, I think it's not ideal since they mean different things. Can be misleading to have haagent.Enabled() check in IsHaIntegration() since a check can be flagged as HA, but HA Agent feature can be disabled.

AlexandreYang · 2025-02-03T14:53:31Z

/merge

dd-devflow · 2025-02-03T14:53:46Z

Devflow running: `/merge`

View all feedbacks in Devflow UI.

2025-02-03 14:53:45 UTC ℹ️ MergeQueue: pull request added to the queue

The median merge time in main is 27m.

2025-02-03 15:28:38 UTC ℹ️ MergeQueue: Readding this merge request to the queue because another merge request processed with yours failed. No action is needed from your side.

2025-02-03 16:00:33 UTC ℹ️ MergeQueue: This merge request was merged

coignetp added 7 commits January 20, 2025 15:02

Add config version tag in metric

6203116

Use fleet policies dir instead

027ba6a

Cleaner

b5634e7

Fix

7e52c66

Rename

b198953

Change to configid

bf0993b

Fixed tag

3c0593c

github-actions bot added medium review PR review might take time team/ndm-core labels Jan 22, 2025

AlexandreYang added 3 commits January 22, 2025 15:55

[HA Agent] add config_id tag to datadog.agent.ha_agent.running metric

609e43c

Revert "Revert "update e2e test to assert config_id""

b772211

This reverts commit 228fc9b.

rename agent_state to ha_agent_state

6a06cd3

AlexandreYang commented Jan 23, 2025

View reviewed changes

AlexandreYang changed the title ~~[HA Agent] draft integration metrics~~ [HA Agent] Add datadog.agent.ha_agent.integration_runs Jan 23, 2025

AlexandreYang added 2 commits January 23, 2025 10:04

add config_id tag

e66ec05

fix test

28d02d4

AlexandreYang changed the base branch from main to NDMII-3313-datadog-agent-ha-agent-running January 23, 2025 09:09

AlexandreYang changed the base branch from NDMII-3313-datadog-agent-ha-agent-running to main January 23, 2025 09:09

AlexandreYang changed the base branch from main to NDMII-3313-datadog-agent-ha-agent-running January 23, 2025 09:11

AlexandreYang changed the base branch from NDMII-3313-datadog-agent-ha-agent-running to main January 23, 2025 09:11

AlexandreYang force-pushed the NDMII-3313-integration-metric branch 2 times, most recently from af5e6f0 to ae78648 Compare January 23, 2025 09:23

[HA Agent] add integration metrics

a1e484a

AlexandreYang force-pushed the NDMII-3313-integration-metric branch from ae78648 to a1e484a Compare January 23, 2025 09:26

AlexandreYang changed the base branch from main to NDMII-3313-datadog-agent-ha-agent-running January 23, 2025 09:26

AlexandreYang marked this pull request as ready for review January 23, 2025 09:28

AlexandreYang requested a review from a team as a code owner January 23, 2025 09:28

Merge branch 'NDMII-3313-datadog-agent-ha-agent-running' into NDMII-3…

ed96e97

…313-integration-metric

AlexandreYang requested review from a team as code owners February 1, 2025 16:27

AlexandreYang removed request for a team February 1, 2025 16:29

AlexandreYang added 2 commits February 1, 2025 17:58

Merge branch 'main' into NDMII-3313-datadog-agent-ha-agent-running

e18aa87

Merge branch 'NDMII-3313-datadog-agent-ha-agent-running' into NDMII-3…

050dfa0

…313-integration-metric

Base automatically changed from NDMII-3313-datadog-agent-ha-agent-running to main February 1, 2025 19:06

AlexandreYang added changelog/no-changelog qa/done QA done before merge and regressions are covered by tests team/ndm-core and removed changelog/no-changelog qa/done QA done before merge and regressions are covered by tests team/ndm-core labels Feb 1, 2025

Merge branch 'main' into NDMII-3313-integration-metric

b5001c3

AlexandreYang requested review from a team and hush-hush February 1, 2025 21:10

hush-hush approved these changes Feb 3, 2025

View reviewed changes

github-actions bot added long review PR is complex, plan time to review it and removed medium review PR review might take time labels Feb 3, 2025

dd-mergequeue bot merged commit a85f505 into main Feb 3, 2025
231 checks passed

dd-mergequeue bot deleted the NDMII-3313-integration-metric branch February 3, 2025 16:00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[HA Agent] Add datadog.agent.ha_agent.integration_runs #33233

[HA Agent] Add datadog.agent.ha_agent.integration_runs #33233

AlexandreYang commented Jan 22, 2025 •

edited

Loading

agent-platform-auto-pr bot commented Jan 22, 2025 •

edited

Loading

agent-platform-auto-pr bot commented Jan 22, 2025 •

edited

Loading

cit-pr-commenter bot commented Jan 22, 2025 •

edited

Loading

Fine details of change detection per experiment

Bounds Checks: ✅ Passed

Explanation

AlexandreYang Jan 23, 2025 •

edited

Loading

AlexandreYang commented Feb 1, 2025

dd-devflow bot commented Feb 1, 2025 •

edited

Loading

AlexandreYang commented Feb 1, 2025

dd-devflow bot commented Feb 1, 2025

AlexandreYang commented Feb 1, 2025

dd-devflow bot commented Feb 1, 2025

hush-hush left a comment

hush-hush Feb 3, 2025

AlexandreYang Feb 3, 2025

AlexandreYang commented Feb 3, 2025

dd-devflow bot commented Feb 3, 2025 •

edited

Loading

[HA Agent] Add datadog.agent.ha_agent.integration_runs #33233

[HA Agent] Add datadog.agent.ha_agent.integration_runs #33233

Conversation

AlexandreYang commented Jan 22, 2025 • edited Loading

What does this PR do?

Motivation

Describe how you validated your changes

Possible Drawbacks / Trade-offs

Additional Notes

agent-platform-auto-pr bot commented Jan 22, 2025 • edited Loading

Uncompressed package size comparison

Decision

agent-platform-auto-pr bot commented Jan 22, 2025 • edited Loading

Test changes on VM

cit-pr-commenter bot commented Jan 22, 2025 • edited Loading

Regression Detector

Regression Detector Results

Optimization Goals: ✅ No significant changes detected

Fine details of change detection per experiment

Bounds Checks: ✅ Passed

Explanation

CI Pass/Fail Decision

AlexandreYang Jan 23, 2025 • edited Loading

Choose a reason for hiding this comment

AlexandreYang commented Feb 1, 2025

dd-devflow bot commented Feb 1, 2025 • edited Loading

Devflow running: /merge

AlexandreYang commented Feb 1, 2025

dd-devflow bot commented Feb 1, 2025

Devflow running: /remove

AlexandreYang commented Feb 1, 2025

dd-devflow bot commented Feb 1, 2025

Devflow running: /remove

hush-hush left a comment

Choose a reason for hiding this comment

hush-hush Feb 3, 2025

Choose a reason for hiding this comment

AlexandreYang Feb 3, 2025

Choose a reason for hiding this comment

AlexandreYang commented Feb 3, 2025

dd-devflow bot commented Feb 3, 2025 • edited Loading

Devflow running: /merge

AlexandreYang commented Jan 22, 2025 •

edited

Loading

agent-platform-auto-pr bot commented Jan 22, 2025 •

edited

Loading

agent-platform-auto-pr bot commented Jan 22, 2025 •

edited

Loading

cit-pr-commenter bot commented Jan 22, 2025 •

edited

Loading

AlexandreYang Jan 23, 2025 •

edited

Loading

dd-devflow bot commented Feb 1, 2025 •

edited

Loading

Devflow running: `/merge`

Devflow running: `/remove`

Devflow running: `/remove`

dd-devflow bot commented Feb 3, 2025 •

edited

Loading

Devflow running: `/merge`