Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add new OTel pipeline for scraping Kueue metrics and publishing to CloudWatch #1426

Merged
merged 34 commits into from
Nov 14, 2024
Merged

Add new OTel pipeline for scraping Kueue metrics and publishing to CloudWatch #1426

Show file tree
Hide file tree
Changes from 33 commits
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
71c57f6
add processor and translator for kueue metrics (#844)
rohangujarathi Oct 25, 2024
39ad36f
Add Kueue Metrics Translator Definitions (#847)
transtv Oct 25, 2024
5242cb4
add NodeName to kueueLabelFilter and fix unit tests (#862)
rohangujarathi Oct 25, 2024
4c9b6b6
drop ResourceMetricAttributes for kueue (#866)
rohangujarathi Oct 30, 2024
97581d4
change: separate kueue metrics into its own container insights pipeline
rvasahu-amazon Oct 30, 2024
41f1403
change: add separate emf exporter config for kueue metrics
rvasahu-amazon Oct 30, 2024
1a79607
refactor: remove unnecessary kueue config field from emf exporter
rvasahu-amazon Nov 6, 2024
edcf26b
change: add awscontainerinsightskueuereceiver as a new receiver type …
rvasahu-amazon Nov 6, 2024
71d8f8f
change: adjust kueue pipeline translator to use awscontainerinsightsk…
rvasahu-amazon Nov 6, 2024
beb8e4f
refactor: only create kueue pipeline translator if kueue_container_in…
rvasahu-amazon Nov 6, 2024
ea4c0f0
change: remove kueue flag from container insights receiver translator
rvasahu-amazon Nov 6, 2024
024f0b3
change: add default emf config for kueue
rvasahu-amazon Nov 6, 2024
bf8e923
test: update tocw sample configs to reflect a separate receiver for k…
rvasahu-amazon Nov 6, 2024
10492d6
change: add placeholder version for new receiver
rvasahu-amazon Nov 6, 2024
84f5d9f
doc: add licence to containerinsights/translators.go
rvasahu-amazon Nov 6, 2024
b5b0511
fix: add another placeholder for new awscontainerinsightskueuereceive…
rvasahu-amazon Nov 6, 2024
47db1d8
refactor: simplify emf exporter helper functions
rvasahu-amazon Nov 7, 2024
fef26a5
refactor: no need to always return nil error from containerinsights/t…
rvasahu-amazon Nov 7, 2024
481e9e9
refactor: move KueueContainerInsightsEnabled to containerinsights pac…
rvasahu-amazon Nov 7, 2024
7e54555
doc: remove commented code
rvasahu-amazon Nov 7, 2024
f387805
change: remove redundant helper function
rvasahu-amazon Nov 7, 2024
204f0a2
fix: move KueueContainerInsightsEnabled to common package
rvasahu-amazon Nov 7, 2024
db141b6
refactor: remove unneeded KueueContainerInsightsCall
rvasahu-amazon Nov 7, 2024
8515308
doc: lint fix to awsemf/kueue.go
rvasahu-amazon Nov 7, 2024
35ce89c
Merge branch 'main-kueue' into kueue-for-cw
rvasahu-amazon Nov 12, 2024
dfd17ce
build: fill placeholder for kueue receiver in go.mod
rvasahu-amazon Nov 12, 2024
2e1e1f3
fix: merge conflict for AWS EMF exporter translator
rvasahu-amazon Nov 12, 2024
3b80984
Merge remote-tracking branch 'public/main' into kueue-for-cw
rvasahu-amazon Nov 14, 2024
0d24e7d
doc: move kueue receiver dependency to require block with other recei…
rvasahu-amazon Nov 14, 2024
6ab1cea
build: update versions for otel dependencies
rvasahu-amazon Nov 14, 2024
f6fad06
build: re-enable github PR build actions
rvasahu-amazon Nov 14, 2024
ffaebc5
fix: add pipeline name back as a parameter
rvasahu-amazon Nov 14, 2024
8cf5193
fix: add new entitystore extension to kueue translated yamls
rvasahu-amazon Nov 14, 2024
fa60db6
build: update version for awsemfexporter
rvasahu-amazon Nov 14, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion .github/workflows/PR-build.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -136,4 +136,3 @@ jobs:
- name: Build
if: steps.cached_binaries.outputs.cache-hit != 'true' && needs.changes.outputs.build == 'true'
run: make amazon-cloudwatch-agent-${{ matrix.family }}

46 changes: 24 additions & 22 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,40 +7,41 @@ replace github.com/influxdata/telegraf => github.com/aws/telegraf v0.10.2-0.2024
// Replace with https://github.com/amazon-contributing/opentelemetry-collector-contrib, there are no requirements for all receivers/processors/exporters
// to be all replaced since there are some changes that will always be from upstream
replace (
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/awscloudwatchlogsexporter => github.com/amazon-contributing/opentelemetry-collector-contrib/exporter/awscloudwatchlogsexporter v0.0.0-20241104203805-20919412150d
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/awscloudwatchlogsexporter => github.com/amazon-contributing/opentelemetry-collector-contrib/exporter/awscloudwatchlogsexporter v0.0.0-20241114192544-cb251fe99cbb
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/awsemfexporter => github.com/amazon-contributing/opentelemetry-collector-contrib/exporter/awsemfexporter v0.0.0-20241024185216-f0b80d5c67a5
rvasahu-amazon marked this conversation as resolved.
Show resolved Hide resolved
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/awsxrayexporter => github.com/amazon-contributing/opentelemetry-collector-contrib/exporter/awsxrayexporter v0.0.0-20241104203805-20919412150d
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/awsxrayexporter => github.com/amazon-contributing/opentelemetry-collector-contrib/exporter/awsxrayexporter v0.0.0-20241114192544-cb251fe99cbb
)

replace github.com/open-telemetry/opentelemetry-collector-contrib/extension/awsproxy => github.com/amazon-contributing/opentelemetry-collector-contrib/extension/awsproxy v0.0.0-20241104203805-20919412150d
replace github.com/open-telemetry/opentelemetry-collector-contrib/extension/awsproxy => github.com/amazon-contributing/opentelemetry-collector-contrib/extension/awsproxy v0.0.0-20241114192544-cb251fe99cbb

replace (
github.com/open-telemetry/opentelemetry-collector-contrib/internal/aws/awsutil => github.com/amazon-contributing/opentelemetry-collector-contrib/internal/aws/awsutil v0.0.0-20241104203805-20919412150d
github.com/open-telemetry/opentelemetry-collector-contrib/internal/aws/containerinsight => github.com/amazon-contributing/opentelemetry-collector-contrib/internal/aws/containerinsight v0.0.0-20241104203805-20919412150d
github.com/open-telemetry/opentelemetry-collector-contrib/internal/aws/cwlogs => github.com/amazon-contributing/opentelemetry-collector-contrib/internal/aws/cwlogs v0.0.0-20241104203805-20919412150d
github.com/open-telemetry/opentelemetry-collector-contrib/internal/aws/k8s => github.com/amazon-contributing/opentelemetry-collector-contrib/internal/aws/k8s v0.0.0-20241104203805-20919412150d
github.com/open-telemetry/opentelemetry-collector-contrib/internal/aws/proxy => github.com/amazon-contributing/opentelemetry-collector-contrib/internal/aws/proxy v0.0.0-20241104203805-20919412150d
github.com/open-telemetry/opentelemetry-collector-contrib/internal/aws/xray => github.com/amazon-contributing/opentelemetry-collector-contrib/internal/aws/xray v0.0.0-20241104203805-20919412150d
github.com/open-telemetry/opentelemetry-collector-contrib/internal/coreinternal => github.com/amazon-contributing/opentelemetry-collector-contrib/internal/coreinternal v0.0.0-20241104203805-20919412150d
github.com/open-telemetry/opentelemetry-collector-contrib/internal/k8sconfig => github.com/amazon-contributing/opentelemetry-collector-contrib/internal/k8sconfig v0.0.0-20241104203805-20919412150d
github.com/open-telemetry/opentelemetry-collector-contrib/internal/kubelet => github.com/amazon-contributing/opentelemetry-collector-contrib/internal/kubelet v0.0.0-20241104203805-20919412150d
github.com/open-telemetry/opentelemetry-collector-contrib/internal/aws/awsutil => github.com/amazon-contributing/opentelemetry-collector-contrib/internal/aws/awsutil v0.0.0-20241114192544-cb251fe99cbb
github.com/open-telemetry/opentelemetry-collector-contrib/internal/aws/containerinsight => github.com/amazon-contributing/opentelemetry-collector-contrib/internal/aws/containerinsight v0.0.0-20241114192544-cb251fe99cbb
github.com/open-telemetry/opentelemetry-collector-contrib/internal/aws/cwlogs => github.com/amazon-contributing/opentelemetry-collector-contrib/internal/aws/cwlogs v0.0.0-20241114192544-cb251fe99cbb
github.com/open-telemetry/opentelemetry-collector-contrib/internal/aws/k8s => github.com/amazon-contributing/opentelemetry-collector-contrib/internal/aws/k8s v0.0.0-20241114192544-cb251fe99cbb
github.com/open-telemetry/opentelemetry-collector-contrib/internal/aws/proxy => github.com/amazon-contributing/opentelemetry-collector-contrib/internal/aws/proxy v0.0.0-20241114192544-cb251fe99cbb
github.com/open-telemetry/opentelemetry-collector-contrib/internal/aws/xray => github.com/amazon-contributing/opentelemetry-collector-contrib/internal/aws/xray v0.0.0-20241114192544-cb251fe99cbb
github.com/open-telemetry/opentelemetry-collector-contrib/internal/coreinternal => github.com/amazon-contributing/opentelemetry-collector-contrib/internal/coreinternal v0.0.0-20241114192544-cb251fe99cbb
github.com/open-telemetry/opentelemetry-collector-contrib/internal/k8sconfig => github.com/amazon-contributing/opentelemetry-collector-contrib/internal/k8sconfig v0.0.0-20241114192544-cb251fe99cbb
github.com/open-telemetry/opentelemetry-collector-contrib/internal/kubelet => github.com/amazon-contributing/opentelemetry-collector-contrib/internal/kubelet v0.0.0-20241114192544-cb251fe99cbb
)

replace (
// For clear resource attributes after copy functionality https://github.com/amazon-contributing/opentelemetry-collector-contrib/pull/148
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/resourcetotelemetry => github.com/amazon-contributing/opentelemetry-collector-contrib/pkg/resourcetotelemetry v0.0.0-20241104203805-20919412150d
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza => github.com/amazon-contributing/opentelemetry-collector-contrib/pkg/stanza v0.0.0-20241104203805-20919412150d
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/resourcetotelemetry => github.com/amazon-contributing/opentelemetry-collector-contrib/pkg/resourcetotelemetry v0.0.0-20241114192544-cb251fe99cbb
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza => github.com/amazon-contributing/opentelemetry-collector-contrib/pkg/stanza v0.0.0-20241114192544-cb251fe99cbb
// Replace with contrib to revert upstream change https://github.com/open-telemetry/opentelemetry-collector-contrib/pull/20519
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/translator/prometheus => github.com/amazon-contributing/opentelemetry-collector-contrib/pkg/translator/prometheus v0.0.0-20241104203805-20919412150d
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/translator/prometheus => github.com/amazon-contributing/opentelemetry-collector-contrib/pkg/translator/prometheus v0.0.0-20241114192544-cb251fe99cbb
)

replace github.com/open-telemetry/opentelemetry-collector-contrib/processor/resourcedetectionprocessor => github.com/amazon-contributing/opentelemetry-collector-contrib/processor/resourcedetectionprocessor v0.0.0-20241104203805-20919412150d
replace github.com/open-telemetry/opentelemetry-collector-contrib/processor/resourcedetectionprocessor => github.com/amazon-contributing/opentelemetry-collector-contrib/processor/resourcedetectionprocessor v0.0.0-20241114192544-cb251fe99cbb

replace (
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/awscontainerinsightreceiver => github.com/amazon-contributing/opentelemetry-collector-contrib/receiver/awscontainerinsightreceiver v0.0.0-20241104203805-20919412150d
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/awsxrayreceiver => github.com/amazon-contributing/opentelemetry-collector-contrib/receiver/awsxrayreceiver v0.0.0-20241104203805-20919412150d
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/jmxreceiver => github.com/amazon-contributing/opentelemetry-collector-contrib/receiver/jmxreceiver v0.0.0-20241104203805-20919412150d
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/prometheusreceiver => github.com/amazon-contributing/opentelemetry-collector-contrib/receiver/prometheusreceiver v0.0.0-20241104203805-20919412150d
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/awscontainerinsightreceiver => github.com/amazon-contributing/opentelemetry-collector-contrib/receiver/awscontainerinsightreceiver v0.0.0-20241114192544-cb251fe99cbb
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/awscontainerinsightskueuereceiver => github.com/amazon-contributing/opentelemetry-collector-contrib/receiver/awscontainerinsightskueuereceiver v0.0.0-20241114192544-cb251fe99cbb
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/awsxrayreceiver => github.com/amazon-contributing/opentelemetry-collector-contrib/receiver/awsxrayreceiver v0.0.0-20241114192544-cb251fe99cbb
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/jmxreceiver => github.com/amazon-contributing/opentelemetry-collector-contrib/receiver/jmxreceiver v0.0.0-20241114192544-cb251fe99cbb
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/prometheusreceiver => github.com/amazon-contributing/opentelemetry-collector-contrib/receiver/prometheusreceiver v0.0.0-20241114192544-cb251fe99cbb
)

// Temporary fix, pending PR https://github.com/shirou/gopsutil/pull/957
Expand Down Expand Up @@ -91,7 +92,7 @@ replace github.com/aws/aws-sdk-go => github.com/aws/aws-sdk-go v1.48.6
require (
github.com/BurntSushi/toml v1.3.2
github.com/Jeffail/gabs v1.4.0
github.com/amazon-contributing/opentelemetry-collector-contrib/extension/awsmiddleware v0.0.0-20241104203805-20919412150d
github.com/amazon-contributing/opentelemetry-collector-contrib/extension/awsmiddleware v0.0.0-20241114192544-cb251fe99cbb
github.com/aws/aws-sdk-go v1.53.11
github.com/aws/aws-sdk-go-v2/service/cloudwatch v1.30.2
github.com/bigkevmcd/go-configparser v0.0.0-20200217161103-d137835d2579
Expand Down Expand Up @@ -143,6 +144,7 @@ require (
github.com/open-telemetry/opentelemetry-collector-contrib/processor/tailsamplingprocessor v0.103.0
github.com/open-telemetry/opentelemetry-collector-contrib/processor/transformprocessor v0.103.0
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/awscontainerinsightreceiver v0.103.0
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/awscontainerinsightskueuereceiver v0.0.0-20241114192544-cb251fe99cbb
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/awsecscontainermetricsreceiver v0.103.0
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/awsxrayreceiver v0.103.0
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/filelogreceiver v0.103.0
Expand Down Expand Up @@ -230,7 +232,7 @@ require (
github.com/alecthomas/participle v0.4.1 // indirect
github.com/alecthomas/participle/v2 v2.1.1 // indirect
github.com/alecthomas/units v0.0.0-20231202071711-9a357b53e9c9 // indirect
github.com/amazon-contributing/opentelemetry-collector-contrib/override/aws v0.0.0-20241104203805-20919412150d // indirect
github.com/amazon-contributing/opentelemetry-collector-contrib/override/aws v0.0.0-20241114192544-cb251fe99cbb // indirect
github.com/antchfx/jsonquery v1.1.5 // indirect
github.com/antchfx/xmlquery v1.3.9 // indirect
github.com/antchfx/xpath v1.2.0 // indirect
Expand Down
Loading
Loading