Ci neuron #1

sam6134 · 2024-02-20T12:15:59Z

CR for initial Reviews on the NeuronScraper

New revision:

Rebased the GPU code, and made the decorator generic
Added a new decorator to add Pod-Attributes to the metric
Make both the Dcgm scraper and NeuronScraper implement the SimpleScraper

…cks package

update dcgm relabeling rules

receiver/awscontainerinsightreceiver/config.go

receiver/awscontainerinsightreceiver/internal/neuron/neuron_monitor_scraper.go

straussb · 2024-02-21T18:50:23Z

receiver/awscontainerinsightreceiver/internal/neuron/neuron_monitor_scraper.go

+			{
+				SourceLabels: model.LabelNames{"__address__"},
+				Regex:        relabel.MustNewRegexp("([^:]+)(?::\\d+)?"),
+				Replacement:  "${1}:8000",


My ignorance, why do you need this one?

Still missing a comment in the new code about why we do this.

Can be removed , I think it doesnt affect testing

receiver/awscontainerinsightreceiver/internal/neuron/neuron_monitor_scraper.go

receiver/awscontainerinsightreceiver/internal/neuron/neuron_monitor_scraper_test.go

receiver/awscontainerinsightreceiver/config.go

receiver/awscontainerinsightreceiver/internal/neuron/neuron_monitor_scraper_test.go

receiver/awscontainerinsightreceiver/internal/neuron/neuron_monitor_scraper.go

sam6134 · 2024-02-23T18:19:45Z

Made changes, if the initial structure for neuron looks fine I will make changes to make dcgm also use the simple class

straussb

Looks great, the code and tests are so much easier to read now! Please go ahead with converting DCGM exporter as well.

straussb · 2024-02-26T19:13:00Z

receiver/awscontainerinsightreceiver/internal/neuron/neuron_monitor_scraper_config.go

+				AttachMetadata: kubernetes.AttachMetadataConfig{
+					Node: true,
+				},


Sorry didn't notice this before, but do you need this? What effect does it have?

straussb · 2024-02-26T19:13:59Z

receiver/awscontainerinsightreceiver/internal/neuron/neuron_monitor_scraper.go

+			{
+				SourceLabels: model.LabelNames{"__address__"},
+				Regex:        relabel.MustNewRegexp("([^:]+)(?::\\d+)?"),
+				Replacement:  "${1}:8000",


Still missing a comment in the new code about why we do this.

receiver/awscontainerinsightreceiver/internal/neuron/neuron_monitor_scraper_test.go

straussb · 2024-02-26T19:20:04Z

receiver/awscontainerinsightreceiver/internal/neuron/neuron_monitor_scraper_config.go

+	jobName            = "containerInsightsNeuronMonitorScraper"
+)
+
+func GetNueronScrapeConfig(opts prometheusscraper.SimplePromethuesScraperOpts) *config.ScrapeConfig {


typo "Nueron" - same on GetNueronMetricRelabelConfigs below and in the package name itself.

receiver/awscontainerinsightreceiver/internal/prometheusscraper/simple_prometheus_scraper.go

straussb · 2024-02-26T19:51:06Z

receiver/awscontainerinsightreceiver/internal/neuron/neuron_monitor_scraper_test.go

+		scraper.Shutdown()
+	})
+
+	// wait for 2 scrapes, one initiated by us, another by the new scraper process


Still wondering about this one.

receiver/awscontainerinsightreceiver/internal/neuron/neuron_monitor_scraper_test.go

)

sam6134 · 2024-03-06T14:25:56Z

receiver/awscontainerinsightreceiver/internal/stores/kubeletutil/podresourcesclient.go

You can ignore this file, this will be merged in this PR - amazon-contributing#167

sam6134 · 2024-03-06T14:26:07Z

receiver/awscontainerinsightreceiver/internal/stores/podresourcesstore.go

sam6134 · 2024-03-06T14:26:19Z

receiver/awscontainerinsightreceiver/internal/stores/podresourcesstore_test.go

sam6134 · 2024-03-06T14:27:33Z

receiver/awscontainerinsightreceiver/internal/stores/staticPodResourcesRefresher.go

…emetry#24676) **Description:** The metadata.yml for the SSH check receiver currently documents a resource attribute containing the SSH endpoint but this is not emitted. This PR updates the receiver to include this resource attribute. **Link to tracking Issue:** open-telemetry#24441 **Testing:** Example collector config: ```yaml receivers: sshcheck: endpoint: 13.245.150.131:22 username: ec2-user key_file: /Users/dewald.dejager/.ssh/sandbox.pem collection_interval: 15s known_hosts: /Users/dewald.dejager/.ssh/known_hosts ignore_host_key: false resource_attributes: "ssh.endpoint": enabled: true exporters: logging: verbosity: detailed prometheus: endpoint: 0.0.0.0:8081 resource_to_telemetry_conversion: enabled: true service: pipelines: metrics: receivers: [sshcheck] exporters: [logging, prometheus] ``` The log output looks like this: ``` 2023-07-30T16:52:38.724+0200 info MetricsExporter {"kind": "exporter", "data_type": "metrics", "name": "logging", "resource metrics": 1, "metrics": 2, "data points": 2} 2023-07-30T16:52:38.724+0200 info ResourceMetrics #0 Resource SchemaURL: Resource attributes: -> ssh.endpoint: Str(13.245.150.131:22) ScopeMetrics #0 ScopeMetrics SchemaURL: InstrumentationScope otelcol/sshcheckreceiver 0.82.0-dev Metric #0 Descriptor: -> Name: sshcheck.duration -> Description: Measures the duration of SSH connection. -> Unit: ms -> DataType: Gauge NumberDataPoints #0 StartTimestamp: 2023-07-30 14:52:22.381672 +0000 UTC Timestamp: 2023-07-30 14:52:38.404003 +0000 UTC Value: 319 Metric #1 Descriptor: -> Name: sshcheck.status -> Description: 1 if the SSH client successfully connected, otherwise 0. -> Unit: 1 -> DataType: Sum -> IsMonotonic: false -> AggregationTemporality: Cumulative NumberDataPoints #0 StartTimestamp: 2023-07-30 14:52:22.381672 +0000 UTC Timestamp: 2023-07-30 14:52:38.404003 +0000 UTC Value: 1 ``` And the Prometheus metrics look like this: ``` # HELP sshcheck_duration Measures the duration of SSH connection. # TYPE sshcheck_duration gauge sshcheck_duration{ssh_endpoint="13.245.150.131:22"} 311 # HELP sshcheck_status 1 if the SSH client successfully connected, otherwise 0. # TYPE sshcheck_status gauge sshcheck_status{ssh_endpoint="13.245.150.131:22"} 1 ```

) **Description:** Adding command line argument `--status-code` to `telemetrygen traces`, which accepts `(Unset,Error,Ok)` (case sensitive) or the enum equivalent of `(0,1,2)`. Running ```shell telemetrygen traces --otlp-insecure --traces 1 --status-code 1 ``` against a minimal local collector yields ```txt 2023-07-29T21:27:57.862+0100 info ResourceSpans #0 Resource SchemaURL: https://opentelemetry.io/schemas/1.4.0 Resource attributes: -> service.name: Str(telemetrygen) ScopeSpans #0 ScopeSpans SchemaURL: InstrumentationScope telemetrygen Span #0 Trace ID : f6dc4be32c78b9999c69d504a79e68c1 Parent ID : 4e2cd6e0e90cf2ea ID : 20835413e32d26a5 Name : okey-dokey Kind : Server Start time : 2023-07-29 20:27:57.861602 +0000 UTC End time : 2023-07-29 20:27:57.861726 +0000 UTC Status code : Error Status message : Attributes: -> net.peer.ip: Str(1.2.3.4) -> peer.service: Str(telemetrygen-client) Span #1 Trace ID : f6dc4be32c78b9999c69d504a79e68c1 Parent ID : ID : 4e2cd6e0e90cf2ea Name : lets-go Kind : Client Start time : 2023-07-29 20:27:57.861584 +0000 UTC End time : 2023-07-29 20:27:57.861726 +0000 UTC Status code : Error Status message : Attributes: -> net.peer.ip: Str(1.2.3.4) -> peer.service: Str(telemetrygen-server) ``` and similarly (the string version) ```shell telemetrygen traces --otlp-insecure --traces 1 --status-code '"Ok"' ``` produces ```txt Resource SchemaURL: https://opentelemetry.io/schemas/1.4.0 Resource attributes: -> service.name: Str(telemetrygen) ScopeSpans #0 ScopeSpans SchemaURL: InstrumentationScope telemetrygen Span #0 Trace ID : dfd830da170acfe567b12f87685d7917 Parent ID : 8e15b390dc6a1ccc ID : 165c300130532072 Name : okey-dokey Kind : Server Start time : 2023-07-29 20:29:16.026965 +0000 UTC End time : 2023-07-29 20:29:16.027089 +0000 UTC Status code : Ok Status message : Attributes: -> net.peer.ip: Str(1.2.3.4) -> peer.service: Str(telemetrygen-client) Span #1 Trace ID : dfd830da170acfe567b12f87685d7917 Parent ID : ID : 8e15b390dc6a1ccc Name : lets-go Kind : Client Start time : 2023-07-29 20:29:16.026956 +0000 UTC End time : 2023-07-29 20:29:16.027089 +0000 UTC Status code : Ok Status message : Attributes: -> net.peer.ip: Str(1.2.3.4) -> peer.service: Str(telemetrygen-server) ``` The default is `Unset` which is the current behaviour. **Link to tracking Issue:** 24286 **Testing:** Added unit tests which covers both valid and invalid inputs. **Documentation:** Command line arguments are self documenting via the usage info in the flag. Co-authored-by: Pablo Baeyens <[email protected]>

Co-authored-by: matianjun1 <[email protected]>

straussb · 2024-03-07T03:04:33Z

internal/aws/proxy/server_test.go

-	assert.Contains(t, err.Error(), "failed to parse proxy URL")
+	assert.Contains(t, err.Error(), "invalid control character in URL")


Why this change?

Merge error will fix

receiver/awscontainerinsightreceiver/config.go

straussb · 2024-03-07T03:06:14Z

receiver/awscontainerinsightreceiver/internal/gpu/dcgmscraper.go

-	caFile                    = "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"
+	caFile                    = "/etc/amazon-cloudwatch-observability-agent-cert/tls-ca.crt"


I guess the changes in this file are coming from 9cb314e - can we merge it to ci-nvidia-gpu first so they don't show up as diffs in your PR?

straussb · 2024-03-07T03:10:12Z

receiver/awscontainerinsightreceiver/internal/gpu/dcgmscraper_config.go

+	GetInstanceID() string
+}
+
+func GetScraperConfig(hostInfoProvider hostInfoProvider) *config.ScrapeConfig {


If you're moving it here, shouldn't it be removed from internal/gpu/dcgmscraper.go?

Yes will delete it correct, again mad merge

straussb · 2024-03-07T03:47:41Z

receiver/awscontainerinsightreceiver/receiver.go

+	acir.podResourcesStore.AddResourceName("aws.amazon.com/neuroncore")
+	acir.podResourcesStore.AddResourceName("aws.amazon.com/neuron")
+	acir.podResourcesStore.AddResourceName("aws.amazon.com/neurondevice")


Can these share the constants from PodAttributesDecoratorConsumer?

straussb · 2024-03-07T03:49:03Z

receiver/awscontainerinsightreceiver/receiver.go

-	if acir.dcgmScraper != nil {
-		acir.dcgmScraper.GetMetrics() //nolint:errcheck
+	// if acir.dcgmScraper != nil {
+	// 	acir.dcgmScraper.GetMetrics() //nolint:errcheck
+	// }


straussb · 2024-03-07T03:49:13Z

receiver/awscontainerinsightreceiver/receiver.go

+	acir.settings.Logger.Info("We will start the Neuron Scraper")
+
+	if acir.neuronMonitorScraper != nil {
+		acir.settings.Logger.Info("Neuron Scraper is not NIL")


straussb · 2024-03-07T03:49:22Z

receiver/awscontainerinsightreceiver/receiver.go

 	}

+	acir.settings.Logger.Info("If this happened Neuron is started or not")


straussb · 2024-03-07T03:49:41Z

receiver/awsxrayreceiver/factory_test.go

-	t.Setenv(defaultRegionEnvName, mockRegion)
+	t.Setenv(regionEnvName, mockRegion)


Why all the changes in awsxrayreceiver?

movence and others added 11 commits February 6, 2024 09:02

add dcgm exporter scraper and move prometheus scraper test mock to mo…

8920742

…cks package

update emf exporter to handle GPU metrics with different metric types

e0f5bcd

update dcgm relabeling rules

remove custom logic in emf exporter

51fe859

update gpu flag comment

cee2244

remove comments and test codes

3debd28

add neuron monitor scraper

3d9de49

remove unused codes and rename scraper init funcs

9e70069

remove comments

b9a0e03

add changelog for gpu

52cd972

Merge branch 'ci-nvidia-gpu' into ci-neuron

eeb90e2

Update Scraper for new metrics

1747a50

straussb reviewed Feb 21, 2024

View reviewed changes

straussb reviewed Feb 22, 2024

View reviewed changes

receiver/awscontainerinsightreceiver/internal/neuron/neuron_monitor_scraper.go Outdated Show resolved Hide resolved

sam6134 added 3 commits February 23, 2024 18:00

Make Neuron Scraper extension for simple prometheus scraper

4f0e3e1

Minor fixes

c95f590

EnableFlag default to false

609198b

straussb reviewed Feb 26, 2024

View reviewed changes

movence and others added 2 commits February 26, 2024 15:54

add gpu metric consumer that uses k8s decorator for attributes

1444acd

Merge branch 'ci-nvidia-gpu' into ci-neuron

a89378e

straussb mentioned this pull request Feb 27, 2024

Add dcgm scraper to collect nvidia GPU metrics amazon-contributing/opentelemetry-collector-contrib#160

Merged

sam6134 and others added 6 commits March 1, 2024 13:38

testing support

d2c417d

debugging pod resources store

00a12dc

Add dcgm scraper to collect nvidia GPU metrics (amazon-contributing#160)

d3bf111

[internal/aws/proxy] Fix proxy server unit test (amazon-contributing#177

622200a

)

Adding default TLS to dcgmscraper (amazon-contributing#178)

9cb314e

add podresource scrapper and metric data printer

a821803

sam6134 force-pushed the ci-nvidia-gpu branch from 1444acd to 852fff7 Compare March 4, 2024 10:53

aditya-purang and others added 2 commits March 4, 2024 11:07

refactor logMd

83896ab

Merge branch 'ci-nvidia-gpu' into ci-neuron

69969dd

sam6134 added 10 commits March 4, 2024 12:12

Merge conflicts

3267653

More cleanups

164bd84

Remove unused imports

19223b1

Add decorator to neuron scraper

c65ad64

Merge branch 'ci-neuron' into docker-testing

1f60d15

Add decorator to add podResources

c6966db

Unified the decorator and added podResources decorator

05b1c75

Minor fixes

76e05aa

remove unused file

9e2f849

Making Dcgm implement SimplePrometheusScraper

3168bb2

sam6134 commented Mar 6, 2024

View reviewed changes

receiver/awscontainerinsightreceiver/internal/stores/podresourcesstore.go Outdated

Copy link

Collaborator Author

sam6134 Mar 6, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same

sam6134 commented Mar 6, 2024

View reviewed changes

receiver/awscontainerinsightreceiver/internal/stores/podresourcesstore_test.go Outdated

Copy link

Collaborator Author

sam6134 Mar 6, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same

sam6134 commented Mar 6, 2024

View reviewed changes

receiver/awscontainerinsightreceiver/internal/stores/staticPodResourcesRefresher.go Outdated

Copy link

Collaborator Author

sam6134 Mar 6, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same

Merge branch 'ci-neuron' into ci-neuron

0c8eac2

aditya-purang pushed a commit that referenced this pull request Mar 6, 2024

Add support AWS Embedded Metric Format Version 0 (#1)

52ade39

Co-authored-by: matianjun1 <[email protected]>

straussb reviewed Mar 7, 2024

View reviewed changes

sam6134 closed this Mar 7, 2024

straussb mentioned this pull request Mar 13, 2024

Add Neuron Scraper for scraping neuron monitor metrics amazon-contributing/opentelemetry-collector-contrib#184

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Ci neuron #1

Ci neuron #1

sam6134 commented Feb 20, 2024 •

edited

Loading

straussb Feb 21, 2024

straussb Feb 26, 2024

sam6134 Mar 1, 2024

sam6134 commented Feb 23, 2024

straussb left a comment

straussb Feb 26, 2024

straussb Feb 26, 2024

straussb Feb 26, 2024

straussb Feb 26, 2024

sam6134 Mar 6, 2024

sam6134 Mar 6, 2024

sam6134 Mar 6, 2024

sam6134 Mar 6, 2024

straussb Mar 7, 2024

sam6134 Mar 7, 2024

straussb Mar 7, 2024

straussb Mar 7, 2024

sam6134 Mar 7, 2024

straussb Mar 7, 2024

straussb Mar 7, 2024

straussb Mar 7, 2024

straussb Mar 7, 2024

straussb Mar 7, 2024

		assert.Contains(t, err.Error(), "failed to parse proxy URL")
		assert.Contains(t, err.Error(), "invalid control character in URL")

		caFile = "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"
		caFile = "/etc/amazon-cloudwatch-observability-agent-cert/tls-ca.crt"

		}

		acir.settings.Logger.Info("If this happened Neuron is started or not")

		t.Setenv(defaultRegionEnvName, mockRegion)
		t.Setenv(regionEnvName, mockRegion)

Ci neuron #1

Ci neuron #1

Conversation

sam6134 commented Feb 20, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sam6134 commented Feb 23, 2024

straussb left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sam6134 commented Feb 20, 2024 •

edited

Loading