Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Unable to export custom application metric using the datadog exporter #36905

Closed
cbright opened this issue Dec 19, 2024 · 3 comments
Closed

Unable to export custom application metric using the datadog exporter #36905

cbright opened this issue Dec 19, 2024 · 3 comments
Labels
data:metrics Metric related issues exporter/datadog Datadog components waiting for author

Comments

@cbright
Copy link

cbright commented Dec 19, 2024

Component(s)

exporter/datadog

What happened?

Description

I'm attempting to verify a custom open telemetry metric in my .net framework application using the .net SDK though the otel collector and into datadog exporter. I've verified that I can export traces and that the collect is receiving my metric using the debug exporter.

Steps to Reproduce

  1. Start collector inside docker.
  2. Start application
  3. Observe metric reporting through console exporter
  4. Observe metric collected inside of collector through debug exporter
  5. DataDog exporter does not export metric.

Expected Result

Custom application metric exported to DataDog

Actual Result

Custom metric is not exported to datadog and no error is reported.

Collector version

0.113.0

Environment information

Environment

container: otel/opentelemetry-collector-contrib:0.113.0 thru docker desktop
OS: windows 11

OpenTelemetry Collector configuration

receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318
      grpc:
        endpoint: 0.0.0.0:4317
  # The hostmetrics receiver is required to get correct infrastructure metrics in Datadog.
  # hostmetrics:
  #   collection_interval: 10s
  #   scrapers:
  #     paging:
  #       metrics:
  #         system.paging.utilization:
  #           enabled: true
  #     cpu:
  #       metrics:
  #         system.cpu.utilization:
  #           enabled: true
  #     disk:
  #     filesystem:
  #       metrics:
  #         system.filesystem.utilization:
  #           enabled: true
  #     load:
  #     memory:
  #     network:
  #     processes:
  # The prometheus receiver scrapes metrics needed for the OpenTelemetry Collector Dashboard.
  prometheus:
    config:
      scrape_configs:
      - job_name: 'otelcol'
        scrape_interval: 10s
        static_configs:
        - targets: ['0.0.0.0:8888']

  filelog:
    include_file_path: true
    poll_interval: 500ms
    include:
      - /var/log/**/*example*/*.log

processors:
  batch:
    send_batch_max_size: 1000
    send_batch_size: 100
    timeout: 10s

connectors:
  datadog/connector:

exporters:
  datadog:
    hostname: ******
    api:
      site: ${DataDog Site}
      key: ${DataDog Api Key}
    traces:
    metrics:
    host_metadata:
      enabled: true
      tags: ["env:localdev"]
  debug:
    verbosity: detailed
    sampling_initial: 5
    sampling_thereafter: 200

service:
  pipelines:
    metrics:
      receivers: [otlp, datadog/connector]
      processors: [batch]
      exporters: [datadog, debug]
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [datadog/connector, datadog, debug]
  telemetry:
    logs:
      level: "debug"

Log output

2024-12-19 16:06:29 2024-12-19T21:06:29.271Z    info    [email protected]/service.go:166 Setting up own telemetry...
2024-12-19 16:06:29 2024-12-19T21:06:29.271Z    info    telemetry/metrics.go:70 Serving metrics {"address": "localhost:8888", "metrics level": "Normal"}
2024-12-19 16:06:29 2024-12-19T21:06:29.271Z    info    builders/builders.go:26 Development component. May change in the future.        {"kind": "exporter", "data_type": "traces", "name": "debug"}
2024-12-19 16:06:29 2024-12-19T21:06:29.272Z    debug   builders/builders.go:24 Beta component. May change in the future.       {"kind": "exporter", "data_type": "metrics", "name": "datadog"}
2024-12-19 16:06:29 2024-12-19T21:06:29.272Z    debug   provider/provider.go:52 Trying out source provider      {"kind": "exporter", "data_type": "metrics", "name": "datadog", "provider": "config"}
2024-12-19 16:06:29 2024-12-19T21:06:29.272Z    debug   provider/provider.go:52 Trying out source provider      {"kind": "exporter", "data_type": "metrics", "name": "datadog", "provider": "azure"}
2024-12-19 16:06:29 2024-12-19T21:06:29.272Z    debug   provider/provider.go:52 Trying out source provider      {"kind": "exporter", "data_type": "metrics", "name": "datadog", "provider": "ecs"}
2024-12-19 16:06:29 2024-12-19T21:06:29.272Z    debug   provider/provider.go:52 Trying out source provider      {"kind": "exporter", "data_type": "metrics", "name": "datadog", "provider": "ec2"}
2024-12-19 16:06:29 2024-12-19T21:06:29.272Z    debug   provider/provider.go:52 Trying out source provider      {"kind": "exporter", "data_type": "metrics", "name": "datadog", "provider": "gcp"}
2024-12-19 16:06:29 2024-12-19T21:06:29.272Z    debug   provider/provider.go:52 Trying out source provider      {"kind": "exporter", "data_type": "metrics", "name": "datadog", "provider": "kubernetes"}
2024-12-19 16:06:29 2024-12-19T21:06:29.272Z    debug   provider/provider.go:52 Trying out source provider      {"kind": "exporter", "data_type": "metrics", "name": "datadog", "provider": "system"}
2024-12-19 16:06:29 2024-12-19T21:06:29.272Z    info    provider/provider.go:71 Resolved source {"kind": "exporter", "data_type": "metrics", "name": "datadog", "provider": "config", "source": {"Kind":"host","Identifier":"******"}}
2024-12-19 16:06:29 2024-12-19T21:06:29.272Z    debug   [email protected]/zaplogger.go:28        Stats writer initialized (climit=5 qsize=83)    {"kind": "exporter", "data_type": "metrics", "name": "datadog"}
2024-12-19 16:06:29 2024-12-19T21:06:29.273Z    debug   [email protected]/factory.go:280 Starting Datadog Trace-Agent StatsWriter        {"kind": "exporter", "data_type": "metrics", "name": "datadog"}
2024-12-19 16:06:29 2024-12-19T21:06:29.273Z    debug   ec2/ec2.go:58   EC2 Metadata not available      {"kind": "exporter", "data_type": "metrics", "name": "datadog"}
2024-12-19 16:06:29 2024-12-19T21:06:29.274Z    info    builders/builders.go:26 Development component. May change in the future.        {"kind": "exporter", "data_type": "metrics", "name": "debug"}
2024-12-19 16:06:29 2024-12-19T21:06:29.274Z    debug   builders/builders.go:24 Beta component. May change in the future.       {"kind": "processor", "name": "batch", "pipeline": "metrics"}
2024-12-19 16:06:29 2024-12-19T21:06:29.274Z    info    clientutil/api.go:41    Validating API key.     {"kind": "exporter", "data_type": "metrics", "name": "datadog"}
2024-12-19 16:06:29 2024-12-19T21:06:29.275Z    debug   builders/builders.go:24 Beta component. May change in the future.       {"kind": "connector", "name": "datadog/connector", "exporter_in_pipeline": "traces", "receiver_in_pipeline": "metrics"}
2024-12-19 16:06:29 2024-12-19T21:06:29.275Z    info    [email protected]/factory.go:62 Datadog connector using the native OTel API to ingest OTel spans and produce APM stats. To revert to the legacy processing pipeline, disable the feature gate        {"kind": "connector", "name": "datadog/connector", "exporter_in_pipeline": "traces", "receiver_in_pipeline": "metrics", "feature gate": "connector.datadogconnector.NativeIngest"}
2024-12-19 16:06:29 2024-12-19T21:06:29.275Z    info    [email protected]/connector_native.go:62        Building datadog connector for traces to metrics        {"kind": "connector", "name": "datadog/connector", "exporter_in_pipeline": "traces", "receiver_in_pipeline": "metrics"}
2024-12-19 16:06:29 2024-12-19T21:06:29.275Z    debug   builders/builders.go:24 Beta component. May change in the future.       {"kind": "exporter", "data_type": "traces", "name": "datadog"}
2024-12-19 16:06:29 2024-12-19T21:06:29.275Z    info    [email protected]/factory.go:359 Trace metrics are now disabled in the Datadog Exporter by default. To continue receiving Trace Metrics, configure the Datadog Connector or disable the feature gate. {"kind": "exporter", "data_type": "traces", "name": "datadog", "documentation": "https://docs.datadoghq.com/opentelemetry/guide/migration/", "feature gate ID": "exporter.datadogexporter.DisableAPMStats"}
2024-12-19 16:06:29 2024-12-19T21:06:29.275Z    info    [email protected]/zaplogger.go:38        Starting Agent with processor trace buffer of size 0    {"kind": "exporter", "data_type": "traces", "name": "datadog"}
2024-12-19 16:06:29 2024-12-19T21:06:29.275Z    debug   [email protected]/zaplogger.go:28        Stats writer initialized (climit=5 qsize=83)    {"kind": "exporter", "data_type": "traces", "name": "datadog"}
2024-12-19 16:06:29 2024-12-19T21:06:29.275Z    info    [email protected]/zaplogger.go:38        Receiver configured with 6 decoders and a timeout of 0ms        {"kind": "exporter", "data_type": "traces", "name": "datadog"}
2024-12-19 16:06:29 2024-12-19T21:06:29.277Z    info    [email protected]/zaplogger.go:38        Trace writer initialized (climit=5 qsize=1 compression=gzip)    {"kind": "exporter", "data_type": "traces", "name": "datadog"}
2024-12-19 16:06:29 2024-12-19T21:06:29.277Z    debug   [email protected]/zaplogger.go:23        HTTP Server is off: HTTPReceiver is disabled.   {"kind": "exporter", "data_type": "traces", "name": "datadog"}
2024-12-19 16:06:29 2024-12-19T21:06:29.277Z    debug   [email protected]/zaplogger.go:23        Debug server is disabled by config (apm_config.debug.port: 0).  {"kind": "exporter", "data_type": "traces", "name": "datadog"}
2024-12-19 16:06:29 2024-12-19T21:06:29.277Z    debug   [email protected]/zaplogger.go:23        Starting concentrator   {"kind": "exporter", "data_type": "traces", "name": "datadog"}
2024-12-19 16:06:29 2024-12-19T21:06:29.277Z    info    [email protected]/zaplogger.go:38        Processing Pipeline configured with 12 workers  {"kind": "exporter", "data_type": "traces", "name": "datadog"}
2024-12-19 16:06:29 2024-12-19T21:06:29.278Z    debug   builders/builders.go:24 Beta component. May change in the future.       {"kind": "processor", "name": "batch", "pipeline": "traces"}
2024-12-19 16:06:29 2024-12-19T21:06:29.278Z    info    clientutil/api.go:41    Validating API key.     {"kind": "exporter", "data_type": "traces", "name": "datadog"}
2024-12-19 16:06:29 2024-12-19T21:06:29.278Z    debug   builders/builders.go:24 Stable component.       {"kind": "receiver", "name": "otlp", "data_type": "traces"}
2024-12-19 16:06:29 2024-12-19T21:06:29.278Z    debug   builders/builders.go:24 Stable component.       {"kind": "receiver", "name": "otlp", "data_type": "metrics"}
2024-12-19 16:06:29 2024-12-19T21:06:29.279Z    info    [email protected]/service.go:238 Starting otelcol-contrib...     {"Version": "0.113.0", "NumCPU": 12}
2024-12-19 16:06:29 2024-12-19T21:06:29.279Z    info    extensions/extensions.go:39     Starting extensions...
2024-12-19 16:06:29 2024-12-19T21:06:29.279Z    warn    [email protected]/warning.go:40 Using the 0.0.0.0 address exposes this server to every network interface, which may facilitate Denial of Service attacks.    {"kind": "receiver", "name": "otlp", "data_type": "traces", "documentation": "https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks"}
2024-12-19 16:06:29 2024-12-19T21:06:29.279Z    info    [email protected]/server.go:684      [core] [Server #1]Server created        {"grpc_log": true}
2024-12-19 16:06:29 2024-12-19T21:06:29.279Z    info    [email protected]/otlp.go:112       Starting GRPC server    {"kind": "receiver", "name": "otlp", "data_type": "traces", "endpoint": "0.0.0.0:4317"}
2024-12-19 16:06:29 2024-12-19T21:06:29.279Z    warn    [email protected]/warning.go:40 Using the 0.0.0.0 address exposes this server to every network interface, which may facilitate Denial of Service attacks.    {"kind": "receiver", "name": "otlp", "data_type": "traces", "documentation": "https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks"}
2024-12-19 16:06:29 2024-12-19T21:06:29.279Z    info    [email protected]/otlp.go:169       Starting HTTP server    {"kind": "receiver", "name": "otlp", "data_type": "traces", "endpoint": "0.0.0.0:4318"}
2024-12-19 16:06:29 2024-12-19T21:06:29.280Z    info    [email protected]/server.go:880      [core] [Server #1 ListenSocket #2]ListenSocket created  {"grpc_log": true}
2024-12-19 16:06:29 2024-12-19T21:06:29.280Z    info    [email protected]/connector_native.go:91        Starting datadogconnector       {"kind": "connector", "name": "datadog/connector", "exporter_in_pipeline": "traces", "receiver_in_pipeline": "metrics"}
2024-12-19 16:06:29 2024-12-19T21:06:29.280Z    info    [email protected]/service.go:261 Everything is ready. Begin running and processing data.
2024-12-19 16:06:29 2024-12-19T21:06:29.280Z    debug   [email protected]/zaplogger.go:23        Starting concentrator   {"kind": "exporter", "data_type": "traces", "name": "datadog"}
2024-12-19 16:06:29 2024-12-19T21:06:29.543Z    info    clientutil/api.go:45    API key validation successful.  {"kind": "exporter", "data_type": "traces", "name": "datadog"}
2024-12-19 16:06:29 2024-12-19T21:06:29.546Z    info    clientutil/api.go:45    API key validation successful.  {"kind": "exporter", "data_type": "metrics", "name": "datadog"}
2024-12-19 16:06:49 2024-12-19T21:06:49.278Z    debug   [email protected]/zaplogger.go:28        Update oldestTs to 1734642390000000000  {"kind": "exporter", "data_type": "traces", "name": "datadog"}
2024-12-19 16:06:49 2024-12-19T21:06:49.281Z    debug   [email protected]/zaplogger.go:28        Update oldestTs to 1734642390000000000  {"kind": "exporter", "data_type": "traces", "name": "datadog"}
2024-12-19 16:06:59 2024-12-19T21:06:59.276Z    debug   [email protected]/zaplogger.go:28        Update oldestTs to 1734642400000000000  {"kind": "exporter", "data_type": "traces", "name": "datadog"}
2024-12-19 16:06:59 2024-12-19T21:06:59.280Z    debug   [email protected]/zaplogger.go:28        Update oldestTs to 1734642400000000000  {"kind": "exporter", "data_type": "traces", "name": "datadog"}
2024-12-19 16:07:09 2024-12-19T21:07:09.277Z    debug   [email protected]/zaplogger.go:28        Update oldestTs to 1734642410000000000  {"kind": "exporter", "data_type": "traces", "name": "datadog"}
2024-12-19 16:07:09 2024-12-19T21:07:09.279Z    debug   [email protected]/zaplogger.go:28        Update oldestTs to 1734642410000000000  {"kind": "exporter", "data_type": "traces", "name": "datadog"}
2024-12-19 16:07:19 2024-12-19T21:07:19.277Z    debug   [email protected]/zaplogger.go:28        Update oldestTs to 1734642420000000000  {"kind": "exporter", "data_type": "traces", "name": "datadog"}
2024-12-19 16:07:19 2024-12-19T21:07:19.280Z    debug   [email protected]/zaplogger.go:28        Update oldestTs to 1734642420000000000  {"kind": "exporter", "data_type": "traces", "name": "datadog"}
2024-12-19 16:07:19 2024-12-19T21:07:19.280Z    info    Metrics {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 1, "metrics": 2, "data points": 4}
2024-12-19 16:07:19 2024-12-19T21:07:19.280Z    info    ResourceMetrics #0
2024-12-19 16:07:19 Resource SchemaURL: 
2024-12-19 16:07:19 Resource attributes:
2024-12-19 16:07:19      -> service.name: Str(Definitive-ServiceBus)
2024-12-19 16:07:19      -> service.instance.id: Str(1578abe4-15ed-484d-8b94-dd4e334efe9c)
2024-12-19 16:07:19      -> telemetry.sdk.name: Str(opentelemetry)
2024-12-19 16:07:19      -> telemetry.sdk.language: Str(dotnet)
2024-12-19 16:07:19      -> telemetry.sdk.version: Str(1.8.1)
2024-12-19 16:07:19 ScopeMetrics #0
2024-12-19 16:07:19 ScopeMetrics SchemaURL: 
2024-12-19 16:07:19 InstrumentationScope OpenTelemetry.Instrumentation.Http 1.0.0.0
2024-12-19 16:07:19 Metric #0
2024-12-19 16:07:19 Descriptor:
2024-12-19 16:07:19      -> Name: http.client.request.duration
2024-12-19 16:07:19      -> Description: Duration of HTTP client requests.
2024-12-19 16:07:19      -> Unit: s
2024-12-19 16:07:19      -> DataType: Histogram
2024-12-19 16:07:19      -> AggregationTemporality: Cumulative
2024-12-19 16:07:19 HistogramDataPoints #0
2024-12-19 16:07:19 Data point attributes:
2024-12-19 16:07:19      -> http.request.method: Str(POST)
2024-12-19 16:07:19      -> http.response.status_code: Int(200)
2024-12-19 16:07:19      -> network.protocol.version: Str(1.1)
2024-12-19 16:07:19      -> server.address: Str(localhost)
2024-12-19 16:07:19      -> server.port: Int(8000)
2024-12-19 16:07:19      -> url.scheme: Str(http)
2024-12-19 16:07:19 StartTimestamp: 2024-12-19 19:46:15.2466738 +0000 UTC
2024-12-19 16:07:19 Timestamp: 2024-12-19 21:07:15.4339923 +0000 UTC
2024-12-19 16:07:19 Count: 15
2024-12-19 16:07:19 Sum: 1.055737
2024-12-19 16:07:19 Min: 0.012407
2024-12-19 16:07:19 Max: 0.187386
2024-12-19 16:07:19 ExplicitBounds #0: 0.005000
2024-12-19 16:07:19 ExplicitBounds #1: 0.010000
2024-12-19 16:07:19 ExplicitBounds #2: 0.025000
2024-12-19 16:07:19 ExplicitBounds #3: 0.050000
2024-12-19 16:07:19 ExplicitBounds #4: 0.075000
2024-12-19 16:07:19 ExplicitBounds #5: 0.100000
2024-12-19 16:07:19 ExplicitBounds #6: 0.250000
2024-12-19 16:07:19 ExplicitBounds #7: 0.500000
2024-12-19 16:07:19 ExplicitBounds #8: 0.750000
2024-12-19 16:07:19 ExplicitBounds #9: 1.000000
2024-12-19 16:07:19 ExplicitBounds #10: 2.500000
2024-12-19 16:07:19 ExplicitBounds #11: 5.000000
2024-12-19 16:07:19 ExplicitBounds #12: 7.500000
2024-12-19 16:07:19 ExplicitBounds #13: 10.000000
2024-12-19 16:07:19 Buckets #0, Count: 0
2024-12-19 16:07:19 Buckets #1, Count: 0
2024-12-19 16:07:19 Buckets #2, Count: 3
2024-12-19 16:07:19 Buckets #3, Count: 0
2024-12-19 16:07:19 Buckets #4, Count: 9
2024-12-19 16:07:19 Buckets #5, Count: 1
2024-12-19 16:07:19 Buckets #6, Count: 2
2024-12-19 16:07:19 Buckets #7, Count: 0
2024-12-19 16:07:19 Buckets #8, Count: 0
2024-12-19 16:07:19 Buckets #9, Count: 0
2024-12-19 16:07:19 Buckets #10, Count: 0
2024-12-19 16:07:19 Buckets #11, Count: 0
2024-12-19 16:07:19 Buckets #12, Count: 0
2024-12-19 16:07:19 Buckets #13, Count: 0
2024-12-19 16:07:19 Buckets #14, Count: 0
2024-12-19 16:07:19 HistogramDataPoints #1
2024-12-19 16:07:19 Data point attributes:
2024-12-19 16:07:19      -> http.request.method: Str(GET)
2024-12-19 16:07:19      -> http.response.status_code: Int(200)
2024-12-19 16:07:19      -> network.protocol.version: Str(1.1)
2024-12-19 16:07:19      -> server.address: Str(localhost)
2024-12-19 16:07:19      -> server.port: Int(4572)
2024-12-19 16:07:19      -> url.scheme: Str(http)
2024-12-19 16:07:19 StartTimestamp: 2024-12-19 19:46:15.2466738 +0000 UTC
2024-12-19 16:07:19 Timestamp: 2024-12-19 21:07:15.4339923 +0000 UTC
2024-12-19 16:07:19 Count: 2
2024-12-19 16:07:19 Sum: 0.036416
2024-12-19 16:07:19 Min: 0.017558
2024-12-19 16:07:19 Max: 0.018858
2024-12-19 16:07:19 ExplicitBounds #0: 0.005000
2024-12-19 16:07:19 ExplicitBounds #1: 0.010000
2024-12-19 16:07:19 ExplicitBounds #2: 0.025000
2024-12-19 16:07:19 ExplicitBounds #3: 0.050000
2024-12-19 16:07:19 ExplicitBounds #4: 0.075000
2024-12-19 16:07:19 ExplicitBounds #5: 0.100000
2024-12-19 16:07:19 ExplicitBounds #6: 0.250000
2024-12-19 16:07:19 ExplicitBounds #7: 0.500000
2024-12-19 16:07:19 ExplicitBounds #8: 0.750000
2024-12-19 16:07:19 ExplicitBounds #9: 1.000000
2024-12-19 16:07:19 ExplicitBounds #10: 2.500000
2024-12-19 16:07:19 ExplicitBounds #11: 5.000000
2024-12-19 16:07:19 ExplicitBounds #12: 7.500000
2024-12-19 16:07:19 ExplicitBounds #13: 10.000000
2024-12-19 16:07:19 Buckets #0, Count: 0
2024-12-19 16:07:19 Buckets #1, Count: 0
2024-12-19 16:07:19 Buckets #2, Count: 2
2024-12-19 16:07:19 Buckets #3, Count: 0
2024-12-19 16:07:19 Buckets #4, Count: 0
2024-12-19 16:07:19 Buckets #5, Count: 0
2024-12-19 16:07:19 Buckets #6, Count: 0
2024-12-19 16:07:19 Buckets #7, Count: 0
2024-12-19 16:07:19 Buckets #8, Count: 0
2024-12-19 16:07:19 Buckets #9, Count: 0
2024-12-19 16:07:19 Buckets #10, Count: 0
2024-12-19 16:07:19 Buckets #11, Count: 0
2024-12-19 16:07:19 Buckets #12, Count: 0
2024-12-19 16:07:19 Buckets #13, Count: 0
2024-12-19 16:07:19 Buckets #14, Count: 0
2024-12-19 16:07:19 HistogramDataPoints #2
2024-12-19 16:07:19 Data point attributes:
2024-12-19 16:07:19      -> http.request.method: Str(PUT)
2024-12-19 16:07:19      -> http.response.status_code: Int(200)
2024-12-19 16:07:19      -> network.protocol.version: Str(1.1)
2024-12-19 16:07:19      -> server.address: Str(localhost)
2024-12-19 16:07:19      -> server.port: Int(4572)
2024-12-19 16:07:19      -> url.scheme: Str(http)
2024-12-19 16:07:19 StartTimestamp: 2024-12-19 19:46:15.2466738 +0000 UTC
2024-12-19 16:07:19 Timestamp: 2024-12-19 21:07:15.4339923 +0000 UTC
2024-12-19 16:07:19 Count: 2
2024-12-19 16:07:19 Sum: 0.043939
2024-12-19 16:07:19 Min: 0.021098
2024-12-19 16:07:19 Max: 0.022841
2024-12-19 16:07:19 ExplicitBounds #0: 0.005000
2024-12-19 16:07:19 ExplicitBounds #1: 0.010000
2024-12-19 16:07:19 ExplicitBounds #2: 0.025000
2024-12-19 16:07:19 ExplicitBounds #3: 0.050000
2024-12-19 16:07:19 ExplicitBounds #4: 0.075000
2024-12-19 16:07:19 ExplicitBounds #5: 0.100000
2024-12-19 16:07:19 ExplicitBounds #6: 0.250000
2024-12-19 16:07:19 ExplicitBounds #7: 0.500000
2024-12-19 16:07:19 ExplicitBounds #8: 0.750000
2024-12-19 16:07:19 ExplicitBounds #9: 1.000000
2024-12-19 16:07:19 ExplicitBounds #10: 2.500000
2024-12-19 16:07:19 ExplicitBounds #11: 5.000000
2024-12-19 16:07:19 ExplicitBounds #12: 7.500000
2024-12-19 16:07:19 ExplicitBounds #13: 10.000000
2024-12-19 16:07:19 Buckets #0, Count: 0
2024-12-19 16:07:19 Buckets #1, Count: 0
2024-12-19 16:07:19 Buckets #2, Count: 2
2024-12-19 16:07:19 Buckets #3, Count: 0
2024-12-19 16:07:19 Buckets #4, Count: 0
2024-12-19 16:07:19 Buckets #5, Count: 0
2024-12-19 16:07:19 Buckets #6, Count: 0
2024-12-19 16:07:19 Buckets #7, Count: 0
2024-12-19 16:07:19 Buckets #8, Count: 0
2024-12-19 16:07:19 Buckets #9, Count: 0
2024-12-19 16:07:19 Buckets #10, Count: 0
2024-12-19 16:07:19 Buckets #11, Count: 0
2024-12-19 16:07:19 Buckets #12, Count: 0
2024-12-19 16:07:19 Buckets #13, Count: 0
2024-12-19 16:07:19 Buckets #14, Count: 0
2024-12-19 16:07:19 ScopeMetrics #1
2024-12-19 16:07:19 ScopeMetrics SchemaURL: 
2024-12-19 16:07:19 InstrumentationScope Defi.InititalDecisionTime 
2024-12-19 16:07:19 Metric #0
2024-12-19 16:07:19 Descriptor:
2024-12-19 16:07:19      -> Name: defi.initial_decision_time
2024-12-19 16:07:19      -> Description: The amount of time for defi to respond to the source system with an initial decision on the loan.
2024-12-19 16:07:19      -> Unit: seconds
2024-12-19 16:07:19      -> DataType: Histogram
2024-12-19 16:07:19      -> AggregationTemporality: Cumulative
2024-12-19 16:07:19 HistogramDataPoints #0
2024-12-19 16:07:19 Data point attributes:
2024-12-19 16:07:19      -> defi.applicants: Str(1)
2024-12-19 16:07:19      -> defi.appStatus: Str(AutoApprove)
2024-12-19 16:07:19      -> defi.clientId: Int(1)
2024-12-19 16:07:19 StartTimestamp: 2024-12-19 19:47:03.5721674 +0000 UTC
2024-12-19 16:07:19 Timestamp: 2024-12-19 21:07:15.4339923 +0000 UTC
2024-12-19 16:07:19 Count: 2
2024-12-19 16:07:19 Sum: 307.250000
2024-12-19 16:07:19 Min: 124.670000
2024-12-19 16:07:19 Max: 182.580000
2024-12-19 16:07:19 ExplicitBounds #0: 0.000000
2024-12-19 16:07:19 ExplicitBounds #1: 5.000000
2024-12-19 16:07:19 ExplicitBounds #2: 10.000000
2024-12-19 16:07:19 ExplicitBounds #3: 25.000000
2024-12-19 16:07:19 ExplicitBounds #4: 50.000000
2024-12-19 16:07:19 ExplicitBounds #5: 75.000000
2024-12-19 16:07:19 ExplicitBounds #6: 100.000000
2024-12-19 16:07:19 ExplicitBounds #7: 250.000000
2024-12-19 16:07:19 ExplicitBounds #8: 500.000000
2024-12-19 16:07:19 ExplicitBounds #9: 750.000000
2024-12-19 16:07:19 ExplicitBounds #10: 1000.000000
2024-12-19 16:07:19 ExplicitBounds #11: 2500.000000
2024-12-19 16:07:19 ExplicitBounds #12: 5000.000000
2024-12-19 16:07:19 ExplicitBounds #13: 7500.000000
2024-12-19 16:07:19 ExplicitBounds #14: 10000.000000
2024-12-19 16:07:19 Buckets #0, Count: 0
2024-12-19 16:07:19 Buckets #1, Count: 0
2024-12-19 16:07:19 Buckets #2, Count: 0
2024-12-19 16:07:19 Buckets #3, Count: 0
2024-12-19 16:07:19 Buckets #4, Count: 0
2024-12-19 16:07:19 Buckets #5, Count: 0
2024-12-19 16:07:19 Buckets #6, Count: 0
2024-12-19 16:07:19 Buckets #7, Count: 2
2024-12-19 16:07:19 Buckets #8, Count: 0
2024-12-19 16:07:19 Buckets #9, Count: 0
2024-12-19 16:07:19 Buckets #10, Count: 0
2024-12-19 16:07:19 Buckets #11, Count: 0
2024-12-19 16:07:19 Buckets #12, Count: 0
2024-12-19 16:07:19 Buckets #13, Count: 0
2024-12-19 16:07:19 Buckets #14, Count: 0
2024-12-19 16:07:19 Buckets #15, Count: 0
2024-12-19 16:07:19     {"kind": "exporter", "data_type": "metrics", "name": "debug"}
2024-12-19 16:07:19 2024-12-19T21:07:19.281Z    debug   [email protected]/metrics_exporter.go:231        exporting native Datadog payload        {"kind": "exporter", "data_type": "metrics", "name": "datadog", "metric": [{"metric":"otel.datadog_exporter.metrics.running","points":[{"timestamp":1734642439,"value":1}],"resources":[{"name":"*********","type":"host"}],"tags":["version:0.113.0","command:otelcol-contrib"],"type":3}]}
2024-12-19 16:07:19 2024-12-19T21:07:19.282Z    debug   gohai/gohai.go:37       Failed to retrieve filesystem metadata  {"kind": "exporter", "data_type": "metrics", "name": "datadog", "error": "df failed to collect filesystem data: exec: \"df\": executable file not found in $PATH"}
2024-12-19 16:07:19 2024-12-19T21:07:19.283Z    debug   gohai/gohai.go:62       Failed to retrieve platform metadata    {"kind": "exporter", "data_type": "metrics", "name": "datadog", "error": "exec: \"uname\": executable file not found in $PATH"}
2024-12-19 16:07:19 2024-12-19T21:07:19.796Z    debug   ec2/ec2.go:58   EC2 Metadata not available      {"kind": "exporter", "data_type": "metrics", "name": "datadog"}
2024-12-19 16:07:19 2024-12-19T21:07:19.797Z    debug   hostmetadata/metadata.go:142    Sending host metadata payload   {"kind": "exporter", "data_type": "metrics", "name": "datadog", "payload": {"meta":{"hostname":"********","socket-hostname":"*******"},"internalHostname":"********","otel_version":"0.113.0","agent-flavor":"otelcol-contrib","host-tags":{"otel":["env:localdev"]},"gohai":"{\"cpu\":{\"cache_size\":\"12288 KB\",\"cpu_cores\":\"6\",\"cpu_logical_processors\":\"12\",\"family\":\"6\",\"mhz\":\"2304.007\",\"model\":\"165\",\"model_name\":\"Intel(R) Core(TM) i5-10500T CPU @ 2.30GHz\",\"stepping\":\"3\",\"vendor_id\":\"GenuineIntel\"},\"filesystem\":null,\"memory\":{\"swap_total\":\"2097152kB\",\"total\":\"8019136kB\"},\"network\":{\"interfaces\":[{\"ipv4\":[\"*.*.*.*\"],\"ipv4-network\":\"*.*.*.*/*\",\"ipv6\":[],\"macaddress\":\"*:*:*:*:*:*\",\"name\":\"eth0\"}],\"ipaddress\":\"*.*.*.*\",\"macaddress\":\"*:*:*:*:*:*\"},\"platform\":null}","resources":{"processes":{"snaps":[[1734642439,[]]]},"meta":{"host":"*******"}}}}
2024-12-19 16:07:29 2024-12-19T21:07:29.276Z    debug   [email protected]/zaplogger.go:28        Update oldestTs to 1734642430000000000  {"kind": "exporter", "data_type": "traces", "name": "datadog"}
2024-12-19 16:07:29 2024-12-19T21:07:29.279Z    debug   [email protected]/zaplogger.go:28        Update oldestTs to 1734642430000000000  {"kind": "exporter", "data_type": "traces", "name": "datadog"}
2024-12-19 16:07:39 2024-12-19T21:07:39.276Z    debug   [email protected]/zaplogger.go:28        Update oldestTs to 1734642440000000000  {"kind": "exporter", "data_type": "traces", "name": "datadog"}
2024-12-19 16:07:39 2024-12-19T21:07:39.279Z    debug   [email protected]/zaplogger.go:28        Update oldestTs to 1734642440000000000  {"kind": "exporter", "data_type": "traces", "name": "datadog"}
2024-12-19 16:07:49 2024-12-19T21:07:49.276Z    debug   [email protected]/zaplogger.go:28        Update oldestTs to 1734642450000000000  {"kind": "exporter", "data_type": "traces", "name": "datadog"}
2024-12-19 16:07:49 2024-12-19T21:07:49.278Z    debug   [email protected]/zaplogger.go:28        Update oldestTs to 1734642450000000000  {"kind": "exporter", "data_type": "traces", "name": "datadog"}
2024-12-19 16:07:59 2024-12-19T21:07:59.275Z    debug   [email protected]/zaplogger.go:28        Update oldestTs to 1734642460000000000  {"kind": "exporter", "data_type": "traces", "name": "datadog"}
2024-12-19 16:07:59 2024-12-19T21:07:59.278Z    debug   [email protected]/zaplogger.go:28        Update oldestTs to 1734642460000000000  {"kind": "exporter", "data_type": "traces", "name": "datadog"}
2024-12-19 16:08:09 2024-12-19T21:08:09.276Z    debug   [email protected]/zaplogger.go:28        Update oldestTs to 1734642470000000000  {"kind": "exporter", "data_type": "traces", "name": "datadog"}
2024-12-19 16:08:09 2024-12-19T21:08:09.278Z    debug   [email protected]/zaplogger.go:28        Update oldestTs to 1734642470000000000  {"kind": "exporter", "data_type": "traces", "name": "datadog"}
2024-12-19 16:08:19 2024-12-19T21:08:19.275Z    debug   [email protected]/zaplogger.go:28        Update oldestTs to 1734642480000000000  {"kind": "exporter", "data_type": "traces", "name": "datadog"}
2024-12-19 16:08:19 2024-12-19T21:08:19.278Z    debug   [email protected]/zaplogger.go:28        Update oldestTs to 1734642480000000000  {"kind": "exporter", "data_type": "traces", "name": "datadog"}
2024-12-19 16:08:19 2024-12-19T21:08:19.282Z    info    Metrics {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 1, "metrics": 2, "data points": 4}

Additional context

No response
@cbright cbright added bug Something isn't working needs triage New item requiring triage labels Dec 19, 2024
@github-actions github-actions bot added the exporter/datadog Datadog components label Dec 19, 2024
@github-actions GitHub Actions
Copy link
Contributor

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

@songy23
Copy link
Member

songy23 commented Dec 19, 2024

@cbright From the logs it seems your metrics are cumulative metrics. You may want to follow https://docs.datadoghq.com/opentelemetry/guide/otlp_delta_temporality/ and have the SDK emit delta metrics instead. Delta works better with Datadog than cumulative.

@songy23 songy23 added waiting for author and removed bug Something isn't working needs triage New item requiring triage labels Dec 19, 2024
@mx-psi mx-psi added the data:metrics Metric related issues label Dec 20, 2024
@cbright
Copy link
Author

cbright commented Dec 20, 2024

@songy23 - Using delta over cumulative did resolve my issue. Thanks. I'm guessing the underlying issue was the following from the linked docs or something similar.

"Datadog might not send the first point it receives from a given timeseries if it cannot ensure this point is the true start of the timeseries. This may lead to missing points upon restarts."

@cbright cbright closed this as completed Dec 20, 2024
@github-actions github-actions bot mentioned this issue Dec 24, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
data:metrics Metric related issues exporter/datadog Datadog components waiting for author
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@cbright @mx-psi @songy23