Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(instrumentation-aws-sdk): add gen ai metrics for bedrock #2771

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

anuraaga
Copy link
Contributor

@anuraaga anuraaga commented Mar 27, 2025

Which problem is this PR solving?

  • Gen AI defines metrics that should be applied for operations like bedrock runtime

Short description of the changes

  • Allows propagating meter to service extensions
  • Passes start time to service extensions to allow them to compute operation duration as needed
  • Implement metrics for bedrock service extension
  • Refactor tests to share a single instrumentation instance

/cc @trentm @codefromthecrypt

@@ -0,0 +1,25 @@
/*
Copy link
Contributor Author

@anuraaga anuraaga Mar 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried many attempts to have the meter-enabled instrumentation running for bedrock tests, including stuff like beforeEach(() => registerInstrumentationForTesting(); instrumentation.enable()) but nothing worked, so I ended up refactoring to make sure there's only a single instrumentation instance among all these tests (since mocha doesn't isolate tests). Let me know if there's any better approach

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yah, mocha running all tests in the same single process is a pain, IMHO. (Test frameworks often go this way though because speed.)

Could you add a block comment at the top of the file (after the license block) that explains basically what you've said above.

Copy link

codecov bot commented Mar 27, 2025

Codecov Report

Attention: Patch coverage is 96.55172% with 1 line in your changes missing coverage. Please review.

Project coverage is 89.51%. Comparing base (9d84216) to head (00e7ac6).
Report is 2 commits behind head on main.

Files with missing lines Patch % Lines
...entelemetry-instrumentation-aws-sdk/src/aws-sdk.ts 87.50% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2771      +/-   ##
==========================================
- Coverage   90.55%   89.51%   -1.05%     
==========================================
  Files         168      172       +4     
  Lines        8076     8332     +256     
  Branches     1547     1593      +46     
==========================================
+ Hits         7313     7458     +145     
- Misses        763      874     +111     
Files with missing lines Coverage Δ
...entelemetry-instrumentation-aws-sdk/src/semconv.ts 100.00% <100.00%> (ø)
...ntation-aws-sdk/src/services/ServicesExtensions.ts 96.87% <100.00%> (+0.32%) ⬆️
...umentation-aws-sdk/src/services/bedrock-runtime.ts 98.11% <100.00%> (+0.43%) ⬆️
...entelemetry-instrumentation-aws-sdk/src/aws-sdk.ts 92.21% <87.50%> (-0.34%) ⬇️

... and 6 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

private operationDuration!: Histogram;

updateMetricInstruments(meter: Meter) {
this.tokenUsage = meter.createHistogram('gen_ai.client.token.usage', {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: A comment linking to https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-metrics/#metric-gen_aiclienttokenusage could be helpful. Ditto for the other metric below.


this.tokenUsage.record(outputTokens, {
...sharedMetricAttrs,
[ATTR_GEN_AI_TOKEN_TYPE]: GEN_AI_TOKEN_TYPE_VALUE_COMPLETION,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

export const GEN_AI_TOKEN_TYPE_VALUE_COMPLETION = "output" is weird: "COMPLETION" rather than "OUTPUT".
Ah, looks like that'll be fixed by https://github.com/open-telemetry/semantic-conventions/pull/1757/files#diff-0936f2641601c5b370b10dfb33c8f82d6d731a2663cf042eef2aa5114c720902R177
which is probably in semconv v1.31.0, for which the OTel JS semconv package doesn't have a release yet.
So eventually this'll be GEN_AI_TOKEN_TYPE_VALUE_OUTPUT.

@@ -0,0 +1,25 @@
/*
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yah, mocha running all tests in the same single process is a pain, IMHO. (Test frameworks often go this way though because speed.)

Could you add a block comment at the top of the file (after the license block) that explains basically what you've said above.

registerInstrumentationTesting,
} from '@opentelemetry/contrib-test-utils';
registerInstrumentationTesting(new AwsInstrumentation());
import { getTestSpans } from '@opentelemetry/contrib-test-utils';
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This import from contrib-test-utils doesn't absolutely need to be before ./load-instrumentation does it?


export const instrumentation = new AwsInstrumentation();
export const metricReader = initMeterProvider(instrumentation);
instrumentation._updateMetricInstruments();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This method is called in InstrumentationAbstract's constructor. Is it necessary to call it again?
The method is also called in initMeterProvider (that calls instrumentation.setMeterProvider(meterProvider);, which I think calls instr._updateMetricInstruments()).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants