-
Notifications
You must be signed in to change notification settings - Fork 561
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(instrumentation-aws-sdk): add gen ai metrics for bedrock #2771
base: main
Are you sure you want to change the base?
Conversation
@@ -0,0 +1,25 @@ | |||
/* |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I tried many attempts to have the meter-enabled instrumentation running for bedrock tests, including stuff like beforeEach(() => registerInstrumentationForTesting(); instrumentation.enable())
but nothing worked, so I ended up refactoring to make sure there's only a single instrumentation instance among all these tests (since mocha doesn't isolate tests). Let me know if there's any better approach
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yah, mocha running all tests in the same single process is a pain, IMHO. (Test frameworks often go this way though because speed.)
Could you add a block comment at the top of the file (after the license block) that explains basically what you've said above.
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #2771 +/- ##
==========================================
- Coverage 90.55% 89.51% -1.05%
==========================================
Files 168 172 +4
Lines 8076 8332 +256
Branches 1547 1593 +46
==========================================
+ Hits 7313 7458 +145
- Misses 763 874 +111
🚀 New features to boost your workflow:
|
private operationDuration!: Histogram; | ||
|
||
updateMetricInstruments(meter: Meter) { | ||
this.tokenUsage = meter.createHistogram('gen_ai.client.token.usage', { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: A comment linking to https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-metrics/#metric-gen_aiclienttokenusage could be helpful. Ditto for the other metric below.
|
||
this.tokenUsage.record(outputTokens, { | ||
...sharedMetricAttrs, | ||
[ATTR_GEN_AI_TOKEN_TYPE]: GEN_AI_TOKEN_TYPE_VALUE_COMPLETION, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
export const GEN_AI_TOKEN_TYPE_VALUE_COMPLETION = "output"
is weird: "COMPLETION" rather than "OUTPUT".
Ah, looks like that'll be fixed by https://github.com/open-telemetry/semantic-conventions/pull/1757/files#diff-0936f2641601c5b370b10dfb33c8f82d6d731a2663cf042eef2aa5114c720902R177
which is probably in semconv v1.31.0, for which the OTel JS semconv package doesn't have a release yet.
So eventually this'll be GEN_AI_TOKEN_TYPE_VALUE_OUTPUT
.
@@ -0,0 +1,25 @@ | |||
/* |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yah, mocha running all tests in the same single process is a pain, IMHO. (Test frameworks often go this way though because speed.)
Could you add a block comment at the top of the file (after the license block) that explains basically what you've said above.
registerInstrumentationTesting, | ||
} from '@opentelemetry/contrib-test-utils'; | ||
registerInstrumentationTesting(new AwsInstrumentation()); | ||
import { getTestSpans } from '@opentelemetry/contrib-test-utils'; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This import from contrib-test-utils doesn't absolutely need to be before ./load-instrumentation
does it?
|
||
export const instrumentation = new AwsInstrumentation(); | ||
export const metricReader = initMeterProvider(instrumentation); | ||
instrumentation._updateMetricInstruments(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This method is called in InstrumentationAbstract's constructor. Is it necessary to call it again?
The method is also called in initMeterProvider
(that calls instrumentation.setMeterProvider(meterProvider);
, which I think calls instr._updateMetricInstruments()
).
Which problem is this PR solving?
Short description of the changes
/cc @trentm @codefromthecrypt