Skip to content

Commit

Permalink
[Obs AI Assistant] Send custom telemetry event on Insight response (e…
Browse files Browse the repository at this point in the history
…lastic#202353)

## Summary

### Problem
There is no custom Telemetry event for Insights at the moment. Therefore
the clicks are differentiated based on the HTML of the elements of the
UI, however sometimes these clicks can come from clusters with a
connector or without a connector.

Because of this behaviour there is no way to differentiate which
clusters have a GenAI connector and which clusters don't.

### Solution
- Introduce a custom Telemetry event - `InsightResponse =
'observability_ai_assistant_insight_response'`
- Send this event when the LLM generates a response for an insight.

### Checklist

- [x] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
  • Loading branch information
viduni94 authored and CAWilson94 committed Dec 9, 2024
1 parent 05c83f5 commit 419433e
Show file tree
Hide file tree
Showing 4 changed files with 48 additions and 1 deletion.
Original file line number Diff line number Diff line change
Expand Up @@ -10,17 +10,27 @@ import { AssistantScope } from '@kbn/ai-assistant-common';
import type { Message } from '../../common';
import { chatFeedbackEventSchema, ChatFeedback } from './schemas/chat_feedback';
import { insightFeedbackEventSchema, InsightFeedback } from './schemas/insight_feedback';
import { insightResponseEventSchema, InsightResponse } from './schemas/insight_response';
import { userSentPromptEventSchema } from './schemas/user_sent_prompt';
import { ObservabilityAIAssistantTelemetryEventType } from './telemetry_event_type';

const schemas = [chatFeedbackEventSchema, insightFeedbackEventSchema, userSentPromptEventSchema];
const schemas = [
chatFeedbackEventSchema,
insightFeedbackEventSchema,
userSentPromptEventSchema,
insightResponseEventSchema,
];

export type TelemetryEventTypeWithPayload =
| { type: ObservabilityAIAssistantTelemetryEventType.ChatFeedback; payload: ChatFeedback }
| { type: ObservabilityAIAssistantTelemetryEventType.InsightFeedback; payload: InsightFeedback }
| {
type: ObservabilityAIAssistantTelemetryEventType.UserSentPromptInChat;
payload: Message & { scopes: AssistantScope[] };
}
| {
type: ObservabilityAIAssistantTelemetryEventType.InsightResponse;
payload: InsightResponse;
};

export const registerTelemetryEventTypes = (analytics: AnalyticsServiceSetup) => {
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/

import type { EventTypeOpts } from '@kbn/core/public';
import { ObservabilityAIAssistantTelemetryEventType } from '../telemetry_event_type';

export interface InsightResponse {
'@timestamp': string;
}

export const insightResponseEventSchema: EventTypeOpts<InsightResponse> = {
eventType: ObservabilityAIAssistantTelemetryEventType.InsightResponse,
schema: {
'@timestamp': {
type: 'text',
_meta: {
description: 'The timestamp of the last response from the LLM.',
},
},
},
};
Original file line number Diff line number Diff line change
Expand Up @@ -9,4 +9,5 @@ export enum ObservabilityAIAssistantTelemetryEventType {
ChatFeedback = 'observability_ai_assistant_chat_feedback',
InsightFeedback = 'observability_ai_assistant_insight_feedback',
UserSentPromptInChat = 'observability_ai_assistant_user_sent_prompt_in_chat',
InsightResponse = 'observability_ai_assistant_insight_response',
}
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,17 @@ function ChatContent({
next(initialMessagesRef.current);
}, [next]);

useEffect(() => {
if (state !== ChatState.Loading && lastAssistantResponse) {
chatService.sendAnalyticsEvent({
type: ObservabilityAIAssistantTelemetryEventType.InsightResponse,
payload: {
'@timestamp': lastAssistantResponse['@timestamp'],
},
});
}
}, [state, lastAssistantResponse, chatService]);

return (
<>
<MessagePanel
Expand Down

0 comments on commit 419433e

Please sign in to comment.