Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: support sum + count aggregation for histogram metrics + merge histogram metric name #481

Open
wants to merge 14 commits into
base: main
Choose a base branch
from
Open

feat: support sum + count aggregation for histogram metrics + merge histogram metric name #481

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
76745fd
fix: parse histogram metric name + tags
wrn14897 Jul 24, 2024
3c4d138
feat: query histogram sum and count metrics
wrn14897 Jul 24, 2024
18053a5
docs: add changeset
wrn14897 Jul 24, 2024
76e3c01
fix: histogram metric name in tests
wrn14897 Aug 7, 2024
21eb158
fix: select last value (sum and count are both monotonic)
wrn14897 Aug 7, 2024
ad7b7c8
test: add histogram sum + count tests
wrn14897 Aug 7, 2024
d565eab
feat: histogram metric name backward compatibility
wrn14897 Aug 7, 2024
1ca0858
docs: update changeset
wrn14897 Aug 7, 2024
8155c2c
test: update endTime
wrn14897 Aug 7, 2024
fa9355c
feat: suppert histogram sum + count rate metrics
wrn14897 Aug 7, 2024
7900d8a
test: sum + count rate functions
wrn14897 Aug 7, 2024
dc09603
feat: histogram metric should be rate only
wrn14897 Aug 7, 2024
e2a9809
Merge branch 'main' into warren/query-histogram-sum-count-metrics
wrn14897 Aug 27, 2024
175f8c3
Merge branch 'main' into warren/query-histogram-sum-count-metrics
wrn14897 Aug 28, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions .changeset/chatty-maps-march.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
---
'@hyperdx/api': minor
'@hyperdx/app': minor
---

feat: support sum + count aggregation for histogram metrics + merge histogram
metric names
239 changes: 226 additions & 13 deletions packages/api/src/clickhouse/__tests__/clickhouseGetMultiSeriesChart.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -117,7 +117,41 @@ describe('clickhouse - getMultiSeriesChart', () => {
),
clickhouse.bulkInsertTeamMetricStream(
buildMetricSeries({
name: 'test.two_timestamps_lower_bound',
name: 'test.three_timestamps_sum',
tags: { host: 'test2', runId },
data_type: clickhouse.MetricsDataType.Histogram,
is_monotonic: false,
is_delta: false,
unit: '',
points: [
{ value: 0, timestamp: now },
{ value: 1, timestamp: now + ms('1m') },
{ value: 2, timestamp: now + ms('1m') },
{ value: 3, timestamp: now + ms('2m') },
],
team_id: teamId,
}),
),
clickhouse.bulkInsertTeamMetricStream(
buildMetricSeries({
name: 'test.three_timestamps_count',
tags: { host: 'test2', runId },
data_type: clickhouse.MetricsDataType.Histogram,
is_monotonic: false,
is_delta: false,
unit: '',
points: [
{ value: 0, timestamp: now },
{ value: 1, timestamp: now + ms('1m') },
{ value: 2, timestamp: now + ms('1m') },
{ value: 3, timestamp: now + ms('2m') },
],
team_id: teamId,
}),
),
clickhouse.bulkInsertTeamMetricStream(
buildMetricSeries({
name: 'test.two_timestamps_lower_bound_bucket',
tags: { host: 'test2', runId },
data_type: clickhouse.MetricsDataType.Histogram,
is_monotonic: false,
Expand All @@ -137,7 +171,7 @@ describe('clickhouse - getMultiSeriesChart', () => {
),
clickhouse.bulkInsertTeamMetricStream(
buildMetricSeries({
name: 'test.two_timestamps_lower_bound_inf',
name: 'test.two_timestamps_lower_bound_inf_bucket',
tags: { host: 'test2', runId },
data_type: clickhouse.MetricsDataType.Histogram,
is_monotonic: false,
Expand All @@ -157,7 +191,7 @@ describe('clickhouse - getMultiSeriesChart', () => {
),
clickhouse.bulkInsertTeamMetricStream(
buildMetricSeries({
name: 'test.two_timestamps_higher_bound',
name: 'test.two_timestamps_higher_bound_bucket',
tags: { host: 'test2', runId },
data_type: clickhouse.MetricsDataType.Histogram,
is_monotonic: false,
Expand All @@ -177,7 +211,7 @@ describe('clickhouse - getMultiSeriesChart', () => {
),
clickhouse.bulkInsertTeamMetricStream(
buildMetricSeries({
name: 'test.two_timestamps_higher_bound_inf',
name: 'test.two_timestamps_higher_bound_inf_bucket',
tags: { host: 'test2', runId },
data_type: clickhouse.MetricsDataType.Histogram,
is_monotonic: false,
Expand All @@ -197,7 +231,7 @@ describe('clickhouse - getMultiSeriesChart', () => {
),
clickhouse.bulkInsertTeamMetricStream(
buildMetricSeries({
name: 'test.two_timestamps_zero_offset',
name: 'test.two_timestamps_zero_offset_bucket',
tags: { host: 'test2', runId },
data_type: clickhouse.MetricsDataType.Histogram,
is_monotonic: false,
Expand All @@ -217,7 +251,7 @@ describe('clickhouse - getMultiSeriesChart', () => {
),
clickhouse.bulkInsertTeamMetricStream(
buildMetricSeries({
name: 'test.two_timestamps_non_zero_offset',
name: 'test.two_timestamps_non_zero_offset_bucket',
tags: { host: 'test2', runId },
data_type: clickhouse.MetricsDataType.Histogram,
is_monotonic: false,
Expand All @@ -237,7 +271,7 @@ describe('clickhouse - getMultiSeriesChart', () => {
),
clickhouse.bulkInsertTeamMetricStream(
buildMetricSeries({
name: 'test.three_timestamps',
name: 'test.three_timestamps_bucket',
tags: { host: 'test2', runId },
data_type: clickhouse.MetricsDataType.Histogram,
is_monotonic: false,
Expand All @@ -261,7 +295,7 @@ describe('clickhouse - getMultiSeriesChart', () => {
),
clickhouse.bulkInsertTeamMetricStream(
buildMetricSeries({
name: 'test.three_timestamps_group_by',
name: 'test.three_timestamps_group_by_bucket',
tags: { host: 'host-a', region: 'region-a' },
data_type: clickhouse.MetricsDataType.Histogram,
is_monotonic: false,
Expand All @@ -285,7 +319,7 @@ describe('clickhouse - getMultiSeriesChart', () => {
),
clickhouse.bulkInsertTeamMetricStream(
buildMetricSeries({
name: 'test.three_timestamps_group_by',
name: 'test.three_timestamps_group_by_bucket',
tags: { host: 'host-b', region: 'region-a' },
data_type: clickhouse.MetricsDataType.Histogram,
is_monotonic: false,
Expand All @@ -309,7 +343,7 @@ describe('clickhouse - getMultiSeriesChart', () => {
),
clickhouse.bulkInsertTeamMetricStream(
buildMetricSeries({
name: 'test.four_timestamps',
name: 'test.four_timestamps_bucket',
tags: { host: 'test2', runId },
data_type: clickhouse.MetricsDataType.Histogram,
is_monotonic: false,
Expand All @@ -336,7 +370,7 @@ describe('clickhouse - getMultiSeriesChart', () => {
),
clickhouse.bulkInsertTeamMetricStream(
buildMetricSeries({
name: 'test.counter_reset_with_inconsistent_bucket_resolution',
name: 'test.counter_reset_with_inconsistent_bucket_resolution_bucket',
tags: { host: 'test2', runId },
data_type: clickhouse.MetricsDataType.Histogram,
is_monotonic: false,
Expand Down Expand Up @@ -367,7 +401,7 @@ describe('clickhouse - getMultiSeriesChart', () => {
),
clickhouse.bulkInsertTeamMetricStream([
...buildMetricSeries({
name: 'test.counter_might_reset',
name: 'test.counter_might_reset_bucket',
tags: { host: 'host-a', region: 'region-a' },
data_type: clickhouse.MetricsDataType.Histogram,
is_monotonic: false,
Expand Down Expand Up @@ -397,7 +431,7 @@ describe('clickhouse - getMultiSeriesChart', () => {
team_id: teamId,
}),
...buildMetricSeries({
name: 'test.counter_might_reset',
name: 'test.counter_might_reset_bucket',
tags: { host: 'host-b', region: 'region-a' },
data_type: clickhouse.MetricsDataType.Histogram,
is_monotonic: false,
Expand Down Expand Up @@ -839,6 +873,185 @@ Array [
`);
});

it('three_timestamps histogram (sum + count)', async () => {
const [sumData, sumRateData, countData, countRateData] = await Promise.all([
(
await clickhouse.getMultiSeriesChart({
series: [
{
type: 'time',
table: 'metrics',
aggFn: clickhouse.AggFn.Sum,
field: 'test.three_timestamps',
where: `runId:${runId}`,
groupBy: [],
metricDataType: clickhouse.MetricsDataType.Histogram,
},
],
tableVersion: undefined,
teamId,
startTime: now,
endTime: now + ms('3m'),
granularity: '1 minute',
maxNumGroups: 20,
seriesReturnType: clickhouse.SeriesReturnType.Column,
})
).data.map(d => {
return _.pick(d, ['group', 'series_0.data', 'ts_bucket']);
}),
(
await clickhouse.getMultiSeriesChart({
series: [
{
type: 'time',
table: 'metrics',
aggFn: clickhouse.AggFn.SumRate,
field: 'test.three_timestamps',
where: `runId:${runId}`,
groupBy: [],
metricDataType: clickhouse.MetricsDataType.Histogram,
},
],
tableVersion: undefined,
teamId,
startTime: now,
endTime: now + ms('3m'),
granularity: '1 minute',
maxNumGroups: 20,
seriesReturnType: clickhouse.SeriesReturnType.Column,
})
).data.map(d => {
return _.pick(d, ['group', 'series_0.data', 'ts_bucket']);
}),
(
await clickhouse.getMultiSeriesChart({
series: [
{
type: 'time',
table: 'metrics',
aggFn: clickhouse.AggFn.Count,
field: 'test.three_timestamps',
where: `runId:${runId}`,
groupBy: [],
metricDataType: clickhouse.MetricsDataType.Histogram,
},
],
tableVersion: undefined,
teamId,
startTime: now,
endTime: now + ms('3m'),
granularity: '1 minute',
maxNumGroups: 20,
seriesReturnType: clickhouse.SeriesReturnType.Column,
})
).data.map(d => {
return _.pick(d, ['group', 'series_0.data', 'ts_bucket']);
}),
(
await clickhouse.getMultiSeriesChart({
series: [
{
type: 'time',
table: 'metrics',
aggFn: clickhouse.AggFn.CountRate,
field: 'test.three_timestamps',
where: `runId:${runId}`,
groupBy: [],
metricDataType: clickhouse.MetricsDataType.Histogram,
},
],
tableVersion: undefined,
teamId,
startTime: now,
endTime: now + ms('3m'),
granularity: '1 minute',
maxNumGroups: 20,
seriesReturnType: clickhouse.SeriesReturnType.Column,
})
).data.map(d => {
return _.pick(d, ['group', 'series_0.data', 'ts_bucket']);
}),
]);

expect(sumData).toMatchInlineSnapshot(`
Array [
Object {
"group": Array [],
"series_0.data": 0,
"ts_bucket": 1641340800,
},
Object {
"group": Array [],
"series_0.data": 2,
"ts_bucket": 1641340860,
},
Object {
"group": Array [],
"series_0.data": 3,
"ts_bucket": 1641340920,
},
]
`);
expect(sumRateData).toMatchInlineSnapshot(`
Array [
Object {
"group": Array [],
"series_0.data": 0,
"ts_bucket": 1641340800,
},
Object {
"group": Array [],
"series_0.data": 2,
"ts_bucket": 1641340860,
},
Object {
"group": Array [],
"series_0.data": 1,
"ts_bucket": 1641340920,
},
]
`);

expect(countData).toMatchInlineSnapshot(`
Array [
Object {
"group": Array [],
"series_0.data": 0,
"ts_bucket": 1641340800,
},
Object {
"group": Array [],
"series_0.data": 2,
"ts_bucket": 1641340860,
},
Object {
"group": Array [],
"series_0.data": 3,
"ts_bucket": 1641340920,
},
]
`);
expect(countRateData).toMatchInlineSnapshot(`
Array [
Object {
"group": Array [],
"series_0.data": 0,
"ts_bucket": 1641340800,
},
Object {
"group": Array [],
"series_0.data": 2,
"ts_bucket": 1641340860,
},
Object {
"group": Array [],
"series_0.data": 1,
"ts_bucket": 1641340920,
},
]
`);
});

it('two_timestamps_lower_bound histogram (p50)', async () => {
const p50Data = (
await clickhouse.getMultiSeriesChart({
Expand Down
Loading
Loading