Skip to content

Commit

Permalink
Avoid duplicate countitems metrics
Browse files Browse the repository at this point in the history
Metrics from both workers & managers are instantiated in both kind of
processes, leading to duplicates when merging (naively) the metrics.

Improving merge is not trivial, for now we simply are more careful about
instantiating the metrics: `bucketProcessingDuration` and
`consolidationDuration` are only instantiated on master process. Since
they have labels, `bucketsCount` and `objectsCount` are not affected, so
we can keep them as is for now (since they are used directly from
S3UtilsMongoClient, and would need a significant refactor to make it
work both in prod and tests).

Issue: S3UTILS-175
  • Loading branch information
francoisferrand committed Aug 12, 2024
1 parent e96a3d1 commit 30ac3cc
Showing 1 changed file with 5 additions and 4 deletions.
9 changes: 5 additions & 4 deletions utils/monitoring.js
Original file line number Diff line number Diff line change
@@ -1,22 +1,23 @@
const { errors } = require('arsenal');
const promClient = require('prom-client');
const { http } = require('httpagent');
const cluster = require('cluster');

const aggregatorRegistry = new promClient.AggregatorRegistry();
const { collectDefaultMetrics } = promClient;

// Histogram of the bucket processing duration, by the utilization service.
const bucketProcessingDuration = new promClient.Histogram({
const bucketProcessingDuration = cluster.isMaster ? new promClient.Histogram({
name: 's3_countitems_bucket_listing_duration_seconds',
help: 'Bucket processing duration',
buckets: [1, 10, 60, 600, 3600, 18000, 36000],
});
}) : null;

const consolidationDuration = new promClient.Histogram({
const consolidationDuration = cluster.isMaster ? new promClient.Histogram({
name: 's3_countitems_bucket_merge_duration_seconds',
help: 'Duration of metrics consolidation in seconds',
buckets: [0.01, 0.05, 0.1, 0.2, 0.5, 1, 2, 5, 10],
});
}) : null;

const bucketsCount = new promClient.Counter({
name: 's3_countitems_total_buckets_count',
Expand Down

0 comments on commit 30ac3cc

Please sign in to comment.