Skip to content

Commit

Permalink
report GC stats even in the event of a GC worker error
Browse files Browse the repository at this point in the history
When customers seed their SpiceDB clusters and
use idempotent TOUCH operations, large amounts of garbage
may be generated that all will expire around the same time.

This makes the GC workers take longer to process, to the point
they may exceed the deadline. In these scenarios the code is
not reporting metrics, which hides how much garbage was being collected.

This changes the code so that GC metrics are issued, even in the event
of an error.
  • Loading branch information
vroldanbet committed Mar 21, 2024
1 parent d539511 commit 34e6988
Showing 1 changed file with 9 additions and 5 deletions.
14 changes: 9 additions & 5 deletions internal/datastore/common/gc.go
Original file line number Diff line number Diff line change
Expand Up @@ -184,22 +184,26 @@ func RunGarbageCollection(gc GarbageCollector, window, timeout time.Duration) er
}

collected, err := gc.DeleteBeforeTx(ctx, watermark)

// even if an error happened, garbage would have been collected. This makes sure these are reflected even if the
// worker eventually fails or times out.
gcRelationshipsCounter.Add(float64(collected.Relationships))
gcTransactionsCounter.Add(float64(collected.Transactions))
gcNamespacesCounter.Add(float64(collected.Namespaces))
collectionDuration := time.Since(startTime)
gcDurationHistogram.Observe(collectionDuration.Seconds())

if err != nil {
return fmt.Errorf("error deleting in gc: %w", err)
}

collectionDuration := time.Since(startTime)
log.Ctx(ctx).Info().
Stringer("highestTxID", watermark).
Dur("duration", collectionDuration).
Time("nowTime", now).
Interface("collected", collected).
Msg("datastore garbage collection completed successfully")

gcDurationHistogram.Observe(collectionDuration.Seconds())
gcRelationshipsCounter.Add(float64(collected.Relationships))
gcTransactionsCounter.Add(float64(collected.Transactions))
gcNamespacesCounter.Add(float64(collected.Namespaces))
gc.MarkGCCompleted()
return nil
}

0 comments on commit 34e6988

Please sign in to comment.