Micrometer 404s can be abused to trigger an out of memory

[checketts]

With Micrometer, each timer is written to memory and retained. This means it could be a sort of memory leak, so you need to constrain the tag cardinality. (A good rule of thumb is 100 per metric)

If a ktor application with Micrometer is hit and triggers a 404, it creates a new metric for the new url. This can cause memory usage to climb until the application runs out of memory.

A meter filter can fix it as a work around (unifying all 404 request as a 404 route tag):

class Filter404: MeterFilter {
    override fun map(id: Meter.Id): Meter.Id {
        if(id.name == "ktor.http.server.requests" && id.getTag("status") == "404") {
            return id.withTag(Tag.of("route", "404"));
        }
        return id;
    }
}

Add it to the registry being used:

val appMicrometerRegistry = SimpleMeterRegistry().apply {
        config().meterFilter(Filter404())
    }

However, it would be great to proactively group all 404 meters into the tag., so others don't need to encounter this. Would a PR with that functionality be worthwhile?

[Stexxe]

Unfortunately, I cannot reproduce the potential OOM problem. Can you please share a code snippet for the server to reproduce the problem by making many requests leading to a 404 status code?

[checketts]

Sorry I wasn't clear. The 404s need to be unique URLs. I've created a sample app

Here is a test that fails:

    @Test
    fun `test Unique 404s Create Unbounded Metrics Growth`() = runTest {
        testApplication {
            val appMicrometerRegistry = SimpleMeterRegistry()

            install(MicrometerMetrics) { registry = appMicrometerRegistry }
            routing {
                get("/metrics-micrometer") {
                    call.respond(appMicrometerRegistry.metersAsString)
                }
            }

            (1..1000).forEach {
                (1..10_000).map {
                    client.get("/${UUID.randomUUID()}")
                }
                println("Metrics page size: ${client.get("/metrics-micrometer").bodyAsText().length}")
            }

        }
    }

Console output:

Metrics page size: 9980868
Metrics page size: 19954234
2025-03-04 09:25:05.440 [DefaultDispatcher-worker-2 @request#90021] ERROR io.ktor.test - Unhandled: GET - /metrics-micrometer
java.lang.OutOfMemoryError: Java heap space

[Stexxe]

Thank you for the test. I've reproduced the problem by executing your test. Since we manage issues in YouTrack, I've created an issue there.