Memory increase due to too many traces

[ChrisTerBeke]

Hi,

We're using this lib in several APIs running on GKE and using StackDriver. We noticed that when sending all traces to StackDriver, the amount of traces collected grows faster than the amount that can be exported to StackDriver. Because of this, a list containing all trace spans that still need to be sent grows over time, to the point where Kubernetes kills our pod because it uses too much memory.

While this is expected behavior from the current code, I wonder if there would be a cleaner way to do this. For example by dropping the spans when the list reaches a maximum and then logging a warning or error? This will prevent apps from 'leaking' memory.

What are your thoughts about this?

p.s. this is probably the same as reported on #334, but it's not really resolved there.

[c24t]

Configuring samplers to have a global rate limit may help (see #458), but I agree that memory shouldn't grow unbounded by default. Dropping traces seems like a good solution to me.

As for your application: are you sampling every trace? You might want to use the ProbabilitySampler instead.

#334 is a different issue, the library wasn't cleaning up monitoring clients.

[c24t]

See census-instrumentation/opencensus-java#1813 for a similar issue in the java client.

[ChrisTerBeke]

We were sampling every trace, but since then have switched to the probability sampler (which solves the problem).

Glad to hear you also think this is something to improve :)

[c24t]

@reyang's work on #642 should fix the memory issue by dropping spans once the queue is full.