harvest 23.08.0-1 memory consumption

[Pengng88]

Hi, we have an environment of more than 360 cluster and I had just upgrade harvest from 23.05.0-1 to 23.08.0-1 .
I have observed that memory usage from the previous version 40GB had increase to 100GB .

I do not know if there is any memory leak that's causing the high utilization. Anyone experiencing the same issue?

[cgrinds]

hi @Pengng88 We are not aware of any memory leaks. Happy to take a look at your poller logs. Maybe pick the poller with the largest resident set size (RSS) and email its logs to [email protected]?

Are you running Prometheus and Grafana on the same machine that's hosting the 360 pollers or are Prometheus and Grafana on a different machine?

If anything, we would have expected memory to be reduced in 23.08 since we disabled some templates in 23.08 that were enabled out of the box in 23.05. We did that for a variety of reasons but performance was one of them for XX and XX.

In terms of changes between 23.05 and 23.08 - here's a summary of template changes?

Zapi

• 23.08 includes new template EmsDestination
• 23.08 includes new template QosWorkload

ZapiPerf

• 23.08 includes new template Iwarp

Rest

• 23.08 includes new template EmsDestination
• 23.08 includes new template QosWorkload
• 23.08 includes new template SupportAutoUpdate
• 23.08 disables NetConnections
• 23.08 disables NFSClients

RestPerf

• No additions or disables

[Pengng88]

Hi @cgrinds , Prometheus and Grafana are as dockers on the same machine. I have tried disabling both QosWorkload and EmsDestination on harvest but the CPU and memory consumption didn't go down at all.

I have sent the logs. Please help to take a look.

[cgrinds]

The increase in memory ended up being caused by enabling the ZapiPerf:ExternalServiceOperation template. For one of the clusters, that resulted in a large addition of metrics

              Collector             | NumInstances | AvgApiMs | NumMetrics 
------------------------------------+--------------+----------+-------------
  ZapiPerf:ExternalServiceOperation |       42,800 |  215,535 |  2,011,600