MM-58729: Improve proxy cache configuration

[agnivade]

Tested with various combinations of settings. None of them showed a major improvement. Therefore, settled on the very basic changes.

Full thread here: https://community.mattermost.com/core/pl/rjbg3yr3mfrrfrcq7zk3eo6gky

https://mattermost.atlassian.net/browse/MM-58729
https://mattermost.atlassian.net/browse/MM-59871

[agarciamontoro]

Thanks for the investigation! I wonder that given that none showed improvements, do we want/need to change them? I'm ok with the change of proxy_cache_lock_timeout, since we wanted to make timeouts easier to debug, but what about proxy_cache_min_uses? I'm not opposed to it, but I'm not entirely sure of the benefit.

[agnivade]

The benefit of min_uses would be that it populates the cache faster, and avoids an extra request to the server. At high scales, this should make a difference. Feels like a safe optimization IMO, but only 1/5.

[streamer45]

Thanks. Have you had a chance to review the proxy_cache_path directive as well?

I noticed the below comment seems somewhat inaccurate:

mattermost-load-test-ng/deployment/terraform/create.go

Lines 798 to 799 in 6f5a0c9

	cacheSize = "16g" // Ideally we'd like half of the total server mem. But the mem consumption rarely exceeds 10G
	// from my tests. So there's no point stretching it further.

As far as I understand, max_size controls disk space, not memory. It's also not really required given:

When the size is exceeded or there is not enough free space, it removes the least recently used data.

Another thing to check is whether we have eliminated the [alert] 3892#3892: could not allocate node in cache keys zone "mattermost_cache" log.

Tangential, but it would be great to look into cache instrumentation and get something to show on Grafana. Could be an interesting side project (OSF) or help wanted.

Resources:

https://blog.nginx.org/blog/nginx-caching-guide
https://www.f5.com/company/blog/nginx/nginx-high-performance-caching

[streamer45]

I am supportive of this change (caching slightly more aggressively) especially since it's nginx's default.

[agnivade]

Another thing to check is whether we have eliminated the [alert] 3892#3892: could not allocate node in cache keys zone "mattermost_cache" log.

Ahh, yes I missed to check that. Looks like I need to run more tests.