Debug: write memory profile if heap exceeds threshold

[jtibshirani]

This PR adds adds a debugging flag to periodically check memory usage against a threshold. If it exceeds the threshold, then a memory profile like indexmemory.prof.1 is written to disk. No more than 10 profiles will be written.

I've already found this more useful than the existing -memprofile flag, so I removed that. It's hard to get insights using that flag, since it only takes a single profile per shard, forces GC, and forces parallelism to 1.

[jtibshirani]

Leaving this as a draft for now, since I need to double-check some the concurrency logic. And I'm not sure it will be useful now that we have GCP profiling for zoekt-git-index (#816).

I took it for a spin while indexing sgtest/megarepo locally, and did notice some interesting things like much higher alloc_space than inuse_space:

top alloc_space

Showing top 10 nodes out of 101
      flat  flat%   sum%        cum   cum%
 1472.87MB 24.87% 24.87%  1472.87MB 24.87%  github.com/go-git/go-git/v5/plumbing.(*MemoryObject).Write
 1174.62MB 19.84% 44.71%  1174.62MB 19.84%  bytes.growSlice
  551.09MB  9.31% 54.01%   551.09MB  9.31%  bufio.NewReaderSize (inline)
  536.84MB  9.07% 63.08%   536.84MB  9.07%  github.com/go-git/go-git/v5/plumbing/format/idxfile.(*MemoryIndex).genOffsetHash
  429.91MB  7.26% 70.34%   429.91MB  7.26%  github.com/sourcegraph/zoekt.(*postingsBuilder).newSearchableString
  302.38MB  5.11% 75.44%   302.38MB  5.11%  github.com/go-git/go-git/v5/plumbing/format/idxfile.readObjectNames
  268.23MB  4.53% 79.97%   452.75MB  7.65%  github.com/sourcegraph/go-ctags.(*ctagsProcess).Parse
  185.17MB  3.13% 83.10%  1215.74MB 20.53%  github.com/sourcegraph/zoekt/gitindex.prepareNormalBuild
   96.95MB  1.64% 84.74%  4876.95MB 82.36%  github.com/sourcegraph/zoekt/gitindex.indexGitRepo
   83.80MB  1.42% 86.15%    83.80MB  1.42%  github.com/sourcegraph/zoekt/gitindex.(*repoWalker).handleEntry

top inuse_space

Showing top 10 nodes out of 68
      flat  flat%   sum%        cum   cum%
  806.24MB 36.69% 36.69%   806.24MB 36.69%  bytes.growSlice
  529.86MB 24.11% 60.81%   529.86MB 24.11%  github.com/go-git/go-git/v5/plumbing/format/idxfile.(*MemoryIndex).genOffsetHash
  302.38MB 13.76% 74.57%   302.38MB 13.76%  github.com/go-git/go-git/v5/plumbing/format/idxfile.readObjectNames
  126.53MB  5.76% 80.33%   126.53MB  5.76%  github.com/sourcegraph/zoekt.(*postingsBuilder).newSearchableString
  101.30MB  4.61% 84.94%   101.30MB  4.61%  github.com/go-git/go-git/v5/plumbing.(*MemoryObject).Write
   89.88MB  4.09% 89.03%   642.24MB 29.23%  github.com/sourcegraph/zoekt/gitindex.prepareNormalBuild
   58.84MB  2.68% 91.71%    58.84MB  2.68%  github.com/go-git/go-git/v5/plumbing/format/idxfile.readOffsets
   49.35MB  2.25% 93.95%  1983.50MB 90.27%  github.com/sourcegraph/zoekt/gitindex.indexGitRepo
      30MB  1.37% 95.32%       30MB  1.37%  encoding/json.(*decodeState).literalStore
   29.50MB  1.34% 96.66%    29.50MB  1.34%  github.com/sourcegraph/zoekt/build.(*tagsToSections).Convert

[keegancsmith]

Our use of git is quite vanilla, I wonder if we should instead rely on just shelling out to git? Maybe there is a wrapper which just relies on things like git cat-file? Or we can get cody to implement a go-git.Storer which just does read operations via git cat-file.

[jtibshirani]

@keegancsmith @stefanhengl this is ready for review!

[stefanhengl]

Very nice! Left a question just for my understanding.

[stefanhengl]

Why do we check based on doc count and not, for example, time based? If it's time based we wouldn't have to check in two places?

[jtibshirani]

Good question! That approach could definitely work. What I liked about this: it gives fine-grained control over when we check memory. So we can check exactly when we are most concerned about an impending OOM, to maximize the chance we'll get useful data. In fact I just added another check after my latest round of research into memory usage (I found that loading the list of files to index can be very expensive).