Run GC compaction before applying solutions

[kkeundotnet]

Hello. I am curious about if you are interested in running additional garbage collections for small-RAM machines.

This PR runs garbage collection (Gc.compact) before applying solutions, with an additional option --gc-before-action.

context:
I have a tiny machine with 1 GB of RAM. When opam-installing ppxlib or js_of_ocaml in it, it died with OOM. However, building the libraries directly from their source code was fine.

In my local runnings, this PR saved about 240 MB (from 482 to 242) before starting library builds (e.g. when opam install ppxlib). While the save is not huge (and it is not a big deal for most of modern machines), it was sufficient save for my tiny machine to be able to build ppxlib or js_of_ocaml without OOM.

[dra27]

How much time does the compaction take on your machine (easiest way would just be to add another debug line before it and run with debugging). In the grand scheme of things, I’m wondering if we could just unconditionally do the compaction. It could also be done as a (priority) parallel task at the start of the action graph, which would further mitigate any speed impact.

There’s a slight worry for the future that OCaml 5 at present has no compactor!

[avsm]

You should be able to get the same effect with a best-fit GC (the default in OCaml 4.14) and a Gc.full_major instead of a compact. I'd be most curious to know if that allows compilation to go through on your 1GB machine.

[kkeundotnet]

Thank you for the replies. I will try the suggestions/experiments.

There’s a slight worry for the future that OCaml 5 at present has no compactor!

Ooops, I see. While I don't know much about how the parallel jobs working in opam, with assuming running the job with -j 1, the problem is that once the execution control is passed from the opam process to the library build process, there is not much of chance to run GC when memory is insufficient during the build. Maybe in OCaml 5, where I believe the memory management is being done more ideally through multiple processes, this may not be a problem (I hope). After reading some code opamProcess.ml, I found my understanding was not correct.

[kkeundotnet]

@dra27

How much time does the compaction take on your machine

00:10.084  SOLUTION               GC compact (before heap 482 MB)
00:10.831  SOLUTION               GC compact (after heap 242 MB)

It took 747ms. Hmmm, much bigger than I expected. Is this small enough for us to run it unconditionally?

It could also be done as a (priority) parallel task at the start of the action graph, which would further mitigate any speed impact.

Taking some time for understanding how it works and where it is. Any pointer would be appreciated.

---

@avsm

You should be able to get the same effect with a best-fit GC (the default in OCaml 4.14) and a Gc.full_major instead of a compact. I'd be most curious to know if that allows compilation to go through on your 1GB machine.

Unfortunately, Gc.full_major with OCaml 4.14.0 did not help. Unlikely to Gc.compact, the heap_words sizes were the same befor and after running Gc.full_major and installing ppxlib failed with OOM in my 1GB-RAM machine.

$ opam install ppxlib --gc-before-action --debug
...
00:10.383  SOLUTION               GC compact (heap 482 MB -> 482 MB)
...
Processing  2/3: [ppxlib: dune build]
Killed

[dra27]

That's what I'd expect - best fit might reduce the peak heap requirement (although I expect opam's memory profile is a stream of allocations, followed by a large amount of garbage immediately after the solution is converted to an action graph, so fragmentation is hopefully not causing much problem).

@rjbou - what I was thinking here was that possibly at the point where opamProcess first is going to wait for the completion of running jobs, we could add a Gc.compact? i.e. we set the first tranche of things going (downloads / tar extractions, etc.) and in that I/O gap, do a compaction?

[kkeundotnet]

Hello, @dra27. I tried running GC when the main process is waiting for the children process in #5396. I hope I added the Gc.compact code in the right line. (fingers crossed)