Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Run garbage collection once when the main process is waiting for #5396

Merged
merged 1 commit into from
Nov 27, 2024

Conversation

kkeundotnet
Copy link
Contributor

@kkeundotnet kkeundotnet commented Dec 20, 2022

In #5376, @dra27 suggested running Gc.compact when the main process is waiting for the children processes for the first time.

what I was thinking here was that possibly at the point where opamProcess first is going to wait
for the completion of running jobs, we could add a Gc.compact?

In my local running on opam install ppxlib with this PR, "GC compact" ran in the middle of parallel processing of actions.

The following actions will be performed:
=== install 1 package
  ∗ ppxlib 0.28.0
00:07.216  XSYS                            Adding to env { LC_ALL=C }
00:09.099  STATE                           depexts loaded in 1.883s
00:09.100  SOLUTION                        parallel_apply
00:09.100  SOLUTION                        Regroup shared source packages: {}

<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><><><><>  🐫
00:09.106  PARALLEL                        Iterate over 3 task(s) with 11 process(es)
00:09.106  PARALLEL                        Starting job 444950918 (worker -/11 -/1 1/3): ⬇ ppxlib.0.28.0
00:09.106  SOLUTION                        Fetching sources for ppxlib.0.28.0
00:09.106  ACTION                          download_package: ppxlib.0.28.0
00:09.106  SYSTEM                          rmdir /Users/kkeundotnet/.opam/4.14.0/.opam-switch/sources/ppxlib.0.28.0
00:09.109  SYSTEM                          mkdir /var/folders/gx/6809fgsd3nndyyf69y3h12_h0000gn/T/opam-14220-67d907
00:09.110  PARALLEL                        Next task in job 444950918: /usr/bin/tar xfj /Users/kkeundotnet/.opam/download-cache/sha256/d8/d87ae5f9a081206308ca964809b50a66aeb8e83d254801e8b9675448b60cf377 -C /var/folders/gx/6809fgsd3nndyyf69y3h12_h0000gn/T/opam-14220-67d907
00:09.619  PARALLEL                        GC compact (heap 490 MB -> 328 MB)
00:09.619  PARALLEL                        Collected task for job 444950918 (ret:0)
00:10.158  SYSTEM                          rmdir /var/folders/gx/6809fgsd3nndyyf69y3h12_h0000gn/T/opam-14220-67d907
⬇ retrieved ppxlib.0.28.0  (cached)
00:10.321  PARALLEL                        Job 444950918 finished

Similar to #5376, this PR enabled my 1GB-RAM machine to install ppxlib or js_of_ocaml without OOM.

@rjbou
Copy link
Collaborator

rjbou commented Jan 6, 2023

Thanks for the PR! we'll look at it

@kit-ty-kate
Copy link
Member

@kkeundotnet was there any specific reason why you closed this PR? Sorry we haven’t had time to review this properly but there is only 1 person working on opam full-time and we’ve been extremely busy working on the release of 2.2.0 which includes Windows support (which is not an easy task)

@kkeundotnet kkeundotnet reopened this Nov 1, 2023
@kkeundotnet
Copy link
Contributor Author

No, just because it has been stcuk long time and the maintainers look busy to address this minor issue, supporting 1GB-ram machine by running GC once at starting. Also, I found 1GB is small enough to OOM on building js_of_ocaml even with this PR these days. So, I need to find another way anyhow.

I reopen the PR, but feel free to close it if you have not much of time. Actually, "Windows support" sounds much much cooler, more exciting, and important!!

In ocaml#5376, @dra27 suggested running `Gc.compact` when the main
process is waiting for the children processes for the first time.

> what I was thinking here was that possibly at the point where opamProcess first is going to wait
> for the completion of running jobs, we could add a Gc.compact?

In my local running on `opam install ppxlib`, "GC compact" ran in the middle of parallel processing
of actions.

```
The following actions will be performed:
=== install 1 package
  ∗ ppxlib 0.28.0
00:07.216  XSYS                            Adding to env { LC_ALL=C }
00:09.099  STATE                           depexts loaded in 1.883s
00:09.100  SOLUTION                        parallel_apply
00:09.100  SOLUTION                        Regroup shared source packages: {}

<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><><><><>  🐫
00:09.106  PARALLEL                        Iterate over 3 task(s) with 11 process(es)
00:09.106  PARALLEL                        Starting job 444950918 (worker -/11 -/1 1/3): ⬇ ppxlib.0.28.0
00:09.106  SOLUTION                        Fetching sources for ppxlib.0.28.0
00:09.106  ACTION                          download_package: ppxlib.0.28.0
00:09.106  SYSTEM                          rmdir /Users/scho/.opam/4.14.0/.opam-switch/sources/ppxlib.0.28.0
00:09.109  SYSTEM                          mkdir /var/folders/gx/6809fgsd3nndyyf69y3h12_h0000gn/T/opam-14220-67d907
00:09.110  PARALLEL                        Next task in job 444950918: /usr/bin/tar xfj /Users/scho/.opam/download-cache/sha256/d8/d87ae5f9a081206308ca964809b50a66aeb8e83d254801e8b9675448b60cf377 -C /var/folders/gx/6809fgsd3nndyyf69y3h12_h0000gn/T/opam-14220-67d907
00:09.619  PARALLEL                        GC compact (heap 490 MB -> 328 MB)
00:09.619  PARALLEL                        Collected task for job 444950918 (ret:0)
00:10.158  SYSTEM                          rmdir /var/folders/gx/6809fgsd3nndyyf69y3h12_h0000gn/T/opam-14220-67d907
⬇ retrieved ppxlib.0.28.0  (cached)
00:10.321  PARALLEL                        Job 444950918 finished
```

Similar to ocaml#5376, this PR enabled my 1GB-RAM machine to install `ppxlib` or `js_of_ocaml` without OOM.
Copy link
Collaborator

@rjbou rjbou left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR!

@kit-ty-kate
Copy link
Member

Thanks!

@kit-ty-kate kit-ty-kate merged commit b2a4da8 into ocaml:master Nov 27, 2024
42 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants