How to build in parallel

Overlord supports parallelism by default. Even without parallelism, however, Overlord tries to discourage reliance on side effects by randomizing the order in which dependencies are built.

Support for parallelism is controlled by a flag:

(overlord:use-threads-p) => T   ; The default.
(setf (overlord:use-threads-p) nil) ; Turn off parallelism.

You cannot turn on parallelism during a build, or for part of a build; it is all or nothing.

When parallelism is enabled, it is used in three ways:

1. After a build, the database is saved in the background.
2. A permanent thread pool (2 threads per processor) is allocated for the sole purpose of reading file metadata. This can substantially speed up null builds, especially on slow file systems (and Windows).
3. Targets are built in parallel, using fresh threads. By default, the number of threads to spin up is equal to the number of processors; if you want more (or fewer), you can specify the number as an argument to build:

(overlord:build 'mytarget :jobs 20) ;Start 19 threads (beside the current thread).
(overlord:build 'mytarget :jobs 1)  ;No parallelism please.

Overlord makes efficient use of the provided threads by storing how long each target took to build. In builds after the first, the profiling information is used to schedule tasks onto threads in such a way that the slowest targets are built first. (In the first build, various heuristics are used instead.)

The idea is to avoid a worst-case scenario where the fastest targets monopolize the kernel, leaving an exceptionally slow target to be built last. E.g. instead of this:

.........
......... _______

We want this:

_______ .....
.............

We also try to minimize the number of threads used. E.g. instead of this:

_______
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

We want this:

_______
.......
.......
....

At the end of the build, Overlord prints information about the maximum level of concurrency actually achieved, so you can tune the size of the thread pool for future builds.

A caveat: at the time of this writing (2019), SBCL can deadlock during threaded builds. This is not a problem with Overlord, but a problem with SBCL: if you start a lot of threads in a fresh SBCL session, and those threads call generic functions, the process of initializing SBCL’s CLOS caches can deadlock.