How to build in parallel

Overlord supports parallelism, but it is not on by default. Even without parallelism, it tries to discourage reliance on side effects by, whenever possible, randomizing the order in which targets are built.

Support for parallelism is behind a flag:

(overlord:use-threads-p) => NIL   ; The default.
(setf (overlord:use-threads-p) t) ; Turn on parallelism.

You cannot turn on parallelism during a build, or for part of a build; it is all or nothing.

When parallelism is enabled, it is used in three ways:

1. After a build, the database is saved in background. This doesn't make things any faster, but it makes them feel faster.
2. A permanent thread pool (2 threads per processor) is allocated for the sole purpose of reading file metadata. This can substantially speed up null builds, especially on slow file systems (and Windows).
3. Targets are built in parallel, using fresh threads. The number of threads to spin up defaults to the number of processors; if you want more (or fewer), you can specify the number as an argument to build:

(overlord:build 'mytarget :jobs 20) ;Start 19 threads (beside the current thread).
(overlord:build 'mytarget :jobs 1)  ;No parallelism please.

Overlord makes efficient use of the provided threads by storing how long each target took to build. In builds after the first, the profiling information is used to schedule tasks onto threads in such a way that the slowest targets are built first. (In the first build, various heuristics are used instead.)

The idea is to avoid a worst-case scenario where the fastest targets monopolize the kernel, leaving an exceptionally slow target to be built last. E.g. instead of this:

.........
......... _______

We want this:

_______ .....
.............

We also try to minimize the number of threads used. E.g. instead of this:

_______
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

We want this:

_______
.......
.......
....

At the end of the build, Overlord prints information about the maximum level of concurrency actually achieved, so you can tune the size of the thread pool for future builds.

A caveat: if you interrupt a threaded build, external processes that are spawned by the build are not automatically killed. At the moment, if you have to cancel a threaded build that makes heavy use of external processes, the safest way is simply to kill your Lisp process.

Another caveat: at the time of this writing (2019), SBCL can deadlock during threaded builds. This is not a problem with Overlord, but a problem with SBCL: if you start a lot of threads in a fresh SBCL session, and those threads call generic functions, the process of initializing SBCL’s CLOS caches can deadlock.