Reworked Makefiles to allow for parallel builds via make -j
#76
Conversation
This allows Make to utilize parallelism via jobs when building, and for that matter cleaning, the individual "projects"
Allows for parallel execution (via make -j) by separating the compilation from the archiving phase thus preventing race conditions on the .a files Conflicts: src/Makefile src/ioDevice/Makefile
|
If you don't choose to accept these changes, you should at least consider the changes in my fixed_build branch. There were some targets in src/basic/Makefile that had been moved to the functors directory that were breaking the build. |
|
Thanks for the work on this. I'm getting ready to leave town on travel On 08/31/2015 02:04 PM, Jacob McDonald wrote:
|
Before, builds couldn't be done with more than one Make job because of race conditions on the library files (i.e. multiple jobs trying to write to the same .a file at the same time). In order to prevent this, the compilation and the archiving phases of the build process were broken into two separate phases. The first, compilation, can be done as concurrently as desired because there are no dependencies. The second phase, archiving, must be done sequentially though because the files related to each library are nicely separated into subdirectories of src/, the Makefiles for those subdirectories (both phases) can be run concurrently to each other.
I did some very crude benchmarking of these changes. I used the Linux time(1) utility and averaged the values for three runs.
Current Build Process (
make):real 5m20.482s
user 0m6.782s
sys 0m9.327s
New Build Process (
make -j 4):real 2m28.016s
user 0m7.1297s
sys 0m11.431s
So in my environment, I am getting more than a 50% speedup from these changes. You'll notice that both the user and sys time went up after the changes though. I would venture to say that this is because of the additional overhead related to my method of splitting up the two build phases. Regardless, the real time improvement is significant.
Additionally, I compared the libraries generated using the old build process and the new build process using the procedure described in this Gist. In addition to the shell commands, a tar.gz file containing the outputs is attached. As you can see, the generated libraries are identical (disregarding ordering of object files within the libraries).