bazel compiles everything twice: as -fpic and without ... could this be tweaked ?

[hzeller]

[this is probably more a meta question about how bazel deals with compilations, but it is interesting in this context as XLS is a particularly large project for playing along at home]

Compiling XLS on machines even with many cores is still taking some time; my laptop with 20 cores takes about 2.5 hours.

Let's compile, extract the compile commands, and remember these:

bazel build --noshow_loading_progress --noshow_progress -c opt -s ... > /tmp/allcompile-commands.txt 2>&1

Now, let's fish out all the compiler invocations, extract the source filename after -c, and see how many compiler invocations there are.

grep "cc_wrapper.*std=" /tmp/allcompile-commands.txt | sed 's|.*-c \([^ ]*\).*|\1|p;d' | wc -l
21469

There is one file xls/common/build_embed.cc that is compiled particularly often, but that seems to just be due to the linkstamp setting in
//xls/common:init_xls, so let's ignore that for now; still over 20k compilations:

grep "cc_wrapper.*std=" /tmp/allcompile-commands.txt | sed 's|.*-c \([^ ]*\).*|\1|p;d' | grep -v xls/common/build_embed.cc | wc -l
20890

Looking at the output, it appears that the same file is mentioned more than once... let's get the data and see how many times the same file is mentioned uniquely:

grep "cc_wrapper.*std=" /tmp/allcompile-commands.txt | sed 's|.*-c \([^ ]*\).*|\1|p;d' | grep -v xls/common/build_embed.cc | sort -u | wc -l
9670

That is ... less than half!

Let's just look at just one file, and also capture the rest of the command line to see the -o output, as these are probably different:

grep "cc_wrapper.*std=" /tmp/allcompile-commands.txt | sed 's|.*\(xls/common/file/filesystem.cc.*\)|\1|p;d'
xls/common/file/filesystem.cc -o bazel-out/k8-opt/bin/xls/common/file/_objs/filesystem/filesystem.o)
xls/common/file/filesystem.cc -o bazel-out/k8-opt/bin/xls/common/file/_objs/filesystem/filesystem.pic.o)
xls/common/file/filesystem.cc -o bazel-out/k8-opt-exec-ST-d57f47055a04/bin/xls/common/file/_objs/filesystem/filesystem.o)

yes indeed: apparently every file is compiled 'regular' and as -fPIC position independent code. And this one apparently into yet another location ?

All the binaries are linked statically, so they don't need -fPIC object files to create shared libraries. But looks like all the unit tests are linked dynamically:

bazel build -c opt //xls/common/file:filesystem_test
ldd bazel-bin/xls/common/file/filesystem_test | wc -l
256

This is probably done to avoid linking large binaries for all the tests, however this means every source file is compiled twice!

I wonder if the -fPIC object files could also be used for the statically linked version (just linking them in a static context) to avoid the multi-compilation, and if there are options that I can give to bazel to achieve that.

[proppy]

But looks like all the unit tests are linked dynamically:

https://bazel.build/reference/be/c-cpp#cc_test:~:text=By%20default%2C%20C%2B%2B%20tests%20are%20dynamically%20linked.%0ATo%20statically%20link%20a%20unit%20test%2C%20specify%20linkstatic%3DTrue.%20It%20would%20probably%20be%20good%20to%20comment%20why%20your%20test%20needs%20linkstatic%3B%20this%20is%20probably%20not%20obvious.

Should we set this to True?

[hzeller]

Might be interesting to see the impact, though ideally that would be a global setting not per test (as it might actually also have the impact of slowing down things as now each test binary is very large and needs to be linked).

I am more wondering the other way: typically an object file compiled with -fPIC provides just more information than a regular object file, so it can be linked in a static context. Thus if everything would be compiled with -fPIC, we can have all three: static binaries and dynamically linked tests. as well as half the number of files that need to be compiled.

Ther might be a reason why bazel is not doing that, but I wonder why and if that reason applies ?

[hzeller]

The third compilation of xls/common/file/filesystem.cc starts with SUBCOMMAND: # //xls/common/file:filesystem [action 'Compiling xls/common/file/filesystem.cc [for tool]'... , so it looks like this is as part of some toolchain (possibly because we use dslx itself in *.bzl rules).

This of course only would make sense if there was a reason to compile tools differently than regular binaries, e.g. if host architecture != target architecture. But we're on the same platform, so these things should be coalesced.

So if we count the number of times the same file is compiled multiple times, then aggregate these counts, we see that we actually have 4687 files that are comiled 3 times due to that (and some files that are only compiled once: these are all the *_test.cc and *_main.cc files)

grep "cc_wrapper.*std=" /tmp/allcompile-commands.txt | sed 's|.*-c \([^ ]*\).*|\1|p;d' | grep -v xls/common/build_embed.cc | sort | uniq -c | awk '{print $1}' | sort | uniq -c
   3188 1
   1778 2
   4687 3
     17 5

cross-check and compare with the numbers from #1936 (comment)

$ echo "3188 + 1778 + 4687 + 17" | bc              # number of unique files compiled
9670

$ echo "3188 + 1778 * 2 + 4687 * 3 + 17 * 5" | bc  # number of compilations seen
20890

[trybka]

But looks like all the unit tests are linked dynamically:

https://bazel.build/reference/be/c-cpp#cc_test:~:text=By%20default%2C%20C%2B%2B%20tests%20are%20dynamically%20linked.%0ATo%20statically%20link%20a%20unit%20test%2C%20specify%20linkstatic%3DTrue.%20It%20would%20probably%20be%20good%20to%20comment%20why%20your%20test%20needs%20linkstatic%3B%20this%20is%20probably%20not%20obvious.

Should we set this to True?

and

Might be interesting to see the impact, though ideally that would be a global setting not per test (as it might actually also have the impact of slowing down things as now each test binary is very large and needs to be linked).

You can try to build with, e.g. --dynamic_mode=off in your top-level .bazelrc

A note about [tool] vs. not: The Exec configuration used to build tools typically sanitizes the build configuration so that you get a consistent and cacheable (and optimized) binary for use in your build pipeline. It's not always about architecture. If you turn on debug mode or assertions, you typically don't want your tools to be subject to those options (though sometimes you might).

If you are concerned about tool dependencies, prebuilding is an option.

Are subsequent incremental rebuilds slow as well? (i.e. is the concern primarily about cold builds?)

[hzeller]

Will try dymaic_mode=off, but wouldn't the other suggestion work as well ? Always build -fPIC and use the resulting object files in the static links.

I can understand if I build a debug build and the tool should be compiled in opt mode. However in this particular case I did -c opt, so these should be the same.

Since we're building our own tools as part of the build that then are used, they are affected similar to the rest of the code. (Things not changing might be tools built as part of the environment, protoc comes to mind)

Incremental builds will also have to compile the same file 3 times, so yes it affects them as well of course.

[hzeller]

Compiling with --dynamic_mod=off

bazel build --dynamic_mode=off --noshow_loading_progress --noshow_progress -c opt -s ... > /tmp/allcompile-commands-nodynamic.txt 2>&1

It still runs a compile with and with pic

$ grep "cc_wrapper.*std=" /tmp/allcompile-commands-nodynamic.txt | sed 's|.*\(xls/common/file/filesystem.cc.*\)|\1|p;d'
xls/common/file/filesystem.cc -o bazel-out/k8-opt/bin/xls/common/file/_objs/filesystem/filesystem.o)
xls/common/file/filesystem.cc -o bazel-out/k8-opt/bin/xls/common/file/_objs/filesystem/filesystem.pic.o)
xls/common/file/filesystem.cc -o bazel-out/k8-opt-exec-ST-d57f47055a04/bin/xls/common/file/_objs/filesystem/filesystem.o)

[trybka]

What toolchain are you using? There are unfortunately a number of knobs that could do what you want.

First, maybe try --force_pic? https://bazel.build/reference/command-line-reference#flag--force_pic

[hzeller]

We use the external llvm toolchain: https://github.com/google/xls/blob/main/MODULE.bazel#L7-L16

[trybka]

OK. that toolchain seems to rely on the legacy features. --force_pic should do what you want.

[hzeller]

Nice, that did the trick.

That now essentially reduces triple compilation to dual compilation:

bazel build --force_pic --noshow_loading_progress --noshow_progress -c opt -s ... > /tmp/allcompile-commands-forcepic.txt 2>&1
grep "cc_wrapper.*std=" /tmp/allcompile-commands-forcepic.txt | sed 's|.*-c \([^ ]*\).*|\1|p;d' | grep -v xls/common/build_embed.cc | sort | uniq -c | awk '{print $1}' | sort | uniq -c
   4265 1
   5409 2
      4 3
     17 4

Now we don't need a bit more than double but only +50%-ish of compliations:

echo "4265 + 5409 + 4 + 17" | bc   # unique compilations
9695
echo "4265 + 5409 * 2 + 4 * 3 + 17 * 4" | bc  # total compilations
15163

So --force-pic seems to be a good candidate to put in ~/.bazelrc

Is there a way to further reduce the now double compilation to single compilation for the tools ? We typically compile with -c opt anyway, so it won't hurt; and, conversely, if we build with -c dbg, we actually would like to have our tools also compiled with that.

[hzeller]

here the example of the singled out file: we see that it now only is compiled as PIC, yay.

... and it also shows that the tool building is still there, but it actually does not adhere to the --force_pic flag, but compiles it 'traditionally'. So before we can hope to coalesce tool/non-tool compilation, we need to convince it to also force pic.

$ grep "cc_wrapper.*std=" /tmp/allcompile-commands-forcepic.txt | sed 's|.*\(xls/common/file/filesystem.cc.*\)|\1|p;d'
xls/common/file/filesystem.cc -o bazel-out/k8-opt-exec-ST-d57f47055a04/bin/xls/common/file/_objs/filesystem/filesystem.o)
xls/common/file/filesystem.cc -o bazel-out/k8-opt/bin/xls/common/file/_objs/filesystem/filesystem.pic.o)

[hzeller]

This is now enabled in the .bazelrc

To be tested if this works on macOS; if not, we can make it a platform-specific common:linux feature (known to work) or give it a `--features=prefer_pic_for_opt_binaries ... but first tests seem to indicate that it indeed does not address the triple compliation.

[hzeller]

I can confirm that on three machines I tested at home, --force_pic reduces the compilation time by about 20%, which translates to 20-30min absolute time saving on these.