Specially index the C++ standard library

[varungandhi-src]

Depends on:

• Cross-repo support (part of ☂️ Post-MVP feature checklist #26)
• (Optional) Special Bazel support (part of ☂️ Post-MVP feature checklist #26)

Some things to keep in mind, off the top of my head:

• How do we do it?: Standard libraries largely consist of header files, but there is no meaningful interpretation of indexing/compiling a header file standalone.
  • One hack would be to make a fake C++ project which includes every header in the standard library. This fake C++ project should have multiple files so that indexing runs in parallel.
  • This fake C++ project should be generated instead of being hard-coded so that we can accommodate different stdlib versions and ideally automatically pick up new headers introduced in future stdlib versions.
  • The project root needs to be set to the stdlib root so that stdlib files are treated as in-project and the fake C++ project is ignored.
• How will the stdlib be available in Sourcegraph: Using a special package in Sourcegraph (e.g. so as to accommodate generated headers) seems like a good option, similar to the JDK. However, we don't have support for C++ packages yet in Sourcegraph.
• Which stdlib source code do we use?: The options are (i) release tarballs (ii) directly cloning from Git (iii) distro packages (iv) pre-packaged Bazel toolchains. With distro packages, building the code becomes easier (e.g. building libstdc++ from source seems to require building GCC from source, which would be good to avoid). However, distros may potentially apply their own patches to libstdc++. If we use the distro package, we need to figure out how to map the distro package code to the package in Sourcegraph. The same concerns apply to pre-packaged Bazel toolchains.
• Which stdlibs do we index?: This depends on the answer to the previous question. If we're going with distro packages, one potential problem might be that not every minor stdlib version may be available (or they make take a while to be released).
• Stdlib identity/version detection: We need to make sure we're tracking the versions
• Hermetic toolchains with Bazel: If a customer using using a pre-packaged hermetic toolchain with Bazel, like Aspect's GCC toolchain or grailbio's LLVM toolchain, we need to make sure that we're able to map the stdlib identity/version correctly.
• Custom Bazel toolchains: Are some customers using custom toolchains (e.g. some non-tagged commits in the LLVM repo)? Or their homegrown hermetic toolchain? Depending on the exact setup, we may need some way to pass in extra configuration as a command-line flag or via Bazel.
• Vendoring: Large projects like Chromium vendor all dependencies, down to the exact version of Clang and libstdc++. Are customers vendoring C++ toolchains like Chromium? If so, we need to think about how to map those into Sourcegraph. (If it's a monorepo, that should probably just work, but it's more likely that the build system is downloading the code on-the-fly.)