Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Using rules_cuda with hermetic clang #324

Open
Tracked by #283
udaya2899 opened this issue Feb 14, 2025 · 19 comments · May be fixed by #331
Open
Tracked by #283

Using rules_cuda with hermetic clang #324

udaya2899 opened this issue Feb 14, 2025 · 19 comments · May be fixed by #331

Comments

@udaya2899
Copy link
Contributor

I am trying out #283 from cloudhan/hermetic-ctk-3 branch locally.
We have a hermetic clang+llvm setup following @bazel-contrib/toolchains_llvm:

bazel_dep(name = "toolchains_llvm", version = "1.2.0")

llvm = use_extension(
    "@toolchains_llvm//toolchain/extensions:llvm.bzl",
    "llvm",
    dev_dependency = True,
)
llvm.toolchain(
    llvm_version = "18.1.8",
)

use_repo(llvm, "llvm_toolchain")

register_toolchains(
    "@llvm_toolchain//:all",
    dev_dependency = True,
)

This is not detected by the detect_clang method as it tries to settle for /usr/bin/clang and then fails at:

Invalid Argument: docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "/usr/bin/clang": stat /usr/bin/clang: no such file or directory: unknown.
  Unrecognized error detail: type_url: "type.googleapis.com/google.devtools.remotebuildbot.CommandStatus"

How to let detect_clang know about this? The clang executable is specifically at:
@llvm_toolchain//:bin/clang-cpp. Maybe something like execpath could help?

@cloudhan
Copy link
Collaborator

Those detect_* are for locally install CTK/tools. Clang cuda toolchain is not address in these PRs at the moment.

@udaya2899
Copy link
Contributor Author

If you can guide me with a high-level overview, I can send a PR to use llvm_toolchain's clang. So, right now, the only way is to use nvcc or system non-hermetic clang?

@cloudhan
Copy link
Collaborator

cloudhan commented Feb 16, 2025

@udaya2899 The idea then is not to detect but to configure, maybe manually. Because it seems that we don't have a way to query all configured toolchains and select the clang one as host+device compiler.

The current detected clang use compiler_executable attr with a path as its value. See

compiler_executable = "%{clang_path}",

You can actually configure it to use the newly added compiler_label with the hermetic clang executable as its value. But you also need to fiddle with the compiler_deps. Definitely doable, but not sure if there is a sane way for the user :(

@udaya2899
Copy link
Contributor Author

Yes, an optional arg, maybe passed to cuda.toolkit called "clang_compiler_label" propagated till the clang declaration, maybe?

I can consider sending a PR for that.

Will this label be explicitly only for clang? Because otherwise we have to propagate the same for nvcc which I'm not sure if we want to do.

@cloudhan
Copy link
Collaborator

cloudhan commented Feb 17, 2025

Or maybe just extend

environ = ["CUDA_PATH", "PATH", "CUDA_CLANG_PATH", "BAZEL_LLVM"],
with CUDA_CLANG_LABEL or CUDA_CLANG_LABELS. With the first label for compile executable and remaining for compiler_deps.

@udaya2899
Copy link
Contributor Author

Can you help me with understanding compiler_deps? They are set in BUILD.nvcc only and are used by both nvcc and clang or am I getting it wrong?

@cloudhan
Copy link
Collaborator

My apology, it is attr compiler_files

compiler_files = "@local_cuda//:compiler_deps",
and
compiler_files = "@local_cuda//:compiler_deps",

And is used in

platform_common.ToolchainInfo(
name = ctx.label.name,
compiler_executable = compiler_executable,
all_files = ctx.attr.compiler_files.files if ctx.attr.compiler_files else depset(),

I am not quite sure if we need to add some additional clang deps here.

@udaya2899 udaya2899 mentioned this issue Feb 17, 2025
14 tasks
@udaya2899
Copy link
Contributor Author

udaya2899 commented Feb 17, 2025

You can actually configure it to use the newly added compiler_label with the hermetic clang executable as its value.

This comment helped me. I added the following in my own configuration of cuda_toolchain:

cuda_toolchain(
    name = "clang-local",
    compiler_files = "@local_cuda//:compiler_deps",
    compiler_label = "@@toolchains_llvm++llvm+llvm_toolchain_llvm//:bin/clang++",
    toolchain_config = ":clang-local-config",
)

Although I had to re-declare the whole toolkit for this one line change, this makes the compiler to be the hermetic clang from llvm. That said, once I get my whole setup working (blocked by this error with --cuda-path), I can send a change to improve to use a label passed in as env or from the cuda.toolkit extension call.

@cloudhan
Copy link
Collaborator

cloudhan commented Feb 21, 2025

#283 (comment)

error: cannot specify -o when generating multiple output files

rules_cuda currently relies on the -o to make output name being reliable. We generally have 4 set of outputs: {basename}.o, {basename}.pic.o, {basename}.rdc.o and {basename}.pic.rdc.o where {basename} is parsed from srcs filename, they are all placed in <some_prefix>/_objs/. Also if your target have multiple files with same name, say srcs = ["a.cu", "nested/a.cu"], then they will be _objs/0/a[.pic][.rdc].o and _objs/1/a[.pic][.rdc].o. This behavior is the same as cc rules. So being able to control the output name and path is mandatory.

@udaya2899
Copy link
Contributor Author

udaya2899 commented Feb 21, 2025

I am only compiling one file "foo.cu" which produces two '.o' files I assume. I believe this is specific to clang.

@cloudhan
Copy link
Collaborator

#283 (comment)

fatal error: 'curand_mtgp32_kernel.h' file not found

maybe just add the curand as an mandatory deps to the generated repo, to make the problem much easier to solve.

@cloudhan
Copy link
Collaborator

cloudhan commented Feb 21, 2025

The downloaded repos don't have a version.txt or version.json file.

For version.json, we can just add it, the redistrib.json contains all the information.

@cloudhan
Copy link
Collaborator

clang++-19 -xcu -c --cuda-compile-host-device kernel.cu -o kernel.o produces only one file actually.

@udaya2899
Copy link
Contributor Author

I'll retry on Monday with clang++-19. I had to downgrade to 18.x because the expectation was to be < 19.

@lromor
Copy link

lromor commented Feb 24, 2025

@udaya2899 , I'm also interested in helping out. Do you have a branch/commit I can start looking at?

@udaya2899
Copy link
Contributor Author

@lromor, I've been trying out locally. Which part of this issue are you also facing a similar problem?

@lromor
Copy link

lromor commented Feb 25, 2025

@udaya2899 , I have toolchains_llvm and I want to tell rules_cuda to use that instead of relying on its own heuristics. Something similar to what you did, but I'm interested in adding a proper extensions that allows the user to configure that.

@udaya2899
Copy link
Contributor Author

clang++-19 -xcu -c --cuda-compile-host-device kernel.cu -o kernel.o produces only one file actually.

This hint helped me. I was earlier using 'clang-cpp' from llvm, but now I use 'clang++' and it doesn't complain about the multiple output files anymore. Phew!

@udaya2899
Copy link
Contributor Author

udaya2899 commented Feb 25, 2025

@lromor, that's what I'm trying to do. cuda_toolchain:

"_cc_toolchain": attr.label(default = "@bazel_tools//tools/cpp:current_cc_toolchain"),

used an older way of toolchain resolution which is deprecated. rules_cc mentions using a different find_cc_toolchain from @rules_cc:

https://github.com/bazelbuild/rules_cc/blob/07e3cb66d0d516d5d855c745a3a1b5bae0b67e5e/cc/find_cc_toolchain.bzl#L16C1-L53C4

I tried it but of not much success. For example, my toolchain is configured to use libc++ instead of libstdc++ and I can't get it to work.

Update:

# added to .bazelrc
build:cuda --@rules_cuda//cuda:copts='-stdlib=libc++'
build:cuda --@rules_cuda//cuda:copts="-D_ALLOW_UNSUPPORTED_LIBCPP" # had to explicitly set this for CUDA to be okay with an x86 system using libc++

With this, I'm able to build CUDA code using my own clang++ from toolchains_llvm along with libc++

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants