Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Reactant] Build for aarch64-linux with CUDA #10313

Merged
merged 22 commits into from
Jan 28, 2025

Conversation

giordano
Copy link
Member

This is only to save my progress along the way, nothing works at the moment.

@giordano giordano force-pushed the mg/reactant-cuda-aarch64 branch from af7b742 to 93dd806 Compare January 27, 2025 15:36
@giordano giordano marked this pull request as ready for review January 27, 2025 15:36
@giordano giordano enabled auto-merge (squash) January 27, 2025 15:44
@giordano
Copy link
Member Author

giordano commented Jan 27, 2025

https://buildkite.com/julialang/yggdrasil/builds/16785#0194a869-1348-441d-a6a6-80a72a8b2a2c/687-1725

[15:46:20] ERROR: /workspace/bazel_root/097636303b1142f44508c1d8e3494e4b/external/xla/xla/backends/cpu/runtime/xnnpack/BUILD:42:11: Compiling xla/backends/cpu/runtime/xnnpack/parallel_loop_runner.cc failed: (Exit 1): aarch64-apple-darwin20-cc failed: error executing command (from target @xla//xla/backends/cpu/runtime/xnnpack:parallel_loop_runner) 
[15:46:20]   (cd /workspace/bazel_root/097636303b1142f44508c1d8e3494e4b/sandbox/processwrapper-sandbox/1303/execroot/__main__ && \
[15:46:20]   exec env - \
[15:46:20]     LD_LIBRARY_PATH=/usr/lib/csl-musl-x86_64:/usr/local/lib64:/usr/local/lib:/usr/lib64:/usr/lib:/lib64:/lib:/workspace/x86_64-linux-musl-cxx11/destdir/lib:/workspace/x86_64-linux-musl-cxx11/destdir/lib64:/opt/x86_64-linux-musl/x86_64-linux-musl/lib64:/opt/x86_64-linux-musl/x86_64-linux-musl/lib:/opt/aarch64-apple-darwin20/aarch64-apple-darwin20/lib:/opt/aarch64-apple-darwin20/lib:/workspace/destdir/lib64:/workspace/destdir/lib \
[15:46:20]     MACOSX_DEPLOYMENT_TARGET=10.14 \
[15:46:20]     PATH=/workspace/srcdir/Reactant.jl/deps/ReactantExtra/.local/bin:/opt/bin/aarch64-apple-darwin20-libgfortran5-cxx11-cuda_version+none-gpu+none-mode+opt:/opt/aarch64-apple-darwin20/bin:/opt/bin/x86_64-linux-musl-cxx11:/opt/x86_64-linux-musl/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin:/workspace/x86_64-linux-musl-cxx11/destdir/bin:/workspace/destdir/bin \
[15:46:20]     PWD=/proc/self/cwd \
[15:46:20]     SUPER_VERBOSE='' \
[15:46:20]     TEMP=/workspace/srcdir/Reactant.jl/deps/ReactantExtra/.tmp \
[15:46:20]     TMP=/workspace/srcdir/Reactant.jl/deps/ReactantExtra/.tmp \
[15:46:20]     TMPDIR=/workspace/srcdir/Reactant.jl/deps/ReactantExtra/.tmp \
[15:46:20]   /opt/bin/aarch64-apple-darwin20-libgfortran5-cxx11-cuda_version+none-gpu+none-mode+opt/aarch64-apple-darwin20-cc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wthread-safety -Wself-assign -Wunused-but-set-parameter -Wno-free-nonheap-object -fcolor-diagnostics -fno-omit-frame-pointer -g0 -O2 '-D_FORTIFY_SOURCE=1' -DNDEBUG -ffunction-sections -fdata-sections '-std=c++17' -MD -MF bazel-out/darwin_arm64-opt/bin/external/xla/xla/backends/cpu/runtime/xnnpack/_objs/parallel_loop_runner/parallel_loop_runner.pic.d '-frandom-seed=bazel-out/darwin_arm64-opt/bin/external/xla/xla/backends/cpu/runtime/xnnpack/_objs/parallel_loop_runner/parallel_loop_runner.pic.o' -fPIC '-DEIGEN_MAX_ALIGN_BYTES=64' -DEIGEN_ALLOW_UNALIGNED_SCALARS '-DEIGEN_USE_AVX512_GEMM_KERNELS=0' -DHAVE_SYS_UIO_H -DTF_USE_SNAPPY '-DBAZEL_CURRENT_REPOSITORY="xla"' -iquote external/xla -iquote bazel-out/darwin_arm64-opt/bin/external/xla -iquote external/tsl -iquote bazel-out/darwin_arm64-opt/bin/external/tsl -iquote external/eigen_archive -iquote bazel-out/darwin_arm64-opt/bin/external/eigen_archive -iquote external/ml_dtypes -iquote bazel-out/darwin_arm64-opt/bin/external/ml_dtypes -iquote external/com_google_absl -iquote bazel-out/darwin_arm64-opt/bin/external/com_google_absl -iquote external/com_google_protobuf -iquote bazel-out/darwin_arm64-opt/bin/external/com_google_protobuf -iquote external/snappy -iquote bazel-out/darwin_arm64-opt/bin/external/snappy -iquote external/com_googlesource_code_re2 -iquote bazel-out/darwin_arm64-opt/bin/external/com_googlesource_code_re2 -Ibazel-out/darwin_arm64-opt/bin/external/ml_dtypes/_virtual_includes/float8 -Ibazel-out/darwin_arm64-opt/bin/external/ml_dtypes/_virtual_includes/intn -Ibazel-out/darwin_arm64-opt/bin/external/ml_dtypes/_virtual_includes/mxfloat -isystem external/eigen_archive -isystem bazel-out/darwin_arm64-opt/bin/external/eigen_archive -isystem external/eigen_archive/mkl_include -isystem bazel-out/darwin_arm64-opt/bin/external/eigen_archive/mkl_include -isystem external/ml_dtypes -isystem bazel-out/darwin_arm64-opt/bin/external/ml_dtypes -isystem external/ml_dtypes/ml_dtypes -isystem bazel-out/darwin_arm64-opt/bin/external/ml_dtypes/ml_dtypes -isystem external/com_google_protobuf/src -isystem bazel-out/darwin_arm64-opt/bin/external/com_google_protobuf/src '-D__ARM_FEATURE_AES=1' '-D__ARM_NEON=1' '-D__ARM_FEATURE_SHA2=1' '-DDNNL_ARCH_GENERIC=1' '-std=c++17' -w '-std=c++17' '-DTCP_USER_TIMEOUT=0' -no-canonical-prefixes -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -c external/xla/xla/backends/cpu/runtime/xnnpack/parallel_loop_runner.cc -o bazel-out/darwin_arm64-opt/bin/external/xla/xla/backends/cpu/runtime/xnnpack/_objs/parallel_loop_runner/parallel_loop_runner.pic.o)
[15:46:20] # Configuration: d4932074860c8b5f1a9d033d9dbbff4759679de3a984ff21e2e2b5e2c1625699
[15:46:20] # Execution platform: @local_execution_config_platform//:platform
[15:46:20] 
[15:46:20] Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
[15:46:20] In file included from external/xla/xla/backends/cpu/runtime/xnnpack/parallel_loop_runner.cc:31:
[15:46:20] external/xla/xla/backends/cpu/runtime/work_queue.h:54:12: error: no member named 'hardware_destructive_interference_size' in namespace 'std'
[15:46:20]    54 |       std::hardware_destructive_interference_size;
[15:46:20]       |       ~~~~~^
[15:46:20] 1 error generated.

Sigh. I've seen this error before, but can't remember where.

Edit: that's where I saw it: #10190 (comment)

@giordano
Copy link
Member Author

Doing a quick search in https://github.com/roblabla/MacOSX-SDKs I think that means we need to use at least the 13.3 SDK:

% grep -r 'hardware_destructive_interference_size' .
./MacOSX13.3.sdk/usr/include/c++/v1/new:inline constexpr size_t hardware_destructive_interference_size = __GCC_DESTRUCTIVE_SIZE;
./MacOSX14.0.sdk/usr/include/c++/v1/new:inline constexpr size_t hardware_destructive_interference_size = __GCC_DESTRUCTIVE_SIZE;
./MacOSX14.2.sdk/usr/include/c++/v1/new:inline constexpr size_t hardware_destructive_interference_size = __GCC_DESTRUCTIVE_SIZE;
./MacOSX14.4.sdk/usr/include/c++/v1/new:inline constexpr size_t hardware_destructive_interference_size = __GCC_DESTRUCTIVE_SIZE;
./MacOSX14.5.sdk/usr/include/c++/v1/new:inline constexpr size_t hardware_destructive_interference_size = __GCC_DESTRUCTIVE_SIZE;
./MacOSX15.0.sdk/usr/include/c++/v1/new:inline constexpr size_t hardware_destructive_interference_size  = __GCC_DESTRUCTIVE_SIZE;

I hope that's enough.....

@giordano
Copy link
Member Author

giordano commented Jan 27, 2025

Small progress, with the 13.3 SDK I get to a different error:

ERROR: /workspace/bazel_root/097636303b1142f44508c1d8e3494e4b/external/xla/xla/backends/cpu/runtime/BUILD:83:11: Compiling xla/backends/cpu/runtime/kernel.cc failed: (Exit 1): aarch64-apple-darwin20-cc failed: error executing command (from target @xla//xla/backends/cpu/runtime:kernel) 
  (cd /workspace/bazel_root/097636303b1142f44508c1d8e3494e4b/sandbox/processwrapper-sandbox/212/execroot/__main__ && \
  exec env - \
    LD_LIBRARY_PATH=/usr/lib/csl-musl-x86_64:/usr/local/lib64:/usr/local/lib:/usr/lib64:/usr/lib:/lib64:/lib:/workspace/x86_64-linux-musl-cxx11/destdir/lib:/workspace/x86_64-linux-musl-cxx11/destdir/lib64:/opt/x86_64-linux-musl/x86_64-linux-musl/lib64:/opt/x86_64-linux-musl/x86_64-linux-musl/lib:/opt/aarch64-apple-darwin20/aarch64-apple-darwin20/lib:/opt/aarch64-apple-darwin20/lib:/workspace/destdir/lib64:/workspace/destdir/lib \
    MACOSX_DEPLOYMENT_TARGET=13.3 \
    PATH=/workspace/srcdir/Reactant.jl/deps/ReactantExtra/.local/bin:/opt/bin/aarch64-apple-darwin20-libgfortran5-cxx11-cuda_version+none-gpu+none-mode+opt:/opt/aarch64-apple-darwin20/bin:/opt/bin/x86_64-linux-musl-cxx11:/opt/x86_64-linux-musl/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin:/workspace/x86_64-linux-musl-cxx11/destdir/bin:/workspace/destdir/bin \
    PWD=/proc/self/cwd \
    SUPER_VERBOSE='' \
    TEMP=/workspace/srcdir/Reactant.jl/deps/ReactantExtra/.tmp \
    TMP=/workspace/srcdir/Reactant.jl/deps/ReactantExtra/.tmp \
    TMPDIR=/workspace/srcdir/Reactant.jl/deps/ReactantExtra/.tmp \
  /opt/bin/aarch64-apple-darwin20-libgfortran5-cxx11-cuda_version+none-gpu+none-mode+opt/aarch64-apple-darwin20-cc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wthread-safety -Wself-assign -Wunused-but-set-parameter -Wno-free-nonheap-object -fcolor-diagnostics -fno-omit-frame-pointer -g0 -O2 '-D_FORTIFY_SOURCE=1' -DNDEBUG -ffunction-sections -fdata-sections '-std=c++14' -MD -MF bazel-out/darwin_arm64-opt/bin/external/xla/xla/backends/cpu/runtime/_objs/kernel/kernel.pic.d '-frandom-seed=bazel-out/darwin_arm64-opt/bin/external/xla/xla/backends/cpu/runtime/_objs/kernel/kernel.pic.o' -fPIC '-DEIGEN_MAX_ALIGN_BYTES=64' -DEIGEN_ALLOW_UNALIGNED_SCALARS '-DEIGEN_USE_AVX512_GEMM_KERNELS=0' -DHAVE_SYS_UIO_H -DTF_USE_SNAPPY '-DBAZEL_CURRENT_REPOSITORY="xla"' -iquote external/xla -iquote bazel-out/darwin_arm64-opt/bin/external/xla -iquote external/tsl -iquote bazel-out/darwin_arm64-opt/bin/external/tsl -iquote external/com_google_absl -iquote bazel-out/darwin_arm64-opt/bin/external/com_google_absl -iquote external/eigen_archive -iquote bazel-out/darwin_arm64-opt/bin/external/eigen_archive -iquote external/ml_dtypes -iquote bazel-out/darwin_arm64-opt/bin/external/ml_dtypes -iquote external/com_google_protobuf -iquote bazel-out/darwin_arm64-opt/bin/external/com_google_protobuf -iquote external/snappy -iquote bazel-out/darwin_arm64-opt/bin/external/snappy -iquote external/com_googlesource_code_re2 -iquote bazel-out/darwin_arm64-opt/bin/external/com_googlesource_code_re2 -Ibazel-out/darwin_arm64-opt/bin/external/ml_dtypes/_virtual_includes/float8 -Ibazel-out/darwin_arm64-opt/bin/external/ml_dtypes/_virtual_includes/intn -Ibazel-out/darwin_arm64-opt/bin/external/ml_dtypes/_virtual_includes/mxfloat -isystem external/eigen_archive -isystem bazel-out/darwin_arm64-opt/bin/external/eigen_archive -isystem external/eigen_archive/mkl_include -isystem bazel-out/darwin_arm64-opt/bin/external/eigen_archive/mkl_include -isystem external/ml_dtypes -isystem bazel-out/darwin_arm64-opt/bin/external/ml_dtypes -isystem external/ml_dtypes/ml_dtypes -isystem bazel-out/darwin_arm64-opt/bin/external/ml_dtypes/ml_dtypes -isystem external/com_google_protobuf/src -isystem bazel-out/darwin_arm64-opt/bin/external/com_google_protobuf/src '-D__ARM_FEATURE_AES=1' '-D__ARM_NEON=1' '-D__ARM_FEATURE_SHA2=1' '-DDNNL_ARCH_GENERIC=1' '-std=c++17' -w '-std=c++17' '-DTCP_USER_TIMEOUT=0' -no-canonical-prefixes -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -c external/xla/xla/backends/cpu/runtime/kernel.cc -o bazel-out/darwin_arm64-opt/bin/external/xla/xla/backends/cpu/runtime/_objs/kernel/kernel.pic.o)
# Configuration: 519cd271ee58d2a65a4b73b3190f713220eb6496f8bcfd6a36879c57be6155ae
# Execution platform: @local_execution_config_platform//:platform

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
external/xla/xla/backends/cpu/runtime/kernel.cc:89:36: error: out-of-line definition of 'operator()' does not match any declaration in 'xla::cpu::Kernel::ParallelTask'
   89 | absl::Status Kernel::ParallelTask::operator()(uint64_t task_index) const {
      |                                    ^~~~~~~~
1 error generated.

Edit: git bisect points to openxla/xla@9d95a71.

Edit: this was fixed by openxla/xla#21768, we were building on top of the previous commit 🥲

@giordano giordano force-pushed the mg/reactant-cuda-aarch64 branch from 452d696 to 38aeca6 Compare January 27, 2025 20:38
@giordano
Copy link
Member Author

https://buildkite.com/julialang/yggdrasil/builds/16793#0194a97d-d9de-402f-99e3-30ce6338f7b7/6-20170

[22:52:42] ERROR: /workspace/bazel_root/097636303b1142f44508c1d8e3494e4b/external/enzyme_ad/src/enzyme_ad/jax/BUILD:283:11: Compiling src/enzyme_ad/jax/Passes/ControlFlowToSCF.cpp failed: (Exit 1): x86_64-apple-darwin14-cc failed: error executing command (from target @enzyme_ad//src/enzyme_ad/jax:XLADerivatives) 
[22:52:42]   (cd /workspace/bazel_root/097636303b1142f44508c1d8e3494e4b/sandbox/processwrapper-sandbox/7183/execroot/__main__ && \
[22:52:42]   exec env - \
[22:52:42]     LD_LIBRARY_PATH=/usr/lib/csl-musl-x86_64:/usr/local/lib64:/usr/local/lib:/usr/lib64:/usr/lib:/lib64:/lib:/workspace/x86_64-linux-musl-cxx11/destdir/lib:/workspace/x86_64-linux-musl-cxx11/destdir/lib64:/opt/x86_64-linux-musl/x86_64-linux-musl/lib64:/opt/x86_64-linux-musl/x86_64-linux-musl/lib:/opt/x86_64-apple-darwin14/x86_64-apple-darwin14/lib:/opt/x86_64-apple-darwin14/lib:/workspace/destdir/lib64:/workspace/destdir/lib \
[22:52:42]     MACOSX_DEPLOYMENT_TARGET=13.3 \
[22:52:42]     PATH=/workspace/srcdir/Reactant.jl/deps/ReactantExtra/.local/bin:/opt/bin/x86_64-apple-darwin14-libgfortran5-cxx11-cuda_version+none-gpu+none-mode+opt:/opt/x86_64-apple-darwin14/bin:/opt/bin/x86_64-linux-musl-cxx11:/opt/x86_64-linux-musl/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin:/workspace/x86_64-linux-musl-cxx11/destdir/bin:/workspace/destdir/bin \
[22:52:42]     PWD=/proc/self/cwd \
[22:52:42]     SUPER_VERBOSE='' \
[22:52:42]     TEMP=/workspace/srcdir/Reactant.jl/deps/ReactantExtra/.tmp \
[22:52:42]     TMP=/workspace/srcdir/Reactant.jl/deps/ReactantExtra/.tmp \
[22:52:42]     TMPDIR=/workspace/srcdir/Reactant.jl/deps/ReactantExtra/.tmp \
[22:52:42]   /opt/bin/x86_64-apple-darwin14-libgfortran5-cxx11-cuda_version+none-gpu+none-mode+opt/x86_64-apple-darwin14-cc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wthread-safety -Wself-assign -Wunused-but-set-parameter -Wno-free-nonheap-object -fcolor-diagnostics -fno-omit-frame-pointer -g0 -O2 '-D_FORTIFY_SOURCE=1' -DNDEBUG -ffunction-sections -fdata-sections '-std=c++17' -MD -MF bazel-out/darwin-opt/bin/external/enzyme_ad/src/enzyme_ad/jax/_objs/XLADerivatives/ControlFlowToSCF.pic.d '-frandom-seed=bazel-out/darwin-opt/bin/external/enzyme_ad/src/enzyme_ad/jax/_objs/XLADerivatives/ControlFlowToSCF.pic.o' -fPIC '-DLLVM_ON_UNIX=1' '-DNO_HAVE_BACKTRACE=0' '-DBACKTRACE_HEADER=<execinfo.h>' '-DLTDL_SHLIB_EXT=".so"' '-DLLVM_PLUGIN_EXT=".so"' '-DLLVM_ENABLE_THREADS=0' '-DHAVE_DEREGISTER_FRAME=1' '-DHAVE_LIBPTHREAD=1' '-DFAKE_HAVE_PTHREAD_GETNAME_NP=0' '-DHAVE_PTHREAD_H=1' '-DFAKE_HAVE_PTHREAD_SETNAME_NP=0' '-DHAVE_REGISTER_FRAME=1' '-DHAVE_SETENV_R=1' '-DHAVE_STRERROR_R=1' '-DHAVE_SYSEXITS_H=1' '-DHAVE_UNISTD_H=1' '-DHAVE_MACH_MACH_H=1' '-DHAVE_MALLOC_MALLOC_H=1' '-DHAVE_MALLOC_ZONE_STATISTICS=1' '-DHAVE_PROC_PID_RUSAGE=1' '-DHAVE_UNW_ADD_DYNAMIC_FDE=1' '-DLLVM_NATIVE_ARCH="X86"' '-DLLVM_NATIVE_ASMPARSER=LLVMInitializeX86AsmParser' '-DLLVM_NATIVE_ASMPRINTER=LLVMInitializeX86AsmPrinter' '-DLLVM_NATIVE_DISASSEMBLER=LLVMInitializeX86Disassembler' '-DLLVM_NATIVE_TARGET=LLVMInitializeX86Target' '-DLLVM_NATIVE_TARGETINFO=LLVMInitializeX86TargetInfo' '-DLLVM_NATIVE_TARGETMC=LLVMInitializeX86TargetMC' '-DLLVM_NATIVE_TARGETMCA=LLVMInitializeX86TargetMCA' '-DLLVM_HOST_TRIPLE="x86_64-unknown-darwin"' '-DLLVM_DEFAULT_TARGET_TRIPLE="x86_64-unknown-darwin"' '-DLLVM_VERSION_MAJOR=20' '-DLLVM_VERSION_MINOR=0' '-DLLVM_VERSION_PATCH=0' '-DLLVM_VERSION_STRING="20.0.0git"' -D__STDC_LIMIT_MACROS -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS '-DLLVM_HAS_AArch64_TARGET=1' '-DLLVM_HAS_AMDGPU_TARGET=1' '-DLLVM_HAS_ARM_TARGET=1' '-DLLVM_HAS_NVPTX_TARGET=1' '-DLLVM_HAS_PowerPC_TARGET=1' '-DLLVM_HAS_RISCV_TARGET=1' '-DLLVM_HAS_SystemZ_TARGET=1' '-DLLVM_HAS_X86_TARGET=1' '-DBLAKE3_USE_NEON=0' -DBLAKE3_NO_AVX2 -DBLAKE3_NO_AVX512 -DBLAKE3_NO_SSE2 -DBLAKE3_NO_SSE41 '-DBAZEL_CURRENT_REPOSITORY="enzyme_ad"' -iquote external/enzyme_ad -iquote bazel-out/darwin-opt/bin/external/enzyme_ad -iquote external/llvm-project -iquote bazel-out/darwin-opt/bin/external/llvm-project -iquote external/enzyme -iquote bazel-out/darwin-opt/bin/external/enzyme -iquote external/stablehlo -iquote bazel-out/darwin-opt/bin/external/stablehlo -iquote external/xla -iquote bazel-out/darwin-opt/bin/external/xla -iquote external/tsl -iquote bazel-out/darwin-opt/bin/external/tsl -Ibazel-out/darwin-opt/bin/external/llvm-project/mlir/_virtual_includes/ArithCanonicalizationIncGen -Ibazel-out/darwin-opt/bin/external/llvm-project/mlir/_virtual_includes/AsmParserTokenKinds -Ibazel-out/darwin-opt/bin/external/llvm-project/llvm/_virtual_includes/NVPTXCodeGen -Ibazel-out/darwin-opt/bin/external/llvm-project/llvm/_virtual_includes/NVPTXCommonTableGen -Ibazel-out/darwin-opt/bin/external/llvm-project/llvm/_virtual_includes/NVPTXInfo -Ibazel-out/darwin-opt/bin/external/llvm-project/llvm/_virtual_includes/NVPTXUtilsAndDesc -Ibazel-out/darwin-opt/bin/external/llvm-project/mlir/_virtual_includes/SPIRVCanonicalizationIncGen -Ibazel-out/darwin-opt/bin/external/llvm-project/mlir/_virtual_includes/GPUToNVVMGen -Ibazel-out/darwin-opt/bin/external/llvm-project/llvm/_virtual_includes/JITLinkTableGen -Ibazel-out/darwin-opt/bin/external/llvm-project/mlir/_virtual_includes/GPUToROCDLTGen -Ibazel-out/darwin-opt/bin/external/llvm-project/mlir/_virtual_includes/MLIRShapeCanonicalizationIncGen -Ibazel-out/darwin-opt/bin/external/llvm-project/mlir/_virtual_includes/ShapeToStandardGen -Ibazel-out/darwin-opt/bin/external/llvm-project/mlir/_virtual_includes/TosaDialectBytecodeGen -Ibazel-out/darwin-opt/bin/external/stablehlo/_virtual_includes/reference_ops -Ibazel-out/darwin-opt/bin/external/stablehlo/_virtual_includes/base -Ibazel-out/darwin-opt/bin/external/stablehlo/_virtual_includes/base_attr_interfaces_inc_gen -Ibazel-out/darwin-opt/bin/external/stablehlo/_virtual_includes/reference_axes -Ibazel-out/darwin-opt/bin/external/stablehlo/_virtual_includes/reference_configuration -Ibazel-out/darwin-opt/bin/external/stablehlo/_virtual_includes/reference_errors -Ibazel-out/darwin-opt/bin/external/stablehlo/_virtual_includes/reference_process -Ibazel-out/darwin-opt/bin/external/stablehlo/_virtual_includes/reference_process_grid -Ibazel-out/darwin-opt/bin/external/stablehlo/_virtual_includes/reference_tensor -Ibazel-out/darwin-opt/bin/external/stablehlo/_virtual_includes/reference_element -Ibazel-out/darwin-opt/bin/external/stablehlo/_virtual_includes/reference_types -Ibazel-out/darwin-opt/bin/external/stablehlo/_virtual_includes/reference_index -Ibazel-out/darwin-opt/bin/external/stablehlo/_virtual_includes/reference_scope -Ibazel-out/darwin-opt/bin/external/stablehlo/_virtual_includes/reference_token -Ibazel-out/darwin-opt/bin/external/stablehlo/_virtual_includes/stablehlo_ops -Ibazel-out/darwin-opt/bin/external/stablehlo/_virtual_includes/stablehlo_assembly_format -Ibazel-out/darwin-opt/bin/external/stablehlo/_virtual_includes/stablehlo_attrs_inc_gen -Ibazel-out/darwin-opt/bin/external/stablehlo/_virtual_includes/stablehlo_enums_inc_gen -Ibazel-out/darwin-opt/bin/external/stablehlo/_virtual_includes/stablehlo_ops_inc_gen -Ibazel-out/darwin-opt/bin/external/stablehlo/_virtual_includes/stablehlo_type_inference -Ibazel-out/darwin-opt/bin/external/stablehlo/_virtual_includes/stablehlo_types_inc_gen -Ibazel-out/darwin-opt/bin/external/stablehlo/_virtual_includes/version -Ibazel-out/darwin-opt/bin/external/stablehlo/_virtual_includes/reference_value -Ibazel-out/darwin-opt/bin/external/stablehlo/_virtual_includes/chlo_ops -Ibazel-out/darwin-opt/bin/external/stablehlo/_virtual_includes/broadcast_utils -Ibazel-out/darwin-opt/bin/external/stablehlo/_virtual_includes/chlo_attrs_inc_gen -Ibazel-out/darwin-opt/bin/external/stablehlo/_virtual_includes/chlo_enums_inc_gen -Ibazel-out/darwin-opt/bin/external/stablehlo/_virtual_includes/chlo_ops_inc_gen -Ibazel-out/darwin-opt/bin/external/stablehlo/_virtual_includes/stablehlo_passes -Ibazel-out/darwin-opt/bin/external/stablehlo/_virtual_includes/linalg_passes -Ibazel-out/darwin-opt/bin/external/stablehlo/_virtual_includes/linalg_pass_inc_gen -Ibazel-out/darwin-opt/bin/external/stablehlo/_virtual_includes/stablehlo_pass_inc_gen -Ibazel-out/darwin-opt/bin/external/stablehlo/_virtual_includes/vhlo_ops -Ibazel-out/darwin-opt/bin/external/stablehlo/_virtual_includes/vhlo_attr_interfaces_inc_gen -Ibazel-out/darwin-opt/bin/external/stablehlo/_virtual_includes/vhlo_attrs_inc_gen -Ibazel-out/darwin-opt/bin/external/stablehlo/_virtual_includes/vhlo_enums_inc_gen -Ibazel-out/darwin-opt/bin/external/stablehlo/_virtual_includes/vhlo_op_interfaces_inc_gen -Ibazel-out/darwin-opt/bin/external/stablehlo/_virtual_includes/vhlo_ops_inc_gen -Ibazel-out/darwin-opt/bin/external/stablehlo/_virtual_includes/vhlo_types -Ibazel-out/darwin-opt/bin/external/stablehlo/_virtual_includes/vhlo_type_interfaces_inc_gen -Ibazel-out/darwin-opt/bin/external/stablehlo/_virtual_includes/vhlo_types_inc_gen -Ibazel-out/darwin-opt/bin/external/xla/xla/mlir_hlo/_virtual_includes/mlir_hlo -Ibazel-out/darwin-opt/bin/external/xla/xla/mlir_hlo/_virtual_includes/canonicalize_inc_gen -Ibazel-out/darwin-opt/bin/external/xla/xla/mlir_hlo/_virtual_includes/convert_op_folder -Ibazel-out/darwin-opt/bin/external/xla/xla/mlir_hlo/_virtual_includes/hlo_ops_attrs_inc_gen -Ibazel-out/darwin-opt/bin/external/xla/xla/mlir_hlo/_virtual_includes/hlo_ops_common -Ibazel-out/darwin-opt/bin/external/xla/xla/mlir_hlo/_virtual_includes/hlo_ops_enums_inc_gen -Ibazel-out/darwin-opt/bin/external/xla/xla/mlir_hlo/_virtual_includes/hlo_ops_inc_gen -Ibazel-out/darwin-opt/bin/external/xla/xla/mlir_hlo/_virtual_includes/hlo_ops_pattern_inc_gen -Ibazel-out/darwin-opt/bin/external/xla/xla/mlir_hlo/_virtual_includes/hlo_ops_typedefs_inc_gen -isystem external/llvm-project/mlir/include -isystem bazel-out/darwin-opt/bin/external/llvm-project/mlir/include -isystem external/llvm-project/llvm/include -isystem bazel-out/darwin-opt/bin/external/llvm-project/llvm/include -isystem external/llvm-project/llvm/lib/Target/NVPTX -isystem bazel-out/darwin-opt/bin/external/llvm-project/llvm/lib/Target/NVPTX -isystem external/enzyme/Enzyme -isystem bazel-out/darwin-opt/bin/external/enzyme/Enzyme -isystem external/enzyme/Enzyme/MLIR -isystem bazel-out/darwin-opt/bin/external/enzyme/Enzyme/MLIR -isystem external/llvm-project/mlir/lib/Conversion/ArithToEmitC -isystem bazel-out/darwin-opt/bin/external/llvm-project/mlir/lib/Conversion/ArithToEmitC -isystem external/llvm-project/mlir/lib/Conversion/FuncToSPIRV -isystem bazel-out/darwin-opt/bin/external/llvm-project/mlir/lib/Conversion/FuncToSPIRV -isystem external/llvm-project/mlir/lib/Conversions/GPUToSPIRV -isystem bazel-out/darwin-opt/bin/external/llvm-project/mlir/lib/Conversions/GPUToSPIRV -isystem external/llvm-project/mlir/lib/Conversion/MemRefToSPIRV -isystem bazel-out/darwin-opt/bin/external/llvm-project/mlir/lib/Conversion/MemRefToSPIRV -isystem external/llvm-project/mlir/lib/Conversion/FuncToEmitC -isystem bazel-out/darwin-opt/bin/external/llvm-project/mlir/lib/Conversion/FuncToEmitC -isystem external/llvm-project/mlir/lib/Conversion/MathToEmitC -isystem bazel-out/darwin-opt/bin/external/llvm-project/mlir/lib/Conversion/MathToEmitC -isystem external/llvm-project/mlir/lib/Conversion/MathToSPIRV -isystem bazel-out/darwin-opt/bin/external/llvm-project/mlir/lib/Conversion/MathToSPIRV -isystem external/llvm-project/mlir/lib/Conversion/MemRefToEmitC -isystem bazel-out/darwin-opt/bin/external/llvm-project/mlir/lib/Conversion/MemRefToEmitC -isystem external/llvm-project/mlir/lib/Conversion/TensorToLinalg -isystem bazel-out/darwin-opt/bin/external/llvm-project/mlir/lib/Conversion/TensorToLinalg -isystem external/llvm-project/mlir/lib/Conversion/TensorToSPIRV -isystem bazel-out/darwin-opt/bin/external/llvm-project/mlir/lib/Conversion/TensorToSPIRV -isystem external/llvm-project/mlir/lib/Conversion/TosaToArith -isystem bazel-out/darwin-opt/bin/external/llvm-project/mlir/lib/Conversion/TosaToArith -isystem external/llvm-project/mlir/lib/Conversion/TosaToLinalg -isystem bazel-out/darwin-opt/bin/external/llvm-project/mlir/lib/Conversion/TosaToLinalg -isystem external/llvm-project/mlir/lib/Conversion/TosaToMLProgram -isystem bazel-out/darwin-opt/bin/external/llvm-project/mlir/lib/Conversion/TosaToMLProgram -isystem external/llvm-project/mlir/lib/Conversion/TosaToSCF -isystem bazel-out/darwin-opt/bin/external/llvm-project/mlir/lib/Conversion/TosaToSCF -isystem external/llvm-project/mlir/lib/Conversion/TosaToTensor -isystem bazel-out/darwin-opt/bin/external/llvm-project/mlir/lib/Conversion/TosaToTensor '-std=c++17' -w '-std=c++17' '-DTCP_USER_TIMEOUT=0' '-Werror=unused-variable' '-Werror=unused-but-set-variable' '-Werror=return-type' '-Werror=unused-result' -no-canonical-prefixes -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' '-fmodule-name=@enzyme_ad//src/enzyme_ad/jax:XLADerivatives' '-fmodule-map-file=bazel-out/darwin-opt/bin/external/enzyme_ad/src/enzyme_ad/jax/XLADerivatives.cppmap' -fmodules-strict-decluse -Wprivate-header '-fmodule-map-file=external/local_config_cc/module.modulemap' '-fmodule-map-file=bazel-out/darwin-opt/bin/external/enzyme_ad/src/enzyme_ad/jax/EnzymeXLAOpsIncGen.cppmap' '-fmodule-map-file=bazel-out/darwin-opt/bin/external/enzyme_ad/src/enzyme_ad/jax/EnzymeXLAPassesIncGen.cppmap' '-fmodule-map-file=bazel-out/darwin-opt/bin/external/enzyme_ad/src/enzyme_ad/jax/EnzyeHLOPatternsIncGen.cppmap' '-fmodule-map-file=bazel-out/darwin-opt/bin/external/llvm-project/mlir/DLTIDialect.cppmap' '-fmodule-map-file=bazel-out/darwin-opt/bin/external/llvm-project/mlir/GPUPipelines.cppmap' '-fmodule-map-file=bazel-out/darwin-opt/bin/external/llvm-project/llvm/Core.cppmap' '-fmodule-map-file=bazel-out/darwin-opt/bin/external/llvm-project/llvm/ExecutionEngine.cppmap' '-fmodule-map-file=bazel-out/darwin-opt/bin/external/llvm-project/llvm/OrcJIT.cppmap' '-fmodule-map-file=bazel-out/darwin-opt/bin/external/llvm-project/llvm/OrcTargetProcess.cppmap' '-fmodule-map-file=bazel-out/darwin-opt/bin/external/llvm-project/llvm/Passes.cppmap' '-fmodule-map-file=bazel-out/darwin-opt/bin/external/llvm-project/llvm/Scalar.cppmap' '-fmodule-map-file=bazel-out/darwin-opt/bin/external/llvm-project/llvm/InstCombine.cppmap' '-fmodule-map-file=bazel-out/darwin-opt/bin/external/llvm-project/llvm/IPO.cppmap' '-fmodule-map-file=bazel-out/darwin-opt/bin/external/enzyme_ad/src/enzyme_ad/jax/mhlo-derivatives.cppmap' '-fmodule-map-file=bazel-out/darwin-opt/bin/external/enzyme_ad/src/enzyme_ad/jax/stablehlo-derivatives.cppmap' '-fmodule-map-file=bazel-out/darwin-opt/bin/external/enzyme_ad/src/enzyme_ad/jax/chlo-derivatives.cppmap' '-fmodule-map-file=bazel-out/darwin-opt/bin/external/enzyme/EnzymeMLIR.cppmap' '-fmodule-map-file=bazel-out/darwin-opt/bin/external/llvm-project/llvm/Support.cppmap' '-fmodule-map-file=bazel-out/darwin-opt/bin/external/llvm-project/mlir/MemorySlotInterfaces.cppmap' '-fmodule-map-file=bazel-out/darwin-opt/bin/external/llvm-project/mlir/AffineDialect.cppmap' '-fmodule-map-file=bazel-out/darwin-opt/bin/external/llvm-project/mlir/ArithDialect.cppmap' '-fmodule-map-file=bazel-out/darwin-opt/bin/external/llvm-project/mlir/ArithUtils.cppmap' '-fmodule-map-file=bazel-out/darwin-opt/bin/external/llvm-project/mlir/CommonFolders.cppmap' '-fmodule-map-file=bazel-out/darwin-opt/bin/external/llvm-project/mlir/ComplexDialect.cppmap' '-fmodule-map-file=bazel-out/darwin-opt/bin/external/llvm-project/mlir/ControlFlowInterfaces.cppmap' '-fmodule-map-file=bazel-out/darwin-opt/bin/external/llvm-project/mlir/FuncDialect.cppmap' '-fmodule-map-file=bazel-out/darwin-opt/bin/external/llvm-project/mlir/LLVMDialect.cppmap' '-fmodule-map-file=bazel-out/darwin-opt/bin/external/llvm-project/mlir/AffineToStandard.cppmap' '-fmodule-map-file=bazel-out/darwin-opt/bin/external/llvm-project/mlir/ArithToLLVM.cppmap' '-fmodule-map-file=bazel-out/darwin-opt/bin/external/llvm-project/mlir/MemRefTransforms.cppmap' '-fmodule-map-file=bazel-out/darwin-opt/bin/external/llvm-project/mlir/TransformDialect.cppmap' '-fmodule-map-file=bazel-out/darwin-opt/bin/external/llvm-project/mlir/TransformDialectInterfaces.cppmap' '-fmodule-map-file=bazel-out/darwin-opt/bin/external/llvm-project/mlir/TransformDialectTransforms.cppmap' '-fmodule-map-file=bazel-out/darwin-opt/bin/external/llvm-project/mlir/GPUToGPURuntimeTransforms.cppmap' '-fmodule-map-file=bazel-out/darwin-opt/bin/external/llvm-project/mlir/GPUCommonTransforms.cppmap' '-fmodule-map-file=bazel-out/darwin-opt/bin/external/llvm-project/mlir/GPUToNVVMTransforms.cppmap' '-fmodule-map-file=bazel-out/darwin-opt/bin/external/llvm-project/mlir/GPUTransforms.cppmap' '-fmodule-map-file=bazel-out/darwin-opt/bin/external/llvm-project/mlir/LinalgTransforms.cppmap' '-fmodule-map-file=bazel-out/darwin-opt/bin/external/llvm-project/mlir/ToLLVMIRTranslation.cppmap' '-fmodule-map-file=bazel-out/darwin-opt/bin/external/llvm-project/mlir/FuncToLLVM.cppmap' '-fmodule-map-file=bazel-out/darwin-opt/bin/external/llvm-project/mlir/SCFDialect.cppmap' '-fmodule-map-file=bazel-out/darwin-opt/bin/external/llvm-project/mlir/GPUDialect.cppmap' '-fmodule-map-file=bazel-out/darwin-opt/bin/external/llvm-project/mlir/Transforms.cppmap' '-fmodule-map-file=bazel-out/darwin-opt/bin/external/llvm-project/mlir/NVVMDialect.cppmap' '-fmodule-map-file=bazel-out/darwin-opt/bin/external/llvm-project/mlir/IndexToLLVM.cppmap' '-fmodule-map-file=bazel-out/darwin-opt/bin/external/llvm-project/mlir/NVVMToLLVM.cppmap' '-fmodule-map-file=bazel-out/darwin-opt/bin/external/llvm-project/mlir/ReconcileUnrealizedCasts.cppmap' '-fmodule-map-file=bazel-out/darwin-opt/bin/external/llvm-project/mlir/SCFToControlFlow.cppmap' '-fmodule-map-file=bazel-out/darwin-opt/bin/external/llvm-project/mlir/MathToLLVM.cppmap' '-fmodule-map-file=bazel-out/darwin-opt/bin/external/llvm-project/mlir/VectorToLLVM.cppmap' '-fmodule-map-file=bazel-out/darwin-opt/bin/external/llvm-project/mlir/VectorToSCF.cppmap' '-fmodule-map-file=bazel-out/darwin-opt/bin/external/llvm-project/mlir/OpenMPDialect.cppmap' '-fmodule-map-file=bazel-out/darwin-opt/bin/external/llvm-project/mlir/MemRefToLLVM.cppmap' '-fmodule-map-file=bazel-out/darwin-opt/bin/external/llvm-project/mlir/NVGPUToNVVM.cppmap' '-fmodule-map-file=bazel-out/darwin-opt/bin/external/llvm-project/mlir/MathDialect.cppmap' '-fmodule-map-file=bazel-out/darwin-opt/bin/external/llvm-project/mlir/NVGPUDialect.cppmap' '-fmodule-map-file=bazel-out/darwin-opt/bin/external/llvm-project/mlir/VectorDialect.cppmap' '-fmodule-map-file=bazel-out/darwin-opt/bin/external/llvm-project/mlir/MemRefDialect.cppmap' '-fmodule-map-file=bazel-out/darwin-opt/bin/external/llvm-project/mlir/FunctionInterfaces.cppmap' '-fmodule-map-file=bazel-out/darwin-opt/bin/external/llvm-project/mlir/IR.cppmap' '-fmodule-map-file=bazel-out/darwin-opt/bin/external/llvm-project/mlir/LLVMCommonConversion.cppmap' '-fmodule-map-file=bazel-out/darwin-opt/bin/external/llvm-project/mlir/SideEffectInterfaces.cppmap' '-fmodule-map-file=bazel-out/darwin-opt/bin/external/llvm-project/mlir/BytecodeOpInterface.cppmap' '-fmodule-map-file=bazel-out/darwin-opt/bin/external/llvm-project/mlir/Pass.cppmap' '-fmodule-map-file=bazel-out/darwin-opt/bin/external/llvm-project/mlir/Support.cppmap' '-fmodule-map-file=bazel-out/darwin-opt/bin/external/llvm-project/mlir/TensorDialect.cppmap' '-fmodule-map-file=bazel-out/darwin-opt/bin/external/llvm-project/mlir/TransformUtils.cppmap' '-fmodule-map-file=bazel-out/darwin-opt/bin/external/llvm-project/mlir/TranslateLib.cppmap' '-fmodule-map-file=bazel-out/darwin-opt/bin/external/llvm-project/mlir/FromLLVMIRTranslation.cppmap' '-fmodule-map-file=bazel-out/darwin-opt/bin/external/llvm-project/mlir/ToLLVMIRTranslationRegistration.cppmap' '-fmodule-map-file=bazel-out/darwin-opt/bin/external/llvm-project/mlir/FromLLVMIRTranslationRegistration.cppmap' '-fmodule-map-file=bazel-out/darwin-opt/bin/external/stablehlo/reference_ops.cppmap' '-fmodule-map-file=bazel-out/darwin-opt/bin/external/stablehlo/stablehlo_ops.cppmap' '-fmodule-map-file=bazel-out/darwin-opt/bin/external/stablehlo/chlo_ops.cppmap' '-fmodule-map-file=bazel-out/darwin-opt/bin/external/stablehlo/stablehlo_passes.cppmap' '-fmodule-map-file=bazel-out/darwin-opt/bin/external/stablehlo/stablehlo_type_inference.cppmap' '-fmodule-map-file=bazel-out/darwin-opt/bin/external/xla/xla/mlir_hlo/mlir_hlo.cppmap' -c external/enzyme_ad/src/enzyme_ad/jax/Passes/ControlFlowToSCF.cpp -o bazel-out/darwin-opt/bin/external/enzyme_ad/src/enzyme_ad/jax/_objs/XLADerivatives/ControlFlowToSCF.pic.o)
[22:52:42] # Configuration: 4e11616f7d44db1bb957c4829a8972cb7ddd8ad2f9c100b3ac94d6dcfac86b15
[22:52:42] # Execution platform: @local_execution_config_platform//:platform
[22:52:42] 
[22:52:42] Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
[22:52:42] external/enzyme_ad/src/enzyme_ad/jax/Passes/ControlFlowToSCF.cpp:15:10: error: module @enzyme_ad//src/enzyme_ad/jax:XLADerivatives does not depend on a module exporting 'mlir/Conversion/ControlFlowToSCF/ControlFlowToSCF.h'
[22:52:42]    15 | #include "mlir/Conversion/ControlFlowToSCF/ControlFlowToSCF.h"
[22:52:42]       |          ^
[22:52:42] external/enzyme_ad/src/enzyme_ad/jax/Passes/ControlFlowToSCF.cpp:17:10: error: module @enzyme_ad//src/enzyme_ad/jax:XLADerivatives does not depend on a module exporting 'mlir/Dialect/ControlFlow/IR/ControlFlow.h'
[22:52:42]    17 | #include "mlir/Dialect/ControlFlow/IR/ControlFlow.h"
[22:52:42]       |          ^
[22:52:42] external/enzyme_ad/src/enzyme_ad/jax/Passes/ControlFlowToSCF.cpp:18:10: error: module @enzyme_ad//src/enzyme_ad/jax:XLADerivatives does not depend on a module exporting 'mlir/Dialect/ControlFlow/IR/ControlFlowOps.h'
[22:52:42]    18 | #include "mlir/Dialect/ControlFlow/IR/ControlFlowOps.h"
[22:52:42]       |          ^
[22:52:42] external/enzyme_ad/src/enzyme_ad/jax/Passes/ControlFlowToSCF.cpp:21:10: error: module @enzyme_ad//src/enzyme_ad/jax:XLADerivatives does not depend on a module exporting 'mlir/Dialect/UB/IR/UBOps.h'
[22:52:42]    21 | #include "mlir/Dialect/UB/IR/UBOps.h"
[22:52:42]       |          ^
[22:52:42] 4 errors generated.
[22:52:43] [9,397 / 9,451] Compiling xla/mlir_hlo/mhlo/transforms/hlo_legalize_to_arithmetic/hlo_legalize_to_arithmetic.cc; 40s processwrapper-sandbox ... (15 actions running)
[22:52:46] [9,398 / 9,451] Compiling xla/mlir_hlo/mhlo/transforms/hlo_legalize_to_arithmetic/hlo_legalize_to_arithmetic.cc; 43s processwrapper-sandbox ... (14 actions running)
[22:52:47] [9,408 / 9,451] Compiling src/enzyme_ad/jax/Passes/ArithRaising.cpp; 42s processwrapper-sandbox ... (4 actions running)
[22:52:47] Target //:libReactantExtra.so failed to build
[22:52:48] INFO: Elapsed time: 7670.177s, Critical Path: 695.11s
[22:52:48] INFO: 9412 processes: 2239 internal, 7173 processwrapper-sandbox.
[22:52:48] FAILED: Build did NOT complete successfully

@wsmoses this is your stuff, it was working before EnzymeAD/Reactant.jl@728794f (#640) 🥲

# Compiling LLVM components within XLA requires macOS SDK 10.14.
pushd $WORKSPACE/srcdir/MacOSX10.*.sdk
# But then we need 13.3 for another `std::hardware_destructive_interference_size`:
# <https://github.com/JuliaPackaging/Yggdrasil/pull/10313#issuecomment-2616125018>.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@giordano giordano mentioned this pull request Jan 28, 2025
@giordano giordano force-pushed the mg/reactant-cuda-aarch64 branch from 76d6cb8 to b189eda Compare January 28, 2025 18:54
@giordano giordano merged commit 6a91fce into JuliaPackaging:master Jan 28, 2025
7 checks passed
@giordano giordano deleted the mg/reactant-cuda-aarch64 branch January 28, 2025 21:53
@giordano
Copy link
Member Author

🚀

1 similar comment
@wsmoses
Copy link
Contributor

wsmoses commented Jan 28, 2025

🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants