Reduction with Bools fails on ARM

[joachimbrand]

I'm trying to get our codes to run on my new laptop with an Apple M2 chip, which I use for code development. It is great to see that most things just work for these relatively new processors 😄

I ran into errors related to issue #404 at a number of places. I understand that custom reduction operators defined by Julia functions are currently not available for ARM processors because the mechanism of implementation is only available for Intel LLVM. This affects part of our code where we found it convenient to reduce over structs.

The one place where I don't understand why I get this error is when trying to reduce Bool values with | (OR). Binary boolean operations are known to MPI, and it would make a lot of sense to me if they worked with Bool values.

Here is an example:

julia> using MPI; MPI.Init(); comm = MPI.COMM_WORLD;

julia> res = MPI.Allreduce(1, +, comm)
1

julia> res = MPI.Allreduce(true, |, comm)
ERROR: User-defined reduction operators are currently not supported on non-Intel architectures.
See https://github.com/JuliaParallel/MPI.jl/issues/404 for more details.
Stacktrace:
 [1] error(s::String)
   @ Base ./error.jl:35
 [2] MPI.Op(f::Function, T::Type; iscommutative::Bool)
   @ MPI ~/.julia/packages/MPI/APiiL/src/operators.jl:95
 [3] MPI.Op(f::Function, T::Type)
   @ MPI ~/.julia/packages/MPI/APiiL/src/operators.jl:91
 [4] Allreduce!(rbuf::MPI.RBuffer{Base.RefValue{Bool}, Base.RefValue{Bool}}, op::Function, comm::MPI.Comm)
   @ MPI ~/.julia/packages/MPI/APiiL/src/collective.jl:668
 [5] Allreduce!(sendbuf::Base.RefValue{Bool}, recvbuf::Base.RefValue{Bool}, op::Function, comm::MPI.Comm)
   @ MPI ~/.julia/packages/MPI/APiiL/src/collective.jl:670
 [6] Allreduce(obj::Bool, op::Function, comm::MPI.Comm)
   @ MPI ~/.julia/packages/MPI/APiiL/src/collective.jl:695
 [7] top-level scope
   @ REPL[10]:1

julia> versioninfo()
Julia Version 1.8.5
Commit 17cfb8e65ea (2023-01-08 06:45 UTC)
Platform Info:
  OS: macOS (arm64-apple-darwin21.5.0)
  CPU: 10 × Apple M2 Pro
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-13.0.1 (ORCJIT, apple-m1)
  Threads: 1 on 6 virtual cores

Reading the code in operators.jl I see that | is translated into an MPI reduction operator when operating on standard integer types , but not for Bool. The latter triggers the "User-defined reduction operator" mechanism, which throws an error on ARM.

While I can work around this (i.e. casting Bool values to UInt8 before using MPI.Allreduce on them), I am wondering whether the general behaviour makes sense. Wouldn't it make more sense (and be more performant) to use the built-in MPI reduction operators here instead of generating a user-defined operator?

[simonbyrne]

I think this might just be an oversight: we only switched to using predefined MPI datatypes for Bool relatively recently (#522); I think we forgot to extend operators to those.

According to the MPI spec, MPI_C_BOOL is a "logical type", and so you can use "logical or" MPI_LOR operator in reductions. In MPI.jl, this means that you want

res = MPI.Allreduce(true, MPI.LOR, comm)

Can you check that works and gives the desired results? If so, we can change it so that | will map to MPI.LOR when a Bool type is used.

[joachimbrand]

Yes, thank you! Using the logical MPI operators works, so mapping the Julia operators makes a lot of sense.
Obviously, this should be extended to MPI_LAND and 'MPI_LXOR`.