Compile/test error on Apple M1 #155

JakobAsslaender · 2021-10-19T18:19:49Z

Hi!
I just quickly wanted to draw your attention to an error I get when trying to install FastTransforms.jl (or ApproxFun.jl) on an Apple M1 CPU natively (Julia v1.7.0-rc1). I know this platform is only Tier-3 and has still many issues, but maybe there is an easy fix for this? The issue seems to be related to BinaryProvider.jl, yet, I am posting this here since BinaryProvider.jl does not seem to maintained based on the last commit being 1.5 years old.

Thanks a ton for looking into this!

The error message when running test FastTransforms message is:

Precompiling project...
  ✗ BinaryProvider
  0 dependencies successfully precompiled in 1 seconds (49 already precompiled)
  1 dependency errored. To see a full report either run `import Pkg; Pkg.precompile()` or load the package
     Testing Running tests...
ERROR: LoadError: InitError: UndefVarError: libfasttransforms not defined
Stacktrace:
 [1] ft_set_num_threads(n::Int64)
   @ FastTransforms ~/.julia/packages/FastTransforms/BaTvR/src/libfasttransforms.jl:13
 [2] __init__()
   @ FastTransforms ~/.julia/packages/FastTransforms/BaTvR/src/libfasttransforms.jl:18
 [3] _include_from_serialized(path::String, depmods::Vector{Any})
   @ Base ./loading.jl:768
 [4] _require_search_from_serialized(pkg::Base.PkgId, sourcepath::String)
   @ Base ./loading.jl:854
 [5] _require(pkg::Base.PkgId)
   @ Base ./loading.jl:1097
 [6] require(uuidkey::Base.PkgId)
   @ Base ./loading.jl:1013
 [7] require(into::Module, mod::Symbol)
   @ Base ./loading.jl:997
 [8] include(fname::String)
   @ Base.MainInclude ./client.jl:451
 [9] top-level scope
   @ none:6
during initialization of module FastTransforms
in expression starting at /Users/asslaj01/.julia/packages/FastTransforms/BaTvR/test/runtests.jl:1
ERROR: Package FastTransforms errored during testing

The text was updated successfully, but these errors were encountered:

MikaelSlevinsky · 2021-10-21T12:55:48Z

Hi @JakobAsslaender, I think this issue can be worked on now that all binary dependencies are available on aarch64-apple-darwin.

The "easy fix" would be to see if it builds from source locally https://juliaapproximation.github.io/FastTransforms.jl/dev/dev/#The-developer's-right-to-build-from-source-1 and if not report any errors and possibly warnings too. (I don't have an M1 yet, and I don't know if there's any CI that has it, so that might actually speed things up.)

JakobAsslaender · 2021-10-21T13:42:35Z

Thanks, @MikaelSlevinsky for the quick response and the kind support! The error I am getting again related to the BinaryProvider.jl package and the issue #205.

Here is the error message:

(@v1.7) pkg> build FastTransforms
    Building TimeZones ─────→ `~/.julia/scratchspaces/44cfe95a-1eb2-52ea-b672-e2afdf69b78f/b4c6460412b1db0b4f1679ab2d5ef72568a14a57/build.log`
    Building FastTransforms → `~/.julia/dev/FastTransforms/deps/build.log`
ERROR: Error building `FastTransforms`: 
┌ Warning: Platform `arm64-apple-darwin20.5.0` is not an officially supported platform
└ @ BinaryProvider ~/.julia/packages/BinaryProvider/U2dKK/src/PlatformNames.jl:450
ERROR: LoadError: KeyError: key "unknown" not found
Stacktrace:
  [1] getindex
    @ ./dict.jl:481 [inlined]
  [2] parse_dl_name_version
    @ ~/.julia/packages/BinaryProvider/U2dKK/src/PlatformNames.jl:511 [inlined]
  [3] detect_libgfortran_abi(libgfortran_name::String, platform::BinaryProvider.UnknownPlatform) (repeats 2 times)
    @ BinaryProvider ~/.julia/packages/BinaryProvider/U2dKK/src/PlatformNames.jl:555
  [4] detect_libgfortran_abi
    @ ~/.julia/packages/BinaryProvider/U2dKK/src/PlatformNames.jl:584 [inlined]
  [5] detect_compiler_abi()
    @ BinaryProvider ~/.julia/packages/BinaryProvider/U2dKK/src/PlatformNames.jl:667
  [6] top-level scope
    @ ~/.julia/packages/BinaryProvider/U2dKK/src/PlatformNames.jl:685
  [7] include(mod::Module, _path::String)
    @ Base ./Base.jl:420
  [8] include(x::String)
    @ BinaryProvider ~/.julia/packages/BinaryProvider/U2dKK/src/BinaryProvider.jl:1
  [9] top-level scope
    @ ~/.julia/packages/BinaryProvider/U2dKK/src/BinaryProvider.jl:12
 [10] include
    @ ./Base.jl:420 [inlined]
 [11] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt64}}, source::String)
    @ Base ./loading.jl:1318
 [12] top-level scope
    @ none:1
 [13] eval
    @ ./boot.jl:373 [inlined]
 [14] eval(x::Expr)
    @ Base.MainInclude ./client.jl:453
 [15] top-level scope
    @ none:1
in expression starting at /Users/asslaj01/.julia/packages/BinaryProvider/U2dKK/src/PlatformNames.jl:685
in expression starting at /Users/asslaj01/.julia/packages/BinaryProvider/U2dKK/src/BinaryProvider.jl:1
ERROR: LoadError: Failed to precompile BinaryProvider [b99e7846-7c00-51b0-8f62-c81ae34c0232] to /Users/asslaj01/.julia/compiled/v1.7/BinaryProvider/jl_efaTyB.
Stacktrace:
 [1] error(s::String)
   @ Base ./error.jl:33
 [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_stdout::IO, ignore_loaded_modules::Bool)
   @ Base ./loading.jl:1466
 [3] compilecache(pkg::Base.PkgId, path::String)
   @ Base ./loading.jl:1410
 [4] _require(pkg::Base.PkgId)
   @ Base ./loading.jl:1120
 [5] require(uuidkey::Base.PkgId)
   @ Base ./loading.jl:1013
 [6] require(into::Module, mod::Symbol)
   @ Base ./loading.jl:997
 [7] include(fname::String)
   @ Base.MainInclude ./client.jl:451
 [8] top-level scope
   @ none:5
in expression starting at /Users/asslaj01/.julia/dev/FastTransforms/deps/build.jl:1

MikaelSlevinsky · 2021-10-21T14:57:20Z

I see. Well, it looks like the only use of BinaryProvider here is to configure the build script, which is technically independent (just a convenience) to checking whether or not the library compiles on a particular platform.

JakobAsslaender · 2021-10-22T14:08:35Z

On my quest of debugging this, I removed BinaryBuilder.jl form the Project file and the build.jl--as well as I could.

The latter now looks the following:

# using BinaryProvider
import Libdl

const extension = Sys.isapple() ? "dylib" : Sys.islinux() ? "so" : Sys.iswindows() ? "dll" : ""

print_error() = error(
    "FastTransforms could not be properly installed.\n Please check that you have all dependencies installed. " *
    "Sample installation of dependencies:\n" *
    print_platform_error(platform_key_abi())
)

# print_platform_error(p::Platform) = "On $(BinaryProvider.platform_name(p)), please consider opening a pull request to add support to build from source.\n"
print_platform_error(p) = "On MacOS\n\tbrew install libomp fftw mpfr\n"
# print_platform_error(p::Linux) = "On Linux\n\tsudo apt-get install libomp-dev libblas-dev libopenblas-base libfftw3-dev libmpfr-dev\n"
# print_platform_error(p::Windows) = "On Windows\n\tvcpkg install openblas:x64-windows fftw3[core,threads]:x64-windows mpir:x64-windows mpfr:x64-windows\n"

Compiling now throws the following error message

./src/fasttransforms.h:9:10: fatal error: 'fftw3.h' file not found
#include <fftw3.h>
         ^~~~~~~~~
1 error generated.
In file included from src/transforms.c:3:
./src/fasttransforms.h:9:10: fatal error: 'fftw3.h' file not found
#include <fftw3.h>

Note that I did run brew install libomp fftw mpfr and it exited without errors.

Again, thanks a ton for your support!

JakobAsslaender · 2021-10-22T14:51:09Z

Next step: I hacked the makefile (~/.julia/dev/FastTransforms/deps/FastTransforms/Make.inc) -- really not the way to go, I know -- to have the following includes and links instead of the original ones:


        CFLAGS += -I/opt/homebrew/Cellar/libomp/13.0.0/include
        LDFLAGS += -L/opt/homebrew/Cellar/libomp/13.0.0/lib
        CFLAGS += -I/opt/homebrew/Cellar/fftw/3.3.10/include
        LDFLAGS += -L/opt/homebrew/Cellar/fftw/3.3.10/lib
        CFLAGS += -I/opt/homebrew/Cellar/mpfr/4.1.0/include
        LDFLAGS += -L/opt/homebrew/Cellar/mpfr/4.1.0/lib
        CFLAGS += -I/opt/homebrew/Cellar/gmp/6.2.1_1/include
        LDFLAGS += -L/opt/homebrew/Cellar/gmp/6.2.1_1/lib

With this, FastTransforms compiles! Yet, I still cannot compile Approxjfun.jl, it still throws the following error:

ERROR: The following 1 direct dependency failed to precompile:

ApproxFun [28f2ccd6-bb30-5033-b560-165f7b14dc2f]

Failed to precompile ApproxFun [28f2ccd6-bb30-5033-b560-165f7b14dc2f] to /Users/asslaj01/.julia/compiled/v1.7/ApproxFun/jl_UnJw4v.
ERROR: LoadError: InitError: UndefVarError: libfasttransforms not defined

Once again, thanks for your help @MikaelSlevinsky !

MikaelSlevinsky · 2021-10-22T14:57:41Z

Ok, that's really great news: thank you for testing this! (A while ago, I made sure that the library compiled on AArch64, but never tested specifically with macOS + AArch64.) This means that the release cycle of a new build with precompiled binaries should be relatively straightforward. Let me take a look over the next week or two.

JakobAsslaender · 2021-10-22T15:11:38Z

I get the same error ERROR: UndefVarError: libfasttransforms not defined when trying to run any FastTransforms function.

When trying to run

p = dlopen(expander("~/.julia/dev/deps/libfasttransforms"), RTLD_LAZY; throw_error=true)

I get the following error:

ERROR: could not load library "~/.julia/dev/deps/libfasttransforms"
dlopen(/Users/asslaj01/.julia/dev/deps/libfasttransforms.dylib, 1): image not found

Does that help to trace the error, @MikaelSlevinsky ?

MikaelSlevinsky · 2021-10-22T16:23:56Z

Try moving the library to ~/.julia/dev/FastTransforms/deps/libfasttransforms.dylib

JakobAsslaender · 2021-10-27T16:38:15Z

Thanks! Sorry, the last one was a stupid mistake of mine. I got it to install alright -- and I can use the package. Yet, the test routine throws a segmentation fault:

     Testing Running tests...
Test Summary:     | Pass  Total
Special functions |   19     19
Test Summary:       | Pass  Total
Chebyshev transform |  369    369
Test Summary:                        | Pass  Total
Fejér and Clenshaw–Curtis quadrature |    9      9

signal (11): Segmentation fault: 11
in expression starting at /Users/asslaj01/.julia/dev/FastTransforms/test/libfasttransformstests.jl:5
unknown function (ip: 0x401)
Allocations: 100650954 (Pool: 100629067; Big: 21887); GC: 112
ERROR: Package FastTransforms errored during testing (exit code: 139)

Again, many thanks for your help!

abhayap · 2021-12-06T19:05:11Z

@MikaelSlevinsky I am interested in using FastTransforms on an Apple M1 CPU too. I currently get this error while trying to install it:

[057dd010] ↑ FastTransforms v0.13.1 ⇒ v0.13.2
    Building FastTransforms → `~/.julia/scratchspaces/44cfe95a-1eb2-52ea-b672-e2afdf69b78f/848570d9a053b0e270b79a9cea95b409b317bb07/build.log`
ERROR: Error building `FastTransforms`: 
┌ Warning: Platform `arm64-apple-darwin21.1.0` is not an officially supported platform
└ @ BinaryProvider ~/.julia/packages/BinaryProvider/U2dKK/src/PlatformNames.jl:450

MikaelSlevinsky · 2021-12-07T04:06:16Z

Hi, in principle this would be feasible by a small change to the build script in Julia's Yggdrasil. I'm personally hesitant because

I think there is an issue of lacking testing machines Support for VMs on Apple M1 actions/runner-images#2187
I don't have an M1 computer personally, so couldn't provide any nontrivial support
It's great that it compiles, but I'm stumped by the segmentation fault above. (I could understand an illegal instruction, but memory issues are more important and harder to debug. See point 2.).

All that said, enthusiastic pull requests from folks with the right tools would not be ignored!

abhayap · 2022-01-05T07:08:42Z

I sent a PR here that should work on Apple M1 using Julia v1.7: JuliaPackaging/Yggdrasil#4161

dlfivefifty · 2022-01-05T12:32:21Z

This is awesome, is it time to get an M1?? (Or maybe wait til M2 at this point)

MikaelSlevinsky · 2022-01-05T17:44:26Z

Work bought me an M1 pro for Christmas though it won't come in for a couple more weeks. (I didn't want to mention it to falsely get people's hopes up, but yeah this is great progress @abhayap)

abhayap · 2022-01-05T20:47:10Z

Thanks @MikaelSlevinsky!

MikaelSlevinsky · 2022-01-06T15:59:09Z

Let's assume this issue is closed now with v0.13.3 on Julia v1.7

MikaelSlevinsky closed this as completed Jan 6, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Compile/test error on Apple M1 #155

Compile/test error on Apple M1 #155

JakobAsslaender commented Oct 19, 2021

MikaelSlevinsky commented Oct 21, 2021

JakobAsslaender commented Oct 21, 2021

MikaelSlevinsky commented Oct 21, 2021

JakobAsslaender commented Oct 22, 2021

JakobAsslaender commented Oct 22, 2021 •

edited

Loading

MikaelSlevinsky commented Oct 22, 2021

JakobAsslaender commented Oct 22, 2021 •

edited

Loading

MikaelSlevinsky commented Oct 22, 2021 •

edited

Loading

JakobAsslaender commented Oct 27, 2021

abhayap commented Dec 6, 2021

MikaelSlevinsky commented Dec 7, 2021

abhayap commented Jan 5, 2022

dlfivefifty commented Jan 5, 2022

MikaelSlevinsky commented Jan 5, 2022

abhayap commented Jan 5, 2022

MikaelSlevinsky commented Jan 6, 2022

Compile/test error on Apple M1 #155

Compile/test error on Apple M1 #155

Comments

JakobAsslaender commented Oct 19, 2021

MikaelSlevinsky commented Oct 21, 2021

JakobAsslaender commented Oct 21, 2021

MikaelSlevinsky commented Oct 21, 2021

JakobAsslaender commented Oct 22, 2021

JakobAsslaender commented Oct 22, 2021 • edited Loading

MikaelSlevinsky commented Oct 22, 2021

JakobAsslaender commented Oct 22, 2021 • edited Loading

MikaelSlevinsky commented Oct 22, 2021 • edited Loading

JakobAsslaender commented Oct 27, 2021

abhayap commented Dec 6, 2021

MikaelSlevinsky commented Dec 7, 2021

abhayap commented Jan 5, 2022

dlfivefifty commented Jan 5, 2022

MikaelSlevinsky commented Jan 5, 2022

abhayap commented Jan 5, 2022

MikaelSlevinsky commented Jan 6, 2022

JakobAsslaender commented Oct 22, 2021 •

edited

Loading

JakobAsslaender commented Oct 22, 2021 •

edited

Loading

MikaelSlevinsky commented Oct 22, 2021 •

edited

Loading