Compilation error when adding version "~> 5.0" to existing mix project

[drumusician]

Hi Andrea, as promised after our quick chat on slack, here is the issue I'm experiencing with the latest version of ex-portmidi added to a fresh mix project,

---

When adding ex-portmidi to a fresh mix project portmidi fails to compile.

Steps to reproduce:

1. mix new midi_test --sup
2. follow the exact setup instructions provided in the readme
3. mix compile
   This results in the following error output:

==> portmidi
make: Nothing to be done for `all'.
Compiling 12 files (.ex)

== Compilation error in file lib/portmidi/devices.ex ==
** (CompileError) lib/portmidi/devices.ex:2: module PortMidi.Nifs.Devices is not loaded and could not be found


21:39:44.465 [warn]  The on_load function for module Elixir.PortMidi.Nifs.Devices returned:
{:error, {:load_failed, 'Failed to load NIF library: \'dlopen(/Users/tjaco/code/elixir/midi_test/_build/dev/lib/portmidi/priv/portmidi_devices.so, 2): no suitable image found.  Did find:\n\t/Users/tjaco/code/elixir/midi_test/_build/dev/lib/portmidi/priv/portmidi_devices.so: unknown file type, first eight bytes: 0x7F 0x45 0x4C 0x46 0x02 0x01 0x01 0x00\n\t/Users/tjaco/code/elixir/midi_test/_build/dev/lib/portmidi/priv/portmidi_devices.so: stat() failed with errno=35\''}}


21:39:44.471 [warn]  The on_load function for module Elixir.PortMidi.Nifs.Input returned:
{{:badmatch, {:error, {:load_failed, 'Failed to load NIF library: \'dlopen(/Users/tjaco/code/elixir/midi_test/_build/dev/lib/portmidi/priv/portmidi_in.so, 2): no suitable image found.  Did find:\n\t/Users/tjaco/code/elixir/midi_test/_build/dev/lib/portmidi/priv/portmidi_in.so: unknown file type, first eight bytes: 0x7F 0x45 0x4C 0x46 0x02 0x01 0x01 0x00\n\t/Users/tjaco/code/elixir/midi_test/_build/dev/lib/portmidi/priv/portmidi_in.so: stat() failed with errno=35\''}}}, [{PortMidi.Nifs.Input, :init, 0, [file: 'lib/portmidi/nifs/input.ex', line: 4]}, {:code_server, :"-handle_on_load/5-fun-0-", 1, [file: 'code_server.erl', line: 1340]}]}

could not compile dependency :portmidi, "mix compile" failed. You can recompile this dependency with "mix deps.compile portmidi", update it with "mix deps.update portmidi" or clean it with "mix deps.clean portmidi"

Elixir version: 1.8.1
Erlang/OTP: 21.0

When doing the exact same as above using version "~> 4.2.0" it works as expected.

Let me know if you need any more input or if you'd like me to try anything else.

[lucidstack]

Mmhh I can't replicate, unfortunately. I can see in your pasted output that you have

==> portmidi
make: Nothing to be done for `all'.

Which should mean that the ex-portmidi NIFs were already compiled. Have you tried nuking the _build folder and running mix compile again? Could you post the output of that command, if there are any errors/warnings? Also (just throwing hypotheses at you now, sorry 😅)... do you have portmidi installed on your machine?

Thank you! 🙇‍♂️

[drumusician]

Ok, I have managed to pinpoint this a little further. The problem seems to be specifically version 5.1.2, because 5.1.1 works fine. I believe the .so files that are included in the hex package for version 5.1.2 might not be the correct files. Or maybe you don't actually want these files included in the package because simply removing them and recompiling fixed the issue for me.

So these are the steps I took to get 5.1.2 to work.

rm -rf _build
rm deps/priv/portmidi_*
mix compile

That triggered make to compile the files again and fixed the issue I had.

[thbar]

Just wanted to report I see the same problem with 5.1.2:

20:18:18.301 [warn]  The on_load function for module Elixir.PortMidi.Nifs.Devices returned:
{:error, {:load_failed, 'Failed to load NIF library: \'dlopen(/Users/thbar/git/music-playground/widgets/_build/dev/lib/portmidi/priv/portmidi_devices.so, 2): no suitable image found.  Did find:\n\t/Users/thbar/git/music-playground/widgets/_build/dev/lib/portmidi/priv/portmidi_devices.so: unknown file type, first eight bytes: 0x7F 0x45 0x4C 0x46 0x02 0x01 0x01 0x00\n\t/Users/thbar/git/music-playground/widgets/_build/dev/lib/portmidi/priv/portmidi_devices.so: stat() failed with errno=35\''}}


== Compilation error in file lib/portmidi/devices.ex ==
** (CompileError) lib/portmidi/devices.ex:2: module PortMidi.Nifs.Devices is not loaded and could not be found

I was able to install 5.1.1, though.

[thbar]

I dug a bit deeper, installing the development version of hex which supports downloading packages tarballs:

git clone [email protected]:hexpm/hex.git
mix install

Then I downloaded the 2 versions locally:

$ mix hex.package fetch portmidi 5.1.1 --unpack
portmidi v5.1.1 extracted to /Users/thbar/git/music-playground/portmidi-5.1.1

$ mix hex.package fetch portmidi 5.1.2 --unpack
portmidi v5.1.2 extracted to /Users/thbar/git/music-playground/portmidi-5.1.2

Here is the recursive diff I got with diff -r:

diff -r portmidi-5.1.1/hex_metadata.config portmidi-5.1.2/hex_metadata.config
6,16c6,10
<  [<<"priv/portmidi_devices.so">>,
<   <<"priv/portmidi_devices.so.dSYM/Contents/Info.plist">>,
<   <<"priv/portmidi_devices.so.dSYM/Contents/Resources/DWARF/portmidi_devices.so">>,
<   <<"priv/portmidi_in.so">>,
<   <<"priv/portmidi_in.so.dSYM/Contents/Info.plist">>,
<   <<"priv/portmidi_in.so.dSYM/Contents/Resources/DWARF/portmidi_in.so">>,
<   <<"priv/portmidi_out.so">>,
<   <<"priv/portmidi_out.so.dSYM/Contents/Info.plist">>,
<   <<"priv/portmidi_out.so.dSYM/Contents/Resources/DWARF/portmidi_out.so">>,
<   <<"lib/mix/tasks/portmidi.devices.ex">>,<<"lib/portmidi.ex">>,
<   <<"lib/portmidi/device.ex">>,<<"lib/portmidi/devices.ex">>,
---
>  [<<"priv">>,<<"priv/.gitkeep">>,<<"priv/portmidi_devices.so">>,
>   <<"priv/portmidi_in.so">>,<<"priv/portmidi_out.so">>,<<"lib">>,
>   <<"lib/mix">>,<<"lib/mix/tasks">>,<<"lib/mix/tasks/portmidi.devices.ex">>,
>   <<"lib/portmidi">>,<<"lib/portmidi.ex">>,<<"lib/portmidi/device.ex">>,
>   <<"lib/portmidi/devices.ex">>,<<"lib/portmidi/input">>,
19,21c13,16
<   <<"lib/portmidi/nifs/devices.ex">>,<<"lib/portmidi/nifs/input.ex">>,
<   <<"lib/portmidi/nifs/output.ex">>,<<"lib/portmidi/output.ex">>,
<   <<"src/erl_comm.c">>,<<"src/portmidi_devices.c">>,<<"src/portmidi_in.c">>,
---
>   <<"lib/portmidi/nifs">>,<<"lib/portmidi/nifs/devices.ex">>,
>   <<"lib/portmidi/nifs/input.ex">>,<<"lib/portmidi/nifs/output.ex">>,
>   <<"lib/portmidi/output.ex">>,<<"src">>,<<"src/erl_comm.c">>,
>   <<"src/portmidi_devices.c">>,<<"src/portmidi_in.c">>,
29c24
< {<<"version">>,<<"5.1.1">>}.
---
> {<<"version">>,<<"5.1.2">>}.
diff -r portmidi-5.1.1/lib/portmidi/device.ex portmidi-5.1.2/lib/portmidi/device.ex
8c8
<     |> make_struct
---
>     |> make_struct()
diff -r portmidi-5.1.1/lib/portmidi/devices.ex portmidi-5.1.2/lib/portmidi/devices.ex
6c6
<     do_list
---
>     do_list()
diff -r portmidi-5.1.1/lib/portmidi/input/server.ex portmidi-5.1.2/lib/portmidi/input/server.ex
24c24
<     case Reader.start_link(self, device_name) do
---
>     case Reader.start_link(self(), device_name) do
34c34
<     self
---
>     self()
36c36
<     |> Enum.each(&(send(&1, {self, messages})))
---
>     |> Enum.each(&(send(&1, {self(), messages})))
diff -r portmidi-5.1.1/lib/portmidi/input.ex portmidi-5.1.2/lib/portmidi/input.ex
13,27d12
<   def stream!(input) do
<     Listeners.register(input, self)
< 
<     Stream.resource(fn ->
<       input
<     end, fn(input) ->
<       receive do
<         {^input, events} -> {events, input}
<       end
<     end, fn(input) ->
<       stop(input)
<     end)
< 
<   end
< 
diff -r portmidi-5.1.1/lib/portmidi/listeners.ex portmidi-5.1.2/lib/portmidi/listeners.ex
54a55
> 
diff -r portmidi-5.1.1/mix.exs portmidi-5.1.2/mix.exs
3c3
<   @version "5.1.1"
---
>   @version "5.1.2"
10c10
<      package: package,
---
>      package: package(),
14c14
<      deps: deps,
---
>      deps: deps(),
31c31
<      {:ex_doc, github: "elixir-lang/ex_doc", only: :dev},
---
>      {:ex_doc, "~> 0.19", only: :dev, runtime: false},
44c44
<   @shortdoc "Compiles portmidi bindings"
---
>   @moduledoc "Compiles portmidi bindings"
48a49,50
> 
>     :ok
Only in portmidi-5.1.2/priv: .gitkeep
Binary files portmidi-5.1.1/priv/portmidi_devices.so and portmidi-5.1.2/priv/portmidi_devices.so differ
Only in portmidi-5.1.1/priv: portmidi_devices.so.dSYM
Binary files portmidi-5.1.1/priv/portmidi_in.so and portmidi-5.1.2/priv/portmidi_in.so differ
Only in portmidi-5.1.1/priv: portmidi_in.so.dSYM
Binary files portmidi-5.1.1/priv/portmidi_out.so and portmidi-5.1.2/priv/portmidi_out.so differ
Only in portmidi-5.1.1/priv: portmidi_out.so.dSYM
diff -r portmidi-5.1.1/src/portmidi_devices.c portmidi-5.1.2/src/portmidi_devices.c
14,16d13
<   int* hey;
< 
< 

I'm not yet well-versed in how the C extensions are compiled & loaded, yet, but at least a few differences stand out:

• the .so.dSYM files are only there in 5.1.1
• hex_metadata.config show the same differences

It looks like the published version has been constructed a bit differently.

I'm on a Mac, fwiw.

@lucidstack if you have some guidance, I can go a bit deeper :-)

[thbar]

I investigated a bit more. Here are my findings.

Explanation of the failure

If one installs portmidi version 5.1.2 on a Mac, they will find, under _build/dev/lib/portmidi/priv, compiled output for the C code (e.g. portmidi_devices.so).

It's possible to determine the architecture of the dynamic library this way:

$ cd _build/dev/lib/portmidi/priv
$ file portmidi_devices.so 
portmidi_devices.so: ELF 64-bit LSB pie executable x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=cd366b9c2bd1e9e1f04193d80401c79a122ffe6f, with debug_info, not stripped

This seem to indicate it's a Linux-based compiled file.

If I run the fix provided by @drumusician, we get a different output:

$ file _build/dev/lib/portmidi/priv/portmidi_in.so
_build/dev/lib/portmidi/priv/portmidi_in.so: Mach-O 64-bit dynamically linked shared library x86_64

Mach-O are Apple / Darwin executables.

So this definitely confirms what @drumusician has hinted: the compiled binary hasn't got the right architecture.

But how does this get compiled, at all?

I was trying to determine what triggers the compilation, and it turns out it's not very explicit.

It is simply mix deps.compile which, when a Makefile is found, will call make on its own.

This is documented here:

https://hexdocs.pm/mix/Mix.Tasks.Deps.Compile.html

How can we fix it in ex-portmidi itself?

The first solution would be to make sure that the release process just strips out the content of the priv folder, when a release is created.

I believe the :exclude_patterns option now available in Hex could do the trick (see https://hexdocs.pm/hex/Mix.Tasks.Hex.Build.html).

In the longer run, if we need something more elaborated, I've discovered that the Membrane project provides a tool called Bundlex to finetune that process:

https://blog.swmansion.com/bundlex-compiling-c-with-elixir-made-easy-9ac5d7d24da5

[thbar]

It's also possible to use the latest version by just telling mix to pull the dependency from GitHub, rather than Hex, with this in mix.exs:

defp deps do
  [
    {:portmidi, git: "https://github.com/lucidstack/ex-portmidi.git"}

[pizzapim]

I encountered this issue as well but the above suggestions did not fix it... Did anybody find a reliable fix after this time?

[type1fool]

Hey all!

Getting PortMidi to work on M1 has been daunting. I had almost zero experience modifying Makefiles or working with C code, but I really wanted to get it working for an app I'm working on. In order to get portmidi to compile, I had to do things a little differently.

1. Run brew install portmidi
2. Clone ex-portmidi
3. Run cd ex-portmidi
4. Run rm -rf priv/portmidi_*
5. Add -target arm64-apple-macos11 -I /opt/homebrew/Cellar/portmidi/217_2/include -L /opt/homebrew/Cellar/portmidi/217_2/lib to CFLAGS in Makefile
6. Run make

➜  ex-portmidi git:(master) ✗ make
cc -g -std=c99 -O3 -pedantic -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -target arm64-apple-macos11 -I /opt/homebrew/Cellar/portmidi/217_2/include -L /opt/homebrew/Cellar/portmidi/217_2/lib -I/Users/owen/.asdf/installs/erlang/24.1.4/erts-12.1.4/include -fPIC -shared -dynamiclib -undefined dynamic_lookup -o priv/portmidi_in.so -lportmidi src/portmidi_in.c src/portmidi_shared.c
cc -g -std=c99 -O3 -pedantic -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -target arm64-apple-macos11 -I /opt/homebrew/Cellar/portmidi/217_2/include -L /opt/homebrew/Cellar/portmidi/217_2/lib -I/Users/owen/.asdf/installs/erlang/24.1.4/erts-12.1.4/include -fPIC -shared -dynamiclib -undefined dynamic_lookup -o priv/portmidi_out.so -lportmidi src/portmidi_out.c src/portmidi_shared.c
cc -g -std=c99 -O3 -pedantic -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -target arm64-apple-macos11 -I /opt/homebrew/Cellar/portmidi/217_2/include -L /opt/homebrew/Cellar/portmidi/217_2/lib -I/Users/owen/.asdf/installs/erlang/24.1.4/erts-12.1.4/include -fPIC -shared -dynamiclib -undefined dynamic_lookup -o priv/portmidi_devices.so -lportmidi src/portmidi_devices.c src/portmidi_shared.c

For now, my app points to the local version of ex-portmidi. There appear to be a few changes required to make the package platform agnostic:

• Remove pre-compiled files from priv/
• Automatically detect the machine's architecture
• Make homebrew libs visible to the compiler

@thbar @lucidstack I'm happy to push a PR, but I'm not sure how to solve the problem aside from hardcoding the flags I need in Makefile.

[thbar]

Hi @type1fool!

Thanks for sharing this ; on my side (also Mac M1), I believe I have it handled in thbar#10.

The -I and -L part are handled explicitly, but not the -target like you did (yet it seems to work). I presume maybe the flag is kind of assumed (I didn't dive tonight, but just wanted to share my bits).

At some point also I will have to bring back the various commits I did on my fork (master...thbar:master), or transfer ownership so I can publish the packages.

[type1fool]

Nice! I switched to your git version and it just works™️. It would be great to see those changes merged upstream.

[type1fool]

@thbar I'm not sure what to do with this error:
https://github.com/type1fool/loomer/runs/4255188055?check_suite_focus=true#step:7:36

I noticed you're using MacOS for your fork, but I don't see any special portmidi steps aside from brew install. I tried a few things, but this error has stumped me. 🤦‍♂️