Problems linking against native libraries

[noncom]

With my current project I've stepped into a problem with native libraries. The issue is that lwjgl cannot find its natives although the jar with the natives is properly linked to the project and appears in Leiningen Dependencies.

I have made a little research and found out that this problem is known for lwjgl:

1. http://wiki.lwjgl.org/wiki/Downloading_and_Setting_Up_LWJGL look at the "I keep getting an java.lang.UnsatisfiedLinkError: no lwjgl in java.library.path" paragraph

2. this is how it is solved normally in eclipse: http://stackoverflow.com/questions/19344914/getting-java-lang-unsatisfiedlinkerror-no-lwjgl-in-java-library-path

3. and the official guide: http://wiki.lwjgl.org/index.php?title=Setting_Up_LWJGL_with_Eclipse (mostly same as point 2), but with neat pictures ).

So, in my project there is the lwjgl natives jar, however, the java lwjgl jar native library path points to "project/target/native/windows/x86_64" and in the project build path setup it says "not modifiable".
The interesting part is that the natives are actually do get extracted and are there ready to be picked up, but they are located at "project/target/native/windows".

Resume: if I manually copy the dlls from "project/target/native/windows" to "project/target/native/windows/x86_64", all starts working okay. I know leiningen and java in general have their quirks in working with native libraries, but what is different in this case, is that the path "project/target/native/windows/x86_64" is incorrect and cannot be changed. Is there something in CCW that is getting in touch with this?

---

A small GitHub project illustrating the issue: https://github.com/noncom/test-natives
Upon loading the core ns into the repl, you'll get:

;; Clojure 1.6.0
<Namespace test-natives.core>
CompilerException java.lang.UnsatisfiedLinkError: no lwjgl in java.library.path, compiling:(test_natives\core.clj:4:3)

and if you move the natives as I've described above, it will compile fine. even calling (go) will work ok and return nil. It does not matter that you have no actual opengl context active

[laurentpetit]

thanks, I will take a look at it

[laurentpetit]

I have the same problem with leiningen on the command line, no need to reach for ccw to get the UnstatisfiedLinkError exception.

[laurentpetit]

And as you suggested, moving my target/natives/macosx/ files to target/natives/macosx/x86_84 files did the trick and now leiningen on the command line works again

[laurentpetit]

There's a word about lwjgl in the leiningen samples project.clj, by the way:

https://github.com/technomancy/leiningen/blob/master/sample.project.clj#L55

[laurentpetit]

Please, talk about this issue with the Leiningen folks first. There's a lot on my plate these days, it would be good if you could confirm that you also get the same problem with leiningen, and then maybe find a solution, or some workaround.
Then we'll be able to see if this work-around can or cannot be embedded into CCW.
Deal?

[noncom]

Alright! I will do experiments with Leiningen first and see if it is possible to get a fix from some manipulations with it. Eh, I am sorry that I have loaded you with this one, since it apparently is a core Leiningen issue... on the other hand, I think a user expects his IDE to handle such things automatically... the case seems vague, I'll explore more and report when I have some results on this.

[laurentpetit]

You did the right thing, don't worry. Thanks for exploring the case.

2015-08-17 12:37 GMT+02:00 noncom [email protected]:

Alright! I will do experiments with Leiningen first and see if it is
possible to get a fix from some manipulations with it. Eh, I am sorry that
I have loaded you with this one, since it apparently is a core Leiningen
issue... on the other hand, I think a user expects his IDE to handle such
things automatically... the case seems vague, I'll explore more and report
when I have some results on this.

—
Reply to this email directly or view it on GitHub
#836 (comment).

Laurent Petit

[noncom]

Small update: specifying :native-path "target/native/native/windows" in project.clj makes it work. Not sure what that means though. With this parameter specified and even Leiningen project reset in CCW, the Eclipse dependencies window still shows project/target/native/windows/x86_64, non modifiable, but this time compilation works.

I am still contemplating on this...

[laurentpetit]

So to summarize :

• now it works on windows with leiningen on the command line
• but CCW is unable to derive the right native path

2015-08-20 15:23 GMT+02:00 noncom [email protected]:

Small update: specifying :native-path "target/native/native/windows" in
project.clj makes it work.Not sure what that means though. With this
parameter specified and even Leiningen project reset in CCW, the Eclipse
dependencies window still shows project/target/native/windows/x86_64, non
modifiable, but this time compilation works.

I am still contemplating on this...

—
Reply to this email directly or view it on GitHub
#836 (comment).

Laurent Petit

[noncom]

Yes, it works in Windows command line Leiningen, as well as from CCW if "launch projects with Leiningen" is enabled - same. However, if I uncheck "launch projects with Leiningen", it looks like that CCW goes with what Eclipse is saying in the deps and does not find the libs, failing as before.

Idk if there is even a solution for that. Probably this is a quirk of dealing with the lwjgl library in particular.. As hyPiRion said on #leiningen, not many people work with native deps and this area is highly underdocumented and it is unknown if some uniformity (i.e. a generalized solution) can be applied to the area at all. If it will become clear that it is impossible to formalize a universal approach for this situation, then maybe a plugin for CCW+LWJGL could be considered. Because the problem is there and other people can stumble upon it and there is not much info on this..

[laurentpetit]

@noncom which version of leiningen are you using on the command line?

[laurentpetit]

@noncom also, can you please update the github project test-natives with the fix you found?

[laurentpetit]

I still cannot make it work on the command line.
Here's what I do:

• clone the test-natives repository
• cd into test-natives
• immediately invoke lein repl
• then (require 'test-natives.core) and it fails saying CompilerException java.lang.UnsatisfiedLinkError: no lwjgl in java.library.path, compiling:(test_natives/core.clj:4:3)

Then I check the target directory, and there's nothing inside it.

• Same thing whether I add or do not add the :native-path "target/native/native/macosx" to project.clj (I changed windows to macosx for my platform).
• Same thing with leiningen version 2.5.0 2.5.1 2.5.2.

So I may be missing the obvious, which is ... ?

[laurentpetit]

I will try to work this out, once I get the right configuration for making
it work in OS X command line Leiningen :-)

2015-08-20 15:29 GMT+02:00 noncom [email protected]:

Yes, it works in Windows command line Leiningen, as well as from CCW if
"launch projects with Leiningen" is enabled - same. However, if I uncheck
"launch projects with Leiningen", it looks like that CCW goes with what
Eclipse is saying in the deps and does not find the libs, failing as before.

—
Reply to this email directly or view it on GitHub
#836 (comment).

Laurent Petit

[noncom]

Whoa... everything is more complicated than I originally thought. After I saw your last comments I went and tried to repeat the scenario again, doing this in the command line:

• manually remove the target and bin directories
• lein clean
• add :native-path "target/native/native/windows" to the project.clj
• lein repl
• (require 'test-natives.core)
  and.. it failed.

So I went to see what's up in the target dir and saw weird things. The dlls were lying in this actual location: target\native\native\windows\native\windows\.. oh my this looked very wrong... it didn't work however I tried to make it to. Then I did:

• manually remove the target and bin directories
• lein clean
• lein repl
• exit lein repl back to command line
• add :native-path "target/native/native/windows" to the project.clj
• lein repl
• (require 'test-natives.core)
  and.. it worked.

So, there is a mystery going on. Somehow specifying :natives-path before the first creation of target wreaks havoc on the directory structure.

Alright, so I decided to go check what's really going on! Here was my experiment:

• manually remove the target and bin directories
• lein clean
• add :native-path "target/native/a/b/c/d/native/windows" to the project.clj
• lein repl
• (require 'test-natives.core) fail!
• exit lein repl back to command line
• examine the directory structure and find dlls at target\native\a\b\c\d\native\windows\native\windows\ - the actual path differs from the one specified in project.clj by having an additional \native\windows\ at the end. alright.
• change :native-path "target/native/a/b/c/d/native/windows" to :native-path "target/native/a/b/c/d/native/windows/native/windows" the project.clj
• lein repl
• (require 'test-natives.core)
  and.. it worked.

Obviously this is a bug in Leiningen. It treats the :native-paths parameter differently in cases of the first and subsequent runs. Maybe I should first file an issue on Leiningen?

My current Leiningen version is 2.5.1... looking at https://github.com/technomancy/leiningen/blob/master/NEWS.md I can conclude that 2.5.2 makes no difference.

Also, I have updated the repository.

[noncom]

Ah, sorry, the fact that the solution did not work made me embarrassed and I went checking it out right away. Now I re-read what you're saying and I see you mention that the target directory is just empty in your case? How weird... I do not even know what to attribute this to.. surely it is because of Mac.. becaues the directory is not empty on Windows, Leiningen manages to unpack things there.. Mrhh, this is one more mystery to solve... but i am sure it is a Leiningens bug.

UPDATE: I tried what your described on a Mac on my job, the target directory was not empty (without even adding :natives-path to project.clj). It gets populated just like on Windows... hmmmm... are you sure it's empty on your machine, after you do lein repl?

[laurentpetit]

Yes, this seems like a bug in leiningen, and I'd be more than happy if you
can manage the leiningen part of this.
In the meantime, I think the workaround you found should be enough, right?

2015-08-21 17:08 GMT+02:00 noncom [email protected]:

Whoa... everything is more complicated than I originally thought. After I
saw your last comments I went and tried to repeat the scenario again, doing
this in the command line:

• manually remove the target and bin directories
• lein clean
• add :native-path "target/native/native/windows" to the project.clj
• lein repl
• (require 'test-natives.core) and.. it failed.

So I went to see what's up in the target dir and saw weird things. The
dlls were lying in this actual location:
target\native\native\windows\native\windows.. oh my this looked very
wrong... it didn't work however I tried to make it to. Then I did:

• manually remove the target and bin directories
• lein clean
• lein repl
• exit lein repl back to command line
• add :native-path "target/native/native/windows" to the project.clj
• lein repl
• (require 'test-natives.core) and.. it worked.

So, there is a mystery going on. Somehow specifying :natives-path before
the first creation of target wreaks havoc on the directory structure.

Alright, so I decided to go check what's really going on! Here was my
experiment:

• manually remove the target and bin directories
• lein clean
• add :native-path "target/native/a/b/c/d/native/windows" to the
  project.clj
• lein repl
• (require 'test-natives.core) fail!
• exit lein repl back to command line
• examine the directory structure and find dlls at
  target\native\a\b\c\d\native\windows\native\windows\ - the actual path
  differs from the one specified in project.clj by having an additional
  \native\windows\ at the end. alright.
• change :native-path "target/native/a/b/c/d/native/windows" to :native-path
  "target/native/a/b/c/d/native/windows/native/windows" the project.clj
• lein repl
• (require 'test-natives.core) and.. it worked.

Obviously this is a bug in Leiningen. It treats the :native-paths
parameter differently in cases of the first and subsequent runs. Maybe I
should first file an issue on Leiningen?

My current Leiningen version is 2.5.1...

—
Reply to this email directly or view it on GitHub
#836 (comment).

Laurent Petit

[noncom]

Yes, the available workaround allows me to continue the work, it is not a blocking problem. And also, it is pointless to do any adjustments to CCW until Leiningen is fixed. Maybe CCW will not even need any fixes in the end. Okay, then I will work with the Leiningen part of this issue and come back to this one here when we have some info or a fix from Leiningen!

[noncom]

Leiningen people still did not pick up the issue. Probably it is my bad and it was lousely formulated (although I don't think so, considering the amount of information that needs to be presented for complete description). I have reformulated it. I hope it will receive attention.

[laurentpetit]

Ok, thanks for the update. I also hope it will receive the attention it
deserves.

2015-10-13 14:29 GMT+02:00 Alex Fowler [email protected]:

Leiningen people still did not pick up the issue. Probably it is my bad
and it was lousely formulated (although I don't think so, considering the
amount of information that needs to be presented for complete description).
I have reformulated it. I hope it will receive attention.

—
Reply to this email directly or view it on GitHub
#836 (comment).

Laurent Petit

[noncom]

Yeah, well, you know, some long time ago, even before having this exact issue, I've been talking to hyPiRion about Leiningen and native libraries. He said that as there are really not many people who work with this kind of programs at all, including themselves (the Leiningen team), nobody even know or check how it all works because they have no immediate need to do this while these cases really require some dedication.

[noncom]

Good news on this one! The issue has been resolved: technomancy/leiningen@adb7b99 I did not have a chance yet to test it myself though.. I'll get to it as soon as possible.

[laurentpetit]

Great ! If possible I'll include it in version 0.35.0 !