Building Planetiler with GraalVM native

[cmahnke]

Has anyone tried to build Planetiler with GraalVM native image to see if this can further improve performance?

If not, is there anyone interested in trying this? Just a warning: Due to usage of Log4J2, JNR and certainly other it's certainly not straight forward...

[msbarry]

I did a quick try a while back but didn't spend much time getting it working. I'd be very curious to see the results! It would be cool to be able to generate platform-specific executables people can run without installing Java.

Apart from this, my rough understanding of remaining opportunities performance improvements are

• the same exact algorithm is maybe 10-20% faster in native than Java
• planetiler is optimized for complete profiles running on the planet but there's some low hanging fruit to make running with small profiles/small extracts faster
• most of the runtime now is driven by expensive (but necessary) jts snap node rounding/polygon merging code so any improvements there or alternatives like Wagyu would have a big impact

[cmahnke]

I tryed several different configurations yesterday:

• First by using a run with java -agentlib:native-image-agent to generate the used metadata for planetiler-dist-0.7-SNAPSHOT-with-deps.jar and feeding the results into native-image, but this way it's not so easy to control the input.
• Secondly by hooking directly into the build system:

Using maven to (try to) build a native image has the advantage to be able to configure included dependencies:

• Replacing log4j with logback - did't cleanly work since there seem do be (in-)direct dependencies as well, not only thru slf4j - One is at planetiler-core/src/main/java/com/onthegomap/planetiler/expression/Expression.java for org.apache.logging.log4j.util.Strings - it should certainly be removed, since it's documentation states: "Consider this class private." The Guava Joiner class might be a replacement. But there are others as well, not sure yet, how easy it would be to get rid of them:
  • planetiler-core/src/main/java/com/onthegomap/planetiler/stream/WriteableStreamArchive.java - uses CloseShieldOutputStream which could be provided by Apache Commons IO, but this would introduce a new dependency, maybe Guava also has something similar in store.
  • planetiler-core/src/main/java/com/onthegomap/planetiler/util/log4j/ElapsedTimeLookupPlugin.java
• Configuring the use of proxy classes for
  • com.onthegomap.planetiler.util.Madvise$NativeC
  • jnr.ffi.provider.LoadedLibrary - which works
  • This also seems to fix an issue with com.onthegomap.planetiler.util.Madvise$NativeC$jnr$ffi$0 - but here i'm not completely sure.
• Currently I'm stuck at the point where org.geotools.util.ConverterFactory tries to load org.geotools.data.util.ColorConverterFactory

My background is to be able to generate parts of the planet, Europe for now, in a very limited (Execution time, storage and RAM, see #666 ) environment.

[msbarry]

Ok got it. You should definitely be able to remove those log4j dependencies, they are most likely added accidentally. There's one place where I define a plugin needed to print elapsed time in each log line but maybe just try removing that for testing. I'm not sure about the native wrappers/jnr ffi stuff. Can you tell what it's complaining about with them?

Sorry I didn't see #666 at first! I'll comment over there...

[cmahnke]

Yes, I've hacked something together here, just in case anyone want to take a look:
main...cmahnke:planetiler:main

My solution for the Log4J things isn't very elegant, I came to the realisation, that it should be possible to use the Maven shade plugin to filter the dependencies instead of changing the source files...

I'm not sure yet if I'll have the time to continue on this in the near future...

[cmahnke]

@msbarry I haven't looked into this since Christmas, but remember that I got stuck with the JNR part that is used to handle madvice for mmap. By chance I found the new way Lucene handles this:
apache/lucene#13196

Might that be a worthwhile approach for Planetiler as well?
It would at least require Java 21 and doesn't need any additional dependencies.

[msbarry]

Nice! The foreign function and memory API have been on my radar for a while. That would simplify mmap handling quite a bit and also let us remove the JNR dependency. Unfortunately they weren't fully released until Java 22. I would definitely start using them when planetiler upgrades to requiring the next LTS release (Java 25 around this time next year) but to adopt before then would either need to require people use a non-LTS release or a JVM flag at startup...