Hints about profiling planetiler or similar OSM software about CPU or I/O performance

[stephan75]

Hello @msbarry ,
I have been watching progress in source code and issues of planetiler for many month now, indeed you and the other contributors did a nice work about this piece of software!

Some issues were about finding bottlenecks in CPU performance or data reading/writing ... you also added some screenshots.

Although I have no such programming skills on my own, but:

Can you give some general hints (or how to start) about doing such a software profiling with other OSM related software that are time consuming in processing raw data, and when those are written in Java?

(or what about that profiling in other programming languages?)

[msbarry]

Thanks! I've been using visualvm to sample CPU usage, find hot spots, fix, and repeat. It gets the job done, but yourkit is probably better (and free for open source projects). You need to interpret the output of that in context of what the threads are doing, for example if two things are running in parallel then it won't help to optimize the one that's already taking lest time.

Those custom plots I generated were from parsing log output with this observable notebook that shows what threads are running and how much CPU they each are using over time to help identify the spots to dig into - but then I go back to visualvm to see what it's doing then before optimizing. The plots need the exact same output format as planetiler, so they are not very repeatable for other projects.