[BUG] osm_pass1 dying during parallel operation

[hyperknot]

Describe the bug
If I start a planet run, and while it's working I start a monaco run, the monaco run terminates.

To Reproduce
planetiler version: cf49b86

Start both processes with this logic:

    java_memory_gb = 30 if area == 'planet' else 1

    command = [
        'java',
        f'-Xmx{java_memory_gb}g',
        '-jar',
        config.planetiler_path,
        f'--area={area}',
        '--download',
        '--download-threads=10',
        '--download-chunk-size-mb=1000',
        '--fetch-wikidata',
        '--output=tiles.mbtiles',
        '--nodemap-type=array',
        '--storage=mmap',
        '--force',
    ]

    if area == 'planet':
        command.append('--bounds=planet')

Logs:
planetiler.out.txt
planetiler.err.txt

Expected behavior
Parallel processes shouldn't effect each other.

Environment (please complete the following information):

• Hardware: Physical server with 128 GB memory
• OS: Ubuntu 22.04
• Java version and distribution:
  openjdk version "21.0.4" 2024-07-16
  OpenJDK Runtime Environment (build 21.0.4+7-Ubuntu-1ubuntu222.04)
  OpenJDK 64-Bit Server VM (build 21.0.4+7-Ubuntu-1ubuntu222.04, mixed mode, sharing)
• Maven version: ?

[msbarry]

How was the monaco run kicked off? Are they running in different subdirectories? The temp files will collide otherwise. Also for a small extract like monaco you should use --nodemap-type=sortedtable which will take much less time and storage.

[hyperknot]

Yes, they are running in different subdirectories. It's not important now, as I made monaco run in a non-conflicting time, or do you think with sortedtable it won't happen?

[msbarry]

Actually this might not be related at all to them running at the same time. --nodemap-type=array is meant for storing node locations for dense node IDs like a planet extract and what you saw might be a bug in trying to use it for very sparse node IDs like a monaco extract. You could get it to work using osmium renumber on the monaco extract first, but I'd suggest just switching to --nodemap-type=sortedtable which is meant for smaller extracts.

[hyperknot]

No, monaco definitely works fine on it's own. My testing cron setup was to run monaco once per hour and planet once per day. It was very obvious that monaco only failed when planet was running on the same host.

[msbarry]

Just following up here, have you seen this at all since switching to sortedtable for monaco?

[hyperknot]

No, I made sure that they are not running at the same time. I might just move back Monaco to a conflicting time, but my worry is what if the planet process get terminated.

So basically I'd only do this for debugging, I'm actually happy that they are running in non-conflicting times now.