Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Skip invalid mid in Sweeper::clearMultis #6

Merged
merged 3 commits into from
May 10, 2024
Merged

Skip invalid mid in Sweeper::clearMultis #6

merged 3 commits into from
May 10, 2024

Conversation

hannahbast
Copy link
Member

Not sure if this is an appropriate fix, but spatialjoin then runs through without errors and the results look reasonable.

Not sure if this is an appropriate fix, but spatialjoin then runs
through without errors and the results look reasonable.
@hannahbast hannahbast requested a review from patrickbr May 9, 2024 21:55
@hannahbast hannahbast changed the title Skip invalid mid in clearMultis Skip invalid mid in Sweeper::clearMultis May 9, 2024
patrickbr added 2 commits May 10, 2024 12:25
…r the invalid mid encountered (which also isn't a valid geometry cache offset because it translates to a file offset of 130 TB, so it must originate from memory corruption). Without the lock, concurrent write and read access to _subNotOverlaps is possible, which also invalidates the iterators used in multiOut() - after that, anything goes. Also remove the silent skip if an invalid mid is encounted (this should never happen) and add a WARN log instead.
@patrickbr
Copy link
Member

Thanks, @hannahbast. An invalid mid should never happen here. I strongly suspect a missing lock for the _subOverlaps maps is the cause here. The mid here comes directly from the event file - it is either an offset into the cache file (for normal events) or an offset into the multi-geometry ID vector. An mid of 130033639634376183 is obviously not a valid offset into the file, as we do not have 130 PB of cache data. So memory corruption must be the cause, and this was most like caused by concurrent read/write access to _subOverlaps invalidating all iterators in multiOut(). After that, we are in the purgatory of undefined behavior :)

@patrickbr patrickbr merged commit 3ccce6d into master May 10, 2024
0 of 6 checks passed
@patrickbr patrickbr deleted the invalid-mid branch May 10, 2024 10:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants