Skip to content

Make TestIndexWriterDelete.testDeleteAllRepeated a monster test and force FSDirectory to prevent OOM. #14526

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Apr 23, 2025

Conversation

dweiss
Copy link
Contributor

@dweiss dweiss commented Apr 19, 2025

Workaround for #14508

Copy link
Member

@rmuir rmuir left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It should not impact jenkins as it is now labelled @Monster.

To try to address @uschindler concerns about disks, we also have @SuppressFsync annotation available in LuceneTestCase, for tests that may need to do tons of commits. real fsyncs are suppressed for a given test class 99% of the time by default, but the annotation can make that 100% of the time.

See logic:

// speed up tests by omitting actual fsync calls to the hardware most of the time.
if (targetClass.isAnnotationPresent(SuppressFsync.class) || random.nextInt(100) > 0) {
if (allowed(avoid, DisableFsyncFS.class)) {
fs = new DisableFsyncFS(fs).getFileSystem(null);
}
}

This won't help much if the test truly writes tons of actual data that will get synced in background: in such a situation you need tmpfs or nobarrier, or tone down the test :)

@rmuir
Copy link
Member

rmuir commented Apr 19, 2025

fwiw: here is my tmpfs setup, which avoids breaking my SSD. This one is 7 years old now. I've got 5MB of trash from lucene builds after 9 days uptime, plenty of trash from other less-behaved programs:

# tmpfs mount with a 3GB limit for /tmp
tmpfs  /tmp  tmpfs  rw,nodev,nosuid,size=3G 0 0
# ~/.gradle/gradle.properties or probably possible via -D or -p or something
# the subdirectory will get created automatically by the build if needed
tests.workDir=/tmp/lucene_gradle

@dweiss dweiss merged commit 1753717 into apache:main Apr 23, 2025
7 checks passed
@dweiss dweiss deleted the 14508-testDeleteAllRepeated-oom branch April 23, 2025 07:32
dweiss added a commit that referenced this pull request Apr 23, 2025
jpountz pushed a commit to jpountz/lucene that referenced this pull request Apr 24, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants