Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

OutOfMemoryError reported by jenkins validation build after org.eclipse.ui.tests.UiTestSuite #2432

Open
iloveeclipse opened this issue Oct 21, 2024 · 9 comments
Open

OutOfMemoryError reported by jenkins validation build after org.eclipse.ui.tests.UiTestSuite #2432

iloveeclipse opened this issue Oct 21, 2024 · 9 comments
Labels
bug Something isn't working test junit test related things

Comments

@iloveeclipse
Copy link
Member

See https://ci.eclipse.org/platform/job/eclipse.platform.ui/job/PR-2431/1/consoleFull that didn't changed code, just updated bundle versions, so the problem must be coming from either environment or code changes before.

The question is: what is causing that, and which heap space is actually used on jenkins?

Exception in thread "Active Thread: Equinox Container: aa943991-8bd5-44f7-a55a-1a7bc7647820" java.lang.OutOfMemoryError: Java heap space
Exception in thread "WorkbenchTestable" org.eclipse.swt.SWTException: Failed to execute runnable (java.lang.OutOfMemoryError: Java heap space)
	at org.eclipse.swt.SWT.error(SWT.java:4922)
	at org.eclipse.swt.SWT.error(SWT.java:4837)
	at org.eclipse.swt.widgets.Synchronizer.syncExec(Synchronizer.java:209)
	at org.eclipse.ui.internal.UISynchronizer.syncExec(UISynchronizer.java:133)
	at org.eclipse.swt.widgets.Display.syncExec(Unknown Source)
	at org.eclipse.e4.ui.internal.workbench.swt.E4Testable.runTest(E4Testable.java:118)
	at org.eclipse.tycho.surefire.osgibooter.AbstractUITestApplication.runTests(AbstractUITestApplication.java:38)
	at org.eclipse.e4.ui.internal.workbench.swt.E4Testable.lambda$0(E4Testable.java:79)
	at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.lang.OutOfMemoryError: Java heap space

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Worker-11"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Worker-22"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Worker-23"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Worker-24"
@iloveeclipse iloveeclipse added the bug Something isn't working label Oct 21, 2024
@iloveeclipse iloveeclipse mentioned this issue Oct 21, 2024
Merged
@iloveeclipse iloveeclipse changed the title OutOfMemoryError reported by jenkins validation build OutOfMemoryError reported by jenkins validation build after org.eclipse.ui.tests.UiTestSuite Oct 21, 2024
@iloveeclipse
Copy link
Member Author

Also in https://ci.eclipse.org/platform/job/eclipse.platform.ui/view/change-requests/job/PR-2224/11/consoleFull

Exception in thread "WorkbenchTestable" org.eclipse.swt.SWTException: Failed to execute runnable (java.lang.OutOfMemoryError: Java heap space)
	at org.eclipse.swt.SWT.error(SWT.java:4922)
	at org.eclipse.swt.SWT.error(SWT.java:4837)
	at org.eclipse.swt.widgets.Synchronizer.syncExec(Synchronizer.java:209)
	at org.eclipse.ui.internal.UISynchronizer.syncExec(UISynchronizer.java:133)
	at org.eclipse.swt.widgets.Display.syncExec(Unknown Source)
	at org.eclipse.e4.ui.internal.workbench.swt.E4Testable.runTest(E4Testable.java:118)
	at org.eclipse.tycho.surefire.osgibooter.AbstractUITestApplication.runTests(AbstractUITestApplication.java:38)
	at org.eclipse.e4.ui.internal.workbench.swt.E4Testable.lambda$0(E4Testable.java:79)
	at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.lang.OutOfMemoryError: Java heap space
	at org.mockito.internal.util.concurrent.WeakConcurrentMap.containsKey(WeakConcurrentMap.java:81)
	at org.mockito.internal.creation.bytebuddy.MockMethodAdvice.isMock(MockMethodAdvice.java:166)
	at java.base/java.lang.Object.equals(Object.java:163)
	at org.eclipse.e4.core.internal.di.ObjectDescriptor.hasQualifier(ObjectDescriptor.java:44)
	at org.eclipse.e4.core.internal.di.InjectorImpl.resolveArgs(InjectorImpl.java:573)
	at org.eclipse.e4.core.internal.di.InjectorImpl.invokeUsingClass(InjectorImpl.java:291)
	at org.eclipse.e4.core.internal.di.InjectorImpl.invoke(InjectorImpl.java:233)
	at org.eclipse.e4.core.contexts.ContextInjectionFactory.invoke(ContextInjectionFactory.java:174)
	at org.eclipse.e4.core.commands.internal.HandlerServiceHandler.isEnabled(HandlerServiceHandler.java:68)
	at org.eclipse.core.commands.Command.isEnabled(Command.java:832)
	at org.eclipse.ui.internal.actions.CommandAction.lambda$0(CommandAction.java:91)
	at org.eclipse.ui.internal.actions.CommandAction$$Lambda$384/0x0000000801000d78.commandChanged(Unknown Source)
	at org.eclipse.core.commands.Command$1.run(Command.java:529)
	at org.eclipse.core.runtime.SafeRunner.run(SafeRunner.java:47)
	at org.eclipse.core.commands.Command.fireCommandChanged(Command.java:522)
	at org.eclipse.core.commands.Command.lambda$0(Command.java:1000)
	at org.eclipse.core.commands.Command$$Lambda$216/0x0000000800edd2b8.handlerChanged(Unknown Source)
	at org.eclipse.core.commands.AbstractHandler.fireHandlerChanged(AbstractHandler.java:77)
	at org.eclipse.e4.core.commands.internal.HandlerServiceHandler.fireHandlerChanged(HandlerServiceHandler.java:189)
	at org.eclipse.core.commands.AbstractHandler.setBaseEnabled(AbstractHandler.java:111)
	at org.eclipse.e4.core.commands.internal.HandlerServiceHandler.isEnabled(HandlerServiceHandler.java:70)
	at org.eclipse.core.commands.Command.isEnabled(Command.java:832)
	at org.eclipse.ui.internal.actions.CommandAction.lambda$0(CommandAction.java:91)
	at org.eclipse.ui.internal.actions.CommandAction$$Lambda$384/0x0000000801000d78.commandChanged(Unknown Source)
	at org.eclipse.core.commands.Command$1.run(Command.java:529)
	at org.eclipse.core.runtime.SafeRunner.run(SafeRunner.java:47)
	at org.eclipse.core.commands.Command.fireCommandChanged(Command.java:522)
	at org.eclipse.core.commands.Command.lambda$0(Command.java:1000)
	at org.eclipse.core.commands.Command$$Lambda$216/0x0000000800edd2b8.handlerChanged(Unknown Source)
	at org.eclipse.core.commands.AbstractHandler.fireHandlerChanged(AbstractHandler.java:77)
	at org.eclipse.e4.core.commands.internal.HandlerServiceHandler.fireHandlerChanged(HandlerServiceHandler.java:189)
	at org.eclipse.ui.internal.handlers.E4HandlerProxy.handlerChanged(E4HandlerProxy.java:116)

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Active Thread: Equinox Container: 882cd8a6-664e-4aea-971d-49c2b2d63bef"
Exception in thread "Worker-4" java.lang.OutOfMemoryError: Java heap space
An error has occurred. See the log file
/home/jenkins/agent/workspace/eclipse.platform.ui_PR-2224/tests/org.eclipse.ui.tests/target/work/data/.metadata/.log.

Also in https://ci.eclipse.org/platform/job/eclipse.platform.ui/view/change-requests/job/PR-2424/11/consoleFull

  WorkbookEditorsHandlerTest.doTearDown:73 » Resource Problems encountered while deleting resources.

Tests run: 1659, Failures: 1, Errors: 5, Skipped: 196

Exception in thread "Active Thread: Equinox Container: 5e5cc6b7-0a59-4770-ba00-8313910b610d" java.lang.OutOfMemoryError: Java heap space

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Worker-5"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Worker-15"

iloveeclipse added a commit to iloveeclipse/eclipse.platform.ui that referenced this issue Oct 21, 2024
@iloveeclipse
Add memory reporting for UI tests and exit for E4Testable on OOM
55d06ea 
Maybe this could help understanding OOM errors on jenkins.

See eclipse-platform#2432
This was referenced Oct 21, 2024
Open
Merged
@merks
Copy link
Contributor

merks commented Oct 21, 2024

Maybe this is related:

https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/issues/5056

@laeubi
Copy link
Contributor

laeubi commented Oct 21, 2024

If I look at this where we exchaust thread pools:

I just wanted to mention that each thread requires a small amount of ram as well....

@iloveeclipse
Copy link
Member Author

I just wanted to mention that each thread requires a small amount of ram as well....

I would expect jenkins runs on a very small VM with very few (2-4?) cores, so thread pool size would be also very small and shouldn't be the root cause here.

@iloveeclipse
Copy link
Member Author

Maybe this is related:
https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/issues/5056

Hmm. I don't see OOM's on JDT tests which are known to be memory sensitive, and OOMs in UI tests seem to happen on / after specific testsuite, so it look like a code regression on our (platform / UI) side.

@laeubi
Copy link
Contributor

laeubi commented Oct 21, 2024

I would expect jenkins runs on a very small VM with very few (2-4?) cores, so thread pool size would be also very small and shouldn't be the root cause here.

This is a common misunderstanding with ForkJoinPools... If threads are blocked (e.g. the are waiting for I/O or waiting for other threads to join), then new threads are spwaned untill the configured concurrency level is fulfilled again, see here

The pool attempts to maintain enough active (or available) threads by dynamically adding, suspending, or resuming internal worker threads, even if some tasks are stalled waiting to join others.

The exception given here says "Thread limit exceeded replacing blocked worker" and the limit is by default 32767 (!)

@iloveeclipse
Copy link
Member Author

This is a common misunderstanding with ForkJoinPools

Cool, thanks. So one part of the problem in eclipse-platform/eclipse.platform#1592 is that the used pool is not limited in size (beside that the LocalFile.internalDelete() doesn't do what it supposed to do)?

@laeubi
Copy link
Contributor

laeubi commented Oct 21, 2024

It is just a guess, if we see Thread explosion somehwere and memmory problems elsewhere, In general I/O bound tasks don'T play well with fork-join because the whole fork-join was introduced because of CPU bound (!) task usually suffer from context switch penalities. For I/O non blocking I/O is usually most faster if we asumme a shared device (e.g. harddisk / network / ...).

@iloveeclipse
Copy link
Member Author

https://ci.eclipse.org/platform/job/eclipse.platform.ui/job/PR-2433/1/console shows no major memory increase after multiple Thread limit exceeded replacing blocked worker errors.
So something else must be the problem here.

iloveeclipse added a commit to iloveeclipse/eclipse.platform.ui that referenced this issue Oct 21, 2024
@iloveeclipse
Add memory reporting for UI tests and exit for E4Testable on OOM
cc079cf 
Maybe this could help understanding OOM errors on jenkins.

Note: moved OpenCloseTest to the begin, just in case if that causes OOM.

See eclipse-platform#2432
iloveeclipse added a commit that referenced this issue Oct 21, 2024
@iloveeclipse
Dummy change to check OOMs in jenkins
ee32bb4 
See #2432 and eclipse-platform/eclipse.platform#1592

This change shouldn't affect any test at all, it is not a code change.
@iloveeclipse iloveeclipse mentioned this issue Oct 21, 2024
Closed
iloveeclipse added a commit to iloveeclipse/eclipse.platform.ui that referenced this issue Oct 22, 2024
@iloveeclipse
Add explicit gc and memory reporting for UI tests
6037f01 
Also exit E4Testable on OOM.

This is supposed to workaround and to understand OOM errors on jenkins.

See eclipse-platform#2432
@iloveeclipse iloveeclipse mentioned this issue Oct 22, 2024
Merged
iloveeclipse added a commit to iloveeclipse/eclipse.platform.ui that referenced this issue Oct 22, 2024
@iloveeclipse
Add explicit gc and memory reporting for UI tests
820b875 
Also exit E4Testable on OOM.

This is supposed to workaround and to understand OOM errors on jenkins.

See eclipse-platform#2432
@jukzi jukzi added the test junit test related things label Oct 28, 2024
@fedejeanne fedejeanne mentioned this issue Oct 29, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working test junit test related things
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@merks @iloveeclipse @laeubi @jukzi