OutOfMemoryError reported by jenkins validation build after org.eclipse.ui.tests.UiTestSuite #2432

iloveeclipse · 2024-10-21T08:27:44Z

See https://ci.eclipse.org/platform/job/eclipse.platform.ui/job/PR-2431/1/consoleFull that didn't changed code, just updated bundle versions, so the problem must be coming from either environment or code changes before.

The question is: what is causing that, and which heap space is actually used on jenkins?

Exception in thread "Active Thread: Equinox Container: aa943991-8bd5-44f7-a55a-1a7bc7647820" java.lang.OutOfMemoryError: Java heap space
Exception in thread "WorkbenchTestable" org.eclipse.swt.SWTException: Failed to execute runnable (java.lang.OutOfMemoryError: Java heap space)
	at org.eclipse.swt.SWT.error(SWT.java:4922)
	at org.eclipse.swt.SWT.error(SWT.java:4837)
	at org.eclipse.swt.widgets.Synchronizer.syncExec(Synchronizer.java:209)
	at org.eclipse.ui.internal.UISynchronizer.syncExec(UISynchronizer.java:133)
	at org.eclipse.swt.widgets.Display.syncExec(Unknown Source)
	at org.eclipse.e4.ui.internal.workbench.swt.E4Testable.runTest(E4Testable.java:118)
	at org.eclipse.tycho.surefire.osgibooter.AbstractUITestApplication.runTests(AbstractUITestApplication.java:38)
	at org.eclipse.e4.ui.internal.workbench.swt.E4Testable.lambda$0(E4Testable.java:79)
	at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.lang.OutOfMemoryError: Java heap space

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Worker-11"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Worker-22"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Worker-23"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Worker-24"

The text was updated successfully, but these errors were encountered:

iloveeclipse · 2024-10-21T08:42:26Z

Also in https://ci.eclipse.org/platform/job/eclipse.platform.ui/view/change-requests/job/PR-2224/11/consoleFull

Exception in thread "WorkbenchTestable" org.eclipse.swt.SWTException: Failed to execute runnable (java.lang.OutOfMemoryError: Java heap space)
	at org.eclipse.swt.SWT.error(SWT.java:4922)
	at org.eclipse.swt.SWT.error(SWT.java:4837)
	at org.eclipse.swt.widgets.Synchronizer.syncExec(Synchronizer.java:209)
	at org.eclipse.ui.internal.UISynchronizer.syncExec(UISynchronizer.java:133)
	at org.eclipse.swt.widgets.Display.syncExec(Unknown Source)
	at org.eclipse.e4.ui.internal.workbench.swt.E4Testable.runTest(E4Testable.java:118)
	at org.eclipse.tycho.surefire.osgibooter.AbstractUITestApplication.runTests(AbstractUITestApplication.java:38)
	at org.eclipse.e4.ui.internal.workbench.swt.E4Testable.lambda$0(E4Testable.java:79)
	at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.lang.OutOfMemoryError: Java heap space
	at org.mockito.internal.util.concurrent.WeakConcurrentMap.containsKey(WeakConcurrentMap.java:81)
	at org.mockito.internal.creation.bytebuddy.MockMethodAdvice.isMock(MockMethodAdvice.java:166)
	at java.base/java.lang.Object.equals(Object.java:163)
	at org.eclipse.e4.core.internal.di.ObjectDescriptor.hasQualifier(ObjectDescriptor.java:44)
	at org.eclipse.e4.core.internal.di.InjectorImpl.resolveArgs(InjectorImpl.java:573)
	at org.eclipse.e4.core.internal.di.InjectorImpl.invokeUsingClass(InjectorImpl.java:291)
	at org.eclipse.e4.core.internal.di.InjectorImpl.invoke(InjectorImpl.java:233)
	at org.eclipse.e4.core.contexts.ContextInjectionFactory.invoke(ContextInjectionFactory.java:174)
	at org.eclipse.e4.core.commands.internal.HandlerServiceHandler.isEnabled(HandlerServiceHandler.java:68)
	at org.eclipse.core.commands.Command.isEnabled(Command.java:832)
	at org.eclipse.ui.internal.actions.CommandAction.lambda$0(CommandAction.java:91)
	at org.eclipse.ui.internal.actions.CommandAction$$Lambda$384/0x0000000801000d78.commandChanged(Unknown Source)
	at org.eclipse.core.commands.Command$1.run(Command.java:529)
	at org.eclipse.core.runtime.SafeRunner.run(SafeRunner.java:47)
	at org.eclipse.core.commands.Command.fireCommandChanged(Command.java:522)
	at org.eclipse.core.commands.Command.lambda$0(Command.java:1000)
	at org.eclipse.core.commands.Command$$Lambda$216/0x0000000800edd2b8.handlerChanged(Unknown Source)
	at org.eclipse.core.commands.AbstractHandler.fireHandlerChanged(AbstractHandler.java:77)
	at org.eclipse.e4.core.commands.internal.HandlerServiceHandler.fireHandlerChanged(HandlerServiceHandler.java:189)
	at org.eclipse.core.commands.AbstractHandler.setBaseEnabled(AbstractHandler.java:111)
	at org.eclipse.e4.core.commands.internal.HandlerServiceHandler.isEnabled(HandlerServiceHandler.java:70)
	at org.eclipse.core.commands.Command.isEnabled(Command.java:832)
	at org.eclipse.ui.internal.actions.CommandAction.lambda$0(CommandAction.java:91)
	at org.eclipse.ui.internal.actions.CommandAction$$Lambda$384/0x0000000801000d78.commandChanged(Unknown Source)
	at org.eclipse.core.commands.Command$1.run(Command.java:529)
	at org.eclipse.core.runtime.SafeRunner.run(SafeRunner.java:47)
	at org.eclipse.core.commands.Command.fireCommandChanged(Command.java:522)
	at org.eclipse.core.commands.Command.lambda$0(Command.java:1000)
	at org.eclipse.core.commands.Command$$Lambda$216/0x0000000800edd2b8.handlerChanged(Unknown Source)
	at org.eclipse.core.commands.AbstractHandler.fireHandlerChanged(AbstractHandler.java:77)
	at org.eclipse.e4.core.commands.internal.HandlerServiceHandler.fireHandlerChanged(HandlerServiceHandler.java:189)
	at org.eclipse.ui.internal.handlers.E4HandlerProxy.handlerChanged(E4HandlerProxy.java:116)

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Active Thread: Equinox Container: 882cd8a6-664e-4aea-971d-49c2b2d63bef"
Exception in thread "Worker-4" java.lang.OutOfMemoryError: Java heap space
An error has occurred. See the log file
/home/jenkins/agent/workspace/eclipse.platform.ui_PR-2224/tests/org.eclipse.ui.tests/target/work/data/.metadata/.log.

Also in https://ci.eclipse.org/platform/job/eclipse.platform.ui/view/change-requests/job/PR-2424/11/consoleFull

  WorkbookEditorsHandlerTest.doTearDown:73 » Resource Problems encountered while deleting resources.

Tests run: 1659, Failures: 1, Errors: 5, Skipped: 196

Exception in thread "Active Thread: Equinox Container: 5e5cc6b7-0a59-4770-ba00-8313910b610d" java.lang.OutOfMemoryError: Java heap space

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Worker-5"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Worker-15"

Maybe this could help understanding OOM errors on jenkins. See eclipse-platform#2432

merks · 2024-10-21T11:31:16Z

Maybe this is related:

https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/issues/5056

laeubi · 2024-10-21T11:38:17Z

If I look at this where we exchaust thread pools:

"Problems encountered while deleting files" due "Thread limit exceeded replacing blocked worker" eclipse.platform#1592

I just wanted to mention that each thread requires a small amount of ram as well....

iloveeclipse · 2024-10-21T11:59:46Z

I just wanted to mention that each thread requires a small amount of ram as well....

I would expect jenkins runs on a very small VM with very few (2-4?) cores, so thread pool size would be also very small and shouldn't be the root cause here.

iloveeclipse · 2024-10-21T12:02:20Z

Maybe this is related:
https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/issues/5056

Hmm. I don't see OOM's on JDT tests which are known to be memory sensitive, and OOMs in UI tests seem to happen on / after specific testsuite, so it look like a code regression on our (platform / UI) side.

laeubi · 2024-10-21T12:36:42Z

I would expect jenkins runs on a very small VM with very few (2-4?) cores, so thread pool size would be also very small and shouldn't be the root cause here.

This is a common misunderstanding with ForkJoinPools... If threads are blocked (e.g. the are waiting for I/O or waiting for other threads to join), then new threads are spwaned untill the configured concurrency level is fulfilled again, see here

The pool attempts to maintain enough active (or available) threads by dynamically adding, suspending, or resuming internal worker threads, even if some tasks are stalled waiting to join others.

The exception given here says "Thread limit exceeded replacing blocked worker" and the limit is by default 32767 (!)

iloveeclipse · 2024-10-21T12:44:38Z

This is a common misunderstanding with ForkJoinPools

Cool, thanks. So one part of the problem in eclipse-platform/eclipse.platform#1592 is that the used pool is not limited in size (beside that the LocalFile.internalDelete() doesn't do what it supposed to do)?

laeubi · 2024-10-21T12:55:21Z

It is just a guess, if we see Thread explosion somehwere and memmory problems elsewhere, In general I/O bound tasks don'T play well with fork-join because the whole fork-join was introduced because of CPU bound (!) task usually suffer from context switch penalities. For I/O non blocking I/O is usually most faster if we asumme a shared device (e.g. harddisk / network / ...).

iloveeclipse · 2024-10-21T13:13:28Z

https://ci.eclipse.org/platform/job/eclipse.platform.ui/job/PR-2433/1/console shows no major memory increase after multiple Thread limit exceeded replacing blocked worker errors.
So something else must be the problem here.

Maybe this could help understanding OOM errors on jenkins. Note: moved OpenCloseTest to the begin, just in case if that causes OOM. See eclipse-platform#2432

See #2432 and eclipse-platform/eclipse.platform#1592 This change shouldn't affect any test at all, it is not a code change.

Also exit E4Testable on OOM. This is supposed to workaround and to understand OOM errors on jenkins. See eclipse-platform#2432

mickaelistria · 2024-12-10T08:31:22Z

Was it tried to revert eclipse-platform/eclipse.platform#1592 in a PR and see if it makes build more successful?

Also exit E4Testable on OOM. This is supposed to workaround and to understand OOM errors on jenkins. See eclipse-platform#2432

iloveeclipse · 2024-12-10T08:43:26Z

Was it tried to revert eclipse-platform/eclipse.platform#1592 in a PR and see if it makes build more successful?

As mentioned, I believe we only see OOM's in platform UI tests so far. The change eclipse-platform/eclipse.platform#1592 would also affect JDT tests, which seem not to be the case.

jukzi · 2024-12-10T08:45:21Z

i am just running UiTestSuite locally, and monitoring the memory. The increasing memory usage feels like some leak between tests:

mickaelistria · 2024-12-10T08:48:09Z

As mentioned, I believe we only see OOM's in platform UI tests so far.

Maybe that one of Platform UI tests do exactly track some scalability related to intense usage of the resource model operation and does lead to OOM with the exact purpose of showing a performance issue.
But it's indeed difficult to test as its different repos...

jukzi · 2024-12-10T08:50:30Z

running UiTestSuit locally, i also see more and more symbols in the taskbar, feels like even whole applications are not closed after tests:

iloveeclipse · 2024-12-10T09:42:08Z

Yes

akurtakov · 2024-12-10T09:44:58Z

My thinking - if the PR fails with OOM itself and gives all the details for investigation there is no need to merge it as doing more would not help with OOM and investigation can continue on the PR itself till it actually improves the situation.

jukzi · 2024-12-10T09:48:37Z

running UiTestSuit locally, i also see more and more symbols in the taskbar

Leak: Example ImportArchiveOperationTest alone opens a lot of windows (using openTestWindow().run) without ever closing them. I guess this is regression from 2ebbbe9 @akurtakov because the superclass UITestCase may have closed those windows using UITestCase.closeTestWindows

akurtakov · 2024-12-10T09:50:33Z

I'll fix ImportArchiveOperationTest now. Funny, that I worked on this one while investigating the problem.

jukzi · 2024-12-10T09:51:15Z

tested that a revert close those windows.

Work towards eclipse-platform#2432 .

akurtakov · 2024-12-10T09:59:34Z

#2610 should reduce the frequency of the problem but we have been seeing OOMs before it too so this issue should stay open till we see no OOMs for couple of weeks at least

Work towards #2432 .

Also exit E4Testable on OOM. This is supposed to workaround and to understand OOM errors on jenkins. See eclipse-platform#2432

jukzi · 2024-12-10T13:10:58Z

There is still heap memory leaked and even some windows not closed after tests.

iloveeclipse · 2024-12-10T13:32:28Z

There is still heap memory leaked and even some windows not closed after tests

Any chance to see which test leaks what?

Also exit E4Testable on OOM. This is supposed to workaround and to understand OOM errors on jenkins. See eclipse-platform#2432

akurtakov · 2024-12-10T15:57:26Z

Seems like #2609 brought memory consumption down enough to make build not OOM anymore (for now).
@jukzi Can you show us heap memory usage after this patch too?

iloveeclipse · 2024-12-10T16:19:52Z

Seems like #2609 brought memory consumption down enough to make build not OOM anymore (for now). @jukzi Can you show us heap memory usage after this patch too?

With the PR #2433 rebased on 78615b8 we see following picture:

before 78615b8

#################################################
testOpenCloseView
########### Memory usage reported by JVM ########
   1.073.741.824 bytes max heap
   1.070.596.096 bytes heap allocated
     681.852.864 bytes free heap
     388.743.232 bytes used heap
#################################################

OOM!

after 78615b8

#################################################
testOpenCloseView
########### Memory usage reported by JVM ########
   1.073.741.824 bytes max heap
     300.941.312 bytes heap allocated
     168.866.408 bytes free heap
     132.074.904 bytes used heap
#################################################

No OOM!

akurtakov · 2024-12-10T16:32:21Z

Removing o.e.jdt.ui from the equation and reducing allocated heap by ~700MB is very suspicious and probably points to something huge not being cleaned up there.

iloveeclipse · 2024-12-10T16:33:39Z

I guess automatic detection of JRE's on startup is one of the pain points, we had to explicitly disable that job in all JDT/PDE repos where Java tooling is present.

merks · 2024-12-10T17:11:04Z

How likely is that to be painful for a user with many JDKs!

jukzi · 2024-12-10T17:18:33Z

@jukzi Can you show us heap memory usage after this patch too?

see #2615 as a continuation

iloveeclipse · 2024-12-10T17:24:20Z

How likely is that to be painful for a user with many JDKs!

Not really, given a reasonable sized RAM or proper JVM arguments. 4GB RAM (or 1GB max heap default == 1/4 RAM) is surely not suitable anymore for a serious work. However with already 8GB max heap you don't see any problem at all also on large workspaces.

When running locally and jdt plugins are available the test walked the whole contributed JDT items in the project explorer, which took much time and Heap memory eclipse-platform#2432

When running locally and jdt plugins are available the test walked the whole contributed JDT items in the project explorer, which took much time and Heap memory #2432

iloveeclipse · 2025-01-07T15:18:22Z

I believe this can be closed now.

iloveeclipse added the bug Something isn't working label Oct 21, 2024

iloveeclipse mentioned this issue Oct 21, 2024

Fix API errors by updating bundle versions #2431

Merged

iloveeclipse changed the title ~~OutOfMemoryError reported by jenkins validation build~~ OutOfMemoryError reported by jenkins validation build after org.eclipse.ui.tests.UiTestSuite Oct 21, 2024

iloveeclipse added a commit to iloveeclipse/eclipse.platform.ui that referenced this issue Oct 21, 2024

Add memory reporting for UI tests and exit for E4Testable on OOM

55d06ea

Maybe this could help understanding OOM errors on jenkins. See eclipse-platform#2432

This was referenced Oct 21, 2024

Add memory reporting for UI tests and exit for E4Testable on OOM #2433

Open

Revert "May fix issue #2716 (#3117)" eclipse-jdt/eclipse.jdt.core#3126

Merged

iloveeclipse added a commit that referenced this issue Oct 21, 2024

Dummy change to check OOMs in jenkins

ee32bb4

See #2432 and eclipse-platform/eclipse.platform#1592 This change shouldn't affect any test at all, it is not a code change.

iloveeclipse mentioned this issue Oct 21, 2024

DO NOT MERGE! Dummy change to check OOMs in jenkins #2438

Closed

iloveeclipse added a commit to iloveeclipse/eclipse.platform.ui that referenced this issue Oct 22, 2024

Add explicit gc and memory reporting for UI tests

6037f01

Also exit E4Testable on OOM. This is supposed to workaround and to understand OOM errors on jenkins. See eclipse-platform#2432

iloveeclipse mentioned this issue Oct 22, 2024

Fix for RejectedExecutionException during delete operations eclipse-platform/eclipse.platform#1593

Merged

iloveeclipse added a commit to iloveeclipse/eclipse.platform.ui that referenced this issue Oct 22, 2024

Add explicit gc and memory reporting for UI tests

820b875

Also exit E4Testable on OOM. This is supposed to workaround and to understand OOM errors on jenkins. See eclipse-platform#2432

jukzi added the test junit test related things label Oct 28, 2024

fedejeanne mentioned this issue Oct 29, 2024

Multi line error message fixed #2407

Merged

mickaelistria mentioned this issue Dec 10, 2024

Refresh option not available for some resources that are not closed p… #2601

Merged

iloveeclipse added a commit to iloveeclipse/eclipse.platform.ui that referenced this issue Dec 10, 2024

Add explicit gc and memory reporting for UI tests

4704f07

Also exit E4Testable on OOM. This is supposed to workaround and to understand OOM errors on jenkins. See eclipse-platform#2432

jukzi added the regression label Dec 10, 2024

akurtakov added a commit to akurtakov/eclipse.platform.ui that referenced this issue Dec 10, 2024

Close test windows in ImportArchiveOperationTest

34263ef

Work towards eclipse-platform#2432 .

akurtakov mentioned this issue Dec 10, 2024

Close test windows in ImportArchiveOperationTest #2610

Merged

akurtakov added a commit that referenced this issue Dec 10, 2024

Close test windows in ImportArchiveOperationTest

b379121

Work towards #2432 .

iloveeclipse added a commit to iloveeclipse/eclipse.platform.ui that referenced this issue Dec 10, 2024

Add explicit gc and memory reporting for UI tests

ce8df9e

Also exit E4Testable on OOM. This is supposed to workaround and to understand OOM errors on jenkins. See eclipse-platform#2432

iloveeclipse added a commit to iloveeclipse/eclipse.platform.ui that referenced this issue Dec 10, 2024

Add explicit gc and memory reporting for UI tests

7427a94

Also exit E4Testable on OOM. This is supposed to workaround and to understand OOM errors on jenkins. See eclipse-platform#2432

jukzi mentioned this issue Dec 10, 2024

Some UI Tests leak Shells #2615

Closed

jukzi mentioned this issue Dec 11, 2024

ImportExistingArchiveProjectFilterTest: skip jdt #2432 #2618

Merged

laeubi mentioned this issue Dec 16, 2024

Do not automatically detect JVMs on CI systems eclipse-jdt/eclipse.jdt.debug#577

Merged

iloveeclipse closed this as completed Jan 7, 2025

fedejeanne mentioned this issue Feb 17, 2025

Improve validation message in WorkspaceRoot::getProject(String) eclipse-platform/eclipse.platform#1740

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

OutOfMemoryError reported by jenkins validation build after org.eclipse.ui.tests.UiTestSuite #2432

OutOfMemoryError reported by jenkins validation build after org.eclipse.ui.tests.UiTestSuite #2432

iloveeclipse commented Oct 21, 2024

iloveeclipse commented Oct 21, 2024

merks commented Oct 21, 2024

laeubi commented Oct 21, 2024

iloveeclipse commented Oct 21, 2024

iloveeclipse commented Oct 21, 2024

laeubi commented Oct 21, 2024

iloveeclipse commented Oct 21, 2024

laeubi commented Oct 21, 2024

iloveeclipse commented Oct 21, 2024

mickaelistria commented Dec 10, 2024

iloveeclipse commented Dec 10, 2024

jukzi commented Dec 10, 2024

mickaelistria commented Dec 10, 2024

jukzi commented Dec 10, 2024

iloveeclipse commented Dec 10, 2024

akurtakov commented Dec 10, 2024

jukzi commented Dec 10, 2024

akurtakov commented Dec 10, 2024 •

edited

Loading

jukzi commented Dec 10, 2024

akurtakov commented Dec 10, 2024

jukzi commented Dec 10, 2024

iloveeclipse commented Dec 10, 2024

akurtakov commented Dec 10, 2024

iloveeclipse commented Dec 10, 2024

akurtakov commented Dec 10, 2024

iloveeclipse commented Dec 10, 2024

merks commented Dec 10, 2024

jukzi commented Dec 10, 2024

iloveeclipse commented Dec 10, 2024

iloveeclipse commented Jan 7, 2025

OutOfMemoryError reported by jenkins validation build after org.eclipse.ui.tests.UiTestSuite #2432

OutOfMemoryError reported by jenkins validation build after org.eclipse.ui.tests.UiTestSuite #2432

Comments

iloveeclipse commented Oct 21, 2024

iloveeclipse commented Oct 21, 2024

merks commented Oct 21, 2024

laeubi commented Oct 21, 2024

iloveeclipse commented Oct 21, 2024

iloveeclipse commented Oct 21, 2024

laeubi commented Oct 21, 2024

iloveeclipse commented Oct 21, 2024

laeubi commented Oct 21, 2024

iloveeclipse commented Oct 21, 2024

mickaelistria commented Dec 10, 2024

iloveeclipse commented Dec 10, 2024

jukzi commented Dec 10, 2024

mickaelistria commented Dec 10, 2024

jukzi commented Dec 10, 2024

iloveeclipse commented Dec 10, 2024

akurtakov commented Dec 10, 2024

jukzi commented Dec 10, 2024

akurtakov commented Dec 10, 2024 • edited Loading

jukzi commented Dec 10, 2024

akurtakov commented Dec 10, 2024

jukzi commented Dec 10, 2024

iloveeclipse commented Dec 10, 2024

akurtakov commented Dec 10, 2024

iloveeclipse commented Dec 10, 2024

akurtakov commented Dec 10, 2024

iloveeclipse commented Dec 10, 2024

merks commented Dec 10, 2024

jukzi commented Dec 10, 2024

iloveeclipse commented Dec 10, 2024

iloveeclipse commented Jan 7, 2025

akurtakov commented Dec 10, 2024 •

edited

Loading