Add Mojos to replace "The Eclipse Test Framework" #3058

laeubi · 2023-11-19T14:54:10Z

The Eclipse Test Framework is a way to specifically run a test inside an eclipse installation itself against a prebuild SDK.

This implements the framework on top of maven to make it much easier to run and setup than with the several (ant) prerequsites of the original description provided here:

as well as offer much more flexibility e.g. to run tests in a local mode where not the SDK but a local build is used as the test base.

github-actions · 2023-11-19T16:32:32Z

Test Results

  570 files ±0   570 suites ±0 3h 46m 17s ⏱️ + 1m 25s
  374 tests ±0   368 ✔️ ±0   6 💤 ±0 0 ❌ ±0
1 122 runs ±0 1 103 ✔️ ±0 19 💤 ±0 0 ❌ ±0

Results for commit e32408c. ± Comparison against base commit d76d71c.

♻️ This comment has been updated with latest results.

mickaelistria · 2023-11-19T19:46:19Z

I think the case of running tests inside a pre-existing application is already covered by https://tycho.eclipseprojects.io/doc/latest/tycho-surefire-plugin/test-mojo.html#testRuntime . What is missing that requires further development in Tycho?

laeubi · 2023-11-20T05:28:44Z

@mickaelistria this is solely to replace the current ant based solution of "The Eclipse Testing Framework" in two cases

There are still some tests that are not run as the regular verification (you can search for pom.model.property.skipTests to find some) and are claimed to not work with Tycho, I have already tried to use Tycho here: https://github.com/eclipse-pde/eclipse.pde/runs/18589661815
Eclipse I-Build still uses that to execute test what leaves us in the uncomfortable situation that sometimes we have test-failures not seen in regular builds because different techniques are used.

Also this requires to have a quite complicated and boilerplate setup with test.xml in each test fragment (not talking about that we need test fragments at all) as well as carefully ensure that the test-suites reference all test as they are otherwise not getting executed with the test-suite.

Another issue is that this "Eclipse Test Framework" seems not very well understood and not maintained anymore (the wiki page still mention Java 1.5) and I like to demystify that and offer a Tycho alternative so we probably can deprecate the old ant-based way one time fr platform.

If you could help to use instead the default tycho-sure-fire that would be great, but every time I asked for that in the past it was only "should work but we have not capacity/knowledge/..." and as it is quite easy now to just call the code in Eclipse itself wit little effort this seems a good way forward (even fro those that might still want to use that Eclipse Testing Framework).

akurtakov · 2023-11-20T06:27:31Z

I am for all that can replace https://github.com/eclipse-platform/eclipse.platform.releng.aggregator/blob/master/JenkinsJobs/AutomatedTests/I_unit_cen64_gtk3_java21.groovy#L94 and https://github.com/eclipse-platform/eclipse.platform.releng.aggregator/tree/master/production/testScripts with a saner script.

laeubi · 2023-11-20T08:24:33Z

This now has reached state of basic operation so I mark this ready as it could potentially be merged...

mickaelistria · 2023-11-20T08:26:58Z

This integration-test https://github.com/eclipse-tycho/tycho/blob/f7d89236fc0fc0ce48a0d972d0e686440aef21bd/tycho-its/projects/surefire.p2InstalledRuntime/extProductTest/pom.xml shows how to setup Tycho so the test is executed in a given product installation. These settings could be part of a test-on-SDK-delivery profile in the parent pom, and the SDK installation path be a property.

But I'm still curious about the overall plan: at the moment, the tests are run exactly without Tycho, because there are no artifacts produced and they're only orchestration, so it doesn't really fit the Maven approach; we also don't re-compile tests as part of the test execution.
How would this scenario work with a Maven-based solution?
I personally believe that the issue with the test process nowadays isn't really what it does (ie install test units than run test application), but more that it is very scattered in different Ant files; while it could usually just be 1 not-too-big bash script.

laeubi · 2023-11-20T08:34:11Z

But I'm still curious about the overall plan: at the moment, the tests are run exactly without Tycho, because there are no artifacts produced

I'm not sure if this is true, these test.xml do ALOT of stuff that needs manual crafting (e.g. what should be installed what test to run in what order with what runner and so on), so actually there is something produced, that is we build a customized eclipse product based on the sdk.

so it doesn't really fit the Maven approach

Also Tycho/Maven can run things without producing artifacts (e.g. with pom packaging).

we also don't re-compile tests as part of the test execution. How would this scenario work with a Maven-based solution?

Recompilation is not required, e.g. equinox uses the precompiled osgi TCKs to run the TCK suite against equinox implementations.

I personally believe that the issue with the test process nowadays isn't really what it does

Just take a look at that monster ...
https://github.com/eclipse-platform/eclipse.platform.releng.aggregator/blob/master/production/testScripts/configuration/sdk.tests/testScripts/test.xml

while it could usually just be 1 not-too-big bash script.

Bash scripts only run on Linux (maybe mac)

mickaelistria · 2023-11-20T09:47:47Z

(e.g. what should be installed what test to run in what order with what runner and so on), so actually there is something produced, that is we build a customized eclipse product based on the sdk.

Can you please share links to such test.xml that perform such extra steps?
Usually they do not install anything: they assume that everything is pre-installed, and the test.xml just defines what to run (the test suite) and whether it's UI or non-UI tests.
The ones that define extra requirements or extra installation could probably be replaced by the right dependencies in MANIFEST.MF or p2.inf.

Also Tycho/Maven can run things without producing artifacts (e.g. with pom packaging).

Yes, but it's kind of clunky as one has to produce at least 1 extra artifact to run such a script: the pom.xml ;) and then decide where to maintain it and so on to be discoverable by the build script.
Those could be discoverable by a in a bash script if the main goal is to ditch most of the Ant orchestration:

# bash pseudo/untested-code
wget https://download.eclipse.org/eclipse/downloads/drops4/I20231119-1800/download.php?dropFile=eclipse-platform-I20231119-1800-linux-gtk-x86_64.tar.gz
unzip eclipse-platform-I20231119-1800-linux-gtk-x86_64.tar.gz
cd eclipse
eclipse -application org.eclipse.p2.equinox.director -repository https://download.eclipse.org/eclipse/updates/4.30-I-builds/ -installIU org.eclipse.sdk.tests.features.group
for testPlugin in $(ls -1 plugins/*.tests.jar); do
  rm text.xml #cleanup
  tax xz $testPlugin test.xml
  testType=xmllint --xpath '//ant/target[value()=="*-tests"]' test.xml
  testApplication=org.eclipse.core.tests.application
  if [ "$testType" == "*ui* ]; then
    testApplication=org.eclipse.ui.tests.application
  fi
  testClass=xmllint --xpath '//ant/target[value()=="*-tests"]/property[name="classname"]/value()' test.xml
  eclipse -application $testApplication -out TEST-$testPlugin.xml
fi

akurtakov · 2023-11-20T09:57:26Z

(e.g. what should be installed what test to run in what order with what runner and so on), so actually there is something produced, that is we build a customized eclipse product based on the sdk.

Can you please share links to such test.xml that perform such extra steps? Usually they do not install anything: they assume that everything is pre-installed, and the test.xml just defines what to run (the test suite) and whether it's UI or non-UI tests. The ones that define extra requirements or extra installation could probably be replaced by the right dependencies in MANIFEST.MF or p2.inf.

It's even worse right now as it's not using test.xml in the bundle to define the extra deps but it's part of the main/share one the https://github.com/eclipse-platform/eclipse.platform.releng.aggregator/blob/master/production/testScripts/configuration/sdk.tests/testScripts/test.xml#L34 .

laeubi · 2023-11-20T09:57:40Z

Can you please share links to such test.xml that perform such extra steps?

https://github.com/eclipse-platform/eclipse.platform.releng.aggregator/blob/4b5fccff1c7878e02430c22ede26c8788c60896e/production/testScripts/configuration/sdk.tests/testScripts/test.xml#L34 and following hundreds of lines

https://github.com/eclipse-pde/eclipse.pde/blob/eeffb962b568b6208d1de4d8b897a78371a4eec2/build/org.eclipse.pde.build.tests/test.xml#L111 ...

Those could be discoverable by a in a bash script if the main goal is to ditch most of the Ant orchestration:

How is this better than an ant/pom beside that it only runs on linux and also has several preconditions to be meet and special knowledge to understand?

laeubi · 2023-11-20T10:00:37Z

It's even worse right now as it's not using test.xml in the bundle to define the extra deps but it's part of the main/share one the

You do you like to indicate we actually can delete all test.xml files from bundles right now?

akurtakov · 2023-11-20T10:01:45Z

It's even worse right now as it's not using test.xml in the bundle to define the extra deps but it's part of the main/share one the

You do you like to indicate we actually can delete all test.xml files from bundles right now?

No, as the bundle test.xml contains the test suite to be actually run.

mickaelistria · 2023-11-20T11:26:58Z

https://github.com/eclipse-platform/eclipse.platform.releng.aggregator/blob/4b5fccff1c7878e02430c22ede26c8788c60896e/production/testScripts/configuration/sdk.tests/testScripts/test.xml#L34 and following hundreds of lines

This one is more or less what I think we can replace with a ~20 lines shell script + better instructions in p2.inf to properly define test dependencies in a portable way.

https://github.com/eclipse-pde/eclipse.pde/blob/eeffb962b568b6208d1de4d8b897a78371a4eec2/build/org.eclipse.pde.build.tests/test.xml#L111

We don't ship delta pack any more and this actually seems more like a precondition that could be setup in the test case directly.

How is this better than an ant/pom beside that it only runs on linux and also has several preconditions to be meet and special knowledge to understand?

There are many more people who can maintain bash than people who can maintain Ant/Pom. Bash is the language of DevOps, the language of orchestration that everyone is taught in university. Bash runs on all OSs (even Windows using the Linux subsystem), and on all cloud infras.
It's the right tool for the right problem.

But let's get back to the initial topic. I've lost track and while going through the discussion, it still remained unclear: what is the problem that we're willing to fix for the SDK build? Some test issues? Too much unmaintainable Ant?... How the current proposal in this PRs fixes it better than discussed alternatives?

laeubi · 2023-11-20T11:42:34Z

This one is more or less what I think we can replace with a ~20 lines shell script + better instructions in p2.inf to properly define test dependencies in a portable way.

Can you then provide a PR for that?

We don't ship delta pack any more and this actually seems more like a precondition that could be setup in the test case directly.

Can you then provide a PR for PDE so this test bundle is executed as part of the regular verification build? Currently we only see failures in ibuild what is too late and time consuming to fix/analyze/verify...

There are many more people who can maintain bash than people who can maintain Ant/Pom.

"bash" is useless without the context of Eclipse/Tycho/... here imo but in the end I don't care if it is the agreement of platfrom.releng to replace maven builds with bash scripts I just don't want:

having to add mysterious non trivial boilerplate "codes" to projects that I need to copy all over again that is never used in verification builds
Having failures in the i build because tests are not executed at verification time because "it is to complicated to setup"
Having verification builds pass and i buils fail because I forgotten to add something somewhere manually that already was provided by the configuration
Having verification builds fail/flaky but i builds pass because we use different setups

what is the problem that we're willing to fix for the SDK build

Get rid of "The Eclipse Testing Framework" in its current ant based unmaintainable state.

How the current proposal in this PRs fixes it better than discussed alternatives?

This PR does not fix anything it just might allows to replace hundred of merely duplicated test.xml deleted and as a step forward in this, the alternatives are already mentioned here and there, but actually not action is taken to apply them because of some issues they do not cover. So in the past it has helped a lot to first replace ant>maven and improve on top of that instead of wait for the big-bang that never happens.

mickaelistria · 2023-11-20T12:15:21Z

Can you then provide a PR for that?
...
Can you then provide a PR for PDE so this test bundle is executed as part of the regular verification build? Currently we only see failures in ibuild what is too late and time consuming to fix/analyze/verify...

I may try, but are there some issues about these particular problems/goals? As I'm looking at many things at the same time I too easily forget things if I don't have a tab open about it.

having to add mysterious non trivial boilerplate "codes" to projects that I need to copy all over again that is never used in verification builds

The testSuite still needs to be discoverable somewhere to run the proper tests; and the file has to be embedded in the bundle (also for it to be discovered later). At the moment, it's test.xml and it's mostly boilerplate/useless. What would be a better alternative? A test.properties file? Or some entries in some other common file? (IIRC, build.properties is not viable because it's not embedded by default), or should we include new files in the bundle jar...?
(Note that I'm really not arguing against any of that, and I think a test.properties file in the bundle makes a lot of sense)

Having failures in the i build because tests are not executed at verification time because "it is to complicated to setup"

I don't think the problem is much related to the test framework here, those are issues specific to some test cases and bundles, for which I expect a specific solution of p2.inf + Java code inside the bundle can solve 90% of cases in a very portable way.
Those should be treated as regular bugs to fix with regular existing tricks for best portability; not something that support the idea of reimplementing an "Eclipse test framework" over Maven.

Having verification builds pass and i buils fail because I forgotten to add something somewhere manually that already was provided by the configuration
[...]
Having verification builds fail/flaky but i builds pass because we use different setups

I think other build parameters are the same story as the suite name above.

If one want the test to be "portable" and reliable, it must contain all instructions for proper execution (test suite to run, specific parameters, dependencies). Then, once all that is easily accessible, it becomes easy to run them reliably with whatever technology that is OK reading those parameters without need for further specific configuration tweak (ideally could be a pom.xml, or a bash script or whatever not Ant; what matters is really that it doesn't do much test-specific config per se).
The Maven-first approach is too specific IMO, as it encourages in configuring Maven artifacts more than actually making the bundle host the information; so the output can be missing other options for other build processes (Ant, or Bash, or CLI, or PDE...) to properly provision the test execution.

laeubi · 2023-11-20T12:28:04Z

I may try, but are there some issues about these particular problems/goals?

There is not a dedicated issue but

missing test execution discovered here New PDE test fails since I20231105-1800 eclipse-pde/eclipse.pde#884
currently not working attempt Enable pde build tests eclipse-pde/eclipse.pde#905

The testSuite still needs to be discoverable somewhere to run the proper tests; and the file has to be embedded in the bundle (also for it to be discovered later). At the moment, it's test.xml and it's mostly boilerplate/useless. What would be a better alternative? A test.properties file? Or some entries in some other common file? (IIRC, build.properties is not viable because it's not embedded by default), or should we include new files in the bundle jar...?

There is some (semi) OSGi standards I try to standardize at the moment used by the OSGi-TCKs that is using a Manifest header (Test-Cases) for that purpose:

OSGi "Test-Bundle-Specification" osgi/osgi#547

If one want the test to be "portable" and reliable, it must contain all instructions for proper execution (test suite to run, specific parameters, dependencies).

That's what Requirements and Capabilities are for
https://blog.osgi.org/2015/12/using-requirements-and-capabilities.html

But I won't mind if there is one "setup the integration test" that configures things, as long as it does those in a way that reuses all information from the build/pom/wherever and only adds the specifics of the test (e.g. using mac with webkit ...)

mickaelistria · 2023-11-20T12:39:31Z

But I won't mind if there is one "setup the integration test" that configures things, as long as it does those in a way that reuses all information from the build/pom/wherever

Exactly, but the issue is in choosing the right "whatever" so it can serve all the flavors of installing and launching tests we care about (ie surefire during build or test execution in a next process).

laeubi · 2023-11-20T12:42:50Z

surefire is just a launcher as this mojo is just a wrapper around the core/ui test launcher of eclipse ... So at the end this PR is only make this launcher available without ant workarounds or shell scripts that call the launcher :-)

Neither a shell script nor this mojo solves the problem that we do more things than "launch the test" in platform....

mickaelistria · 2023-11-20T13:40:30Z

I see the goal is to directly reuse the applications, but while it's good to reuse as much as possible the applications per se wouldn't really solve the problem that some tests have pre-requisite (currently in text.xml), so this proposal wouldn't help in resolving the issue as the test.xml is not read by the org.eclipse..tests.application, it read by the org.eclipse.core.ant.antRunner application which then calls the test application.
What does the org.eclipse.tycho.surefire.osgibooter application does differently enough from the org.eclipse..tests.application to make it worth using it as an alternative?

laeubi · 2023-11-20T14:10:53Z

What does the org.eclipse.tycho.surefire.osgibooter application does differently enough from the org.eclipse..tests.application to make it worth using it as an alternative?

Surefire currently would execute an annotated/configured test with the surefire-junit runner, while as you mentioned this here would use the org.eclipse.core.ant.antRunner (and has some way to configure result format). If it is an alternative seem to heavily depend on the use case, Eclipse seem to have such usecase and the wiki claims other might "reuse that technology":

https://wiki.eclipse.org/Platform-releng/Eclipse_Test_Framework

This framework is used by the Eclipse Project Platform to run their unit and integration tests in an automated way, as described in Platform-releng/Automated_Testing, but may be used by others to automate your testing, if you'd like.

So as Tycho already supports bnd-testing and is an Eclipse project and is used for Platfrom it seems fair to have a way to execute such tests with Tycho as well, not that I would recommend that or think its better its just different.

The Eclipse Test Framework is a way to specifically run a test inside an eclipse installation itself against a prebuild SDK. This implements the framework on top of maven to make it much easier to run and setup than with the several (ant) prerequsites of the original description provided here: - https://wiki.eclipse.org/Platform-releng/Eclipse_Test_Framework - https://wiki.eclipse.org/Platform-releng/Automated_Testing as well as offer much more flexibility e.g. to run tests in a local mode where not the SDK but a local build is used as the test base.

laeubi · 2023-11-20T16:43:34Z

Rebase on master to only include the essentials...

laeubi · 2023-11-28T11:51:56Z

I'm closing this for now even though it was a quite interesting journey and has raveled some bugs/missing features in the Embedded Eclipse Launcher... but it seems note useful to keep this technique alive by making it easier to use with Tycho with no strong demands...

laeubi marked this pull request as draft November 19, 2023 14:54

laeubi force-pushed the add_eclipse_test_mojo branch 2 times, most recently from 31de496 to 24338f6 Compare November 19, 2023 18:43

laeubi force-pushed the add_eclipse_test_mojo branch from 24338f6 to 196c841 Compare November 20, 2023 05:59

laeubi marked this pull request as ready for review November 20, 2023 08:17

laeubi marked this pull request as draft November 20, 2023 12:28

laeubi force-pushed the add_eclipse_test_mojo branch from 196c841 to e32408c Compare November 20, 2023 16:33

laeubi closed this Nov 28, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Mojos to replace "The Eclipse Test Framework" #3058

Add Mojos to replace "The Eclipse Test Framework" #3058

laeubi commented Nov 19, 2023 •

edited

Loading

github-actions bot commented Nov 19, 2023 •

edited

Loading

mickaelistria commented Nov 19, 2023

laeubi commented Nov 20, 2023

akurtakov commented Nov 20, 2023

laeubi commented Nov 20, 2023

mickaelistria commented Nov 20, 2023

laeubi commented Nov 20, 2023 •

edited

Loading

mickaelistria commented Nov 20, 2023

akurtakov commented Nov 20, 2023

laeubi commented Nov 20, 2023

laeubi commented Nov 20, 2023

akurtakov commented Nov 20, 2023

mickaelistria commented Nov 20, 2023

laeubi commented Nov 20, 2023

mickaelistria commented Nov 20, 2023

laeubi commented Nov 20, 2023

mickaelistria commented Nov 20, 2023

laeubi commented Nov 20, 2023

mickaelistria commented Nov 20, 2023

laeubi commented Nov 20, 2023 •

edited

Loading

laeubi commented Nov 20, 2023

laeubi commented Nov 28, 2023

Add Mojos to replace "The Eclipse Test Framework" #3058

Add Mojos to replace "The Eclipse Test Framework" #3058

Conversation

laeubi commented Nov 19, 2023 • edited Loading

github-actions bot commented Nov 19, 2023 • edited Loading

Test Results

mickaelistria commented Nov 19, 2023

laeubi commented Nov 20, 2023

akurtakov commented Nov 20, 2023

laeubi commented Nov 20, 2023

mickaelistria commented Nov 20, 2023

laeubi commented Nov 20, 2023 • edited Loading

mickaelistria commented Nov 20, 2023

akurtakov commented Nov 20, 2023

laeubi commented Nov 20, 2023

laeubi commented Nov 20, 2023

akurtakov commented Nov 20, 2023

mickaelistria commented Nov 20, 2023

laeubi commented Nov 20, 2023

mickaelistria commented Nov 20, 2023

laeubi commented Nov 20, 2023

mickaelistria commented Nov 20, 2023

laeubi commented Nov 20, 2023

mickaelistria commented Nov 20, 2023

laeubi commented Nov 20, 2023 • edited Loading

laeubi commented Nov 20, 2023

laeubi commented Nov 28, 2023

laeubi commented Nov 19, 2023 •

edited

Loading

github-actions bot commented Nov 19, 2023 •

edited

Loading

laeubi commented Nov 20, 2023 •

edited

Loading

laeubi commented Nov 20, 2023 •

edited

Loading