-
Notifications
You must be signed in to change notification settings - Fork 14
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Parallel execution using maven-failsafe-plugin #5
Comments
Hello @wakaleo , I think I might be experiencing this same (or a very similar) difficulty: I'm trying to update to the new version for the Cucumber 5 support, and getting failures any time the -Dparallel.tests arg is set to anything greater than 1. No tests get run, and I get one of these errors for each thread that tries to run:
I've tried going through my pom file and cleaning things up a bit, but there are probably a few comments hanging out from where I was trying to find a combination of settings that worked:
Is there any other information I can provide to help? |
Are you trying to run batches of tests across multiple machines? |
Yeah, the application I have to test is unfortunately a desktop client, and we can only run 1 test at a time per machine. Our solution was to set up some VM's and batch the execution out to them using Jenkins, and programmatically generate the appropriate number of SlicedTestRunner classes as needed to match the value of -Dparallel.tests passed in at runtime. |
Hmmm, I'm not familiar with that particular bit of code: maybe @NickBarrett or @cliviu could shed some light? |
Sorry folks. I would love to help but I haven't done any Java since university. I suspect my basic late-1990's Java skills won't help here. 😅 All the best finding a solution 👍 |
Sorry @NickBarrett , wrong Nick, I meant @nbarrett! |
Hi @NickBarrett - sorry you got lumbered with the responsibility of fixing this problem, I'm taking a look now! 😆 |
Phew that's a huge relief, for everybody 🤣 |
This is fantastic, and after updating to 2.2.5 I was able to start running again, thank you! I'm not sure if this is related (was also happening before I updated to the new version), but there is one other slightly strange behavior with failsafe parallelization; for some reason it's failing to find Example-level tags within Scenario Outlines, but only when using multiple threads. e.g. here
It seems like "expected" behavior in that it isn't producing any errors and even manages to produce an (empty) report as output - it just happens that nothing gets included because all of the |
Hi @nbarrett - does this seem related, or would it make more sense in / as a different Issue? I just tried with serenity core and maven version 2.2.7, and serenity cucumber version at 2.2.5 |
Hello again @azmo-rinsler - sorry I've personally had no experience at all with Cucumber 5, but will try and checkout the latest serenity versions this weekend and take a look. My only observation at this point is that (at least when I last used it) Serenity batching and parallel operation only worked using the system parameters as documented here e.g.
I can see that you've also supplied tags by means of system properties and I'm not sure at what point the tags when applied in this way, would be applied to the filtering of scenarios. As an experiment, what would happen if the tags were instead applied as annotations on the test runner(s) you are using? Hope this helps somewhat? |
@azmo-rinsler I might be able to help if you can provide me with a repeatable code that i can run and reproduce this weird behavior. |
I tried grabbing the cucumber starter repo and just make a few small changes to keep everything as close to default as possible; put it here for easy access Trying to run The most notable difference is to the
|
@azmo-rinsler I didn't look deep into it, but all I changed were
And the test was found. |
I'm not extremely familiar with Maven, so I might be misunderstanding, but isn't that what this part here in the POM file is doing?
This also brings up something I hadn't thought about before - it's possible this particular issue might be related to setting I'm not sure if / how it's managing the batching behind the scenes with this set of runtime args, but at the very least it appears to be correctly finding the tags, so maybe it is the value of Finds tests: Does NOT find tests: @nbarrett Sorry - I somehow managed to miss the second part of your last comment. Here's what the test runners look like right now - how should I apply the additional annotation?
Unless you meant within the Feature files? Tags seem to work reliably at the Feature and Scenario / Scenario Outline levels, while Examples do work, but it seems to only be when
|
Hi, guys! Do you plan the release any time soon? We really need that fix for parallel execution and tags |
Probably later this week. |
Good day @wakaleo,
I would like to ask if how can I configure parallel execution and configuration on this serenity-cucumber5? Because I am using maven-failsafe version 3.0.0-M4. I also tried using profiles but then I am not successful to execute in parallel. Hoping for your response. Thank you.
The text was updated successfully, but these errors were encountered: