NEW Rerun failed features in ci

[emteknetnz]

Issue silverstripe/silverstripe-admin#1671

After merge - re-recreate this PR and target 4 and, create a 4.12 branch after merge and release 4.12.0 so that this also works for for 4.13 runs

Simulated failure run - https://github.com/emteknetnz/silverstripe-admin/actions/runs/8639635863/job/23686313351#step:12:142

When I setup this up on my local to pass on retry with the is_ci config value true, it will show there was a failed step, but importantly it shows the scenario as passed

Code used on local to pass on retry:

// RetryScenarioTester.php
    public function test(Environment $env, FeatureNode $feature, ScenarioInterface $scenario, $skip)
    {
        $result = $this->decoratedTester->test($env, $feature, $scenario, $skip);
        if (!$skip && !$result->isPassed()) {
            // Run the scenario again
            file_put_contents('php://stdout', 'Retrying scenario: ' . $scenario->getTitle() . PHP_EOL);
            $myScenarioThatPasses = new class (
                $scenario->getTitle(),
                $scenario->getTags(),
                $scenario->getSteps(),
                $scenario->getKeyword(),
                $scenario->getLine()
            ) extends ScenarioNode {
                public function __construct($title, array $tags, array $steps, $keyword, $line)
                {
                    // Replaces the last step that fails with one that passes
                    $steps = array_slice($steps, 0, -1);
                    $steps[] = new StepNode('And', 'I click on the "pre" element', [], 7);
                    parent::__construct($title, $tags, $steps, $keyword, $line);
                }
            };
            $result = $this->decoratedTester->test($env, $feature, $myScenarioThatPasses, $skip);
        }
        return $result;
    }

[GuySartorelli]

After merge - re-recreate this PR and target 4 and, create a 4.12 branch after merge and release 4.12.0 so that this also works for for 4.13 runs

I'm going to assume this is a step for you to do - i.e. you will create a new PR and I'll review/merge. Let me know if that's not what you intended.

[emteknetnz]

Have changed it so that it's doing the rerun at the feature level, rather than the scenario level. This seemed the level at which backgrounds would rerun.

I've also made updates to the statistics that are updated via an event dispatcher so there aren't duplicate stats

I get this output locally if I have a 2nd failure on rerun

I get this output locally if I get the 2nd run to pass (I'll explain the 1 failed step shortly)

(The reason for the 1 failure in the stats above was because the $stat->getPath() of the passing step in the scenario won't match an existing failure path, it should work fine in real life though. Even if it doesn't, the scenario's passing so CI should go green here. If not it'll be a matter of just working out what's emitted the non zero exit code (I couldn't find anything though I didn't look too hard)

I used the following code to made the subsequent run pass:

// RerunRuntimeSuiteTester.php
    public function test(Environment $env, SpecificationIterator $iterator, $skip = false)
    {
        $results = array();
        foreach ($iterator as $specification) {
            $setup = $this->specTester->setUp($env, $specification, $skip);
            $localSkip = !$setup->isSuccessful() || $skip;
            $testResult = $this->specTester->test($env, $specification, $localSkip);
            $teardown = $this->specTester->tearDown($env, $specification, $localSkip, $testResult);

            if (!$testResult->isPassed()) {
                file_put_contents('php://stdout', 'Retrying specification' . PHP_EOL);

                /** @var \Behat\Gherkin\Node\FeatureNode $specification */
                /** @var \Behat\Gherkin\Node\ScenarioNode $scenario */
                $scenarios = $specification->getScenarios();
                $scenario = $scenarios[0];
                $scenario = new class (
                    $scenario->getTitle(),
                    $scenario->getTags(),
                    $scenario->getSteps(),
                    $scenario->getKeyword(),
                    $scenario->getLine()
                ) extends \Behat\Gherkin\Node\ScenarioNode {
                    public function __construct($title, array $tags, array $steps, $keyword, $line)
                    {
                        // Replaces the last step that fails with one that passes
                        $steps = array_slice($steps, 0, -1);
                        $steps[] = new \Behat\Gherkin\Node\StepNode('And', 'I click on the "pre" element', [], 7);
                        parent::__construct($title, $tags, $steps, $keyword, $line);
                    }
                };
                $scenarios = [$scenario];

                $specification = new \Behat\Gherkin\Node\FeatureNode(
                    $specification->getTitle(),
                    $specification->getDescription(),
                    $specification->getTags(),
                    $specification->getBackground(),
                    $scenarios,
                    $specification->getKeyword(),
                    $specification->getLanguage(),
                    $specification->getFile(),
                    $specification->getLine()
                );

                $setup = $this->specTester->setUp($env, $specification, $skip);
                $localSkip = !$setup->isSuccessful() || $skip;
                $testResult = $this->specTester->test($env, $specification, $localSkip);
                $teardown = $this->specTester->tearDown($env, $specification, $localSkip, $testResult);
            }

            $integerResult = new IntegerTestResult($testResult->getResultCode());
            $results[] = new TestWithSetupResult($setup, $integerResult, $teardown);
        }

        return new TestResults($results);
    }

[emteknetnz]

After merge - re-recreate this PR and target 4 and, create a 4.12 branch after merge and release 4.12.0 so that this also works for for 4.13 runs

I will do this

(rerun multiple times) It might be a good idea to add (either as a replacement of the is_ci option or as a separate one)

Let's just see how we get on with a single reruns in the real world first, we can update this later if there's a desire to

[GuySartorelli]

Works well locally. A few changes and questions below.

[GuySartorelli]

LGTM, merge on green