Unblock run when cancellation was requested #5267
Conversation
| // They will still run on background but we won't await them. | ||
| var runCancelled = new TaskCompletionSource<object>(); | ||
| _testRunCancellationToken?.Register(() => runCancelled.TrySetCanceled()); | ||
| await Task.WhenAny(Task.WhenAll(tasks), runCancelled.Task); |
There was a problem hiding this comment.
Cancellation token _testRunCancellationToken is not a CancellationToken, so we cannot use Task.WhenAll(tasks, ct) here.
| string fileCreationPath = Path.Combine(testHost.DirectoryName, "fileCreation"); | ||
| File.WriteAllText(fileCreationPath, string.Empty); |
There was a problem hiding this comment.
This was unblocking the loop below, so as soon as we started the app we cancelled tests before any test ever had a chance to run.
So this test was testing almost nothing.
|
|
||
| testHostResult.AssertOutputMatchesRegex("Canceling the test session.*"); |
There was a problem hiding this comment.
Now we abort so fast that this does not have a chance to write to screen.
| await ExecuteTestsWithTestRunnerAsync(nonParallelizableTestSet, frameworkHandle, source, sourceLevelParameters, testRunner, usesAppDomains); | ||
| } | ||
| } | ||
| else | ||
| { | ||
| // What to do? |
There was a problem hiding this comment.
Here and few lines above we run the tests when parallelization is not enabled or test is marked with do not parallelize, this blocks the cancellation until the test finishes (and fails the non-parallel tests), and advice here how elegantly solve this? (it won't block if we have parallelism with 1 worker, because that takes the other parallel path.
There was a problem hiding this comment.
@nohwnd How does it block cancellation? ExecuteTestsWithTestRunnerAsync calls ThrowIfCancellationRequested
There was a problem hiding this comment.
Yes but only before it starts running the test. if we cancel while the test is running we don't have a way to abandon it to background and continue to test run teardown code. Like writing test summary to screen. We still have 1 tests in progress and wait for it to complete.
There was a problem hiding this comment.
You can still do the same Task.WhenAny trick right?
| // This method does a lot of work synchronously and cannot be easily cancelled. Make sure we run asynchronously and | ||
| // can abandon the work here (by calling Task.WhenAny higher in the call stack) when user cancels the test run. | ||
| await Task.Yield(); |
There was a problem hiding this comment.
We are already calling ThrowIfCancellationRequested just before we call this method. I'm failing to understand why we need Task.Yeild here
There was a problem hiding this comment.
ThrowIfCancellationRequested will throw when cancellation was requested before we see that line of code. It has no impact on code that is already running.
So if we cancel before entering this method, all is good and we will cancel the process and will be able to leave the Main method and teardown the process.
But if we enter this method, it will run synchronously, and we are forced to await its completion. So as long as there is a test running (e.g. test that is hanging) we won't be able to leave that stack and won't be able to complete the Main method. The Task.WhenAny that we called this method from will not be effective because the method is running synchronously.
This Yield makes sure that this method runs asynchronously, so we can avoid awaiting its completion.
Another approach would be to merge how tests are run when parallel is enabled and when parallel is disabled. When parallel is enabled (even if there is single worker allowed = serial execution) we offload the work to a Task in a queue, which is what this does as well.
There was a problem hiding this comment.
But I think I am just implementing a partially graceful cancellation here, because we need cooperation from the code that runs the tests (not the test themselves, but the code that runs them), so NUnit would also have to go and make sure they cancel in a way that allows them to finish the process.
Instead I should maybe lift this approach higher, and yield in place where we start the tests in platform. So platform ran re-gain control on ctrl+c, without any cooperation with the test framework.
| @@ -487,16 +490,18 @@ private async Task ExecuteTestsInSourceAsync(IEnumerable<TestCase> tests, IRunCo | |||
| // Queue the non parallel set | |||
| if (nonParallelizableTestSet != null) | |||
| { | |||
| _testRunCancellationToken?.ThrowIfCancellationRequested(); | |||
There was a problem hiding this comment.
nit: As you have the same line in the if/else, consider moving it once before the if
| await Task.WhenAll(tasks); | ||
| // Ensure we can abandon currently running tasks on cancellation, rather than waiting for them to complete. | ||
| // They will still run on background but we won't await them. | ||
| await Task.WhenAny(Task.WhenAll(tasks), _testRunCancellationToken.AsTask()); |
There was a problem hiding this comment.
I would make the Task.WhenAny(task, cancellationToken.AsTask()) part as the helper instead. Also, could we - if not already done - get some blessing from runtime? I saw multiple different suggestions for this online.
| _testRunCancellationToken?.ThrowIfCancellationRequested(); | ||
| // Ensure we can abandon currently running tasks on cancellation, rather than waiting for them to complete. | ||
| // They will still run on background but we won't await them. | ||
| await Task.WhenAny(ExecuteTestsWithTestRunnerAsync(nonParallelizableTestSet, frameworkHandle, source, sourceLevelParameters, testRunner, usesAppDomains), completeWhenTestRunIsCanceled); |
There was a problem hiding this comment.
The more I think of it, the more I think it's probably not the right thing. Cancellation will be requested, then we just proceed while the task of ExecuteTestsWithTestRunnerAsync could still be running.
There was a problem hiding this comment.
I agree...you don't know what the dangling tasks will do and when...at this point it's better Env.Exit(abort code) the result of the run cannot be trusted and some other crash could happen for race for unexpected app state after this return the behavior will be always undefined.
Or you cooperate or you don't cooperate or user should decide what to do --disable-cooperative-cancellation and Env.Exit(abort code), by default cooperative and wait.
Ensure that we can complete the run when non-graceful cancellation is requested (e.b. ctrl+C on commandline), by making sure that the test runner does not wait for all currently running tests to finish.