Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unblock run when cancellation was requested #5267

Draft
wants to merge 6 commits into
base: main
Choose a base branch
from
+88 −11
Draft

Unblock run when cancellation was requested #5267

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
024c3d8
Unblock run when cancellation was requested but there are still runni…
nohwnd Mar 19, 2025
0e7051c
Fix test to test cancellation correctly
nohwnd Mar 19, 2025
a4afd44
what to do ?
nohwnd Mar 19, 2025
c422b39
Will continue later
nohwnd Mar 19, 2025
77355ac
Yield so we can abandon
nohwnd Mar 20, 2025
3ed9e8f
Yield so we can abandon
nohwnd Mar 20, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 16 additions & 3 deletions src/Adapter/MSTest.TestAdapter/Execution/TestExecutionManager.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -413,6 +413,7 @@ private async Task ExecuteTestsInSourceAsync(IEnumerable<TestCase> tests, IRunCo
parallelScope = MSTestSettings.CurrentSettings.ParallelizationScope.Value;
}

Task completeWhenTestRunIsCanceled = _testRunCancellationToken.AsTask();
if (!MSTestSettings.CurrentSettings.DisableParallelization && sourceSettings.CanParallelizeAssembly && parallelWorkers > 0)
{
// Parallelization is enabled. Let's do further classification for sets.
Expand Down Expand Up @@ -471,7 +472,9 @@ private async Task ExecuteTestsInSourceAsync(IEnumerable<TestCase> tests, IRunCo

try
{
await Task.WhenAll(tasks);
// Ensure we can abandon currently running tasks on cancellation, rather than waiting for them to complete.
// They will still run on background but we won't await them.
await Task.WhenAny(Task.WhenAll(tasks), completeWhenTestRunIsCanceled);
}
catch (Exception ex)
{
Expand All @@ -485,12 +488,18 @@ private async Task ExecuteTestsInSourceAsync(IEnumerable<TestCase> tests, IRunCo
// Queue the non parallel set
if (nonParallelizableTestSet != null)
{
await ExecuteTestsWithTestRunnerAsync(nonParallelizableTestSet, frameworkHandle, source, sourceLevelParameters, testRunner, usesAppDomains);
_testRunCancellationToken?.ThrowIfCancellationRequested();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: As you have the same line in the if/else, consider moving it once before the if

// Ensure we can abandon currently running tasks on cancellation, rather than waiting for them to complete.
// They will still run on background but we won't await them.
await Task.WhenAny(ExecuteTestsWithTestRunnerAsync(nonParallelizableTestSet, frameworkHandle, source, sourceLevelParameters, testRunner, usesAppDomains), completeWhenTestRunIsCanceled);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The more I think of it, the more I think it's probably not the right thing. Cancellation will be requested, then we just proceed while the task of ExecuteTestsWithTestRunnerAsync could still be running.

Copy link
Contributor

@MarcoRossignoli MarcoRossignoli Mar 22, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree...you don't know what the dangling tasks will do and when...at this point it's better Env.Exit(abort code) the result of the run cannot be trusted and some other crash could happen for race for unexpected app state after this return the behavior will be always undefined.

Or you cooperate or you don't cooperate or user should decide what to do --disable-cooperative-cancellation and Env.Exit(abort code), by default cooperative and wait.

}
}
else
{
await ExecuteTestsWithTestRunnerAsync(testsToRun, frameworkHandle, source, sourceLevelParameters, testRunner, usesAppDomains);
_testRunCancellationToken?.ThrowIfCancellationRequested();
// Ensure we can abandon currently running tasks on cancellation, rather than waiting for them to complete.
// They will still run on background but we won't await them.
await Task.WhenAny(ExecuteTestsWithTestRunnerAsync(testsToRun, frameworkHandle, source, sourceLevelParameters, testRunner, usesAppDomains), completeWhenTestRunIsCanceled);
}

if (PlatformServiceProvider.Instance.IsGracefulStopRequested)
Expand All @@ -509,6 +518,10 @@ private async Task ExecuteTestsWithTestRunnerAsync(
UnitTestRunner testRunner,
bool usesAppDomains)
{
// This method does a lot of work synchronously and cannot be easily cancelled. Make sure we run asynchronously and
// can abandon the work here (by calling Task.WhenAny higher in the call stack) when user cancels the test run.
await Task.Yield();
Comment on lines +521 to +523
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are already calling ThrowIfCancellationRequested just before we call this method. I'm failing to understand why we need Task.Yeild here

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ThrowIfCancellationRequested will throw when cancellation was requested before we see that line of code. It has no impact on code that is already running.

So if we cancel before entering this method, all is good and we will cancel the process and will be able to leave the Main method and teardown the process.

But if we enter this method, it will run synchronously, and we are forced to await its completion. So as long as there is a test running (e.g. test that is hanging) we won't be able to leave that stack and won't be able to complete the Main method. The Task.WhenAny that we called this method from will not be effective because the method is running synchronously.

This Yield makes sure that this method runs asynchronously, so we can avoid awaiting its completion.

Another approach would be to merge how tests are run when parallel is enabled and when parallel is disabled. When parallel is enabled (even if there is single worker allowed = serial execution) we offload the work to a Task in a queue, which is what this does as well.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But I think I am just implementing a partially graceful cancellation here, because we need cooperation from the code that runs the tests (not the test themselves, but the code that runs them), so NUnit would also have to go and make sure they cancel in a way that allows them to finish the process.

Instead I should maybe lift this approach higher, and yield in place where we start the tests in platform. So platform ran re-gain control on ctrl+c, without any cooperation with the test framework.


bool hasAnyRunnableTests = false;
var fixtureTests = new List<TestCase>();

Expand Down
21 changes: 21 additions & 0 deletions src/Adapter/MSTest.TestAdapter/Execution/TestRunCancellationTokenExtensions.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
// Copyright (c) Microsoft Corporation. All rights reserved.
// Licensed under the MIT license. See LICENSE file in the project root for full license information.

namespace Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter;

internal static class TestRunCancellationTokenExtensions
{
/// <summary>
/// Wraps the cancellation token into a task that will complete when cancellation is requested, so we can combine it with Task.WhenAny, to abandon other tasks in the background.
/// </summary>
// Returns Task but is not doing any async work, and we should not await getting the Task.
#pragma warning disable VSTHRD200 // Use "Async" suffix for async methods
public static Task AsTask(this TestRunCancellationToken? testRunCancellationToken)
#pragma warning restore VSTHRD200 // Use "Async" suffix for async methods
{
var cancellationSource = new TaskCompletionSource<object>();
testRunCancellationToken?.Register(() => cancellationSource.TrySetCanceled());

return cancellationSource.Task;
}
}
59 changes: 51 additions & 8 deletions test/IntegrationTests/MSTest.Acceptance.IntegrationTests/AbortionTests.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,15 @@ public sealed class AbortionTests : AcceptanceTestBase<AbortionTests.TestAssetFi

[TestMethod]
[DynamicData(nameof(TargetFrameworks.AllForDynamicData), typeof(TargetFrameworks))]
public async Task AbortWithCTRLPlusC_CancellingTests(string tfm)
public async Task AbortWithCTRLPlusC_CancellingParallelTests(string tfm)
=> await AbortWithCTRLPlusC_CancellingTests(tfm, parallelize: true);

[TestMethod]
[DynamicData(nameof(TargetFrameworks.AllForDynamicData), typeof(TargetFrameworks))]
public async Task AbortWithCTRLPlusC_CancellingNonParallelTests(string tfm)
=> await AbortWithCTRLPlusC_CancellingTests(tfm, parallelize: false);

internal async Task AbortWithCTRLPlusC_CancellingTests(string tfm, bool parallelize)
{
// We expect the same semantic for Linux, the test setup is not cross and we're using specific
// Windows API because this gesture is not easy xplat.
Expand All @@ -25,17 +33,45 @@ public async Task AbortWithCTRLPlusC_CancellingTests(string tfm)

var testHost = TestHost.LocateFrom(AssetFixture.TargetAssetPath, AssetName, tfm);

string? parameters = null;
if (parallelize)
{
// Providing runSettings even with Parallelize Workers = 1, will "enable" parallelization and will run via different path.
// So providing the settings only to the parallel run.
string runSettingsPath = Path.Combine(testHost.DirectoryName, $"{(parallelize ? "parallel" : "serial")}.runsettings");
File.WriteAllText(runSettingsPath, $"""
<RunSettings>
<MSTest>
<Parallelize>
<Workers>{(parallelize ? 0 : 1)}</Workers>
<Scope>MethodLevel</Scope>
</Parallelize>
</MSTest>
</RunSettings>
""");
parameters = $"--settings {runSettingsPath}";
}

string fileCreationPath = Path.Combine(testHost.DirectoryName, "fileCreation");
File.WriteAllText(fileCreationPath, string.Empty);
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was unblocking the loop below, so as soon as we started the app we cancelled tests before any test ever had a chance to run.

So this test was testing almost nothing.


TestHostResult testHostResult = await testHost.ExecuteAsync(environmentVariables: new()
TestHostResult testHostResult = await testHost.ExecuteAsync(parameters, environmentVariables: new()
{
["FILE_DIRECTORY"] = fileCreationPath,
});

testHostResult.AssertExitCodeIs(ExitCodes.TestSessionAborted);
// To ensure we don't cancel right away, so tests have chance to run, and block our
// cancellation if we do it wrong.
testHostResult.AssertOutputContains("Waiting for file creation.");
if (parallelize)
{
testHostResult.AssertOutputContains("Test Parallelization enabled for");
}
else
{
testHostResult.AssertOutputDoesNotContain("Test Parallelization enabled for");
}

testHostResult.AssertOutputMatchesRegex("Canceling the test session.*");
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now we abort so fast that this does not have a chance to write to screen.

testHostResult.AssertExitCodeIs(ExitCodes.TestSessionAborted);
}

public sealed class TestAssetFixture() : TestAssetFixtureBase(AcceptanceFixture.NuGetGlobalPackagesFolder)
Expand Down Expand Up @@ -92,6 +128,7 @@ public static async Task<int> Main(string[] args)
}
else
{
Console.WriteLine("Waiting for file creation.");
Thread.Sleep(1000);
}
}
Expand Down Expand Up @@ -131,12 +168,18 @@ public async Task TestA()
var fireCtrlCTask = Task.Run(() =>
{
// Delay for a short period before firing CTRL+C to simulate some processing time
Task.Delay(1000).Wait();
File.WriteAllText(Environment.GetEnvironmentVariable("FILE_DIRECTORY")!, string.Empty);
});

// Start a task that represents the infinite delay, which should be canceled
await Task.Delay(Timeout.Infinite, TestContext.CancellationTokenSource.Token);
// Wait for 10s, and after that kill the process.
// When we cancel by CRTL+C we do non-graceful teardown so the Environment.Exit should never be reached,
// because the test process already terminated.
//
// If we do reach it, we will see 11111 exit code, and it will fail the test assertion, because we did not cancel.
// (If we don't exit here, the process will happily run to completion after 10 seconds, but will still report
// cancelled exit code, so that is why we are more aggressive here.)
await Task.Delay(10_000);
Environment.Exit(11111);
}
}
""";
Expand Down
Loading