[8.0] [mono] Test process gets killed #90834

carlossanlop · 2023-08-18T22:08:07Z

Error Blob

{
  "ErrorMessage": "Tests' END OF WORK ITEM LOG: Command timed out, and was killed]",
  "BuildRetry": false,
  "ErrorPattern": "",
  "ExcludeConsoleLog": false
}

Reproduction Steps

This failure happens often in System.Runtime.Tests specifically.

PR: [release/8.0] HTTP & SSL Stress: Run nightlies against release/8.0 #90809
Build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=378797&view=logs&j=1f44e919-17d8-5098-631c-31ef11f9e57b&t=d617d884-da52-5b82-bd49-f7613b408af4&l=87
Queue: Libraries Test Run release mono linux x64 Debug
Log: https://helixre107v0xd1eu3ibi6ka.blob.core.windows.net/dotnet-runtime-refs-pull-90809-merge-956e379e01714658b2/System.Runtime.Tests/1/console.73e1394a.log?helixlogtype=result
Output:

Console log: 'System.Runtime.Tests' from job 956e379e-0171-4658-b2cd-b234086ec9af workitem 50980a74-2595-4c8f-b4a1-bbcc0991bb7b (ubuntu.1804.amd64.open.svc) executed on machine a005SZ1 running Linux-5.4.0-1109-azure-x86_64-with-Ubuntu-18.04-bionic
+ ./RunTests.sh --runtime-path /datadisks/disk1/work/9F780922/p
----- start Fri Aug 18 17:28:49 UTC 2023 =============== To repro directly: =====================================================
pushd .
/datadisks/disk1/work/9F780922/p/dotnet exec --runtimeconfig System.Runtime.Tests.runtimeconfig.json --depsfile System.Runtime.Tests.deps.json xunit.console.dll System.Runtime.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=AdditionalTimezoneChecks -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing 
popd
===========================================================================================================
/datadisks/disk1/work/9F780922/w/A2490973/e /datadisks/disk1/work/9F780922/w/A2490973/e
  Discovering: System.Runtime.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Runtime.Tests (found 9160 of 9278 test cases)
  Starting:    System.Runtime.Tests (parallel test collections = on, max threads = 2)
    System.Tests.ArrayEnumeratorTests_int.ICollection_Generic_Remove_ReferenceRemovedFromCollection [SKIP]
      Condition(s) not met: "IsPreciseGcSupported"
['System.Runtime.Tests' END OF WORK ITEM LOG: Command timed out, and was killed]

Known issue validation

Build: 🔎
Result validation: ⚠️ Validation could not be done without an Azure DevOps build URL on the issue. Please add it to the "Build: 🔎" line.
Validation performed at: 8/29/2023 9:18:10 PM UTC

Report

Build	Definition	Test	Pull Request
390358	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#91314
390315	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#91239
390327	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
390312	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#91240
390304	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#91311
390232	dotnet/runtime	System.Globalization.Tests.WorkItemExecution	#90970
390131	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#91303
389716	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#91293
389636	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#91290
389641	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#91291
389595	dotnet/runtime	System.Net.NetworkInformation.Functional.Tests.WorkItemExecution
389529	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#91283
389491	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
389478	dotnet/runtime	Microsoft.Extensions.Configuration.Functional.Tests.WorkItemExecution	#91239
389287	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#91267
389320	dotnet/runtime	System.Text.RegularExpressions.Tests.WorkItemExecution	#91216
389280	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#91264
389142	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#91262
388978	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
388942	dotnet/runtime	normal-System.Threading.Tasks.Tests.WorkItemExecution
388902	dotnet/runtime	Microsoft.Extensions.HostFactoryResolver.Tests.WorkItemExecution	#91176
388878	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#91240
388706	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#91245
388797	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#91237
388672	dotnet/runtime	System.Net.Sockets.Tests.WorkItemExecution	#91247
388667	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#91246
388556	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#89952
388379	dotnet/runtime	System.IO.Pipes.Tests.WorkItemExecution	#91231
388449	dotnet/runtime	System.Net.Http.Unit.Tests.WorkItemExecution
388406	dotnet/runtime	System.Security.Cryptography.OpenSsl.Tests.WorkItemExecution	#91190
388166	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#91218
388145	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#91154
387880	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#91172
387528	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#91189
387355	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#89952
387323	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#90982
387294	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#91171
387117	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#91160
387116	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
387106	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
386969	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#91154
386929	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#90982
386516	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
386357	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
386352	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#91133
386233	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#91074
386241	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#91107
386139	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#91123
386115	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#91122
386015	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#91116
385733	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#90518
385671	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#91102
385446	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#89405
385484	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
385364	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
385005	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#89102
384990	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#91077
384688	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#91057
384597	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#91050
384450	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#91033
384447	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#91032
384038	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#90790
383772	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#88034
382759	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#90954
378797	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#90809

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
18	64	65

The text was updated successfully, but these errors were encountered:

ghost · 2023-08-18T22:08:16Z

Tagging subscribers to this area: @dotnet/area-system-runtime
See info in area-owners.md if you want to be subscribed.

Issue Details

Error Blob

{
  "ErrorMessage": "System.Runtime.Tests' END OF WORK ITEM LOG: Command timed out, and was killed",
  "BuildRetry": false,
  "ErrorPattern": "",
  "ExcludeConsoleLog": true
}

Reproduction Steps

This failure happens often in System.Runtime.Tests specifically.

PR: [release/8.0] HTTP & SSL Stress: Run nightlies against release/8.0 #90809
Build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=378797&view=logs&j=1f44e919-17d8-5098-631c-31ef11f9e57b&t=d617d884-da52-5b82-bd49-f7613b408af4&l=87
Queue: Libraries Test Run release mono linux x64 Debug
Log: https://helixre107v0xd1eu3ibi6ka.blob.core.windows.net/dotnet-runtime-refs-pull-90809-merge-956e379e01714658b2/System.Runtime.Tests/1/console.73e1394a.log?helixlogtype=result
Output:

Console log: 'System.Runtime.Tests' from job 956e379e-0171-4658-b2cd-b234086ec9af workitem 50980a74-2595-4c8f-b4a1-bbcc0991bb7b (ubuntu.1804.amd64.open.svc) executed on machine a005SZ1 running Linux-5.4.0-1109-azure-x86_64-with-Ubuntu-18.04-bionic
+ ./RunTests.sh --runtime-path /datadisks/disk1/work/9F780922/p
----- start Fri Aug 18 17:28:49 UTC 2023 =============== To repro directly: =====================================================
pushd .
/datadisks/disk1/work/9F780922/p/dotnet exec --runtimeconfig System.Runtime.Tests.runtimeconfig.json --depsfile System.Runtime.Tests.deps.json xunit.console.dll System.Runtime.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=AdditionalTimezoneChecks -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing 
popd
===========================================================================================================
/datadisks/disk1/work/9F780922/w/A2490973/e /datadisks/disk1/work/9F780922/w/A2490973/e
  Discovering: System.Runtime.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Runtime.Tests (found 9160 of 9278 test cases)
  Starting:    System.Runtime.Tests (parallel test collections = on, max threads = 2)
    System.Tests.ArrayEnumeratorTests_int.ICollection_Generic_Remove_ReferenceRemovedFromCollection [SKIP]
      Condition(s) not met: "IsPreciseGcSupported"
['System.Runtime.Tests' END OF WORK ITEM LOG: Command timed out, and was killed]

Author:	carlossanlop
Assignees:	-
Labels:	`area-System.Runtime`, `os-linux`, `arch-x64`, `blocking-clean-ci`, `untriaged`, `runtime-mono`, `Known Build Error`
Milestone:	-

carlossanlop · 2023-08-24T15:34:45Z

Here's another 8.0 repro. This time in osx, but also mono:

Affected PR: [release/8.0] Update dependencies from dotnet/roslyn-analyzers #91057
Queue: Libraries Test Run release mono osx x64 Debug
Build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=384688&view=logs&j=ab5d32de-6d19-5068-e03b-b69cf4db6cd5&t=eb2a8345-733e-5823-7aa4-11fb75e4a1ff&l=55
Log: https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-91057-merge-18f6ab2e6a76443fb6/System.Runtime.Tests/1/console.dbd0bad1.log?helixlogtype=result
Output:

...
    System.Tests.WeakReferenceTests.NonGeneric [SKIP]
      Condition(s) not met: "IsPreciseGcSupported"
    System.Tests.ArrayEnumeratorTests_int.ICollection_Generic_Remove_ReferenceRemovedFromCollection [SKIP]
      Condition(s) not met: "IsPreciseGcSupported"
['System.Runtime.Tests' END OF WORK ITEM LOG: Command timed out, and was killed]

akoeplinger · 2023-08-25T09:42:34Z

I was able to reproduce the deadlock and debug it together with @BrzVlad, he's going to open a PR.

ghost · 2023-08-25T13:33:01Z

Tagging subscribers to this area: @BrzVlad
See info in area-owners.md if you want to be subscribed.

Issue Details

Error Blob

{
  "ErrorMessage": "['System.Runtime.Tests' END OF WORK ITEM LOG: Command timed out, and was killed]",
  "BuildRetry": false,
  "ErrorPattern": "",
  "ExcludeConsoleLog": false
}

Reproduction Steps

This failure happens often in System.Runtime.Tests specifically.

PR: [release/8.0] HTTP & SSL Stress: Run nightlies against release/8.0 #90809
Build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=378797&view=logs&j=1f44e919-17d8-5098-631c-31ef11f9e57b&t=d617d884-da52-5b82-bd49-f7613b408af4&l=87
Queue: Libraries Test Run release mono linux x64 Debug
Log: https://helixre107v0xd1eu3ibi6ka.blob.core.windows.net/dotnet-runtime-refs-pull-90809-merge-956e379e01714658b2/System.Runtime.Tests/1/console.73e1394a.log?helixlogtype=result
Output:

Console log: 'System.Runtime.Tests' from job 956e379e-0171-4658-b2cd-b234086ec9af workitem 50980a74-2595-4c8f-b4a1-bbcc0991bb7b (ubuntu.1804.amd64.open.svc) executed on machine a005SZ1 running Linux-5.4.0-1109-azure-x86_64-with-Ubuntu-18.04-bionic
+ ./RunTests.sh --runtime-path /datadisks/disk1/work/9F780922/p
----- start Fri Aug 18 17:28:49 UTC 2023 =============== To repro directly: =====================================================
pushd .
/datadisks/disk1/work/9F780922/p/dotnet exec --runtimeconfig System.Runtime.Tests.runtimeconfig.json --depsfile System.Runtime.Tests.deps.json xunit.console.dll System.Runtime.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=AdditionalTimezoneChecks -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing 
popd
===========================================================================================================
/datadisks/disk1/work/9F780922/w/A2490973/e /datadisks/disk1/work/9F780922/w/A2490973/e
  Discovering: System.Runtime.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Runtime.Tests (found 9160 of 9278 test cases)
  Starting:    System.Runtime.Tests (parallel test collections = on, max threads = 2)
    System.Tests.ArrayEnumeratorTests_int.ICollection_Generic_Remove_ReferenceRemovedFromCollection [SKIP]
      Condition(s) not met: "IsPreciseGcSupported"
['System.Runtime.Tests' END OF WORK ITEM LOG: Command timed out, and was killed]

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=378797
Error message validated: ['System.Runtime.Tests' END OF WORK ITEM LOG: Command timed out, and was killed]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 8/24/2023 3:32:03 PM UTC

Report

Build	Definition	Test	Pull Request
385733	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#90518
385671	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#91102
385446	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#89405
385484	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
385364	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
385005	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#89102
384990	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#91077
384688	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#91057
384597	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#91050
384450	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#91033
384447	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#91032
384038	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#90790
383772	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#88034
382759	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#90954
378797	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#90809

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
8	15	15

Author:	carlossanlop
Assignees:	BrzVlad
Labels:	`os-linux`, `os-mac-os-x`, `arch-x64`, `blocking-clean-ci`, `area-GC-mono`, `runtime-mono`, `Known Build Error`
Milestone:	8.0.0

carlossanlop · 2023-08-25T14:23:09Z

I was able to reproduce the deadlock and debug it together with @BrzVlad, he's going to open a PR.

Thank you very much!

carlossanlop mentioned this issue Aug 22, 2023

[release/8.0] Update Fedora docker image to Fedora 38 #90930

Merged

carlossanlop changed the title ~~System.Runtime.Tests times out and gets killed~~ [mono] System.Runtime.Tests times out and gets killed Aug 24, 2023

carlossanlop added area-VM-meta-mono and removed area-System.Runtime labels Aug 24, 2023

carlossanlop mentioned this issue Aug 24, 2023

[release/8.0] Update dependencies from dotnet/roslyn #91033

Merged

carlossanlop added this to the 8.0.0 milestone Aug 24, 2023

ghost removed the untriaged New issue has not been triaged by the area owner label Aug 24, 2023

carlossanlop changed the title ~~[mono] System.Runtime.Tests times out and gets killed~~ [8.0] [mono] System.Runtime.Tests times out and gets killed Aug 24, 2023

elinor-fung mentioned this issue Aug 24, 2023

Use RuntimeIdentifierGraphPath if available in runtime copy of Microsoft.NET.CrossGen.targets #91077

Merged

akoeplinger added the os-mac-os-x macOS aka OSX label Aug 25, 2023

akoeplinger assigned BrzVlad Aug 25, 2023

SamMonoRT added area-GC-mono and removed area-VM-meta-mono labels Aug 25, 2023

BrzVlad mentioned this issue Aug 28, 2023

[mono][metadata] Replace use of mem manager lock with loader lock #91190

Merged

ghost added the in-pr There is an active PR which will close this issue when it is merged label Aug 28, 2023

carlossanlop changed the title ~~[8.0] [mono] System.Runtime.Tests times out and gets killed~~ [8.0] [mono] Test process gets killed Aug 29, 2023

carlossanlop mentioned this issue Aug 29, 2023

[release/8.0] Add boolean field to indicate whether or not the Windows thread pool is being used #91246

Merged

BrzVlad closed this as completed in #91190 Aug 30, 2023

ghost removed the in-pr There is an active PR which will close this issue when it is merged label Aug 30, 2023

jeffhandley mentioned this issue Aug 30, 2023

[release/8.0] Support Windows based ECDSA SignedCms #91247

Merged

ghost locked as resolved and limited conversation to collaborators Sep 29, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[8.0] [mono] Test process gets killed #90834

[8.0] [mono] Test process gets killed #90834

carlossanlop commented Aug 18, 2023 •

edited by build-analysis bot

Loading

ghost commented Aug 18, 2023

Error Blob

Reproduction Steps

carlossanlop commented Aug 24, 2023

akoeplinger commented Aug 25, 2023

ghost commented Aug 25, 2023

Error Blob

Reproduction Steps

Known issue validation

Report

Summary

carlossanlop commented Aug 25, 2023

[8.0] [mono] Test process gets killed #90834

[8.0] [mono] Test process gets killed #90834

Comments

carlossanlop commented Aug 18, 2023 • edited by build-analysis bot Loading

Error Blob

Reproduction Steps

Known issue validation

Report

Summary

ghost commented Aug 18, 2023

Error Blob

Reproduction Steps

carlossanlop commented Aug 24, 2023

akoeplinger commented Aug 25, 2023

ghost commented Aug 25, 2023

Error Blob

Reproduction Steps

Known issue validation

Report

Summary

carlossanlop commented Aug 25, 2023

carlossanlop commented Aug 18, 2023 •

edited by build-analysis bot

Loading