The Operation will be canceled. The next steps may not contain expected logs.

[tarekgh]

Build

https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=696281

Build leg reported

Build / browser-wasm linux Release LibraryTests / Send to Helix

Pull Request

dotnet/runtime#103009

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "The Operation will be canceled. The next steps may not contain expected logs.",
    "BuildRetry": false,
    "ErrorPattern": "",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=696281
Error message validated: [The Operation will be canceled. The next steps may not contain expected logs.]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 6/4/2024 8:08:48 PM UTC

Report

Build	Definition	Step Name	Console log	Pull Request
2673622	dotnet-roslyn	Run Unit Tests	Log
996727	dotnet/runtime	Send to Helix	Log	dotnet/runtime#113995
2673603	dotnet-vscode-dotnet-runtime	🔎 Test Mac and Linux	Log
995013	dotnet/aspnetcore	Run build.cmd helix target	Log	dotnet/aspnetcore#61126
996607	dotnet/performance	Send job to Helix (Windows)	Log
996508	dotnet/aspire	Build	Log
996411	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#113988
996373	dotnet/runtime	Send to Helix	Log	dotnet/runtime#113460
996376	dotnet/aspnetcore	Run build.cmd helix target	Log	dotnet/aspnetcore#61193
996125	dotnet/aspnetcore	Run build.cmd helix target	Log	dotnet/aspnetcore#61191
996277	dotnet/runtime	AOT compile CoreCLR tests	Log
996279	dotnet/runtime	Send to Helix	Log
2673338	dotnet-runtime	Send job to Helix (Unix)	Log
996039	dotnet/roslyn	Run Integration Tests	Log	dotnet/roslyn#76906
2673300	dotnet-dotnet	Checkout dotnet-dotnet@attempt-different-method to s	Log
2673279	dotnet-dotnet	Checkout dotnet-dotnet@attempt-different-method to s	Log
2673210	dotnet-runtime	Build product	Log
2673134	dotnet-runtime	Send job to Helix (Unix)	Log
2673106	dotnet-performance	Send job to Helix (Unix)	Log
995727	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#47942
995552	dotnet/runtime	AOT compile CoreCLR tests	Log
995537	dotnet/runtime	Send to Helix	Log
2672952	dotnet-runtime	Send job to Helix (Unix)	Log
995472	dotnet/aspire	Run helix tests	Log
995406	dotnet/roslyn	Run Integration Tests	Log	dotnet/roslyn#76906
2672790	dotnet-runtime	Send job to Helix (Unix)	Log
995227	dotnet/sdk	Prepare dotnet/razor for formatting	Log	dotnet/sdk#47926
2672603	dotnet-dotnet	🔒 Publish Artifacts (On Success)	Log
2672751	dotnet-aspnetcore	Checkout dotnet-aspnetcore@do-not-sign-using-wix-targets to s	Log
995173	dotnet/roslyn	Test	Log
995168	dotnet/aspire	Run helix tests	Log
2672713	dotnet-workload-versions	🛡 Guardian: AntiMalware Scanner (Source)	Log
995094	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#47869
995086	dotnet/sdk	🟣 Run AoT Tests	Log	dotnet/sdk#47744
2672658	dotnet-dotnet	🛡 Guardian: CredScan (Source)	Log
995080	dotnet/sdk	🟣 Run AoT Tests	Log	dotnet/sdk#47873
995056	dotnet/roslyn	Test	Log
995064	dotnet/sdk	🟣 Run AoT Tests	Log	dotnet/sdk#47924
995050	dotnet/sdk	🟣 Run AoT Tests	Log	dotnet/sdk#47923
995041	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#47888
995040	dotnet/sdk	🟣 Run AoT Tests	Log	dotnet/sdk#47847
995004	dotnet/sdk	🟣 Run AoT Tests	Log	dotnet/sdk#47880
994974	dotnet/runtime	AOT compile CoreCLR tests	Log
995003	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#47879
994979	dotnet/roslyn	Test	Log
994865	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#113937
994860	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#113906
994870	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#113938
2672591	dotnet-performance	Agent Pool Validation	Log
2672562	dotnet-runtime	Send job to Helix (Unix)	Log
2672531	dotnet-runtime	Send job to Helix (Unix)	Log
994818	dotnet/performance	Run performance job script	Log
2672519	dotnet-dotnet-monitor	Validate Hosted Pool Information (1ES PT)	Log
2672520	dotnet-dotnet-monitor	Initialize containers	Log
994528	dotnet/roslyn	Test	Log	dotnet/roslyn#77840
994506	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#47873
994501	dotnet/sdk	Run AoT Tests	Log	dotnet/sdk#47913
994372	dotnet/sdk	🟣 Run AoT Tests	Log	dotnet/sdk#47767
994462	dotnet/roslyn	Test	Log
994458	dotnet/roslyn	Test	Log
994444	dotnet/roslyn	Test	Log
2672352	dotnet-runtime	Send job to Helix (Windows)	Log
994402	dotnet/roslyn	Test	Log
994394	dotnet/sdk	🟣 Run AoT Tests	Log	dotnet/sdk#47911
994392	dotnet/sdk	🟣 Run AoT Tests	Log	dotnet/sdk#47910
2672330	dotnet-performance	Run performance job script	Log
994374	dotnet/sdk	🟣 Run AoT Tests	Log	dotnet/sdk#47869
994360	dotnet/roslyn	Test	Log	dotnet/roslyn#77835
994369	dotnet/xharness	Run tests in Helix	Log
994351	dotnet/sdk	🟣 Run AoT Tests	Log	dotnet/sdk#47879
994353	dotnet/sdk	🟣 Run AoT Tests	Log	dotnet/sdk#47880
994337	dotnet/sdk	🟣 Run AoT Tests	Log	dotnet/sdk#47908
994339	dotnet/sdk	🟣 Run AoT Tests	Log	dotnet/sdk#47909
994279	dotnet/sdk	🟣 Run AoT Tests	Log	dotnet/sdk#47907
994274	dotnet/sdk	🟣 Run AoT Tests	Log	dotnet/sdk#47833
994148	dotnet/sdk	🟣 Run AoT Tests	Log	dotnet/sdk#47702
994244	dotnet/sdk	🟣 Run AoT Tests	Log	dotnet/sdk#47874
994242	dotnet/xharness	Run tests in Helix	Log	dotnet/xharness#1368
994233	dotnet/xharness	Run tests in Helix	Log	dotnet/xharness#1384
994199	dotnet/roslyn	Test	Log
994196	dotnet/roslyn	Test	Log
994050	dotnet/sdk	Run AoT Tests	Log	dotnet/sdk#47902
994053	dotnet/sdk	Run AoT Tests	Log	dotnet/sdk#47901
994014	dotnet/sdk	🟣 Run AoT Tests	Log	dotnet/sdk#47893
994023	dotnet/sdk	🟣 Run AoT Tests	Log	dotnet/sdk#47894
994025	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#47895
994047	dotnet/sdk	Run AoT Tests	Log	dotnet/sdk#47899
994008	dotnet/sdk	🟣 Run AoT Tests	Log	dotnet/sdk#47892
994041	dotnet/sdk	🟣 Run AoT Tests	Log	dotnet/sdk#47897
994026	dotnet/sdk	🟣 Run AoT Tests	Log	dotnet/sdk#47896
994156	dotnet/runtime	AOT compile CoreCLR tests	Log
993355	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#47877
993456	dotnet/sdk	🟣 Run AoT Tests	Log	dotnet/sdk#47879
994000	dotnet/sdk	Run AoT Tests	Log	dotnet/sdk#47891
994180	dotnet/sdk	🟣 Run AoT Tests	Log	dotnet/sdk#47888
994178	dotnet/sdk	🟣 Run AoT Tests	Log	dotnet/sdk#47847
994049	dotnet/sdk	Run AoT Tests	Log	dotnet/sdk#47900
2672169	dotnet-runtime	Send job to Helix (Unix)	Log
994129	dotnet/roslyn	Test	Log
993837	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#47869
Displaying 100 of 990 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
21	287	888

[dkurepa]

These appear to be timeout cancellations, and most of them are happening on the same runtime leg that uses OSX.1200.ARM64.Open, looking at grafana, they do indeed appear to be quite high. Will check with the team on what we can do about this

[dkurepa]

For pipelines experiencing timeouts on osx.1200.arm64.open we recommend using osx.14.arm64.open, as this queue has a lot more machines

[dkurepa]

There is also a number of Publish Using Darc timeouts, these are in 99% of cases AzDo Api issues. During the Publish Using Darc job, we queue up a Maestro Promotion pipeline, that downloads the artifacts from the build calling it, and publishes them where they need to go. While publishing is happening, the build pipeline keeps busy waiting for the promotion pipeline to be done, checking the status of it every 60 seconds. In cases where this times out, the build most likely did get promoted, the AzDo API just kept telling us the wrong thing

[ivanpovazan]

@dkurepa for context, the same timeouts are also caught with this tracking #1883
where I asked about potentially bumping the queues to osx.13.amd/arm64, would that suffice?
(bumping to osx.14 is not an option right now as we still want testing on osx.13 which is the minimum supported OS version)