The Operation will be canceled. The next steps may not contain expected logs.

[tarekgh]

Build

https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=696281

Build leg reported

Build / browser-wasm linux Release LibraryTests / Send to Helix

Pull Request

dotnet/runtime#103009

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "The Operation will be canceled. The next steps may not contain expected logs.",
    "BuildRetry": false,
    "ErrorPattern": "",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=696281
Error message validated: [The Operation will be canceled. The next steps may not contain expected logs.]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 6/4/2024 8:08:48 PM UTC

Report

Build	Definition	Step Name	Console log	Pull Request
1406001	dotnet/maui	DeviceTestsMacCatalyst_CoreCLR (Windows)	Log
1405962	dotnet/maui	Controls WebView	Log
1405901	dotnet/runtime	Send to Helix	Log	dotnet/runtime#127676
1405922	dotnet/aspnetcore	Run Quarantined Tests	Log
2965484	dotnet-roslyn	Test	Log
2965452	dotnet-aspnetcore	Run build.cmd helix target	Log	#60694
2965471	dotnet-extensions	Agent Pool Validation	Log
2965442	dotnet-aspnetcore	Run build.cmd helix target	Log
2965456	dotnet-aspnetcore	Checkout dotnet-aspnetcore@refs/pull/60696/merge to s	Log	#60696
1405865	dotnet/maui	Controls WebView	Log
2965439	dotnet-performance	Send job to Helix (Unix)	Log
2965402	dotnet-roslyn	Test	Log
2965390	dotnet-runtime	Send job to Helix (Unix)	Log
1405692	dotnet/runtime	Publish	Log	dotnet/runtime#123920
1405703	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#127613
2965369	dotnet-efcore	Send tests to helix	Log
2965351	dotnet-efcore	Run Cosmos tests	Log
2965335	dotnet-roslyn	Test	Log
2965341	dotnet-efcore	Send job to helix	Log
2965230	dotnet-dotnet	Cache Binary Analysis Scan Optimization (1ES PT)	Log
1405528	dotnet/runtime	Publish Build Assets	Log	dotnet/runtime#127613
2965306	dotnet-runtime	Send job to Helix (Unix)	Log
2965305	dotnet-runtime	Send job to Helix (Unix)	Log
2965292	dotnet-roslyn	Test	Log
1405593	dotnet/aspnetcore	Run Quarantined Tests	Log
2965294	dotnet-roslyn	Test	Log
2965304	dotnet-dotnet	🛡 Guardian: AntiMalware Scanner (Source)	Log
2965280	dotnet-runtime	Install workload using artifacts	Log
1405414	dotnet/runtime	Publish Build Assets	Log	dotnet/runtime#126911
1405486	dotnet/sdk	🟣 Run dotnet-format on dotnet/msbuild repo folder	Log
1405477	dotnet/roslyn	Run Integration Tests	Log	dotnet/roslyn#81758
1405478	dotnet/sdk	🟣 Run dotnet-format on dotnet/sdk repo folder	Log
1405315	dotnet/runtime	Zip Build Assets	Log	dotnet/runtime#127649
2965203	dotnet-dotnet	🛡 Guardian: AntiMalware Scanner (Source)	Log
1405194	dotnet/maui	Controls WebView	Log
2965144	dotnet-roslyn	Test	Log
2965036	dotnet-maui	Controls (vlatest) Shell	Log
2965147	dotnet-dotnet	🛡 Guardian: AntiMalware Scanner (Source)	Log
1405168	dotnet/maui	Controls CollectionView	Log
1405200	dotnet/runtime	Zip Build Assets	Log	dotnet/runtime#127642
2965035	dotnet-maui	Run Integration Tests - RunOniOS_MauiDebug	Log
2965055	dotnet-aspnetcore	Run build.cmd helix target	Log
1405085	dotnet/maui	Controls WebView	Log
1405179	dotnet/aspnetcore	Run Quarantined Tests	Log
2965047	dotnet-performance	Send job to Helix (Windows)	Log
1405169	dotnet/maui	DeviceTestsMacCatalyst_CoreCLR (Windows)	Log
1405163	dotnet/runtime	Build product	Log	dotnet/runtime#127596
1405091	dotnet/runtime	Run Native AOT Test App Tests	Log	dotnet/runtime#127555
1404759	dotnet/runtime	Send tests to Helix (Unix)	Log
2964983	dotnet-dotnet	Build (centOSStreamContainer)	Log
2964973	dotnet-efcore	Send tests to helix	Log
1404998	dotnet/runtime	Send to Helix	Log	dotnet/runtime#123252
2964925	dotnet-roslyn	Test	Log
2964896	dotnet-roslyn	Test	Log
1404983	dotnet/runtime	Build product	Log	dotnet/runtime#127604
1404816	dotnet/runtime	Zip Build Assets	Log	dotnet/runtime#127635
2964886	dotnet-roslyn	Test	Log
1404669	dotnet/maui	Controls WebView	Log
2964843	dotnet-docker-tools	🟪 Clean public/dotnet/* (pruneEol > 15d)	Log
2964810	dotnet-dotnet	Build (centOSStreamContainer)	Log
2964822	dotnet-roslyn	Test	Log
1404795	dotnet/aspnetcore	Run Quarantined Tests	Log
1404798	dotnet/runtime	Build product	Log	dotnet/runtime#127300
2964807	dotnet-runtime	Send job to Helix (Unix)	Log
2964811	dotnet-workload-versions	🟣 Build solution	Log
1404555	dotnet/maui	Controls CollectionView	Log	dotnet/maui#35268
2964802	dotnet-workload-versions	🟣 Build solution	Log
2964797	dotnet-efcore	Run Cosmos tests	Log
2964795	dotnet-dotnet-monitor	Build	Log
1404611	dotnet/runtime	Build native test components	Log	dotnet/runtime#127621
2964796	dotnet-workload-versions	🟣 Build solution	Log
1404672	dotnet/maui	DeviceTestsMacCatalyst_CoreCLR (Windows)	Log
2964768	dotnet-workload-versions	🛡 Guardian: TSAUpload	Log
2964595	dotnet-maui	Controls WebView	Log
2964747	dotnet-workload-versions	🟣 Build solution	Log
2964737	dotnet-roslyn	Test	Log
1404542	dotnet/runtime	Zip Build Assets	Log	dotnet/runtime#126367
2964735	dotnet-workload-versions	🟣 Build solution	Log
2964719	dotnet-workload-versions	🛡 Generate SBoM Manifest	Log
1404501	dotnet/runtime	Run Native AOT Test App Tests	Log	dotnet/runtime#127629
2964702	dotnet-roslyn	Test	Log
2964718	dotnet-roslyn	Test	Log
1404384	dotnet/maui	Controls WebView	Log
1404475	dotnet/runtime	Build native test components	Log	dotnet/runtime#127482
2964684	dotnet-aspnetcore	Run build.cmd helix target	Log
1404450	dotnet/runtime	Zip	Log	dotnet/runtime#127578
1404470	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#127613
1404557	dotnet/roslyn	Run Integration Tests	Log	dotnet/roslyn#81758
1404553	dotnet/maui	DeviceTestsMacCatalyst_CoreCLR (Windows)	Log	dotnet/maui#35268
1404157	dotnet/runtime	Send to Helix	Log
2964653	dotnet-efcore	Send tests to helix	Log
2964617	dotnet-roslyn	Test	Log
2964638	dotnet-efcore	Send job to helix	Log
2964419	dotnet-maui	Controls Layout	Log
1404303	dotnet/runtime	Publish Build Assets	Log	dotnet/runtime#127536
1404391	dotnet/aspnetcore	Run Quarantined Tests	Log
1404310	dotnet/runtime	Publish Build Assets	Log
1404252	dotnet/runtime	Publish Build Assets	Log	dotnet/runtime#127395
1404386	dotnet/maui	DeviceTestsMacCatalyst_CoreCLR (Windows)	Log	dotnet/maui#35268
1404155	dotnet/runtime	Send to Helix	Log
Displaying 100 of 2940 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
57	849	2940

[dkurepa]

These appear to be timeout cancellations, and most of them are happening on the same runtime leg that uses OSX.1200.ARM64.Open, looking at grafana, they do indeed appear to be quite high. Will check with the team on what we can do about this

[dkurepa]

For pipelines experiencing timeouts on osx.1200.arm64.open we recommend using osx.14.arm64.open, as this queue has a lot more machines

[dkurepa]

There is also a number of Publish Using Darc timeouts, these are in 99% of cases AzDo Api issues. During the Publish Using Darc job, we queue up a Maestro Promotion pipeline, that downloads the artifacts from the build calling it, and publishes them where they need to go. While publishing is happening, the build pipeline keeps busy waiting for the promotion pipeline to be done, checking the status of it every 60 seconds. In cases where this times out, the build most likely did get promoted, the AzDo API just kept telling us the wrong thing

[ivanpovazan]

@dkurepa for context, the same timeouts are also caught with this tracking #1883
where I asked about potentially bumping the queues to osx.13.amd/arm64, would that suffice?
(bumping to osx.14 is not an option right now as we still want testing on osx.13 which is the minimum supported OS version)

[CarnaViire]

For the "known issues" folks: the timeout here is reported twice, making BA red despite the tracking issue. E.g.:

1️⃣ dotnet/runtime#117255

2️⃣ dotnet/runtime#117800

cc @AlitzelMendez