The Operation will be canceled. The next steps may not contain expected logs.

[tarekgh]

Build

https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=696281

Build leg reported

Build / browser-wasm linux Release LibraryTests / Send to Helix

Pull Request

dotnet/runtime#103009

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "The Operation will be canceled. The next steps may not contain expected logs.",
    "BuildRetry": false,
    "ErrorPattern": "",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=696281
Error message validated: [The Operation will be canceled. The next steps may not contain expected logs.]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 6/4/2024 8:08:48 PM UTC

Report

Build	Definition	Step Name	Console log	Pull Request
2820655	dotnet-dotnet-buildtools-prereqs-docker	🟪 Build Images	Log
1180939	dotnet/runtime	Send to Helix	Log
2821076	dotnet-runtime	Build product	Log
2821033	dotnet-runtime	Build product	Log
1180601	dotnet/sdk	🟣 Run FullFramework Tests	Log	dotnet/sdk#51355
2820872	dotnet-installer	Build (alpine322)	Log	#54526
2820894	dotnet-sdk	Generate delegation SAS Token for dotnetbuilds/internal	Log	#54527
2820880	dotnet-sdk	🛡 Guardian: 1ES Secret Scanning on Linux (SPMI)	Log
2820896	dotnet-sdk	Checkout dotnet-sdk@refs/pull/54528/merge to s	Log	#54528
1180693	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#120926
2820855	dotnet-sdk	Run TestBuild Tests	Log	#54515
2820844	dotnet-runtime	Publish AOT Mono Artifacts	Log
1180676	dotnet/sdk	Run TestBuild Tests	Log	dotnet/sdk#51382
1180611	dotnet/runtime	Send job to Helix (Unix)	Log
2820789	dotnet-installer	Build	Log	#54520
2820729	dotnet-roslyn	Run Unit Tests	Log
2820719	dotnet-dotnet	Build (centOSStreamContainer)	Log
1180229	dotnet/runtime	Send to Helix	Log
2820698	dotnet-aspire	🟣Build native packages	Log
1180382	dotnet/roslyn	Run Unit Tests	Log
2820647	dotnet-aspnetcore	Build Installers	Log
2820668	dotnet-wcf	Windows Build / Publish	Log
2820458	dotnet-aspnetcore	Build ARM64	Log
2820519	dotnet-dotnet	🔒 Publish Vertical Manifest	Log
1180297	dotnet/runtime	Build product	Log
2820568	dotnet-sdk	🟣 Build	Log
2820515	dotnet-arcade-validation	Component Governance scan	Log
2820476	dotnet-sdk	Run TestBuild Tests	Log	#54515
2820521	dotnet-aspire	🟣Build native packages	Log
2820439	dotnet-dotnet	dotnet --version	Log
2820460	dotnet-dotnet	Prep the Build (centOSStreamContainer)	Log
2820374	dotnet-dotnet	Build (centOSStreamContainer)	Log
2820133	dotnet-docker-tools	🟪 Copy Base Images	Log
2820037	dotnet-installer	Build	Log	#54510
1179574	dotnet/sdk	🟣 Run FullFramework Tests	Log	dotnet/sdk#51207
1179506	dotnet/runtime	Send to Helix	Log
1179456	dotnet/runtime	AOT compile CoreCLR tests	Log
2819746	dotnet-runtime	Send tests to Helix (Unix)	Log
1179318	dotnet/runtime	Send tests to Helix (Unix)	Log
2819649	dotnet-performance	Run performance job script	Log
1179335	dotnet/runtime	Send to Helix	Log
2819529	dotnet-sdk	Run AoT Tests	Log	#54383
1179115	dotnet/roslyn	Test	Log	dotnet/roslyn#80794
2819249	dotnet-runtime	Send tests to Helix (Unix)	Log
1178966	dotnet/runtime	Send to Helix	Log
2819151	dotnet-runtime	Send to Helix	Log
1178790	dotnet/sdk	🟣 Run FullFramework Tests	Log	dotnet/sdk#51307
2818961	dotnet-dotnet	Build (centOSStreamContainer)	Log
2818877	dotnet-sdk	Run AoT Tests	Log	#54417
2818915	dotnet-razor	🛡 Guardian: AntiMalware Scanner (Binary)	Log
1178788	dotnet/runtime	Send job to Helix (Unix)	Log
2818893	dotnet-roslyn	🛡 Component Governance	Log
2818939	dotnet-aspire	🟣Build native packages	Log
1178764	dotnet/roslyn	Run Unit Tests	Log
2818839	dotnet-dotnet	Build	Log
1178615	dotnet/runtime	Build product	Log	dotnet/runtime#119432
2818909	dotnet-arcade-validation	PowerShell@2	Log
2818895	dotnet-interactive-window	Build and Test	Log
2818864	dotnet-installer	Build	Log	#54399
1178614	dotnet/runtime	Build product	Log	dotnet/runtime#119432
2818855	dotnet-sdk	Run Tests in Helix	Log	#54377
2818854	dotnet-sdk	Run Tests in Helix	Log	#54379
2818856	dotnet-sdk	Build	Log	#54376
1178554	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#120853
2818840	dotnet-sdk	🛡 Guardian: 1ES Secret Scanning (SPMI)	Log	#54376
2818842	dotnet-sdk	Agent Pool Validation	Log	#54379
2818841	dotnet-sdk	Download secrets: HelixProdKV	Log	#54377
2818832	dotnet-sdk	Build	Log
2818829	dotnet-installer	Build	Log	#54494
2818743	dotnet-dotnet	Build (centOSStreamContainer)	Log
2818773	dotnet-runtime	Run performance job script	Log
1178493	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#50749
1178488	dotnet/runtime	Send job to Helix (Unix)	Log
2818746	dotnet-aspnetcore	Run build.sh	Log
2818733	dotnet-install-scripts	Run Tests	Log
2818712	dotnet-aspnetcore	Run build.cmd helix target	Log	#54424
1178410	dotnet/aspnetcore	Run build.cmd	Log
1178298	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#120281
1177919	dotnet/runtime	Send job to Helix (Unix)	Log
1178390	dotnet/sdk	🟣 Run FullFramework Tests	Log	dotnet/sdk#51341
1178318	dotnet/runtime	AOT compile CoreCLR tests	Log
2818648	dotnet-installer	Validate MacOS Pool & Image (1ES PT)	Log	#54434
1178307	dotnet/runtime	Send job to Helix (Unix)	Log
2818612	dotnet-sdk	Run Tests in Helix	Log	#54379
2818611	dotnet-sdk	Run Tests in Helix	Log	#54376
2818613	dotnet-efcore	Send job to helix	Log	#54472
2818614	dotnet-sdk	Run Tests in Helix	Log	#54377
1177647	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#120623
2818541	dotnet-dotnet	Build (centOSStreamContainer)	Log
1177851	dotnet/runtime	Send to Helix	Log	dotnet/runtime#120812
1177820	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#120787
1177741	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#120810
1177779	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#120411
1177823	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#120812
1177918	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#51331
1177945	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#120694
1177930	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#120516
1177925	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#118270
1177938	dotnet/runtime	Send to Helix (Unix)	Log	dotnet/runtime#120281
1177920	dotnet/runtime	Send tests to Helix (Unix)	Log
Displaying 100 of 1213 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
32	346	1213

[dkurepa]

These appear to be timeout cancellations, and most of them are happening on the same runtime leg that uses OSX.1200.ARM64.Open, looking at grafana, they do indeed appear to be quite high. Will check with the team on what we can do about this

[dkurepa]

For pipelines experiencing timeouts on osx.1200.arm64.open we recommend using osx.14.arm64.open, as this queue has a lot more machines

[dkurepa]

There is also a number of Publish Using Darc timeouts, these are in 99% of cases AzDo Api issues. During the Publish Using Darc job, we queue up a Maestro Promotion pipeline, that downloads the artifacts from the build calling it, and publishes them where they need to go. While publishing is happening, the build pipeline keeps busy waiting for the promotion pipeline to be done, checking the status of it every 60 seconds. In cases where this times out, the build most likely did get promoted, the AzDo API just kept telling us the wrong thing

[ivanpovazan]

@dkurepa for context, the same timeouts are also caught with this tracking #1883
where I asked about potentially bumping the queues to osx.13.amd/arm64, would that suffice?
(bumping to osx.14 is not an option right now as we still want testing on osx.13 which is the minimum supported OS version)

[CarnaViire]

For the "known issues" folks: the timeout here is reported twice, making BA red despite the tracking issue. E.g.:

1️⃣ dotnet/runtime#117255

2️⃣ dotnet/runtime#117800

cc @AlitzelMendez