The Operation will be canceled. The next steps may not contain expected logs.

[tarekgh]

Build

https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=696281

Build leg reported

Build / browser-wasm linux Release LibraryTests / Send to Helix

Pull Request

dotnet/runtime#103009

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "The Operation will be canceled. The next steps may not contain expected logs.",
    "BuildRetry": false,
    "ErrorPattern": "",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=696281
Error message validated: [The Operation will be canceled. The next steps may not contain expected logs.]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 6/4/2024 8:08:48 PM UTC

Report

Build	Definition	Step Name	Console log	Pull Request
2842481	dotnet-docker-tools	🟪 Generate EOL Data for all images in msdotnetcontainersstaging.azurecr.io	Log
1207203	dotnet/runtime	Send to Helix	Log	dotnet/runtime#121767
1207126	dotnet/runtime	Send to Helix	Log	dotnet/runtime#121366
1207108	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#121767
1207100	dotnet/runtime	Send tests to Helix (Windows)	Log
2841580	dotnet-dotnet	Build	Log
1207052	dotnet/runtime	Send to Helix	Log	dotnet/runtime#119432
1207060	dotnet/runtime	Send to Helix	Log	dotnet/runtime#121646
1207096	dotnet/arcade	Run Helix Tests	Log
2842341	dotnet-aspnetcore	Run build.cmd	Log
1206878	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#50864
2842031	dotnet-dotnet	Build	Log
1206835	dotnet/runtime	Send job to Helix (Unix)	Log	dotnet/runtime#121732
1205930	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#51784
2842064	dotnet-dotnet	Build	Log
2842034	dotnet-dotnet-buildtools-prereqs-docker	🟪 Copy Base Images	Log
1206747	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#51765
2841818	dotnet-runtime	Send job to Helix (Unix)	Log
2841799	dotnet-performance	Send job to Helix (Unix)	Log
2841784	dotnet-maui	Run Integration Tests - Build macOS	Log
1206453	dotnet/runtime	Checkout dotnet/runtime@refs/pull/121739/merge to s	Log	dotnet/runtime#121739
1206452	dotnet/runtime	Checkout dotnet/runtime@refs/pull/121739/merge to s	Log	dotnet/runtime#121739
1206454	dotnet/runtime	Build product	Log	dotnet/runtime#121739
1206447	dotnet/runtime	Install Build Dependencies	Log	dotnet/runtime#121739
1206449	dotnet/runtime	Install Build Dependencies	Log	dotnet/runtime#121739
1206440	dotnet/runtime	Build product	Log	dotnet/runtime#121739
1206442	dotnet/runtime	Build product	Log	dotnet/runtime#121739
1206441	dotnet/runtime	Build	Log	dotnet/runtime#121739
1206402	dotnet/sdk	Checkout dotnet/sdk@refs/pull/51791/merge to s	Log	dotnet/sdk#51791
1206385	dotnet/sdk	🟣 Install wasm-tools Workload	Log	dotnet/sdk#51790
1206377	dotnet/sdk	🟣 Run AoT Tests	Log	dotnet/sdk#51791
1206389	dotnet/arcade-services	Run unit tests	Log	dotnet/arcade-services#5493
1206376	dotnet/arcade-services	Run unit tests	Log	dotnet/arcade-services#5493
1206319	dotnet/runtime	Send to Helix	Log
1206363	dotnet/aspnetcore	Run build.cmd helix target	Log	dotnet/aspnetcore#64034
1206317	dotnet/runtime	Send tests to Helix (Windows)	Log
1206320	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#120688
1206341	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#51411
1206372	dotnet/runtime	Checkout dotnet/runtime@refs/pull/121734/merge to s	Log	dotnet/runtime#121734
1206371	dotnet/runtime	Checkout dotnet/runtime@refs/pull/121734/merge to s	Log	dotnet/runtime#121734
1206370	dotnet/runtime	Checkout dotnet/runtime@refs/pull/121734/merge to s	Log	dotnet/runtime#121734
1206351	dotnet/roslyn	Restore	Log	dotnet/roslyn#81281
1206357	dotnet/aspnetcore	Run build.sh	Log	dotnet/aspnetcore#64034
1206350	dotnet/roslyn	Build	Log	dotnet/roslyn#81281
1206359	dotnet/aspnetcore	Run restore.sh	Log	dotnet/aspnetcore#64034
1206358	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/64034/merge to s	Log	dotnet/aspnetcore#64034
1206298	dotnet/runtime	Send to Helix (Unix)	Log	dotnet/runtime#115335
1206346	dotnet/roslyn	Build with analyzers	Log	dotnet/roslyn#81284
1206337	dotnet/roslyn	Build file-based apps	Log	dotnet/roslyn#81281
1206336	dotnet/roslyn	Build	Log	dotnet/roslyn#81281
1206309	dotnet/runtime	Send to Helix	Log	dotnet/runtime#121491
1206292	dotnet/runtime	Send to Helix	Log
1206297	dotnet/aspire	Build	Log	dotnet/aspire#12981
1206281	dotnet/runtime	Send to Helix (Windows)	Log	dotnet/runtime#121732
1206201	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#51565
2841406	dotnet-performance	Send job to Helix (Unix)	Log
1206262	dotnet/runtime	Send to Helix	Log	dotnet/runtime#100729
1206252	dotnet/runtime	Send to Helix	Log	dotnet/runtime#121730
1206269	dotnet/aspnetcore	Run build.cmd helix target	Log	dotnet/aspnetcore#64416
2841620	dotnet-runtime	Send job to Helix (Windows)	Log
1206237	dotnet/runtime	Send to Helix	Log	dotnet/runtime#121316
2841570	dotnet-docker-tools	🟪 Generate EOL Data for all images in msdotnetcontainersstaging.azurecr.io	Log
1206223	dotnet/aspire	Build	Log	dotnet/aspire#12609
2838960	dotnet-performance	Send job to Helix (Windows)	Log
2841490	dotnet-runtime	Send job to Helix (Unix)	Log
1206159	dotnet/runtime	Send to Helix	Log	dotnet/runtime#121725
1206154	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#121724
1206163	dotnet/aspnetcore	Run build.cmd helix target	Log	dotnet/aspnetcore#64413
1206119	dotnet/runtime	Send tests to Helix (Unix)	Log
2841341	dotnet-diagnostics	Generate SBOM manifest	Log
1206108	dotnet/runtime	Send to Helix	Log	dotnet/runtime#121643
1206094	dotnet/runtime	Send to Helix	Log	dotnet/runtime#121720
2839775	dotnet-performance	Send job to Helix (Windows)	Log
1206134	dotnet/aspire	Build	Log	dotnet/aspire#12829
2841371	dotnet-runtime	Send job to Helix (Windows)	Log
2841370	dotnet-runtime	Send job to Helix (Unix)	Log
1206011	dotnet/aspnetcore	Run build.cmd helix target	Log	dotnet/aspnetcore#64408
2839008	dotnet-runtime	Send job to Helix (Unix)	Log
2839298	dotnet-runtime	Send job to Helix (Unix)	Log
2839297	dotnet-runtime	Send job to Helix (Windows)	Log
1206117	dotnet/aspire	Build	Log	dotnet/aspire#12506
2840367	dotnet-runtime	Send job to Helix (Windows)	Log
2839725	dotnet-runtime	Send job to Helix (Windows)	Log
2840366	dotnet-runtime	Send job to Helix (Windows)	Log
2840970	dotnet-runtime	Send job to Helix (Windows)	Log
2840972	dotnet-runtime	Send job to Helix (Windows)	Log
1205992	dotnet/runtime	Send tests to Helix (Unix)	Log
1206100	dotnet/aspire	Build	Log	dotnet/aspire#12506
2841057	dotnet-runtime	Send job to Helix (Windows)	Log
1206070	dotnet/aspnetcore	Run build.cmd helix target	Log
2840311	dotnet-performance	Send job to Helix (Unix)	Log
2841392	dotnet-aspnetcore	Run build.cmd helix target	Log
1206050	dotnet/runtime	Build Tests	Log	dotnet/runtime#121720
1205995	dotnet/runtime	Send to Helix	Log
1205998	dotnet/runtime	Send to Helix	Log
1206066	dotnet/performance	Send job to Helix (Unix)	Log
1206052	dotnet/runtime	Build product	Log	dotnet/runtime#121720
1206054	dotnet/runtime	Build product	Log	dotnet/runtime#121720
1206018	dotnet/roslyn	Download Test Payload	Log	dotnet/roslyn#81307
1206019	dotnet/roslyn	Run Integration Tests	Log	dotnet/roslyn#81307
Displaying 100 of 2454 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
56	974	2454

[dkurepa]

These appear to be timeout cancellations, and most of them are happening on the same runtime leg that uses OSX.1200.ARM64.Open, looking at grafana, they do indeed appear to be quite high. Will check with the team on what we can do about this

[dkurepa]

For pipelines experiencing timeouts on osx.1200.arm64.open we recommend using osx.14.arm64.open, as this queue has a lot more machines

[dkurepa]

There is also a number of Publish Using Darc timeouts, these are in 99% of cases AzDo Api issues. During the Publish Using Darc job, we queue up a Maestro Promotion pipeline, that downloads the artifacts from the build calling it, and publishes them where they need to go. While publishing is happening, the build pipeline keeps busy waiting for the promotion pipeline to be done, checking the status of it every 60 seconds. In cases where this times out, the build most likely did get promoted, the AzDo API just kept telling us the wrong thing

[ivanpovazan]

@dkurepa for context, the same timeouts are also caught with this tracking #1883
where I asked about potentially bumping the queues to osx.13.amd/arm64, would that suffice?
(bumping to osx.14 is not an option right now as we still want testing on osx.13 which is the minimum supported OS version)

[CarnaViire]

For the "known issues" folks: the timeout here is reported twice, making BA red despite the tracking issue. E.g.:

1️⃣ dotnet/runtime#117255

2️⃣ dotnet/runtime#117800

cc @AlitzelMendez