slow macOS - "##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes."

[jeffschwMSFT]

Build

https://dnceng.visualstudio.com/internal/_build/results?buildId=2360768&view=results

Error

##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes. For more information, see https://go.microsoft.com/fwlink/?linkid=2077134

Build leg reported

vsos

Pull Request

No response

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "",
    "BuildRetry": false,
    "ErrorPattern": "The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng/internal/_build/results?buildId=2360768
Error message validated: [The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 2/7/2024 1:03:13 AM UTC

Report

Build	Definition	Step Name	Console log	Pull Request
2955346	dotnet-dotnet-monitor	Test MacOS x64 Release	Log
1386426	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#127124
1386269	dotnet/maui	Controls CollectionView	Log
1386224	dotnet/maui	Controls CollectionView	Log
1386274	dotnet/maui	Controls CollectionView	Log
1385986	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#124967
2954890	dotnet-sdk	AoT: macOS (x64)	Log	#60250
2954889	dotnet-sdk	AoT: macOS (x64)	Log	#60249
2954896	dotnet-sdk	Darwin	Log	#60251
2954897	dotnet-sdk	Darwin	Log	#60252
1385823	dotnet/maui	Controls CollectionView	Log
2954810	dotnet-maui	RunOniOS_BlazorDebug ARM64	Log
2954811	dotnet-maui	Controls Shell	Log
1385862	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_CoreCLR_RuntimeTests	Log
2954805	dotnet-sdk	AoT: macOS (x64)	Log	#60243
2954804	dotnet-sdk	TestBuild: macOS (x64)	Log	#60242
1385739	dotnet/aspnetcore	Tests: macOS	Log
2954737	dotnet-dotnet-monitor	Test MacOS x64 Release	Log
2954691	dotnet-sdk	TestBuild: macOS (x64)	Log	#60242
1385666	dotnet/maui	Controls CollectionView	Log	dotnet/maui#34844
1385664	dotnet/maui	Controls Accessibility,ActionSheet,ActivityIndicator,Animation,AppLinks	Log	dotnet/maui#31790
2954696	dotnet-dotnet-monitor	Test MacOS x64 Release	Log
2954692	dotnet-sdk	TestBuild: macOS (x64)	Log	#60243
1385617	dotnet/aspnetcore	Tests: macOS	Log
2954407	dotnet-maui	Controls Shell	Log
2954470	dotnet-dotnet-monitor	Test MacOS x64 Release	Log
1385324	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono_RuntimeTests	Log
2954406	dotnet-maui	Blazor macOS	Log
1385278	dotnet/maui	Controls CollectionView	Log
1385275	dotnet/aspnetcore	Tests: macOS	Log	dotnet/aspnetcore#66135
2954400	dotnet-runtime	ios-arm64 Release AllSubsets_NativeAOT_Smoke	Log
2954390	dotnet-runtime	tvossimulator-arm64 release NativeAOT	Log
1385254	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono	Log
2954326	dotnet-sdk	TestBuild: macOS (x64)	Log	#60216
2954319	dotnet-maui	Controls Layout	Log
1385176	dotnet/maui	Controls Shell	Log	dotnet/maui#35028
2954332	dotnet-sdk	Darwin	Log	#60218
2954331	dotnet-sdk	Darwin	Log	#60217
2954327	dotnet-sdk	AoT: macOS (x64)	Log	#59942
1384827	dotnet/maui	Controls Layout	Log	dotnet/maui#34980
1384804	dotnet/maui	Controls CollectionView	Log	dotnet/maui#35001
1385177	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono_RuntimeTests	Log
1384193	dotnet/maui	Controls CollectionView	Log	dotnet/maui#34923
1384155	dotnet/maui	Controls Shell	Log	dotnet/maui#31159
1384850	dotnet/maui	Controls Layout	Log	dotnet/maui#34573
2954059	dotnet-maui	Controls CollectionView	Log
1384769	dotnet/maui	Controls ViewBaseTests,VisualStateManager,Window	Log
1384766	dotnet/maui	Controls CollectionView	Log
1384000	dotnet/maui	Controls Layout	Log	dotnet/maui#34758
1384729	dotnet/runtime	osx-x64 Debug PALTests	Log
1384687	dotnet/runtime	osx-x64 release Runtime_Release	Log	dotnet/runtime#127068
2954096	dotnet-dotnet-monitor	Test MacOS x64 Release	Log
1384675	dotnet/aspnetcore	Tests: macOS	Log
1384547	dotnet/runtime	osx-x64 release Runtime_Release	Log	dotnet/runtime#127089
1384541	dotnet/runtime	osx-x64 release Runtime_Release	Log	dotnet/runtime#127090
1383732	dotnet/runtime	osx-x64 Release Libraries_Release_CoreCLR	Log	dotnet/runtime#127055
1383888	dotnet/maui	Controls Layout	Log
1384448	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#125129
2953788	dotnet-maui	Controls CollectionView	Log
1384115	dotnet/maui	Controls Shell	Log	dotnet/maui#35016
1384069	dotnet/maui	Controls Layout	Log
1383651	dotnet/maui	Controls CollectionView	Log	dotnet/maui#34637
1383960	dotnet/maui	Controls Layout	Log	dotnet/maui#34804
1384035	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono_RuntimeTests	Log
2953787	dotnet-maui	RunOniOS_BlazorRelease x64	Log
1384002	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_CoreCLR_RuntimeTests	Log	dotnet/runtime#127081
2953793	dotnet-maui	RunOniOS_MauiRelease x64	Log
2953794	dotnet-maui	Controls ViewBaseTests,VisualStateManager,Window	Log
1383561	dotnet/maui	Controls Shell	Log
1383922	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#127022
1383543	dotnet/maui	Controls Shell	Log
1383927	dotnet/aspnetcore	Tests: macOS	Log
1383653	dotnet/runtime	Libraries Test Run release coreclr osx arm64 Debug	Log	dotnet/runtime#126547
1383879	dotnet/aspnetcore	Tests: macOS	Log	dotnet/aspnetcore#62759
1383820	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#126245
1383415	dotnet/maui	Controls Shell	Log	dotnet/maui#34996
1383767	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_CoreCLR_RuntimeTests	Log	dotnet/runtime#127059
2953633	dotnet-maui	Controls Layout	Log
1383764	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#127059
1383756	dotnet/runtime	osx-arm64 Debug CoreCLR_Libraries	Log	dotnet/runtime#127068
1383730	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_CoreCLR_RuntimeTests	Log	dotnet/runtime#127054
1382435	dotnet/maui	Controls Layout	Log	dotnet/maui#34710
1383717	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#127054
1383735	dotnet/aspnetcore	Tests: macOS	Log	dotnet/aspnetcore#66007
1383697	dotnet/runtime	osx-x64 Release NativeAOT	Log	dotnet/runtime#125945
1383684	dotnet/sdk	AoT: macOS (x64)	Log	dotnet/sdk#53955
1383679	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#53950
1383624	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#127063
2953632	dotnet-maui	RunOniOS_BlazorDebug_CoreCLR ARM64	Log
1383617	dotnet/runtime	osx-x64 release Runtime_Release	Log	dotnet/runtime#127060
2953653	dotnet-efcore	macOS	Log
1383615	dotnet/runtime	osx-arm64 Debug CoreCLR_Libraries	Log	dotnet/runtime#127060
1383582	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#125491
1383461	dotnet/maui	Controls Shell	Log	dotnet/maui#35001
1383439	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#127050
2953538	dotnet-efcore	macOS	Log
2953480	dotnet-dotnet-monitor	Test MacOS x64 Release	Log
1383360	dotnet/runtime	osx-x64 release Runtime_Release	Log	dotnet/runtime#125129
2953434	dotnet-dotnet-monitor	Test MacOS x64 Release	Log
1382887	dotnet/maui	Controls CollectionView	Log
Displaying 100 of 1296 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
8	355	1296

[lewing]

🤔

[nagilson]

@dotnet/dnceng @dougbu This has impacted a lot of PRs recently, may you PTAL into expanding this Mac resource?

[ivanpovazan]

@dotnet/dnceng we are hitting this again and it seems there is some issue with communication with Helix machines.

More context on timeouts happening in : https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=results

• On success: Run tests in Helix step reports:

Waiting for completion of job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open (Details: https://helix.dot.net/api/jobs/fbec1164-55c5-4fa8-b492-e1ba1b413119/details?api-version=2019-06-17 )
Job 7fdc2720-3d98-4bf3-8113-274dacd69c91 on osx.1200.arm64.open is completed with 6 finished work items.
  Job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open is completed with 6 finished work items.
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.amd64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.arm64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )

Build succeeded.

SENDHELIXJOB : warning : Helix queue osx.1200.amd64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
SENDHELIXJOB : warning : Helix queue osx.1200.arm64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
    2 Warning(s)
    0 Error(s)

Time Elapsed 00:03:14.97
Killing running build processes...

Finishing: Run tests in Helix

ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=logs&j=ccc97bb6-1a23-5e71-fdfa-3cdca4a74749&t=27fc7eb2-ead9-59e1-6679-a637855d40c5

• While on failure - timeout the same step gets stuck with:

Waiting for completion of job 967d92a2-ec10-4332-927f-d28a6563f367 on osx.1200.arm64.open (Details: https://helix.dot.net/api/jobs/967d92a2-ec10-4332-927f-d28a6563f367/details?api-version=2019-06-17 )
  Job 5427159b-500d-49f1-aac0-ec148a492bbe on osx.1200.amd64.open is completed with 6 finished work items.

ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=logs&s=c58bc33c-b825-5bca-90ca-50f6e9293dd8&j=e6966639-fe40-5068-d9ae-681cccecafdf

NOTE: All the tests successfully passed on Helix, but it seems that the communication is lost.

[garath]

Looks like the timeout happened because the osx.1200.arm64.open queue was very busy while the job was running. Right now, I do not think there are any problems with the infrastructure. I will investigate a bit more to see what caused such a back-up.

[garath]

Ah, the queue was consumed with updates and patching. The patching jobs did run longer than necessary and we've communicated with our partner team about the issue. Future jobs will be much shorter and should not overly impact jobs.

[ivanpovazan]

Looks like the timeout happened because the osx.1200.arm64.open queue was very busy while the job was running. Right now, I do not think there are any problems with the infrastructure. I will investigate a bit more to see what caused such a back-up.

We are still experiencing the problem.

Should we try to change the queue to osx.13.arm64?

@dotnet/dnceng

[ilyas1974]

Looks at the failing builds, I'm noticing the jobs that are timing out in the hosted pool (Azure Pipelines) are still using the older hardware. I would recommend moving the workloads to the mac-latest-internal or mac-14-arm64 agent specifications. These have the latest mac hardware associated with them.