slow macOS - "##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes."

[jeffschwMSFT]

Build

https://dnceng.visualstudio.com/internal/_build/results?buildId=2360768&view=results

Error

##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes. For more information, see https://go.microsoft.com/fwlink/?linkid=2077134

Build leg reported

vsos

Pull Request

No response

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "",
    "BuildRetry": false,
    "ErrorPattern": "The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng/internal/_build/results?buildId=2360768
Error message validated: [The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 2/7/2024 1:03:13 AM UTC

Report

Build	Definition	Step Name	Console log	Pull Request
1375208	dotnet/aspnetcore	Tests: macOS	Log	dotnet/aspnetcore#64807
1373372	dotnet/maui	Controls CollectionView	Log	dotnet/maui#34673
1375004	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#126478
1374830	dotnet/maui	Controls Layout	Log
2948416	dotnet-maui	Controls Border,BoxView,Brush,Button	Log
1374903	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_CoreCLR_RuntimeTests	Log
1374810	dotnet/runtime	tvossimulator-arm64 Debug AllSubsets_Mono	Log	dotnet/runtime#126520
1374795	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono_RuntimeTests	Log
2948538	dotnet-sdk	Darwin	Log	#59726
2948537	dotnet-sdk	Darwin	Log	#59725
2948283	dotnet-maui	Controls Layout	Log
1374433	dotnet/maui	Controls Layout	Log
1374659	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#126767
1374630	dotnet/runtime	osx-arm64 Debug CoreCLR_Libraries	Log	dotnet/runtime#126719
1374627	dotnet/runtime	osx-x64 release Runtime_Release	Log	dotnet/runtime#126771
1374619	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#126770
1374340	dotnet/maui	Controls Shell	Log
2948263	dotnet-maui	Controls CollectionView	Log
1374592	dotnet/aspnetcore	Tests: macOS	Log	dotnet/aspnetcore#66239
1374539	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#126691
1374466	dotnet/runtime	osx-arm64 Release Libraries_CheckedCoreCLR	Log
1374464	dotnet/aspnetcore	Tests: macOS	Log
1374455	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#126762
2948286	dotnet-maui	RunOniOS_BlazorDebug x64	Log
2948055	dotnet-runtime	tvos-arm64 Release AllSubsets_NativeAOT	Log	#59967
2948282	dotnet-maui	RunOniOS_MauiNativeAOT x64	Log
2948259	dotnet-maui	RunOniOS_MauiDebug ARM64	Log
1374333	dotnet/runtime	osx-x64 Release AllSubsets_Mono_Interpreter_RuntimeTests monointerpreter	Log	dotnet/runtime#126108
1374294	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono_RuntimeTests	Log
1374197	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#126199
1374035	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	dotnet/runtime#126562
1373922	dotnet/maui	Controls Layout	Log
1374159	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#53809
1374148	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_CoreCLR_RuntimeTests	Log	dotnet/runtime#125439
1374151	dotnet/aspnetcore	Tests: macOS	Log
1374135	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#126745
1374131	dotnet/aspnetcore	Tests: macOS	Log	dotnet/aspnetcore#64807
1374026	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#126647
1373958	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#126512
1373685	dotnet/maui	Controls CollectionView	Log
1373891	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#125436
1372838	dotnet/maui	Controls CollectionView	Log	dotnet/maui#34882
1373853	dotnet/aspnetcore	Tests: macOS	Log
1373847	dotnet/aspnetcore	Tests: macOS	Log
2947974	dotnet-maui	RunOniOS_MauiNativeAOT x64	Log
1373675	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#125607
1373303	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_CoreCLR_RuntimeTests	Log
1373603	dotnet/aspnetcore	Tests: macOS	Log	dotnet/aspnetcore#66261
2947266	dotnet-maui	Controls Layout	Log
1373570	dotnet/aspnetcore	Tests: macOS	Log	dotnet/aspnetcore#66260
1373544	dotnet/runtime	osx-x64 release Runtime_Release	Log	dotnet/runtime#126091
1373532	dotnet/aspnetcore	Tests: macOS	Log	dotnet/aspnetcore#61667
1373510	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#126647
1373439	dotnet/runtime	osx-x64 release Runtime_Release	Log	dotnet/runtime#125333
1373280	dotnet/maui	Controls Layout	Log
1373392	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#126743
2947763	dotnet-maui	Controls (vlatest) CarouselView
2947789	dotnet-sdk	AoT: macOS (x64)	Log	#59955
2947788	dotnet-sdk	AoT: macOS (x64)	Log	#59580
2947782	dotnet-sdk	TestBuild: macOS (x64)	Log
1373339	dotnet/aspnetcore	Tests: macOS	Log
2947776	dotnet-sdk	Darwin	Log	#59725
2947777	dotnet-sdk	Darwin	Log	#59726
2947224	dotnet-maui	Controls Shell	Log
2947732	dotnet-dotnet-monitor	Test MacOS x64 Release	Log
1373210	dotnet/maui	Controls Layout	Log
2947489	dotnet-runtime	tvos-arm64 Release AllSubsets_NativeAOT	Log	#59958
2947619	dotnet-performance	Performance powershell ubuntu 2204 x64 Viper main	Log
1373002	dotnet/runtime	osx-arm64 Debug CoreCLR_Libraries	Log	dotnet/runtime#123969
2947209	dotnet-maui	Controls Shell	Log
1372936	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#126124
1372913	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_NativeAOT_Smoke	Log	dotnet/runtime#125795
1372304	dotnet/maui	Controls CollectionView	Log
1372875	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_CoreCLR_RuntimeTests	Log	dotnet/runtime#125439
1372863	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#125439
1372860	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono_RuntimeTests	Log	dotnet/runtime#126631
2947490	dotnet-runtime	Libraries Test Run checked coreclr osx x64 Debug	Log	#59957
1372788	dotnet/aspnetcore	Tests: macOS	Log
2947533	dotnet-performance	Performance nativeaot_scenarios ubuntu 2204 x64 Viper 9.0	Log
1372247	dotnet/maui	Controls Shell	Log
2947511	dotnet-performance	Performance nativeaot_scenarios ubuntu 2204 x64 Viper 9.0	Log
1372698	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#126719
1372440	dotnet/runtime	iossimulator-x64 Release AllSubsets_Mono	Log
1372621	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#126525
2947186	dotnet-runtime	ios-arm64 Release AllSubsets_NativeAOT	Log	#59616
1372612	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#124851
1372151	dotnet/runtime	Libraries Test Run release coreclr osx x64 Debug	Log	dotnet/runtime#125955
1372566	dotnet/runtime	osx-x64 Release AllSubsets_Mono_Interpreter_RuntimeTests monointerpreter	Log	dotnet/runtime#126520
2947265	dotnet-maui	RunOniOS_BlazorRelease x64	Log
1372550	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#126718
1372581	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#53782
1372475	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#126699
2947092	dotnet-maui	Controls CollectionView	Log
2947064	dotnet-maui	Controls (vlatest) Layout	Log
2947251	dotnet-runtime	Libraries Test Run checked coreclr osx x64 Debug	Log	#59971
2947217	dotnet-runtime	Libraries Test Run release coreclr osx x64 Debug	Log	#59617
2947223	dotnet-maui	RunOniOS_BlazorDebug ARM64	Log
1372343	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#126512
1372340	dotnet/aspnetcore	Tests: macOS	Log	dotnet/aspnetcore#66246
1372240	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#126093
Displaying 100 of 1226 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
6	252	1226

[lewing]

🤔

[nagilson]

@dotnet/dnceng @dougbu This has impacted a lot of PRs recently, may you PTAL into expanding this Mac resource?

[ivanpovazan]

@dotnet/dnceng we are hitting this again and it seems there is some issue with communication with Helix machines.

More context on timeouts happening in : https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=results

• On success: Run tests in Helix step reports:

Waiting for completion of job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open (Details: https://helix.dot.net/api/jobs/fbec1164-55c5-4fa8-b492-e1ba1b413119/details?api-version=2019-06-17 )
Job 7fdc2720-3d98-4bf3-8113-274dacd69c91 on osx.1200.arm64.open is completed with 6 finished work items.
  Job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open is completed with 6 finished work items.
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.amd64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.arm64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )

Build succeeded.

SENDHELIXJOB : warning : Helix queue osx.1200.amd64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
SENDHELIXJOB : warning : Helix queue osx.1200.arm64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
    2 Warning(s)
    0 Error(s)

Time Elapsed 00:03:14.97
Killing running build processes...

Finishing: Run tests in Helix

ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=logs&j=ccc97bb6-1a23-5e71-fdfa-3cdca4a74749&t=27fc7eb2-ead9-59e1-6679-a637855d40c5

• While on failure - timeout the same step gets stuck with:

Waiting for completion of job 967d92a2-ec10-4332-927f-d28a6563f367 on osx.1200.arm64.open (Details: https://helix.dot.net/api/jobs/967d92a2-ec10-4332-927f-d28a6563f367/details?api-version=2019-06-17 )
  Job 5427159b-500d-49f1-aac0-ec148a492bbe on osx.1200.amd64.open is completed with 6 finished work items.

ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=logs&s=c58bc33c-b825-5bca-90ca-50f6e9293dd8&j=e6966639-fe40-5068-d9ae-681cccecafdf

NOTE: All the tests successfully passed on Helix, but it seems that the communication is lost.

[garath]

Looks like the timeout happened because the osx.1200.arm64.open queue was very busy while the job was running. Right now, I do not think there are any problems with the infrastructure. I will investigate a bit more to see what caused such a back-up.

[garath]

Ah, the queue was consumed with updates and patching. The patching jobs did run longer than necessary and we've communicated with our partner team about the issue. Future jobs will be much shorter and should not overly impact jobs.

[ivanpovazan]

Looks like the timeout happened because the osx.1200.arm64.open queue was very busy while the job was running. Right now, I do not think there are any problems with the infrastructure. I will investigate a bit more to see what caused such a back-up.

We are still experiencing the problem.

Should we try to change the queue to osx.13.arm64?

@dotnet/dnceng

[ilyas1974]

Looks at the failing builds, I'm noticing the jobs that are timing out in the hosted pool (Azure Pipelines) are still using the older hardware. I would recommend moving the workloads to the mac-latest-internal or mac-14-arm64 agent specifications. These have the latest mac hardware associated with them.