Checkout failure: "Git fetch failed with exit code 128"

[carlossanlop]

Build

https://dev.azure.com/dnceng/public/_build/results?buildId=1706886&view=results

Build leg reported

Libraries Test Run release coreclr Linux_musl x64 Release

Pull Request

Rolling build for runtime

Action required for the engineering services team

To triage this issue (First Responder / @dotnet/dnceng):

• Open the failing build above and investigate
• Add a comment explaining your findings

If this is an issue that is causing build breaks across multiple builds and would get benefit from being listed on the build analysis check, follow the next steps:

1. Add the label "Known Build Error"
2. Edit this issue and add an error string in the Json below that can help us match this issue with future build breaks. You should use the known issues documentation

{
   "ErrorMessage" : "Git fetch failed with exit code: 128",
   "BuildRetry": true
}

Additional information about the issue reported

The Checkout dotnet/runtime@main to s step is the one that failed. Here is the end of the log output:

remote: Compressing objects:  29% (30070/103687)        
remote: Compressing objects:  30% (31107/103687)        
remote: Compressing objects:  31% (32143/103687)        
remote: Compressing objects:  31% (32322/103687)        
remote: Compressing objects:  32% (33180/103687)        
remote: Compressing objects:  33% (34217/103687)        
remote: Compressing objects:  33% (35247/103687)        
remote: Compressing objects:  34% (35254/103687)        
remote: Compressing objects:  35% (36291/103687)        
remote: Compressing objects:  36% (37328/103687)        
remote: Compressing objects:  37% (38365/103687)        
remote: Compressing objects:  37% (38926/103687)        
remote: Compressing objects:  38% (39402/103687)        
error: RPC failed; curl 18 transfer closed with outstanding read data remaining
fatal: protocol error: bad pack header
##[error]Git fetch failed with exit code: 128
Finishing: Checkout dotnet/runtime@main to s

Report

Build	Definition	Step Name	Console log	Pull Request
1374643	dotnet/sdk	Checkout dotnet/sdk@refs/pull/53464/merge to s	Log	dotnet/sdk#53464
2948283	dotnet-maui	Checkout dotnet-maui@main to s	Log
1374239	dotnet/runtime	Checkout dotnet/runtime@refs/pull/126713/merge to s	Log	dotnet/runtime#126713
1372766	dotnet/sdk	Checkout dotnet/sdk@refs/pull/53776/merge to s	Log	dotnet/sdk#53776
1372400	dotnet/sdk	Checkout dotnet/sdk@refs/pull/53781/merge to s	Log	dotnet/sdk#53781
1372304	dotnet/maui	Checkout dotnet/maui@inflight/current to s	Log
1372351	dotnet/runtime	Checkout dotnet/runtime@refs/pull/126710/merge to s	Log	dotnet/runtime#126710
1371280	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/66237/merge to s	Log	dotnet/aspnetcore#66237
1371282	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/66237/merge to s	Log	dotnet/aspnetcore#66237
1371281	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/66237/merge to s	Log	dotnet/aspnetcore#66237
1370762	dotnet/runtime	Checkout dotnet/runtime@refs/pull/126658/merge to s	Log	dotnet/runtime#126658
1370251	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/66217/merge to s	Log	dotnet/aspnetcore#66217
1369438	dotnet/runtime	Checkout dotnet/runtime@main to s	Log
1368706	dotnet/sdk	Checkout dotnet/sdk@refs/pull/53730/merge to s	Log	dotnet/sdk#53730
1365485	dotnet/runtime	Checkout dotnet/runtime@refs/pull/126506/merge to s	Log	dotnet/runtime#126506
1364885	dotnet/sdk	Checkout dotnet/sdk@refs/pull/53175/merge to s	Log	dotnet/sdk#53175
1364883	dotnet/sdk	Checkout dotnet/sdk@refs/pull/53681/merge to s	Log	dotnet/sdk#53681
1364884	dotnet/sdk	Checkout dotnet/sdk@refs/pull/53175/merge to s	Log	dotnet/sdk#53175
1364162	dotnet/runtime	Checkout dotnet/runtime@refs/pull/124650/merge to s	Log	dotnet/runtime#124650
2942034	dotnet-aspnetcore	Checkout dotnet-aspnetcore@refs/pull/59735/merge to s	Log	#59735
1362725	dotnet/runtime	Checkout dotnet/runtime@main to s	Log
1362435	dotnet/runtime	Checkout dotnet/runtime@refs/pull/126422/merge to s	Log	dotnet/runtime#126422
1362429	dotnet/runtime	Checkout dotnet/runtime@refs/pull/126421/merge to s	Log	dotnet/runtime#126421
1360266	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/65912/merge to s	Log	dotnet/aspnetcore#65912
1360167	dotnet/runtime	Checkout dotnet/runtime@refs/pull/125437/merge to s	Log	dotnet/runtime#125437
2939721	dotnet-dotnet	Checkout dotnet-dotnet@refs/pull/59562/merge to s	Log	#59562
1358347	dotnet/runtime	Checkout dotnet/runtime@refs/pull/124544/merge to s	Log	dotnet/runtime#124544
1357834	dotnet/maui	Checkout dotnet/maui@refs/pull/29344/merge to s	Log	dotnet/maui#29344
1355056	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/66013/merge to s	Log	dotnet/aspnetcore#66013
1355085	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/66025/merge to s	Log	dotnet/aspnetcore#66025
1355460	dotnet/runtime	Checkout dotnet/runtime@refs/pull/126213/merge to s	Log	dotnet/runtime#126213
1355231	dotnet/runtime	Checkout dotnet/runtime@refs/pull/126208/merge to s	Log	dotnet/runtime#126208
1355071	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/66021/merge to s	Log	dotnet/aspnetcore#66021
1355062	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/66017/merge to s	Log	dotnet/aspnetcore#66017
1355066	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/66019/merge to s	Log	dotnet/aspnetcore#66019
1355074	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/66023/merge to s	Log	dotnet/aspnetcore#66023
1355049	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/66011/merge to s	Log	dotnet/aspnetcore#66011
1355060	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/66015/merge to s	Log	dotnet/aspnetcore#66015
1354790	dotnet/arcade	Checkout dotnet/arcade@refs/pull/16636/merge to s	Log	#16636
1354279	dotnet/roslyn	Checkout dotnet/roslyn@refs/pull/82943/merge to s	Log	dotnet/roslyn#82943
1354254	dotnet/sdk	Checkout dotnet/sdk@refs/pull/53608/merge to s	Log	dotnet/sdk#53608
1354234	dotnet/runtime	Checkout dotnet/runtime@refs/pull/126175/merge to s	Log	dotnet/runtime#126175
1354119	dotnet/runtime	Checkout dotnet/runtime@refs/pull/126061/merge to s	Log	dotnet/runtime#126061
1352631	dotnet/sdk	Checkout dotnet/sdk@refs/pull/53592/merge to s	Log	dotnet/sdk#53592
1352884	dotnet/sdk	Checkout dotnet/sdk@refs/pull/53591/merge to s	Log	dotnet/sdk#53591
1351856	dotnet/xharness	Checkout dotnet/xharness@refs/pull/1577/merge to s	Log	dotnet/xharness#1577
1352341	dotnet/runtime	Checkout dotnet/runtime@refs/pull/125793/merge to s	Log	dotnet/runtime#125793
1352275	dotnet/runtime	Checkout dotnet/runtime@refs/pull/123846/merge to s	Log	dotnet/runtime#123846
1352266	dotnet/runtime	Checkout dotnet/runtime@release/10.0 to s	Log
1352164	dotnet/runtime	Checkout dotnet/runtime@refs/pull/126006/merge to s	Log	dotnet/runtime#126006
1352155	dotnet/runtime	Checkout dotnet/runtime@refs/pull/125907/merge to s	Log	dotnet/runtime#125907
1351380	dotnet/runtime	Checkout dotnet/runtime@refs/pull/126057/merge to s	Log	dotnet/runtime#126057
1351379	dotnet/runtime	Checkout dotnet/runtime@refs/pull/126057/merge to s	Log	dotnet/runtime#126057
2935067	dotnet-dotnet	Checkout dotnet-dotnet@refs/pull/59373/merge to s	Log	#59373
1350840	dotnet/runtime	Checkout dotnet/runtime@refs/pull/126049/merge to s	Log	dotnet/runtime#126049
1350859	dotnet/runtime	Checkout dotnet/runtime@refs/pull/114017/merge to s	Log	dotnet/runtime#114017
1350567	dotnet/runtime	Checkout dotnet/runtime@refs/pull/126041/merge to s	Log	dotnet/runtime#126041
1349074	dotnet/runtime	Checkout dotnet/runtime@refs/pull/125892/merge to s	Log	dotnet/runtime#125892
1348927	dotnet/runtime	Checkout dotnet/runtime@refs/pull/125910/merge to s	Log	dotnet/runtime#125910
2933831	dotnet-dotnet	Checkout dotnet-dotnet@refs/pull/59025/merge to s	Log	#59025
1348315	dotnet/runtime	Checkout dotnet/runtime@refs/pull/125961/merge to s	Log	dotnet/runtime#125961
1348422	dotnet/maui	Checkout dotnet/maui@main to s	Log
1348421	dotnet/maui	Checkout dotnet/maui@main to s	Log
1348176	dotnet/runtime	Checkout dotnet/runtime@refs/pull/125554/merge to s	Log	dotnet/runtime#125554
1348426	dotnet/runtime	Checkout dotnet/runtime@refs/pull/125963/merge to s	Log	dotnet/runtime#125963
1347053	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/65898/merge to s	Log	dotnet/aspnetcore#65898
1345592	dotnet/runtime	Checkout dotnet/runtime@refs/pull/125860/merge to s	Log	dotnet/runtime#125860
1345485	dotnet/runtime	Checkout dotnet/runtime@refs/pull/125851/merge to s	Log	dotnet/runtime#125851
2931719	dotnet-dotnet	Checkout dotnet-dotnet@refs/pull/59265/merge to s	Log	#59265
1344911	dotnet/maui	Checkout dotnet/maui@inflight/current to s	Log
1344781	dotnet/runtime	Checkout dotnet/runtime@refs/pull/125829/merge to s	Log	dotnet/runtime#125829
1343547	dotnet/runtime	Checkout dotnet/runtime@refs/pull/125787/merge to s	Log	dotnet/runtime#125787
1343544	dotnet/runtime	Checkout dotnet/runtime@refs/pull/125786/merge to s	Log	dotnet/runtime#125786
1343480	dotnet/runtime	Checkout dotnet/runtime@refs/pull/125782/merge to s	Log	dotnet/runtime#125782
1343214	maestro-auth-test/build-result-analysis-test	Checkout maestro-auth-test/build-result-analysis-test@refs/pull/2482/merge to s	Log	maestro-auth-test/build-result-analysis-test#2482
1343215	maestro-auth-test/build-result-analysis-test	Checkout maestro-auth-test/build-result-analysis-test@refs/pull/2482/merge to s	Log	maestro-auth-test/build-result-analysis-test#2482
1343088	dotnet/maui	Checkout dotnet/maui@inflight/current to s	Log
1342980	dotnet/runtime	Checkout dotnet/runtime@refs/pull/125771/merge to s	Log	dotnet/runtime#125771
1342704	dotnet/maui	Checkout dotnet/maui@inflight/current to s	Log
1343022	dotnet/runtime	Checkout dotnet/runtime@refs/pull/125615/merge to s	Log	dotnet/runtime#125615
1342527	dotnet/maui	Checkout dotnet/maui@refs/pull/33908/merge to s	Log	dotnet/maui#33908
1342201	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/65867/merge to s	Log	dotnet/aspnetcore#65867
1341977	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/65847/merge to s	Log	dotnet/aspnetcore#65847
1341972	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/65845/merge to s	Log	dotnet/aspnetcore#65845
1341981	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/65849/merge to s	Log	dotnet/aspnetcore#65849
1341970	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/65844/merge to s	Log	dotnet/aspnetcore#65844
1341979	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/65848/merge to s	Log	dotnet/aspnetcore#65848
1341973	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/65846/merge to s	Log	dotnet/aspnetcore#65846
1341967	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/65843/merge to s	Log	dotnet/aspnetcore#65843
1341509	dotnet/runtime	Checkout dotnet/runtime@refs/pull/125735/merge to s	Log	dotnet/runtime#125735
1341146	maestro-auth-test/build-result-analysis-test	Checkout maestro-auth-test/build-result-analysis-test@refs/pull/2480/merge to s	Log	maestro-auth-test/build-result-analysis-test#2480
1341147	maestro-auth-test/build-result-analysis-test	Checkout maestro-auth-test/build-result-analysis-test@refs/pull/2480/merge to s	Log	maestro-auth-test/build-result-analysis-test#2480
1340771	dotnet/runtime	Checkout dotnet/runtime@refs/pull/125722/merge to s	Log	dotnet/runtime#125722
1340601	dotnet/runtime	Checkout dotnet/runtime@refs/pull/125617/merge to s	Log	dotnet/runtime#125617
1340156	dotnet/runtime	Checkout dotnet/runtime@refs/pull/125698/merge to s	Log	dotnet/runtime#125698
1339463	maestro-auth-test/build-result-analysis-test	Checkout maestro-auth-test/build-result-analysis-test@refs/pull/2475/merge to s	Log	maestro-auth-test/build-result-analysis-test#2475
1339462	maestro-auth-test/build-result-analysis-test	Checkout maestro-auth-test/build-result-analysis-test@refs/pull/2475/merge to s	Log	maestro-auth-test/build-result-analysis-test#2475
1339451	maestro-auth-test/build-result-analysis-test	Checkout maestro-auth-test/build-result-analysis-test@refs/pull/2474/merge to s	Log	maestro-auth-test/build-result-analysis-test#2474
1339452	maestro-auth-test/build-result-analysis-test	Checkout maestro-auth-test/build-result-analysis-test@refs/pull/2474/merge to s	Log	maestro-auth-test/build-result-analysis-test#2474
1338877	dotnet/arcade-services	Checkout dotnet/arcade-services@refs/pull/6107/merge to s	Log	dotnet/arcade-services#6107
Displaying 100 of 108 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
3	14	108

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=323960
Error message validated: Git fetch failed with exit code: 128
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 6/28/2023 9:15:14 PM UTC

[ChadNedzlek]

So this is... interesting. I agree this seems like essentially a random network failure that is not the fault of the code. Also... it doesn't seem like there is anything to be done other than retry the build.

@ulisesh and @AlitzelMendez. This seems like another instance of a "known issue" that would really benefit from the ability to get retried automatically. I think Carlos did the right thing by opening the known issue, but his PR (and any future PR that hits this) is still going to be blocked, since the build doesn't run at all in this case, because the checkout failed. And it would be great to help folks that hit this.

[ulisesh]

I created the issue to track how often it was happening and measure the impact of the problem. It might be hard to get traction on networking problems but having the number of impacted builds has great value in my opinion.

In the last 24 hours, we had 37 builds failed with the same error. cc @markwilkie who might find this data interesting.

I think automatic retries for known issues will be the next feature after we are done with analyzing builds retroactively, we can discuss this if people think we should change priorities.

[markwilkie]

So 37 builds were caught with the known issues feature @ulisesh ? Could I get the query please? (super exciting)

[ulisesh]

I saw 37 hits in the office but now I only see 33 🤔. There are 33 failed jobs in 2 different builds. Sorry if I misused the terms but I still think it is great that we can track these build breaks 😄

This is the query:

KnownIssues
| where IssueId == 9009 and IssueRepository == "dotnet/arcade"

[ViktorHofer]

This happened 36 times in the last 24 hours. Should we add a retry (assuming that's possible)?