Regression test for range sync CGC race condition

[jimmygchen]

Description

Regression test for a sync race condition where range sync gets stuck due to a CGC (custody group count) metadata timing issue. The original bug was discovered in CI kurtosis sync tests: https://github.com/sigp/lighthouse/actions/runs/17632496975/job/50102884940

The core fix (calling attempt_send_awaiting_download_batches on SyncingChain::resume) was merged in #8230. This PR adds a regression test that reproduces the exact scenario, plus related fixes.

Changes:

1. Regression test (finalized_sync_not_enough_custody_peers_resume_after_peer_cgc_update): reproduces the failing scenario and verifies that sync resumes correctly when CGC updates arrive.

2. Moved to separate PR: Count AwaitingDownload batches in buffer size check (range sync + backfill sync): the in_buffer check previously only counted Downloading and AwaitingProcessing batches. This meant AwaitingDownload batches didn't count toward BATCH_BUFFER_SIZE, so new batches could be created while AwaitingDownload ones accumulated. Now AwaitingDownload is included so the buffer limit applies properly.

3. Moved to separate PR
   : Handle NoPeer error without fake-failing the batch: previously, when send_batch got a NoPeer error, it would fake-start the download (start_downloading(1)), then call download_failed to increment the failure count. This incorrectly penalized the batch for something that wasn't a real download failure. Now NoPeer just logs a debug message and leaves the batch in AwaitingDownload (a valid in-between state since Allow AwaitingDownload to be a valid in-between state #7984), letting it be retried when peers become available.

[dapplion]

Nice catch and good test! Some minor comments

[jimmygchen]

I had a look at the failing checkpoint sync tests (optionally triggered jobs), and I don't believe they are related to changes from this PR, i think what's happening is:

• The sepolia checkpoint server used is currently 4 days behind, so it takes longer to sync
• The devnet was under non finality tests, and it was just taking longer to sync.
• Github-hosted runner struggles to sync supernode on peerdas devnet due to capacity, to be resovled in Use unique enclave names for CI Kurtosis tests on self hosted runners #7804 once we have dedicated kurtosis runners.

[jimmygchen]

@dapplion thanks for the review! I've addressed most of your comments.

[pawanjay176]

FYI, running this on devnet 3 i'm getting a stuck chain with 40 peers.
Need to investigate further

[jimmygchen]

Thanks for raising this Pawan. Sorry i haven't managed to get to this today, will look into this tomorrow.

[jimmygchen]

FYI, running this on devnet 3 i'm getting a stuck chain with 40 peers. Need to investigate further

@pawanjay176 im going to remove the clean ups from this PR so we can merge this, because the flaky sync test is really quite annoying 😅

[mergify]

This pull request has merge conflicts. Could you please resolve them @jimmygchen? 🙏

[mergify]

This pull request has merge conflicts. Could you please resolve them @jimmygchen? 🙏

[jimmygchen]

@dapplion please hold off from merging this. I'm seeing a trend of increasing sync_active_network_requests for components_by_range running this branch. Will investigate further tomorrow.

[jimmygchen]

^ unrelated to this PR i think. I've made a PR to clean it up:

• Ensure consistent cleanup of range sync request tracking #8890

[jimmygchen]

@pawanjay176 @dapplion
I believe I've addressed all the prior comments and this change is still required. Could you please re-review?
This PR doesn't fix the potential leaks, but prevents sending an unbounded number of batches upon resuming sync.
The leak is addressed in the link PR above.

[jimmygchen]

@pawanjay176 I've looked at this PR again with @michaelsproul - this PR contains two changes that can be separated:

1. The regression test for a CGC race condition that has already been fixed a while ago - this is a low risk and low hanging fruit, but not urgent for v8.1.1.
2. The NoPeer clean up, which makes it possible to create a batch in AwaitingDownload state, and therefore needs to be bundled with the in_buffer check AND the stall detection. This is mainly a clean up. On current unstable, the batch never stays in AwaitingDownload after send batch returns, so these batches can't accumulate. Therefore this is not a required fix for v8.1.1 either.

[jimmygchen]

Rebased on unstable and split this PR into two:

• This PR now contains only the regression test for the CGC race condition (change 1).
• The NoPeer cleanup (handle NoPeer without fake-failing the batch, count AwaitingDownload in buffer checks, stall detection, periodic retry) has been moved to Cleans up the NoPeer error handling in range sync's send_batch #8919.

[mergify]

Some required checks have failed. Could you please take a look @jimmygchen? 🙏

[mergify]

Hi @jimmygchen, this pull request has been closed automatically due to 30 days of inactivity. If you’d like to continue working on it, feel free to reopen at any time.

[mergify]

Hi @jimmygchen, this pull request has been closed automatically due to 30 days of inactivity. If you’d like to continue working on it, feel free to reopen at any time.

[pawanjay176]

Good test, lets merge!

[mergify]

Merge Queue Status

• ✅ Entered queue — 2026-06-03 23:49 UTC · Rule: default
• ✅ Checks started · on draft merge queue: embarking unstable (d617c82) and #8039 together #9406
• ❌ Checks failed
• 🚫 Left the queue — 2026-06-04 00:11 UTC · at 2d3680fb71e343c85affea42ab047d7d87aa38c2

This pull request spent 21 minutes 49 seconds in the queue, including 19 minutes 37 seconds running CI.

Waiting for

• check-success=local-testnet-success
• check-success=test-suite-success

All conditions

• check-success=local-testnet-success
• check-success=test-suite-success
• #approved-reviews-by >= 1 [🛡 GitHub branch protection]
  • Regression test for range sync CGC race condition #8039

Reason

The merge conditions cannot be satisfied due to failing checks

Hint

You may have to fix your CI before adding the pull request to the queue again.
If you update this pull request, to fix the CI, it will automatically be requeued once the queue conditions match again.
If you think this was a flaky issue instead, you can requeue the pull request, without updating it, by posting a @mergifyio queue comment.

[mergify]

Merge Queue Status

• ✅ Entered queue — 2026-06-03 23:49 UTC · Rule: default
• ✅ Checks started · on draft merge queue: embarking unstable (d617c82) and #8039 together #9406
• ❌ Checks failed
• 🚫 Left the queue — 2026-06-04 00:11 UTC · at 2d3680fb71e343c85affea42ab047d7d87aa38c2

This pull request spent 21 minutes 49 seconds in the queue, including 19 minutes 37 seconds running CI.

Waiting for

• check-success=local-testnet-success
• check-success=test-suite-success

All conditions

• check-success=local-testnet-success
• check-success=test-suite-success
• #approved-reviews-by >= 1 [🛡 GitHub branch protection]
  • Regression test for range sync CGC race condition #8039

Reason

The merge conditions cannot be satisfied due to failing checks

Hint

You may have to fix your CI before adding the pull request to the queue again.
If you update this pull request, to fix the CI, it will automatically be requeued once the queue conditions match again.
If you think this was a flaky issue instead, you can requeue the pull request, without updating it, by posting a @mergifyio queue comment.

[mergify]

Merge Queue Status

• ✅ Entered queue — 2026-06-04 05:46 UTC · Rule: default
• ✅ Checks started · on draft merge queue: embarking unstable (d617c82) and #8039 together #9407
• 🚫 Left the queue — 2026-06-04 06:16 UTC · at 2d3680fb71e343c85affea42ab047d7d87aa38c2

This pull request spent 30 minutes 7 seconds in the queue, including 28 minutes 42 seconds running CI.

Required conditions to merge

• #approved-reviews-by >= 1 [🛡 GitHub branch protection]
  • Regression test for range sync CGC race condition #8039
• check-success=local-testnet-success
• check-success=test-suite-success

Reason

Pull request #8039 has been dequeued

The pull request could not be merged. This could be related to an activated branch protection or ruleset rule that prevents us from merging. 1 review requesting changes and 1 approving review by reviewers with write access.

Hint

You should look at the reason for the failure and decide if the pull request needs to be fixed or if you want to requeue it.
If you do update this pull request, it will automatically be requeued once the queue conditions match again.
If you think this was a flaky issue instead, you can requeue the pull request, without updating it, by posting a @mergifyio queue comment.

[mergify]

Merge Queue Status

• ✅ Entered queue — 2026-06-04 06:54 UTC · Rule: default
• ✅ Checks passed · on draft merge queue: embarking unstable (d617c82) and #8039 together #9408
• ✅ Merged — 2026-06-04 07:24 UTC · at 2d3680fb71e343c85affea42ab047d7d87aa38c2 · squash

This pull request spent 30 minutes 17 seconds in the queue, including 28 minutes 7 seconds running CI.

Required conditions to merge

• #approved-reviews-by >= 1 [🛡 GitHub branch protection]
  • Regression test for range sync CGC race condition #8039
• check-success=local-testnet-success
• check-success=test-suite-success