fix(replica): bound resumable reader retries

[corylanou]

Description

Bounds reconnect attempts across the complete lifetime of a resumable LTX stream instead of restarting the retry budget on every caller read. Adds an HTTP regression test that truncates every response and counts accepted TCP connections.

Scope: This changes resumable-stream retry accounting only. It does not change the S3 transport configuration, AWS SDK retry policy, compaction scheduling, or retention behavior.

Motivation and Context

The retry counter was local to each Read call. When a response returned bytes together with a premature EOF or transport error, the reader closed that body, suppressed the error so the caller could consume the bytes, and reopened on the next call. That next call started the retry counter at zero again, so a sufficiently large source could reconnect indefinitely without surfacing an error or entering replication backoff.

The release history explains the v0.5.14 regression shape:

• v0.5.12 used resumable streams for explicit restore operations.
• fix(compaction): resume remote source reads #1326 added resumable streams to ongoing remote compaction reads in v0.5.14, exposing the latent per-read retry reset during replication.
• fix(s3): keep retrying through sustained transport flaps #1305 did not create HTTP clients or transports, but its ten-attempt S3 operation retryer could multiply every unbounded reopen during a transport flap.

The S3 client caches one transport per replica, AWS SDK middleware drains and closes failed non-streaming responses, and the resumable reader closes the previous successful GetObject body before reopening. The unbounded cross-read retry reset was the resource-lifecycle defect.

The regression server declares the full remaining Content-Length but returns one byte per connection. Before the fix, a 16-byte read opened 16 TCP connections, returned all data, and produced no error. After the fix, it opens at most four connections (the initial connection plus three retries) and returns the retry-limit error so normal replication backoff can engage.

Fixes #1354

How Has This Been Tested?

go test ./internal -run TestResumableReader_BoundsConnectionsAcrossPartialReads -count=1 -v
go test ./internal ./s3 -count=1
go test . -run 'TestCompactor_CompactResumesRemoteSource|TestReplica_Restore' -count=1 -v
go test ./... -count=1
go test -race ./... -count=1
go vet ./...
pre-commit run --all-files

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (would cause existing functionality to not work as expected)

Checklist

• My code follows the code style of this project (go fmt, go vet)
• I have tested my changes (go test ./...)
• I have updated the documentation accordingly (no documentation change is needed)

[github-actions]

PR Build Metrics

✅ All clear — no issues detected

Check	Status	Summary
Binary size	✅	36.95 MB (0.0 KB / 0.00%)
Dependencies	✅	No changes
Vulnerabilities	✅	None detected
Go toolchain	✅	1.25.12 (latest)
Module graph	✅	1240 edges (0)

Binary Size

	Size	Change
Base (c96c0f4)	36.95 MB
PR (7c4031b)	36.95 MB	0.0 KB (0.00%)

Dependency Changes

No dependency changes.

govulncheck Output

=== Symbol Results ===

No vulnerabilities found.

Your code is affected by 0 vulnerabilities.
This scan also found 0 vulnerabilities in packages you import and 1
vulnerability in modules you require, but your code doesn't appear to call these
vulnerabilities.
Use '-show verbose' for more details.

Build Info

Metric	Value
Build time	44s
Go version	go1.25.12
Commit	7c4031b

---

_{🤖 Updated on each push.}