fix(cleanup): add retry logic to manifest reads and per-manifest error tolerance

[jan-exa]

Summary

Fixes S3 mid-stream body download failures that cause cleanup_old_versions() to fail on large lance tables with thousands of manifests (7,600+ in our case). The cleanup reads every manifest file concurrently, and a single transient S3 error during body streaming kills the entire operation — which then takes ~49 minutes to restart from scratch.

Three targeted changes:

1. read_manifest() now retries get_range() calls (rust/lance-table/src/io/manifest.rs)

Previously, read_manifest() called object_store.inner.get_range() directly — no retry on any failure. Meanwhile, read_manifest_indexes() goes through CloudObjectReader which has do_get_with_outer_retry() (retries body-streaming failures up to download_retry_count times). This was an asymmetry — the function added (get_range_with_retry) follows the same pattern and logs full error details via log::warn/log::info on failure/retry.

2. process_manifests() no longer aborts on first error (rust/lance/src/dataset/cleanup.rs)

Replaced try_for_each_concurrent (which aborts the entire stream on the first error) with for_each_concurrent that collects failures, then retries them in up to 3 additional rounds. With ~7,600 manifests and 64 concurrent reads, the probability of at least one transient failure is high. Now a single failure doesn't waste the ~49 minutes of work from all the successful reads.

3. ObjectStore.download_retry_count() exposed as public (rust/lance-io/src/object_store.rs)

The field was private; added a public getter so lance-table can access the configured retry count for manifest reads.

Review & Testing Checklist for Human

• Verify that the retry in get_range_with_retry handles all object_store::Error variants correctly (not just body-streaming failures) — retrying on non-transient errors (e.g., 404 NotFound) is wasteful but not harmful since the retry count is low (default 3)
• Run cleanup_old_versions() against a real large lance table on S3 (atlas.lance with ~7,600 manifests) to validate end-to-end behavior
• Verify the process_manifests retry rounds work correctly by checking logs for "Retry round" messages when a manifest read fails transiently
• Confirm that manifests already processed in the first pass are NOT re-processed during retry rounds (they're tracked in the CleanupInspection mutex — the retry only processes the failed ManifestLocations)

Notes

The root cause is that the upstream object_store crate strips the actual underlying error (connection reset, timeout, etc.) behind a generic "HTTP error: request or response body error" message. The log::warn! in get_range_with_retry uses {:?} (Debug format) which preserves the full error chain including the underlying cause, providing much better diagnostics than the default Display format.

Link to Devin session: https://app.devin.ai/sessions/bb810ab5769542e0a12b08c2505d2ae1
Requested by: @jan-exa

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment and CI monitoring

[github-actions]

ACTION NEEDED
Lance follows the Conventional Commits specification for release automation.

The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification.

For details on the error please inspect the "PR Title Check" action.