store/blobstore: serve git-backed ranged reads from a materialized file#11264
Open
brandon-fryslie wants to merge 1 commit into
Open
store/blobstore: serve git-backed ranged reads from a materialized file#11264brandon-fryslie wants to merge 1 commit into
brandon-fryslie wants to merge 1 commit into
Conversation
GitBlobstore.getFromCache served a partial (ranged) read of an inline blob by streaming the whole blob through `git cat-file` and discarding the bytes before the requested offset (sliceInlineBlob's io.CopyN(io.Discard, rc, offset)). NBS reads a table file with many small ranged reads -- footer, index, then chunk-by-chunk -- and footer reads use negative offsets, so each read re-spawned git and re-inflated (almost) the entire blob from byte 0. A large fetch/pull/ clone over a git-backed remote was therefore O(reads * blobsize): tens of GB of redundant zlib and thousands of subprocesses for a single ~18MB table file. This is the slowdown the existing TODO at sliceInlineBlob anticipates. Materialize each blob to a local seekable file once -- keyed by its immutable, content-addressed OID -- and serve every range from it with a single seek, so the cost becomes O(blobsize) streamed once plus O(range) per read. Full (AllRange) reads still stream directly, since a sequential pass needs no random access. The temp dir is removed on Close. Adds a regression test asserting that many ranged reads of one blob invoke BlobReader exactly once. Signed-off-by: Brandon Fryslie <530235+brandon-fryslie@users.noreply.github.com>
brandon-fryslie
added a commit
to promptctl/links-issue-tracker
that referenced
this pull request
Jun 30, 2026
…inflating (promptctl-sync-pull-l07p) (#276) A large `lit sync pull`/`fetch` over a git-backed remote was O(reads x blobsize): dolt's GitBlobstore served each NBS ranged read (footer/index/chunk) by streaming and re-inflating the whole ~18.5MB archive blob from byte 0. The footer read uses a negative offset, so it re-inflated almost the entire blob just for the trailer. This is the same git-blob pathology PR #274 fixed for init adopt, here fixed at the source so every sync path (fetch/pull/receive/clone) benefits. The fix lives in dolt's blobstore: materialize each immutable, content-addressed blob to a local file once and serve every range by seek. Pin a transient one-commit fork via `replace` until the upstream PR (dolthub/dolt#11264) merges; then drop the replace and bump the dolthub/dolt/go pin. The full ./internal/... suite passes against the patched dolt (behavior-preserving: same bytes, served from a file instead of a re-inflated stream). The authoritative regression test — BlobReader invoked once per blob, not once per range — ships in the upstream PR with the code it guards. Claude-Session: https://claude.ai/code/session_01TMiknvhzuS1tM8frgeDwmH
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
GitBlobstore.getFromCacheserves a partial (ranged) read of an inline blob by streaming the whole blob throughgit cat-file bloband discarding the bytes before the requested offset —sliceInlineBlob'sio.CopyN(io.Discard, rc, offset).NBS reads a table file with many small ranged reads (footer, index, then chunk-by-chunk), and footer reads use negative offsets (read the last N bytes). So each ranged read re-spawns
git cat-fileand re-inflates almost the entire blob from byte 0. A large fetch/pull/clone over a git-backed remote is therefore O(reads × blobsize) — tens of GB of redundant zlib and thousands of subprocesses for a single ~18 MB table file stored as one inline blob (under the default 50 MB part size).This is exactly the slowness the existing
TODO(gitblobstore)abovesliceInlineBlobanticipates:Fix
Materialize each blob to a local seekable file once, keyed by its immutable, content-addressed OID, and serve every range from it with a single
Seek. The cost becomes O(blobsize) to stream once + O(range) per read.AllRange) reads still stream directly — a single sequential pass needs no random access, so there's no reason to spill it to disk.Close.readychannel so the blob is streamed only once; the materialize I/O runs outside the lock so reads of distinct blobs aren't serialized.Because blobs are content-addressed, a materialized copy is always valid — the OID→file mapping never goes stale.
Test
Adds
TestGitBlobstore_RangedReads_MaterializeBlobOnce: seeds an inline blob, performs many scattered ranged reads (including a negative-offset tail read), and assertsBlobReaderis invoked exactly once. On the pre-fix code this count is N (one full re-inflation per ranged read); after the fix it is 1. All existingstore/blobstoretests pass.Notes
Branched from the commit currently pinned by a downstream consumer for a clean, minimal diff; happy to rebase onto the latest
main. The chunked-tree read path (multiPartReadCloser) has the same shape but is bounded by the part size; this change targets the inline path the TODO calls out. A natural follow-up is to apply the same materialization to chunked parts.