Skip to content

store/blobstore: serve git-backed ranged reads from a materialized file#11264

Open
brandon-fryslie wants to merge 1 commit into
dolthub:mainfrom
brandon-fryslie:gitblobstore-materialize-ranges
Open

store/blobstore: serve git-backed ranged reads from a materialized file#11264
brandon-fryslie wants to merge 1 commit into
dolthub:mainfrom
brandon-fryslie:gitblobstore-materialize-ranges

Conversation

@brandon-fryslie

Copy link
Copy Markdown

Problem

GitBlobstore.getFromCache serves a partial (ranged) read of an inline blob by streaming the whole blob through git cat-file blob and discarding the bytes before the requested offset — sliceInlineBlob's io.CopyN(io.Discard, rc, offset).

NBS reads a table file with many small ranged reads (footer, index, then chunk-by-chunk), and footer reads use negative offsets (read the last N bytes). So each ranged read re-spawns git cat-file and re-inflates almost the entire blob from byte 0. A large fetch/pull/clone over a git-backed remote is therefore O(reads × blobsize) — tens of GB of redundant zlib and thousands of subprocesses for a single ~18 MB table file stored as one inline blob (under the default 50 MB part size).

This is exactly the slowness the existing TODO(gitblobstore) above sliceInlineBlob anticipates:

This streaming implementation is correct but may be slow for workloads that do many small ranged reads (e.g. table index/footer reads). Consider caching/materializing blobs to a local file …

Fix

Materialize each blob to a local seekable file once, keyed by its immutable, content-addressed OID, and serve every range from it with a single Seek. The cost becomes O(blobsize) to stream once + O(range) per read.

  • Partial (ranged) reads go through the materialized file.
  • Full (AllRange) reads still stream directly — a single sequential pass needs no random access, so there's no reason to spill it to disk.
  • The per-blobstore temp directory is removed on Close.
  • Concurrent readers of the same OID block on a per-entry ready channel so the blob is streamed only once; the materialize I/O runs outside the lock so reads of distinct blobs aren't serialized.

Because blobs are content-addressed, a materialized copy is always valid — the OID→file mapping never goes stale.

Test

Adds TestGitBlobstore_RangedReads_MaterializeBlobOnce: seeds an inline blob, performs many scattered ranged reads (including a negative-offset tail read), and asserts BlobReader is invoked exactly once. On the pre-fix code this count is N (one full re-inflation per ranged read); after the fix it is 1. All existing store/blobstore tests pass.

Notes

Branched from the commit currently pinned by a downstream consumer for a clean, minimal diff; happy to rebase onto the latest main. The chunked-tree read path (multiPartReadCloser) has the same shape but is bounded by the part size; this change targets the inline path the TODO calls out. A natural follow-up is to apply the same materialization to chunked parts.

GitBlobstore.getFromCache served a partial (ranged) read of an inline blob by
streaming the whole blob through `git cat-file` and discarding the bytes before
the requested offset (sliceInlineBlob's io.CopyN(io.Discard, rc, offset)). NBS
reads a table file with many small ranged reads -- footer, index, then
chunk-by-chunk -- and footer reads use negative offsets, so each read re-spawned
git and re-inflated (almost) the entire blob from byte 0. A large fetch/pull/
clone over a git-backed remote was therefore O(reads * blobsize): tens of GB of
redundant zlib and thousands of subprocesses for a single ~18MB table file. This
is the slowdown the existing TODO at sliceInlineBlob anticipates.

Materialize each blob to a local seekable file once -- keyed by its immutable,
content-addressed OID -- and serve every range from it with a single seek, so
the cost becomes O(blobsize) streamed once plus O(range) per read. Full
(AllRange) reads still stream directly, since a sequential pass needs no random
access. The temp dir is removed on Close.

Adds a regression test asserting that many ranged reads of one blob invoke
BlobReader exactly once.

Signed-off-by: Brandon Fryslie <530235+brandon-fryslie@users.noreply.github.com>
brandon-fryslie added a commit to promptctl/links-issue-tracker that referenced this pull request Jun 30, 2026
…inflating (promptctl-sync-pull-l07p) (#276)

A large `lit sync pull`/`fetch` over a git-backed remote was O(reads x blobsize):
dolt's GitBlobstore served each NBS ranged read (footer/index/chunk) by streaming
and re-inflating the whole ~18.5MB archive blob from byte 0. The footer read uses a
negative offset, so it re-inflated almost the entire blob just for the trailer. This
is the same git-blob pathology PR #274 fixed for init adopt, here fixed at the source
so every sync path (fetch/pull/receive/clone) benefits.

The fix lives in dolt's blobstore: materialize each immutable, content-addressed blob
to a local file once and serve every range by seek. Pin a transient one-commit fork
via `replace` until the upstream PR (dolthub/dolt#11264) merges; then drop the replace
and bump the dolthub/dolt/go pin.

The full ./internal/... suite passes against the patched dolt (behavior-preserving:
same bytes, served from a file instead of a re-inflated stream). The authoritative
regression test — BlobReader invoked once per blob, not once per range — ships in the
upstream PR with the code it guards.

Claude-Session: https://claude.ai/code/session_01TMiknvhzuS1tM8frgeDwmH
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants