fix(db): surface disk-full staging errors#1322
Open
corylanou wants to merge 4 commits into
Open
Conversation
PR Build Metrics✅ All clear — no issues detected
Binary Size
Dependency ChangesNo dependency changes. govulncheck OutputBuild Info
History (8 previous)
🤖 Updated on each push. |
11627d8 to
f4f0076
Compare
099f814 to
978be41
Compare
f4f0076 to
c7f0777
Compare
978be41 to
ca46d85
Compare
This was referenced Jul 2, 2026
c7f0777 to
66314e3
Compare
ca46d85 to
4b06238
Compare
66314e3 to
cfa0759
Compare
4b06238 to
68c0661
Compare
cfa0759 to
11ae6fc
Compare
68c0661 to
ce69343
Compare
11ae6fc to
f170a0d
Compare
ce69343 to
f2851b1
Compare
This was referenced Jul 9, 2026
f170a0d to
906619b
Compare
Classify ENOSPC from staging-directory creation as a disk-full error so the gauge cannot report recovered while the disk is still full. Use a constant log message with attributes for the disk-full monitor log and surface the backoff ceiling. Replace the package-global openLTXFile seam with a per-DB field, move the error constructor next to its type, drop the always-zero Level field, and cover the open/close failure paths plus gauge reset on non-disk-full errors.
Replace LTXStagingDiskFullError with the established LTXError plus wrapped ErrDiskFull sentinel pattern. Key the disk-full gauge on isDiskFullError to match the monitor cleanup predicate, broaden detection with syscall.ENOSPC and Windows/SQLite phrasings, and log a single hex txid key per house style.
f2851b1 to
9566a1a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds a typed
LTXStagingDiskFullErrorplusErrDiskFullfor local LTX staging failures, and wraps ENOSPC/EDQUOT from the L0 staging open/write/encoder close/fsync/close path before it reaches the generic sync loop.The DB monitor now logs a clear
sync stopped: disk full while staging ltx fileerror with the staging path and LTX identity, then exits instead of retrying the same generic sync error forever.This PR is stacked on the
issue-1164-soak-replicate-restore-teststability cluster branch.Motivation and Context
Issue #1310 reports v0.5.x initial sync staging the full snapshot under
.{db}-litestream/ltx/0/before upload. In the reported case, a 22 GB database with about 20 GB free filled the disk during.ltx.tmpcreation, Litestream stayed active, no object reached the replica, and the journal had no clear ERROR/WARN signal.This PR is intentionally scoped to surfacing and fail-fast handling of the local disk-full staging condition. It does not change the snapshot staging architecture or stream initial snapshots directly to the replica.
Fixes #1310
How Has This Been Tested?
go test -run TestDB_SyncReturnsDiskFullErrorForLTXStaging ./...go build ./...go test -race ./...pre-commit run --all-filesTypes of changes
Checklist
go fmt,go vet)go test ./...)