Skip to content

fix(db): surface disk-full staging errors#1322

Open
corylanou wants to merge 4 commits into
issue-1164-soak-replicate-restore-testfrom
issue-1310-disk-full-staging-wedge
Open

fix(db): surface disk-full staging errors#1322
corylanou wants to merge 4 commits into
issue-1164-soak-replicate-restore-testfrom
issue-1310-disk-full-staging-wedge

Conversation

@corylanou

Copy link
Copy Markdown
Collaborator

Description

Adds a typed LTXStagingDiskFullError plus ErrDiskFull for local LTX staging failures, and wraps ENOSPC/EDQUOT from the L0 staging open/write/encoder close/fsync/close path before it reaches the generic sync loop.

The DB monitor now logs a clear sync stopped: disk full while staging ltx file error with the staging path and LTX identity, then exits instead of retrying the same generic sync error forever.

This PR is stacked on the issue-1164-soak-replicate-restore-test stability cluster branch.

Motivation and Context

Issue #1310 reports v0.5.x initial sync staging the full snapshot under .{db}-litestream/ltx/0/ before upload. In the reported case, a 22 GB database with about 20 GB free filled the disk during .ltx.tmp creation, Litestream stayed active, no object reached the replica, and the journal had no clear ERROR/WARN signal.

This PR is intentionally scoped to surfacing and fail-fast handling of the local disk-full staging condition. It does not change the snapshot staging architecture or stream initial snapshots directly to the replica.

Fixes #1310

How Has This Been Tested?

  • go test -run TestDB_SyncReturnsDiskFullErrorForLTXStaging ./...
  • go build ./...
  • go test -race ./...
  • pre-commit run --all-files

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (would cause existing functionality to not work as expected)

Checklist

  • My code follows the code style of this project (go fmt, go vet)
  • I have tested my changes (go test ./...)
  • I have updated the documentation accordingly (if needed)

@github-actions

github-actions Bot commented Jun 25, 2026

Copy link
Copy Markdown

PR Build Metrics

All clear — no issues detected

Check Status Summary
Binary size 36.98 MB (+8.0 KB / +0.02%)
Dependencies No changes
Vulnerabilities None detected
Go toolchain 1.25.12 (latest)
Module graph 1240 edges (0)

Binary Size

Size Change
Base (906619b) 36.98 MB
PR (14a4dac) 36.98 MB +8.0 KB (+0.02%)

Dependency Changes

No dependency changes.

govulncheck Output

=== Symbol Results ===

No vulnerabilities found.

Your code is affected by 0 vulnerabilities.
This scan also found 0 vulnerabilities in packages you import and 1
vulnerability in modules you require, but your code doesn't appear to call these
vulnerabilities.
Use '-show verbose' for more details.

Build Info

Metric Value
Build time 32s
Go version go1.25.12
Commit 14a4dac

History (8 previous)

Commit Updated Status Summary
1c098fc 2026-07-16 15:08 UTC 36.98 MB (+8.0 KB / +0.02%)
c7488a1 2026-07-08 20:45 UTC 36.94 MB (+4.0 KB / +0.01%)
05faeb0 2026-07-08 20:40 UTC 36.94 MB (+8.0 KB / +0.02%)
0282358 2026-07-08 18:54 UTC 36.94 MB (+8.0 KB / +0.02%)
b814779 2026-07-08 18:50 UTC 36.94 MB (+8.0 KB / +0.02%)
2f795f5 2026-07-02 03:44 UTC 36.93 MB (+8.0 KB / +0.02%)
d361760 2026-06-26 15:06 UTC 36.93 MB (+12.0 KB / +0.03%)
0ae9081 2026-06-25 01:59 UTC 36.93 MB (+8.0 KB / +0.02%)

🤖 Updated on each push.

@corylanou corylanou force-pushed the issue-1164-soak-replicate-restore-test branch from 11627d8 to f4f0076 Compare June 26, 2026 15:02
@corylanou corylanou force-pushed the issue-1310-disk-full-staging-wedge branch from 099f814 to 978be41 Compare June 26, 2026 15:02
@corylanou corylanou force-pushed the issue-1164-soak-replicate-restore-test branch from f4f0076 to c7f0777 Compare July 2, 2026 03:40
@corylanou corylanou force-pushed the issue-1310-disk-full-staging-wedge branch from 978be41 to ca46d85 Compare July 2, 2026 03:40
@corylanou corylanou force-pushed the issue-1164-soak-replicate-restore-test branch from c7f0777 to 66314e3 Compare July 8, 2026 18:48
@corylanou corylanou force-pushed the issue-1310-disk-full-staging-wedge branch from ca46d85 to 4b06238 Compare July 8, 2026 18:48
@github-actions github-actions Bot added metrics: vulns-found govulncheck found vulnerabilities metrics: go-update Go toolchain has a newer patch release labels Jul 8, 2026
@corylanou corylanou force-pushed the issue-1164-soak-replicate-restore-test branch from 66314e3 to cfa0759 Compare July 8, 2026 18:50
@corylanou corylanou force-pushed the issue-1310-disk-full-staging-wedge branch from 4b06238 to 68c0661 Compare July 8, 2026 18:50
@corylanou corylanou force-pushed the issue-1164-soak-replicate-restore-test branch from cfa0759 to 11ae6fc Compare July 8, 2026 20:39
@corylanou corylanou force-pushed the issue-1310-disk-full-staging-wedge branch from 68c0661 to ce69343 Compare July 8, 2026 20:39
@corylanou corylanou force-pushed the issue-1164-soak-replicate-restore-test branch from 11ae6fc to f170a0d Compare July 8, 2026 20:41
@corylanou corylanou force-pushed the issue-1310-disk-full-staging-wedge branch from ce69343 to f2851b1 Compare July 8, 2026 20:41
@github-actions github-actions Bot removed metrics: vulns-found govulncheck found vulnerabilities metrics: go-update Go toolchain has a newer patch release labels Jul 8, 2026
@corylanou corylanou force-pushed the issue-1164-soak-replicate-restore-test branch from f170a0d to 906619b Compare July 16, 2026 15:04
Classify ENOSPC from staging-directory creation as a disk-full error so
the gauge cannot report recovered while the disk is still full. Use a
constant log message with attributes for the disk-full monitor log and
surface the backoff ceiling. Replace the package-global openLTXFile
seam with a per-DB field, move the error constructor next to its type,
drop the always-zero Level field, and cover the open/close failure
paths plus gauge reset on non-disk-full errors.
Replace LTXStagingDiskFullError with the established LTXError plus
wrapped ErrDiskFull sentinel pattern. Key the disk-full gauge on
isDiskFullError to match the monitor cleanup predicate, broaden
detection with syscall.ENOSPC and Windows/SQLite phrasings, and log
a single hex txid key per house style.
@corylanou corylanou force-pushed the issue-1310-disk-full-staging-wedge branch from f2851b1 to 9566a1a Compare July 16, 2026 15:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants