Skip to content

test(integration): add replicate restore soak test#1302

Open
corylanou wants to merge 4 commits into
issue-1164-database-gets-corrupted-randomlyfrom
issue-1164-soak-replicate-restore-test
Open

test(integration): add replicate restore soak test#1302
corylanou wants to merge 4 commits into
issue-1164-database-gets-corrupted-randomlyfrom
issue-1164-soak-replicate-restore-test

Conversation

@corylanou

@corylanou corylanou commented Jun 8, 2026

Copy link
Copy Markdown
Collaborator

📍 Stack map: part of a stacked series — see the canonical PR map for review order, scope, and current blockers.

Description

Adds TestSoakReplicateRestore, an integration soak test that mirrors the issue #1164 replicate/restore corruption scenario against MinIO-backed S3 storage.

This test was moved out of #1166 so #1166 can stay limited to the product fix and focused internal/unit tests.

Motivation and Context

The soak test exercises the longer-running reproduce-and-restore path separately from the smaller product-fix PR. It runs a WAL-mode SQLite database with indexed tables, writes under sustained load, periodically restores from S3-compatible storage, and checks PRAGMA integrity_check on each restored database.

Refs #1164
Refs #1166
Refs #1281

How Has This Been Tested?

gofmt ran on the added test file.

The commit hook passed:

trim trailing whitespace: Passed
fix end of files: Passed
check yaml: Skipped
go-imports-repo: Passed
go-vet-repo-mod: Passed
go-staticcheck-repo-mod: Passed

The full soak test was split out but not run in this branch. On the underlying #1166 stacked branch, the previously red WAL-growth integration test passed:

go test -tags "integration,soak,docker" -run TestWALGrowth -v -timeout 10m ./tests/integration/

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (would cause existing functionality to not work as expected)

Checklist

  • My code follows the code style of this project (go fmt, go vet)
  • I have tested my changes (go test ./...)
  • I have updated the documentation accordingly (if needed)

@github-actions

github-actions Bot commented Jun 8, 2026

Copy link
Copy Markdown

PR Build Metrics

All clear — no issues detected

Check Status Summary
Binary size 36.98 MB (0.0 KB / 0.00%)
Dependencies No changes
Vulnerabilities None detected
Go toolchain 1.25.12 (latest)
Module graph 1240 edges (0)

Binary Size

Size Change
Base (7393941) 36.98 MB
PR (697a111) 36.98 MB 0.0 KB (0.00%)

Dependency Changes

No dependency changes.

govulncheck Output

=== Symbol Results ===

No vulnerabilities found.

Your code is affected by 0 vulnerabilities.
This scan also found 0 vulnerabilities in packages you import and 1
vulnerability in modules you require, but your code doesn't appear to call these
vulnerabilities.
Use '-show verbose' for more details.

Build Info

Metric Value
Build time 41s
Go version go1.25.12
Commit 697a111

History (8 previous)

Commit Updated Status Summary
9383210 2026-07-08 20:44 UTC 36.93 MB (0.0 KB / 0.00%)
9f3d4ce 2026-07-08 20:40 UTC 36.93 MB (0.0 KB / 0.00%)
e61e8f7 2026-07-08 18:54 UTC 36.93 MB (0.0 KB / 0.00%)
19ce31d 2026-07-08 18:50 UTC 36.93 MB (0.0 KB / 0.00%)
e19a923 2026-07-02 03:44 UTC 36.93 MB (0.0 KB / 0.00%)
b300a4c 2026-06-26 15:05 UTC 36.92 MB (0.0 KB / 0.00%)
eee1186 2026-06-09 20:01 UTC 36.92 MB (0.0 KB / 0.00%)
00ba633 2026-06-09 14:00 UTC 36.92 MB (0.0 KB / 0.00%)

🤖 Updated on each push.

@github-actions github-actions Bot added metrics: vulns-found govulncheck found vulnerabilities metrics: go-update Go toolchain has a newer patch release labels Jun 8, 2026
@corylanou corylanou force-pushed the issue-1164-database-gets-corrupted-randomly branch from 4b68110 to d2d5a2f Compare June 9, 2026 13:57
@corylanou corylanou force-pushed the issue-1164-soak-replicate-restore-test branch from 20c704b to 18ed80c Compare June 9, 2026 13:58
@github-actions github-actions Bot removed metrics: vulns-found govulncheck found vulnerabilities metrics: go-update Go toolchain has a newer patch release labels Jun 9, 2026
@corylanou corylanou force-pushed the issue-1164-database-gets-corrupted-randomly branch from d2d5a2f to 374d420 Compare June 9, 2026 19:57
@corylanou corylanou force-pushed the issue-1164-soak-replicate-restore-test branch from 18ed80c to 11627d8 Compare June 9, 2026 19:59
@corylanou corylanou force-pushed the issue-1164-database-gets-corrupted-randomly branch from 374d420 to 2045e7f Compare June 26, 2026 15:02
@corylanou corylanou force-pushed the issue-1164-soak-replicate-restore-test branch from 11627d8 to f4f0076 Compare June 26, 2026 15:02
@corylanou corylanou force-pushed the issue-1164-database-gets-corrupted-randomly branch from 2045e7f to b99709f Compare July 2, 2026 03:40
@corylanou corylanou force-pushed the issue-1164-soak-replicate-restore-test branch from f4f0076 to c7f0777 Compare July 2, 2026 03:40
@corylanou corylanou force-pushed the issue-1164-database-gets-corrupted-randomly branch from b99709f to 5f321ac Compare July 8, 2026 18:48
@corylanou corylanou force-pushed the issue-1164-soak-replicate-restore-test branch from c7f0777 to 66314e3 Compare July 8, 2026 18:48
@github-actions github-actions Bot added metrics: vulns-found govulncheck found vulnerabilities metrics: go-update Go toolchain has a newer patch release labels Jul 8, 2026
@corylanou corylanou force-pushed the issue-1164-database-gets-corrupted-randomly branch from 5f321ac to d41cec3 Compare July 8, 2026 18:50
@corylanou corylanou force-pushed the issue-1164-soak-replicate-restore-test branch from 66314e3 to cfa0759 Compare July 8, 2026 18:50
@corylanou corylanou force-pushed the issue-1164-database-gets-corrupted-randomly branch from d41cec3 to 209f2ba Compare July 8, 2026 20:39
@corylanou corylanou force-pushed the issue-1164-soak-replicate-restore-test branch from cfa0759 to 11ae6fc Compare July 8, 2026 20:39
corylanou added 3 commits July 8, 2026 15:40
The soak only failed on integrity_check corruption: a restore that
failed every cycle, a restored DB that could not be opened, or an
integrity-valid but empty restore all reported PASSED. Now restore,
open, count, and stop-litestream failures fail the test; restored row
counts must be nonzero once the source has data and must never shrink
between cycles; zero successful writes fails the run as applying no
load. The P99 latency threshold is env-tunable via SOAK_P99_THRESHOLD
and an explicit SOAK_DURATION wins over the -short default.
Wire TestSoakReplicateRestore into nightly-stability.yml so the #1164
regression test actually executes, and document it in the soak tables.
Fail when no restore cycle ran, when the restore lags a full cycle
behind the source, or when the source row count query errors. Replace
fixed sleeps with a restore-probe poll and rely on helper-internal
start/stop waits.
Keep restored DB files for cycles that recorded an error so the
nightly artifact upload contains the corruption evidence. Fail the
run when write errors exceed 1% of attempts or the source row count
never grows past the first cycle. Fatal on set-but-unparsable
SOAK_P99_THRESHOLD/SOAK_DURATION instead of silently defaulting.
@corylanou corylanou force-pushed the issue-1164-database-gets-corrupted-randomly branch from 209f2ba to 7393941 Compare July 8, 2026 20:41
@corylanou corylanou force-pushed the issue-1164-soak-replicate-restore-test branch from 11ae6fc to f170a0d Compare July 8, 2026 20:41
@github-actions github-actions Bot removed metrics: vulns-found govulncheck found vulnerabilities metrics: go-update Go toolchain has a newer patch release labels Jul 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants