test(integration): add replicate restore soak test#1302
Open
corylanou wants to merge 4 commits into
Open
Conversation
PR Build Metrics✅ All clear — no issues detected
Binary Size
Dependency ChangesNo dependency changes. govulncheck OutputBuild Info
History (8 previous)
🤖 Updated on each push. |
4b68110 to
d2d5a2f
Compare
20c704b to
18ed80c
Compare
d2d5a2f to
374d420
Compare
18ed80c to
11627d8
Compare
374d420 to
2045e7f
Compare
11627d8 to
f4f0076
Compare
2045e7f to
b99709f
Compare
f4f0076 to
c7f0777
Compare
b99709f to
5f321ac
Compare
c7f0777 to
66314e3
Compare
5f321ac to
d41cec3
Compare
66314e3 to
cfa0759
Compare
d41cec3 to
209f2ba
Compare
cfa0759 to
11ae6fc
Compare
The soak only failed on integrity_check corruption: a restore that failed every cycle, a restored DB that could not be opened, or an integrity-valid but empty restore all reported PASSED. Now restore, open, count, and stop-litestream failures fail the test; restored row counts must be nonzero once the source has data and must never shrink between cycles; zero successful writes fails the run as applying no load. The P99 latency threshold is env-tunable via SOAK_P99_THRESHOLD and an explicit SOAK_DURATION wins over the -short default.
Wire TestSoakReplicateRestore into nightly-stability.yml so the #1164 regression test actually executes, and document it in the soak tables. Fail when no restore cycle ran, when the restore lags a full cycle behind the source, or when the source row count query errors. Replace fixed sleeps with a restore-probe poll and rely on helper-internal start/stop waits.
Keep restored DB files for cycles that recorded an error so the nightly artifact upload contains the corruption evidence. Fail the run when write errors exceed 1% of attempts or the source row count never grows past the first cycle. Fatal on set-but-unparsable SOAK_P99_THRESHOLD/SOAK_DURATION instead of silently defaulting.
209f2ba to
7393941
Compare
11ae6fc to
f170a0d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds
TestSoakReplicateRestore, an integration soak test that mirrors the issue #1164 replicate/restore corruption scenario against MinIO-backed S3 storage.This test was moved out of #1166 so #1166 can stay limited to the product fix and focused internal/unit tests.
Motivation and Context
The soak test exercises the longer-running reproduce-and-restore path separately from the smaller product-fix PR. It runs a WAL-mode SQLite database with indexed tables, writes under sustained load, periodically restores from S3-compatible storage, and checks
PRAGMA integrity_checkon each restored database.Refs #1164
Refs #1166
Refs #1281
How Has This Been Tested?
gofmtran on the added test file.The commit hook passed:
The full soak test was split out but not run in this branch. On the underlying #1166 stacked branch, the previously red WAL-growth integration test passed:
Types of changes
Checklist
go fmt,go vet)go test ./...)