docs: HNSW snapshots are automatic in 1.39#475
Open
g-despot wants to merge 4 commits into
Open
Conversation
In v1.39 the HNSW commit log compactor owns the full on-disk lifecycle of the vector index. Snapshots are always created and managed automatically, and the five PERSISTENCE_HNSW_SNAPSHOT_* environment variables are inert no-ops that log a startup warning when set. Docs serve the latest three supported minors, so the pre-v1.39 configuration guidance is retained and scoped to v1.31 through v1.38 rather than removed. - configuration/hnsw-snapshots.md: lead with the v1.39 behavior and the list of ignored variables; move the existing configuration reference into a section scoped to earlier versions. - concepts/storage.md: describe snapshots as part of the storage engine rather than an opt-in speedup, cover the streaming on-disk merge and the crash-safety mechanisms, and scope the previous conditional creation and snapshot fallback behavior to earlier versions. - env-vars/index.md: mark the five variables deprecated in v1.39 while preserving what they did for readers on earlier versions. - feature-notes/hnsw-snapshots.mdx: note the v1.39 change on both pages that consume the include.
The v1.39 crash-safety paragraph said a compacted segment that cannot be read in full is dropped in favor of the snapshot and the remaining clean segments. That is not what the loader does. For both raw commit logs and compacted .sorted/.condensed segments it calls truncateToLastValidRecord, keeping every record written before the corruption and leaving the file valid for later compaction. Only the damaged tail is lost. The two outcomes differ in recall, so describe the retention explicitly and phrase both cases as the single mechanism they actually are. Note that the struct comment at compact/loader.go:73-74 and the matching one at hnsw/startup.go:101-102 still describe the discard behavior and contradict the implementation. Tracked separately on the core side.
The card read "Configure HNSW index snapshots for faster recovery and
startup." There is nothing to configure in v1.39, but the page still
serves v1.37 and v1.38 readers who can configure it, so the description
now leads with the outcome instead of the action and reads correctly for
both audiences.
Keeps the imperative benefit-first voice of the neighbouring cards
("Reduce memory usage with...", "Enable the built-in..."), and drops the
restatement of the card title, which the title already supplies.
Blocking items: - Move the v1.39 startup-failure behavior out of the historical note and into the main durability prose. It was the last sentence of a box titled "Behavior before v1.39", which told v1.39 readers to skip the highest-consequence fact in the release. - Split the durability paragraph in three so commit logs and snapshots are not conflated. Commit logs, including compacted ones, self-heal by truncation; snapshots are never truncated or repaired and an unreadable one fails the index load. - Stop calling HNSW compaction output a "segment", which contradicts "the vector index does not use segmentation" earlier on the same page. Vocabulary is now commit log, compacted commit log, snapshot, compactor. - Replace en dashes in the env-var rows with "through", matching the other two files. Also: - Compound title on the feature-note include, per the rq-8bit precedent. - Halve the repeated boilerplate in the five env-var rows and point the historical half at a new stable anchor on the pre-v1.39 section, so readers on earlier versions land on the part that applies to them. - Drop "periodically" from the snapshot lead-in (compaction is driven by the compactor, not a timer) and stop framing a snapshot as conditional. - Retitle both scoping notes to name v1.31 through v1.38 explicitly, since snapshots did not exist before v1.31. - Add a caution to the configuration page covering the fail-to-start behavior, which every deprecated env-var row links to. - Name the configuration file that silently accepts these settings. The recovery advice was verified against the compactor deleting its input files after a successful snapshot write, and the loader filtering covered commit logs, which together confirm no local copy survives.
![orca-security-eu[bot]](https://avatars.githubusercontent.com/in/276244?s=60&v=4){kind=small}
There was a problem hiding this comment.
Orca Security Scan Summary
| Status | Check | Issues by priority | |
|---|---|---|---|
 Passed |
Secrets |  0  0  0  0 |
View in Orca |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Compact v2 makes HNSW snapshots always-on and managed by the compactor. Five
PERSISTENCE_HNSW_SNAPSHOT_*variables become inert no-ops that log a startup warning.The docs site serves the latest three minors, and these variables still work on 1.37 and 1.38, so the configuration guidance is scoped to those versions rather than deleted.
hnsw-snapshots.mdsplit into a 1.39 section and a pre-1.39 sectionconcepts/storage.md: snapshots described as part of the storage engine rather than an opt-in speedupenv-vars/index.md: five rows marked deprecated, with what they did preserved and deep-linked to the pre-1.39 sectionBehaviour change worth reviewing closely. In 1.39 a snapshot that cannot be read fails index startup. Before 1.39 it was deleted and Weaviate fell back to replaying commit logs. There is no fallback now because the commit logs a snapshot covers are deleted once it is written (
compactor.go:648-666) and the loader filters them out of the replay set. Verified in both directions against v1.38.5 and main.If that regression is unintended and gets fixed before GA, the corresponding paragraph in
storage.mdmust be reverted.Also documented: the equivalent settings in
weaviate.conf.jsonare ignored too but produce no startup warning.Site build passes.