docs: HNSW snapshots are automatic in 1.39

_includes/feature-notes/hnsw-snapshots.mdx

@@ -1,2 +1,5 @@
-:::info Added in `v1.31`
+:::info Added in `v1.31` · Changed in `v1.39`
+
+Starting in `v1.39`, HNSW snapshots are created and managed automatically, and are no longer configurable.
+
 :::

docs/deploy/configuration/env-vars/index.md

@@ -83,11 +83,11 @@ import APITable from '@site/src/components/APITable';
 | `OPERATIONAL_MODE` | Sets the [mode of operation](../status.md#operational-modes) for the instance. Options: `READ_WRITE` (default), `READ_ONLY`, `WRITE_ONLY`, `SCALE_OUT`. Limits available operations based on the mode selected. | `string` | `READ_WRITE` |
 | `ORIGIN` | Set the http(s) origin for Weaviate | `string - HTTP origin` | `https://my-weaviate-deployment.com` |
 | `PERSISTENCE_DATA_PATH` | Path to the Weaviate data store.<br/>[Note about file systems and performance](/weaviate/concepts/resources.md#file-system). | `string - file path` | `/var/lib/weaviate` <br/> Defaults to `./data`|
-| `PERSISTENCE_HNSW_DISABLE_SNAPSHOTS` | If set, [HNSW snapshotting](/weaviate/concepts/storage.md#persistence-and-crash-recovery) will be disabled. Default: `false` (enabled) as of `v1.36`; `true` (disabled) in `v1.31`–`v1.35`<br/>Added in `v1.31` | `boolean` | `false` |
-| `PERSISTENCE_HNSW_SNAPSHOT_INTERVAL_SECONDS` | The minimum time in seconds that must pass before the next snapshot is created. Default: `21600` seconds (6 hours)<br/>Added in `v1.31` | `string - number` | `3600` |
-| `PERSISTENCE_HNSW_SNAPSHOT_MIN_DELTA_COMMITLOGS_NUMBER` | The minimum number of new commit log files created since the last snapshot. Default: `1`<br/>Added in `v1.31` | `string - number` | `100` |
-| `PERSISTENCE_HNSW_SNAPSHOT_MIN_DELTA_COMMITLOGS_SIZE_PERCENTAGE` | The minimum total size of new commit logs (as a percentage of the previous snapshot's size) required to trigger a new snapshot. Default: `5` (meaning 5% of the previous snapshot's size in new commit logs)<br/>Added in `v1.31` | `string - number` | `15` |
-| `PERSISTENCE_HNSW_SNAPSHOT_ON_STARTUP` | If set, Weaviate will try to create a new snapshot during startup if there are changes in the commit log since the last snapshot. If there are no changes, then the existing snapshot will be loaded. Default: `true`<br/>Added in `v1.31` | `boolean` | `false` |
+| `PERSISTENCE_HNSW_DISABLE_SNAPSHOTS` | **Deprecated in `v1.39`.** Ignored, and logs a startup warning. [In `v1.31` through `v1.38`](/weaviate/configuration/hnsw-snapshots.md#pre-v1-39-configuration): if set, HNSW snapshotting will be disabled. Default: `false` (enabled) as of `v1.36`; `true` (disabled) in `v1.31` through `v1.35`<br/>Added in `v1.31` | `boolean` | `false` |
+| `PERSISTENCE_HNSW_SNAPSHOT_INTERVAL_SECONDS` | **Deprecated in `v1.39`.** Ignored, and logs a startup warning. [In `v1.31` through `v1.38`](/weaviate/configuration/hnsw-snapshots.md#pre-v1-39-configuration): the minimum time in seconds that must pass before the next snapshot is created. Default: `21600` seconds (6 hours)<br/>Added in `v1.31` | `string - number` | `3600` |
+| `PERSISTENCE_HNSW_SNAPSHOT_MIN_DELTA_COMMITLOGS_NUMBER` | **Deprecated in `v1.39`.** Ignored, and logs a startup warning. [In `v1.31` through `v1.38`](/weaviate/configuration/hnsw-snapshots.md#pre-v1-39-configuration): the minimum number of new commit log files created since the last snapshot. Default: `1`<br/>Added in `v1.31` | `string - number` | `100` |
+| `PERSISTENCE_HNSW_SNAPSHOT_MIN_DELTA_COMMITLOGS_SIZE_PERCENTAGE` | **Deprecated in `v1.39`.** Ignored, and logs a startup warning. [In `v1.31` through `v1.38`](/weaviate/configuration/hnsw-snapshots.md#pre-v1-39-configuration): the minimum total size of new commit logs (as a percentage of the previous snapshot's size) required to trigger a new snapshot. Default: `5` (meaning 5% of the previous snapshot's size in new commit logs)<br/>Added in `v1.31` | `string - number` | `15` |
+| `PERSISTENCE_HNSW_SNAPSHOT_ON_STARTUP` | **Deprecated in `v1.39`.** Ignored, and logs a startup warning. [In `v1.31` through `v1.38`](/weaviate/configuration/hnsw-snapshots.md#pre-v1-39-configuration): if set, Weaviate will try to create a new snapshot during startup if there are changes in the commit log since the last snapshot. If there are no changes, then the existing snapshot will be loaded. Default: `true`<br/>Added in `v1.31` | `boolean` | `false` |
 | `PERSISTENCE_HNSW_MAX_LOG_SIZE` | Maximum size of the HNSW [write-ahead-log](/weaviate/concepts/storage.md#hnsw-vector-index-storage). Increase this to improve log compaction efficiency, or decrease to reduce memory requirements. Default: 500MiB | `string` | `4GiB` (IEC units), `4GB` (SI units), `4000000000` (bytes) |
 | `PERSISTENCE_LSM_ACCESS_STRATEGY` | Function used to access disk data in virtual memory. Default: `mmap` | `string` | `mmap` or `pread` |
 | `PERSISTENCE_LSM_MAX_SEGMENT_SIZE` | Maximum size of a segment in the [LSM store](/weaviate/concepts/storage.md#object-and-inverted-index-store). Set this to limit disk usage spikes during compaction to ~2x the segment size. Default: no limit | `string` | `4GiB` (IEC units), `4GB` (SI units), `4000000000` (bytes) |

docs/weaviate/concepts/storage.md

@@ -101,35 +101,35 @@ For the HNSW vector index, the Write-Ahead-Log (WAL) is a critical component for
 
 The entire HNSW index state can be reconstructed by replaying these WAL entries.
 
-For very large indexes of tens or hundreds of millions of objects, this can be time-consuming. If you have a large index and you want to speed up the startup time, you can use the **[HNSW snapshots](../configuration/hnsw-snapshots.md)** feature.
+For very large indexes of tens or hundreds of millions of objects, this can be time-consuming. To avoid replaying the entire WAL on every restart, Weaviate writes **[HNSW snapshots](../configuration/hnsw-snapshots.md)**.
 
 ### HNSW snapshots
 
 import HnswSnapshots from '/_includes/feature-notes/hnsw-snapshots.mdx';
 
 <HnswSnapshots/>
 
-For very large HNSW vector indexes, HNSW snapshots can significantly reduce the startup time.
+For very large HNSW vector indexes, HNSW snapshots significantly reduce the startup time.
 
-A snapshot represents a point-in-time state of the HNSW index. When Weaviate starts, if a valid snapshot exists, it will be loaded into memory first. This significantly reduces startup time, as the number of WAL entries that need to be processed, as only the changes made after the snapshot was taken need to be replayed from the WAL.
+A snapshot represents a point-in-time state of the HNSW index. When Weaviate starts, it loads the most recent snapshot and replays only the commit log entries written after it. This significantly reduces startup time, because the number of entries that have to be replayed no longer grows with the age of the index.
 
-If a snapshot cannot be loaded for any reason, it is safely removed, and Weaviate falls back to the traditional method of loading the full commit log from the beginning, ensuring resilience.
+The commit log still persists every change immediately, guaranteeing that any acknowledged write is durable. Even with a fresh snapshot, the server typically still has to load at least one subsequent commit log file.
 
-Snapshots can be created at startup and periodically based on time passed or changes in the commit log.
+Starting in `v1.39`, snapshots are part of how the vector index is stored rather than an optional speedup, and Weaviate manages them automatically. A background compactor owns the on-disk lifecycle of the index: it compacts newly flushed commit logs, merges them together, and writes a new snapshot when doing so is worthwhile. Snapshots and commit logs live in the same directory, and a snapshot replaces the commit logs it covers rather than duplicating them, which keeps the disk footprint proportional to the size of the index.
 
-Weaviate will try to create a new snapshot during startup if there are changes in the commit log since the last snapshot. If there are no changes, then the existing snapshot will be loaded.
+Because the merge reads its inputs as sorted streams from disk, only the most recently flushed commit log is processed in memory. That step has a fixed cost regardless of how large the graph is, so snapshot creation no longer requires enough memory to hold the previous snapshot plus the commit log delta.
 
-A snapshot will also be created if the corresponding conditions are met, meaning the specified time interval has passed and a sufficient number of new commits exist. This is handled by the same background process that manages commit log combining and condensing, ensuring stability as the commit logs used for snapshot creation are not mutable during this process.
+Weaviate protects this on-disk state in several ways. New files are written to a temporary path and atomically renamed into place, so an interrupted write can never be mistaken for a complete file, and orphaned temporary files are cleaned up on the next startup.
 
-Each new HNSW snapshot is based on the previous snapshot and newer (delta) commit logs.
+Commit logs, including compacted ones, are self-healing. If one cannot be read in full, whether because a crash tore its tail or because the file was damaged on disk, it is truncated back to its last valid entry. The entries written before the damage are retained and the file becomes valid again for later compaction, so only the damaged tail is lost.
 
-It's important to note that even with a fresh snapshot, the server typically still has to load at least one subsequent commit log file.
+Snapshots are handled differently. A snapshot is stored in a checksummed block format, and it is never truncated or repaired. If the current snapshot cannot be read, the affected vector index fails to start rather than loading partial data. Because the commit logs a snapshot covers are deleted once it has been written, nothing remains on the node to replay in its place, so recovering that shard means restoring the data, for example from a [backup](/deploy/configuration/backups.md).
 
-The WAL is still used to persist every change immediately, guaranteeing that any acknowledged write is durable. Over time, the append-only WAL will contain redundant information for operations occurring after the last snapshot. A background process continuously compacts these newer WAL files, removing redundant information. This, combined with snapshotting, keeps the disk footprint manageable and startup times fast.
+See **[the HNSW snapshots configuration](../configuration/hnsw-snapshots.md)** for version-specific details.
 
-See **[the HNSW snapshots configuration](../configuration/hnsw-snapshots.md)** for more details on how to configure this feature.
-
-Starting in `v1.36`, HNSW snapshots are enabled by default. In `v1.31` through `v1.35`, they are disabled by default.
+:::note Behavior in `v1.31` through `v1.38`
+In these versions, snapshots are an optional feature layered on top of the commit log, and are configured with the `PERSISTENCE_HNSW_SNAPSHOT_*` environment variables. They are enabled by default starting in `v1.36`, and disabled by default in `v1.31` through `v1.35`. Weaviate creates a snapshot at startup if the commit log changed since the last snapshot, and periodically once a configured time interval has passed and enough new commit log data has accumulated. If a snapshot cannot be loaded, it is removed and Weaviate falls back to loading the full commit log from the beginning.
+:::
 
 ## Conclusions
 

docs/weaviate/configuration/hnsw-snapshots.md

@@ -2,29 +2,55 @@
 title: HNSW Snapshots
 sidebar_position: 47
 sidebar_label: HNSW Snapshots
-description: Learn about HNSW snapshots in Weaviate for faster startup times and how to manage them.
+description: Learn how Weaviate uses HNSW snapshots for faster startup times, and how they were configured in versions before v1.39.
 ---
 
 import HnswSnapshots from '/_includes/feature-notes/hnsw-snapshots.mdx';
 
 <HnswSnapshots/>
 
-HNSW (Hierarchical Navigable Small World) snapshots can significantly reduce startup times for instances with large vector indexes.
-
-HNSW snapshotting is **enabled by default** starting in `v1.36`. To disable it, set `PERSISTENCE_HNSW_DISABLE_SNAPSHOTS` to `true`.
-
-In versions prior to `v1.36`, HNSW snapshotting is disabled by default. Set `PERSISTENCE_HNSW_DISABLE_SNAPSHOTS` to `false` to enable it.
+HNSW (Hierarchical Navigable Small World) snapshots significantly reduce startup times for instances with large vector indexes.
 
 :::info Concepts: HNSW snapshots
 See this [concepts page](../concepts/storage.md#hnsw-snapshots) for a detailed description.
 :::
 
-## Configuring snapshot creation
+## Snapshots in `v1.39` and later
+
+Starting in `v1.39`, HNSW snapshots are always enabled and are not configurable.
+
+The commit log compactor owns the on-disk lifecycle of the vector index, and snapshots are one of the file formats it produces. Weaviate creates and maintains them automatically, so there is nothing to enable, disable, schedule, or tune.
+
+:::caution Snapshots are now required to start
+Because a snapshot replaces the commit logs it covers, it is the only copy of the index state up to that point. If the current snapshot cannot be read, the affected vector index fails to start rather than loading partial data, and it cannot be rebuilt from the node's remaining files. Recovering that shard means restoring the data, for example from a [backup](/deploy/configuration/backups.md). In `v1.31` through `v1.38`, an unreadable snapshot is discarded and Weaviate replays the full commit log instead.
+:::
+
+### Environment variables that no longer have an effect
+
+The following environment variables are still recognized, so existing deployments continue to start without a configuration error. However, they are ignored:
+
+- `PERSISTENCE_HNSW_DISABLE_SNAPSHOTS`
+- `PERSISTENCE_HNSW_SNAPSHOT_INTERVAL_SECONDS`
+- `PERSISTENCE_HNSW_SNAPSHOT_ON_STARTUP`
+- `PERSISTENCE_HNSW_SNAPSHOT_MIN_DELTA_COMMITLOGS_NUMBER`
+- `PERSISTENCE_HNSW_SNAPSHOT_MIN_DELTA_COMMITLOGS_SIZE_PERCENTAGE`
+
+If any of these variables is set, Weaviate logs a warning at startup that names the variable and states that it has no effect and will be removed in a future version. Variables that are not set produce no warning. To clear the warnings, remove the variables from your deployment configuration.
+
+The same settings in a Weaviate configuration file (`weaviate.conf.json` by default) are accepted and ignored too, but they produce no startup warning.
+
+## Configuring snapshot creation in versions before `v1.39` {#pre-v1-39-configuration}
+
+:::note Applies to `v1.31` through `v1.38`
+In `v1.39` and later, the environment variables below are ignored.
+:::
+
+In these versions, HNSW snapshotting is an optional feature layered on top of the commit log, and the following environment variables control it.
 
-Set the following optional environment variables to configure the snapshotting behavior.
+HNSW snapshotting is **enabled by default** starting in `v1.36`. To disable it, set `PERSISTENCE_HNSW_DISABLE_SNAPSHOTS` to `true`. In versions prior to `v1.36`, HNSW snapshotting is disabled by default. Set `PERSISTENCE_HNSW_DISABLE_SNAPSHOTS` to `false` to enable it.
 
 :::note
-Before creating a new snapshot, the previous snapshot and the commit log difference need to be loaded into memory. Make sure you have enough memory to accommodate this process.
+Before creating a new snapshot, the previous snapshot and the commit log difference need to be loaded into memory. Make sure you have enough memory to accommodate this process. This requirement does not apply in `v1.39` and later, where snapshots are merged from disk as a stream.
 :::
 
 ### Snapshot on startup

docs/weaviate/configuration/index.mdx

@@ -28,7 +28,7 @@ export const configOpsData = [
   {
     title: "HNSW Snapshots",
     description:
-      "Configure HNSW index snapshots for faster recovery and startup.",
+      "Speed up startup and recovery for instances with large vector indexes.",
     link: "/weaviate/configuration/hnsw-snapshots",
     icon: "fas fa-camera",
   },