From 018e776c111142625ee1a2e4fc7daa4a2ebe514a Mon Sep 17 00:00:00 2001 From: claudespice Date: Sat, 11 Jul 2026 03:15:28 -0700 Subject: [PATCH] docs(cayenne): datalake promotion is incremental carry-forward, not whole-table spiceai/spiceai#11745 rewrote cold-tier (datalake) promotion from whole-table re-materialization to incremental carry-forward: - dirty/clean manifest classification via per-PK-column stat rectangles refined by per-file PK bloom filters (conservative in the safe direction) - clean files carried forward by manifest reference, never re-read - per-file PK blooms serve upsert cold-tier key existence without a scan - periodic mark-and-sweep physical GC (cayenne_datalake_gc_interval_ms, default 5m, doubling as the orphan grace period) The published How-it-works section still described whole-table re-materialization; correct the promotion + read-path bullets and add a physical-GC bullet to match current trunk. --- website/docs/components/data-accelerators/cayenne/index.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/website/docs/components/data-accelerators/cayenne/index.md b/website/docs/components/data-accelerators/cayenne/index.md index 755f5a06e..eae43225c 100644 --- a/website/docs/components/data-accelerators/cayenne/index.md +++ b/website/docs/components/data-accelerators/cayenne/index.md @@ -612,8 +612,9 @@ The cold tier lets a table grow beyond local NVMe capacity while keeping recent, ### How it works -- **Promotion (write path).** A dedicated background worker graduates the warm tier once the size or file-count threshold (`cayenne_cold_tier_warm_max_bytes` / `cayenne_cold_tier_warm_max_files`) is crossed, evaluated every `cayenne_cold_tier_background_interval_ms`. Promotion re-materializes the visible table (all deletes applied, one version per key), **Z-order clusters** it for tight multi-column zone maps, writes read-optimized Vortex at the larger `cayenne_cold_target_file_size_mb`, and atomically registers the cold files while clearing the promoted warm files in a single transaction. -- **Cross-tier scan (read path).** Queries span all tiers and push filters, projection, and limits down to each. Cold files are pruned from statistics held in the metastore, so pruning requires **no object-store round-trip on the query path**. A `DELETE` after promotion correctly hides a cold-resident row. +- **Promotion (write path).** A dedicated background worker graduates the warm tier once the size or file-count threshold (`cayenne_cold_tier_warm_max_bytes` / `cayenne_cold_tier_warm_max_files`) is crossed, evaluated every `cayenne_cold_tier_background_interval_ms`. Promotion is **incremental carry-forward**: rather than re-materializing the whole table each cycle, it classifies the existing cold manifest into *dirty* files that may host a tombstoned key — using per-PK-column min/max rectangles from each file's persisted statistics, refined by a per-file PK bloom filter, and conservative in the safe direction (a false positive costs an extra rewrite; a missed tombstone is impossible) — and *clean* files that are provably untouched. Only the warm delta plus the dirty files are re-read (all deletes applied, one version per key), **Z-order clustered** for tight multi-column zone maps, and written as read-optimized Vortex at the larger `cayenne_cold_target_file_size_mb` under a per-promotion prefix; clean files are carried forward by manifest reference and never re-read. The new files are atomically registered while the promoted warm files are cleared in a single transaction, so promotion cost tracks the *changed* data, not total table size. +- **Cross-tier scan (read path).** Queries span all tiers and push filters, projection, and limits down to each. Cold files are pruned from statistics held in the metastore, so pruning requires **no object-store round-trip on the query path**. Each cold file's PK bloom lets an upsert's keyset rebuild answer cold-tier key existence without scanning the object store. A `DELETE` after promotion correctly hides a cold-resident row. +- **Physical GC.** A periodic mark-and-sweep, rooted at the manifest, reclaims cold objects orphaned by carry-forward rewrites. It runs every `cayenne_datalake_gc_interval_ms` (default 5m), which doubles as the orphan grace period: an object no longer referenced by the manifest is deleted only after it has been observed orphaned for at least one interval. - **Clustering.** Cold files are clustered by `cayenne_datalake_clustering_columns` (multi-column Z-order / Morton order), falling back to `cayenne_sort_columns` and then the primary key. Clustering on more than one dimension prunes far better than a single-column sort for selective queries on any clustering column. ### Requirements and v1 constraints