Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions adapter/redis.go
Original file line number Diff line number Diff line change
Expand Up @@ -204,7 +204,7 @@ type RedisServer struct {
// docs/design/2026_05_21_proposed_txn_secondary_idempotency.md). The
// FSM probe ships on every node in production, satisfying R5 (FSM
// determinism across a rolling upgrade), so the gate now defaults on
// per docs/design/2026_06_10_proposed_redis_onephase_dedup_default_on.md.
// per docs/design/2026_06_10_implemented_redis_onephase_dedup_default_on.md.
// Set ELASTICKV_REDIS_ONEPHASE_DEDUP=0 (or WithOnePhaseTxnDedup(false))
// to opt out — kept as a one-env-var operator rollback.
onePhaseTxnDedup bool
Expand All @@ -231,7 +231,7 @@ type RedisServerOption func(*RedisServer)
// WithOnePhaseTxnDedup enables (or disables) the option-2 one-phase
// idempotency dedup on list-push and MULTI/EXEC retries
// (see RedisServer.onePhaseTxnDedup). On by default since the rollout
// recorded in docs/design/2026_06_10_proposed_redis_onephase_dedup_default_on.md;
// recorded in docs/design/2026_06_10_implemented_redis_onephase_dedup_default_on.md;
// pass false to opt out from code, or set ELASTICKV_REDIS_ONEPHASE_DEDUP=0
// to opt out from the environment. The constructor option trumps the env var.
// Standalone SET requires the separate WithStandaloneSetDedup gate; see
Expand Down Expand Up @@ -493,7 +493,7 @@ func NewRedisServer(listen net.Listener, redisAddr string, store store.MVCCStore
// onePhaseTxnDedup defaults on — the parent design's R5 rolling-upgrade
// constraint is discharged (FSM probe shipped on every node months ago,
// 12 consecutive green dedup-mode Jepsen runs 2026-05-31 → 2026-06-10).
// See docs/design/2026_06_10_proposed_redis_onephase_dedup_default_on.md.
// See docs/design/2026_06_10_implemented_redis_onephase_dedup_default_on.md.
// ELASTICKV_REDIS_ONEPHASE_DEDUP=0 opts out; the WithOnePhaseTxnDedup
// constructor option still trumps the env var.
onePhaseTxnDedup: os.Getenv("ELASTICKV_REDIS_ONEPHASE_DEDUP") != "0",
Expand Down
2 changes: 1 addition & 1 deletion adapter/s3_admin.go
Original file line number Diff line number Diff line change
Expand Up @@ -379,7 +379,7 @@ func (s *S3Server) AdminPutBucketAcl(ctx context.Context, principal AdminPrincip
// orphans left by any PutObject that committed
// chunks/manifest between the empty-probe and the
// Phase-1 commit. See design doc
// 2026_04_28_proposed_admin_delete_bucket_safety_net.md
// 2026_04_28_implemented_admin_delete_bucket_safety_net.md
// §6.2 for the original single-OperationGroup design
// and the dispatch-shape rejection that forced the
// two-phase split.
Expand Down
2 changes: 1 addition & 1 deletion adapter/s3_admin_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -348,7 +348,7 @@ func TestS3Server_AdminListBuckets_PaginatesPastSinglePage(t *testing.T) {

// TestS3Server_AdminDeleteBucket_SweepsOrphansAcrossAllPerBucketPrefixes
// is the regression test for the AdminDeleteBucket TOCTOU race
// (design doc 2026_04_28_proposed_admin_delete_bucket_safety_net.md;
// (design doc 2026_04_28_implemented_admin_delete_bucket_safety_net.md;
// coderabbitai 🔴/🟠 on PR #669). The race lands when a concurrent
// PutObject inserts data between AdminDeleteBucket's empty-probe
// scan (at readTS) and its commit (at a later commitTS). Without
Expand Down
2 changes: 1 addition & 1 deletion adapter/sqs_catalog.go
Original file line number Diff line number Diff line change
Expand Up @@ -533,7 +533,7 @@ var sqsAttributeAppliers = map[string]attributeApplier{
"DeduplicationScope must be 'messageGroup' or 'queue'")
},
// Throttle* are non-AWS extensions for per-queue rate limiting,
// see docs/design/2026_04_26_proposed_sqs_per_queue_throttling.md.
// see docs/design/2026_04_26_implemented_sqs_per_queue_throttling.md.
// Each accepts a non-negative float64; the cross-attribute
// validation that enforces both-zero-or-both-positive on each
// (capacity, refill) pair, capacity ≥ refill, hard ceiling, and
Expand Down
2 changes: 1 addition & 1 deletion adapter/sqs_throttle.go
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ import (
)

// Per-queue throttling — token-bucket store that hangs off *SQSServer.
// See docs/design/2026_04_26_proposed_sqs_per_queue_throttling.md for
// See docs/design/2026_04_26_implemented_sqs_per_queue_throttling.md for
// the full design and rollout context. This file implements §3.1 (bucket
// store + token bucket), §3.3 (charging model), §3.4 (Throttling
// envelope helpers) and the cache-invalidation primitives §3.1 calls
Expand Down
4 changes: 2 additions & 2 deletions cmd/elastickv-snapshot-encode/main.go
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
// Command elastickv-snapshot-encode is the Phase 0b M6 snapshot encoder
// described in docs/design/2026_06_01_proposed_snapshot_encode_cli.md
// (parent: docs/design/2026_05_25_partial_snapshot_logical_encoder.md).
// described in docs/design/2026_06_01_implemented_snapshot_encode_cli.md
// (parent: docs/design/2026_05_25_implemented_snapshot_logical_encoder.md).
//
// It reads a vendor-independent per-adapter directory tree (produced by
// elastickv-snapshot-decode or by a future Phase 1 live extractor) and
Expand Down
2 changes: 1 addition & 1 deletion docs/admin_deployment.md
Original file line number Diff line number Diff line change
Expand Up @@ -331,7 +331,7 @@ is tombstoned at the same `commitTS` rather than left as an
unreachable orphan. The behaviour is intentional — the
alternative was orphan objects that no API can enumerate or
remove. See
[`docs/design/2026_04_28_proposed_admin_delete_bucket_safety_net.md`](design/2026_04_28_proposed_admin_delete_bucket_safety_net.md)
[`docs/design/2026_04_28_implemented_admin_delete_bucket_safety_net.md`](design/2026_04_28_implemented_admin_delete_bucket_safety_net.md)
for the full race analysis.

Operationally:
Expand Down
2 changes: 1 addition & 1 deletion docs/admin_ui_key_visualizer_design.md
Original file line number Diff line number Diff line change
Expand Up @@ -320,7 +320,7 @@ Because writes are recorded by Raft leaders and follower-local reads are recorde
| 0 | `cmd/elastickv-admin` skeleton, token-protected `Admin` gRPC service stub, empty SPA shell, CI wiring. | Binary builds, `/api/cluster/overview` returns live data from a real node only when the configured admin token is supplied. |
| 1 | Overview, Routes, Raft Groups, Adapters pages. `LiveSummary` added. No sampler. | All read-only pages match `grpcurl` ground truth. |
| 2-A | Key Visualizer MVP server side: in-memory sampler with adaptive sub-sampling, leader writes, leader/follower reads, static matrix API with virtual-bucket metadata. | Benchmark gate green; ±5% / 95%-CI accuracy SLO holds under synthetic bursts; matrix endpoint returns the local node's view. |
| 2-B | KeyViz SPA integration into `web/admin/`: heatmap page, series picker, row budget, manual + auto refresh. See `docs/design/2026_04_27_proposed_keyviz_spa_integration.md`. | Heatmap shows synthetic hotspot within ~5 s of `make client` driving traffic against `make run`; type check (`tsc -b --noEmit`) clean. |
| 2-B | KeyViz SPA integration into `web/admin/`: heatmap page, series picker, row budget, manual + auto refresh. See `docs/design/2026_04_27_implemented_keyviz_spa_integration.md`. | Heatmap shows synthetic hotspot within ~5 s of `make client` driving traffic against `make run`; type check (`tsc -b --noEmit`) clean. |
| 2-C | Cluster fan-out: admin RPC that aggregates each node's local sampler view so the SPA shows a cluster-wide heatmap rather than the local node's slice. | Fan-out returns complete view with 1 node down; SPA renders aggregate within the §10 budget. |
| 3 | Drill-down, split/merge continuity, namespace-isolated persistence of compacted columns distributed **per owning Raft group**, lineage recovery, and retention GC. | Heatmap remains continuous across a live `SplitRange`; restart preserves last 7 days; expired data and stale lineage records are collected; no single Raft group sees more than its share of KeyViz writes. |
| 4 (deferred) | Mutating admin operations (`SplitRange` from UI), browser login, RBAC, and identity-provider integration. Out of scope for this design; a follow-up design will cover it. | — |
Expand Down
2 changes: 1 addition & 1 deletion docs/design/2026_04_20_implemented_lease_read.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Lease Read Design

Status: Proposed
Status: Implemented
Author: bootjp
Date: 2026-04-20

Expand Down
2 changes: 1 addition & 1 deletion docs/design/2026_04_24_implemented_admin_dashboard.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
| **P3** — React SPA + embed | ✅ shipped | #649, #650 |
| **P4** — TLS, read-only role, CSRF, `docs/admin.md`, deployment runbook + `scripts/rolling-update.sh` admin support | ✅ shipped | TLS / role / CSRF live in P1; operator doc + runbook + script wiring in #674 / #669 / #678 |

The AdminDeleteBucket TOCTOU is fully resolved: see [`2026_04_28_proposed_admin_delete_bucket_safety_net.md`](2026_04_28_proposed_admin_delete_bucket_safety_net.md) for the safety-net design and [`docs/admin_deployment.md`](../admin_deployment.md) §4.6 for the operator-side contract (a `PutObject` 200-OK landing during the race window can be swept by the concurrent admin delete; pause writes before delete to retain in-flight writes).
The AdminDeleteBucket TOCTOU is fully resolved: see [`2026_04_28_implemented_admin_delete_bucket_safety_net.md`](2026_04_28_implemented_admin_delete_bucket_safety_net.md) for the safety-net design and [`docs/admin_deployment.md`](../admin_deployment.md) §4.6 for the operator-side contract (a `PutObject` 200-OK landing during the race window can be swept by the concurrent admin delete; pause writes before delete to retain in-flight writes).

### Out-of-scope follow-ups

Expand Down
Original file line number Diff line number Diff line change
@@ -1,22 +1,22 @@
# Per-Queue Throttling and Tenant Fairness for the SQS Adapter

**Status:** Proposed
**Status:** Implemented
**Author:** bootjp
**Date:** 2026-04-26

---

## 1. Background and Motivation

elastickv's SQS adapter currently has **no per-queue rate limiting**. A single tenant's runaway producer can:
Before this work, elastickv's SQS adapter had **no per-queue rate limiting**. A single tenant's runaway producer could:

1. Saturate the leader's Raft proposal pipeline (one OCC dispatch per `SendMessage`), pushing latency on every other queue's writes through the same shard.
2. Exhaust the receive-path's visibility-index scan budget (`sqsVisScanPageLimit = 1024`), causing other tenants' `ReceiveMessage` calls to time out empty.
3. Fill the message keyspace fast enough that the retention reaper cannot keep up — the keyspace grows unbounded until the next manual purge.

Phase 3.C in [`docs/design/2026_04_24_partial_sqs_compatible_adapter.md`](2026_04_24_partial_sqs_compatible_adapter.md) §16.5 marks this as TODO. AWS itself enforces per-account / per-API limits ("standard request throttle of 3000 RPS per region per AWS account" plus per-API limits like 300 TPS for batch APIs); operators running elastickv as a multi-tenant SQS facade need an equivalent control plane. Without it, the only knobs are (a) shard-level capacity (too coarse — adding a shard requires a Raft membership change) and (b) external load-balancer rate limiting (no visibility into per-queue cost).

This document proposes per-queue token-bucket throttling, configured per-queue in queue meta, evaluated at the SQS-adapter layer on the leader, and surfaced as the same `Throttling.Sender` error AWS uses (so existing SDK retry/backoff logic engages naturally).
This document describes per-queue token-bucket throttling, configured per-queue in queue meta, evaluated at the SQS-adapter layer on the leader, and surfaced as the same `Throttling.Sender` error AWS uses (so existing SDK retry/backoff logic engages naturally).

---

Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
status: proposed
status: implemented
phase: 2-C
parent_design: docs/admin_ui_key_visualizer_design.md
date: 2026-04-27
Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
status: proposed
status: implemented
phase: 2-B
parent_design: docs/admin_ui_key_visualizer_design.md
date: 2026-04-27
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# AdminDeleteBucket Orphan-Object Safety Net

**Status:** Proposed
**Status:** Implemented
**Author:** bootjp
**Date:** 2026-04-28

Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
status: proposed
status: implemented
phase: 2-C+
parent_design: docs/admin_ui_key_visualizer_design.md
author: bootjp
Expand Down Expand Up @@ -579,7 +579,7 @@ from PR #694: Claude bot critical, Gemini high.)
The fan-out aggregator's per-cell merge key gains the label:

- Phase 2-C (current): `(bucketID, raftGroupID, leaderTerm,
windowStart)` per design `2026_04_27_proposed_keyviz_cluster_fanout.md`
windowStart)` per design `2026_04_27_implemented_keyviz_cluster_fanout.md`
§4.
- With labels: same tuple — but `bucketID` itself now carries the
label via the §5 composite (`route:1:dynamo`). The merge key
Expand Down
9 changes: 5 additions & 4 deletions docs/design/2026_04_29_partial_snapshot_logical_decoder.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,10 @@ Date: 2026-04-29

> **Lifecycle (2026-05-25):** Phase 0a (decoder) has fully shipped —
> `internal/backup/` + `cmd/elastickv-snapshot-decode` (PRs #790,
> #791, #792, #806, #810). Phase 0b (encoder) is specified in detail
> in `2026_05_25_proposed_snapshot_logical_encoder.md` and is not yet
> implemented. Phase 0c (operator integration) is open. This doc
> #791, #792, #806, #810). Phase 0b (encoder) has shipped and is
> specified in detail in
> `2026_05_25_implemented_snapshot_logical_encoder.md`. Phase 0c
> (operator integration) is open. This doc
> remains the format owner; the encoder doc owns the reverse-direction
> wire-format reconstruction.

Expand All @@ -30,7 +31,7 @@ terminates on a clean EOF at the start of a key-length field
`ReadSnapshot`). A CRC32C footer exists only on the *MVCC streaming
restore* path (`store/lsm_store.go` `readStreamingMVCCRestoreHeader`),
which is a different framing the decoder/encoder do not touch. See
`2026_05_25_proposed_snapshot_logical_encoder.md` §"Why a separate
`2026_05_25_implemented_snapshot_logical_encoder.md` §"Why a separate
design doc" item 3.

Snapshots are taken automatically every `defaultSnapshotEvery = 10000`
Expand Down
2 changes: 1 addition & 1 deletion docs/design/2026_04_29_proposed_logical_backup.md
Original file line number Diff line number Diff line change
Expand Up @@ -1570,7 +1570,7 @@ Scope: out of this proposal; mentioned only to draw the boundary.
- Concurrent multi-cluster fan-out (one logical backup spanning shards
on different physical clusters) — depends on the `Distribution`
control plane being fan-out-aware (see
`2026_04_27_proposed_keyviz_cluster_fanout.md`).
`2026_04_27_implemented_keyviz_cluster_fanout.md`).

## Required Tests

Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
status: proposed
status: implemented
phase: 2-A+
parent_design: docs/admin_ui_key_visualizer_design.md
author: bootjp
Expand Down
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
# Snapshot Logical Encoder (Phase 0b)

Status: Partial (M1–M5 merged via PRs #807 / #841 / #847 / #849 / #864 / #846 / #892; M6 CLI in review at PR #896)
Status: Implemented (M1-M6 merged; external-sort remains a future optimization)
Author: bootjp
Date: 2026-05-25 (promoted proposed → partial 2026-06-01 in PR #896)
Date: 2026-05-25

## Background

Expand All @@ -22,9 +22,9 @@ specifics the parent doc left at sketch level (parent §"Encoder:
The parent doc is the format owner and remains authoritative for the
directory-tree shape, filename encoding, and `MANIFEST.json`. This
doc owns the **reverse-direction wire-format reconstruction** and the
decisions that only arise on the encode side. On landing this
proposal, the parent doc is promoted `proposed` → `partial` (Phase 0a
shipped) via `git mv`.
decisions that only arise on the encode side. The Phase 0b v1 encoder
is implemented through the CLI milestone; the external-sort path
remains a future optimization outside the implemented v1 surface.

## Why a separate design doc

Expand Down Expand Up @@ -266,14 +266,11 @@ covered by a cross-check test that asserts the encoder's derived index
rows are byte-identical to the live adapter's output for a shared
fixture.

> **Scope note / open question.** GSI and SQS-side-record derivation
> are the heaviest pieces. If the cross-check tests show the live
> builders are impractical to mirror offline within Phase 0b, the
> fallback is to emit only the user records + cheap indexes (TTL,
> generation) and document that GSI/SQS-side-state rebuild lazily on
> first adapter access after restart. The recommended path is full
> reconstruction; the fallback is called out in §"Milestones" as a
> per-adapter decision gate.
> **Scope note.** GSI and SQS-side-record derivation were the heaviest
> pieces. The implemented path follows the full-reconstruction option:
> derived GSI and SQS side rows are emitted offline and pinned by
> cross-check tests against the live key builders. The lazy-rebuild
> fallback below was not chosen for Phase 0b v1.
>
> **The fallback is not zero-cost transparency.** A missing GSI row
> makes a DynamoDB GSI query return empty *silently* (no error); a
Expand Down Expand Up @@ -399,9 +396,8 @@ output file, so a node never receives an unloadable `.fsm`.

## Milestones (per-adapter PRs, mirroring Phase 0a)

Doc-first: this proposal lands as its own PR before any code. Then,
in order (each its own PR, each with the cross-check + round-trip
tests for that adapter):
Phase 0b v1 landed in the following order (each with cross-check and
round-trip coverage for the adapter slice):

1. **Encoder core** — `encode.go`: MANIFEST read/validate, MVCC
re-encoding, in-memory sort, EKVPBBL1 writer, in-process round-trip
Expand Down
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
---
status: proposed
status: implemented
phase: 2-A++
parent_design: docs/admin_ui_key_visualizer_design.md
related_design: docs/design/2026_05_25_proposed_keyviz_subrange_sampling.md
related_design: docs/design/2026_05_25_implemented_keyviz_subrange_sampling.md
author: bootjp
date: 2026-05-28
---
Expand All @@ -13,7 +13,7 @@ date: 2026-05-28

Phase 2-A+ sub-range sampling (#836 + #841) shows hot **sub-ranges**:
the heatmap tells you *which region of the key space* is busy. The
sub-range design (`2026_05_25_proposed_keyviz_subrange_sampling.md` §2.2
sub-range design (`2026_05_25_implemented_keyviz_subrange_sampling.md` §2.2
/ §8) explicitly deferred naming the actual hottest **keys** —
"`Top-K can layer on later as a drill-down`." This is that follow-up.

Expand Down
Loading
Loading