Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
1743981
docs: mark keyviz cluster fanout implemented
bootjp Jul 7, 2026
dfc61d3
docs: update keyviz phase status
bootjp Jul 7, 2026
32a6e4f
docs: align keyviz fanout spec
bootjp Jul 8, 2026
b1a34d5
docs: align keyviz fanout contract
bootjp Jul 8, 2026
b0be263
docs: align keyviz fanout wire sample
bootjp Jul 8, 2026
7d2bca0
Merge remote-tracking branch 'origin/main' into design/promote-keyviz…
bootjp Jul 8, 2026
104eb4f
Reserve retired raftadmin transfer fields
bootjp Jul 8, 2026
ca4f103
Align keyviz fanout docs with shipped merge
bootjp Jul 8, 2026
22f2dc5
Reject unsupported gated transfer requests
bootjp Jul 8, 2026
7e91170
Merge remote-tracking branch 'origin/main' into design/promote-keyviz…
bootjp Jul 8, 2026
e03fb75
Document local-first keyviz metadata merge
bootjp Jul 8, 2026
8235b38
Clarify keyviz catalog divergence status
bootjp Jul 8, 2026
3135d67
Merge remote-tracking branch 'origin/main' into design/promote-keyviz…
bootjp Jul 10, 2026
1dbadac
docs: align keyviz fanout rollout contract
bootjp Jul 10, 2026
d99c490
Merge remote-tracking branch 'origin/main' into design/promote-keyviz…
bootjp Jul 10, 2026
c550501
docs: clarify keyviz fanout rollout semantics
bootjp Jul 10, 2026
3eee023
docs: clarify keyviz fanout rollout requirements
bootjp Jul 10, 2026
cc4e448
docs: align keyviz fanout operations guide
bootjp Jul 10, 2026
cffe865
docs: separate keyviz fanout architecture paths
bootjp Jul 10, 2026
e51f140
docs: align keyviz fanout shipped contract
bootjp Jul 10, 2026
17f5917
docs: align keyviz fanout API contract
bootjp Jul 10, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
135 changes: 93 additions & 42 deletions docs/admin_ui_key_visualizer_design.md

Large diffs are not rendered by default.

493 changes: 493 additions & 0 deletions docs/design/2026_04_27_implemented_keyviz_cluster_fanout.md

Large diffs are not rendered by default.

441 changes: 0 additions & 441 deletions docs/design/2026_04_27_proposed_keyviz_cluster_fanout.md

This file was deleted.

17 changes: 10 additions & 7 deletions docs/design/2026_04_28_implemented_keyviz_adapter_labels.md
Original file line number Diff line number Diff line change
Expand Up @@ -587,20 +587,23 @@ from PR #694: Claude bot critical, Gemini high.)
The fan-out aggregator's per-cell merge key gains the label:

- Phase 2-C (current): `(bucketID, raftGroupID, leaderTerm,
windowStart)` per design `2026_04_27_proposed_keyviz_cluster_fanout.md`
windowStart)` per design `2026_04_27_implemented_keyviz_cluster_fanout.md`
Comment thread
bootjp marked this conversation as resolved.
§4.
- With labels: same tuple — but `bucketID` itself now carries the
label via the §5 composite (`route:1:dynamo`). The merge key
width does **not** change; the new label dimension is
encoded into `bucketID` so the aggregator already separates
same-route different-label rows correctly.

Reads still sum, writes still max-with-conflict; nothing about
the merge **rules** changes other than the wire shape of
`bucketID`. The merge **key** logic is unchanged: composite
`bucketID` already separates same-route different-label rows
correctly, so the aggregator's bucketing/grouping path needs no
edits.
The merge **rules** do not change. Phase 2-C+ already uses the
canonical per-cell identity `(bucketID, raftGroupID, leaderTerm,
column)`: read samples sum across nodes, write samples are deduped
within the same `(bucketID, raftGroupID, leaderTerm, column)` and
summed across distinct terms, and conflicts are marked only for
same-identity disagreements. Adapter labels only change the
`bucketID` dimension by embedding the label into the composite ID, so
same-route different-label rows are already separated by the existing
bucketing/grouping path.

The merge **value** path, however, still needs one explicit
field copy: `mergeRowInto` (`internal/admin/keyviz_fanout.go:509`)
Expand Down
7 changes: 4 additions & 3 deletions docs/design/2026_04_29_proposed_logical_backup.md
Original file line number Diff line number Diff line change
Expand Up @@ -1568,9 +1568,10 @@ Scope: out of this proposal; mentioned only to draw the boundary.
between two `read_ts` values and dumps the delta in the same
per-adapter format under a `delta/<from-ts>-<to-ts>/` subtree.
- Concurrent multi-cluster fan-out (one logical backup spanning shards
on different physical clusters) — depends on the `Distribution`
control plane being fan-out-aware (see
`2026_04_27_proposed_keyviz_cluster_fanout.md`).
on different physical clusters) — depends on a future
`Distribution` control-plane fan-out and membership-discovery design.
The implemented KeyViz fan-out is a static admin-HTTP heatmap
aggregator, not the backup discovery contract.

## Required Tests

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -161,7 +161,7 @@ Options considered:

**Follower-forwarded write caveat (codex P2).** `ShardedCoordinator.observeMutation` runs on the node that received the client request before `ShardRouter.Commit` forwards (`kv/sharded_coordinator.go:1795-1844`), and the follower's `Internal.Forward` handler calls `transactionManager.Commit` directly (`adapter/internal.go:52`) rather than re-entering the leader's `ShardedCoordinator` observation hook. Therefore even a route whose shard group is led by the default-group leader can be **under-observed** if most clients connect through followers. M3 does not treat "missing local rows" as evidence of low load: local evidence can promote a split, but absence of local evidence only means "unknown" and is fail-closed for target selection (§6.2) and conservative for candidacy (fewer splits). Full coverage for follower-forwarded writes needs follower reports / fan-out into the same detector interface and is tracked under OQ-5.

**Why not the keyviz cluster fan-out** (`docs/design/2026_04_27_proposed_keyviz_cluster_fanout.md`): that aggregates across nodes for the admin heatmap UI and dedupes write samples by `(groupID, leaderTerm)`. Pulling cross-node data in would add latency and a dependency on the fan-out being configured, so M3 stays leader-local and fail-closed on unknown load. If a future milestone wants complete cluster-wide scoring, including follower-forwarded writes, it can read the same fan-out aggregate behind the same detector interface (§5) — recorded as OQ-5.
**Why not the keyviz cluster fan-out** (`docs/design/2026_04_27_implemented_keyviz_cluster_fanout.md`): that aggregates across nodes for the admin heatmap UI and dedupes write samples by `(groupID, leaderTerm)`. Pulling cross-node data in would add latency and a dependency on the fan-out being configured, so M3 stays leader-local and fail-closed on unknown load. If a future milestone wants complete cluster-wide scoring, including follower-forwarded writes, it can read the same fan-out aggregate behind the same detector interface (§5) — recorded as OQ-5.

**Known limitation — multi-leader (multi-group) deployments.** Leader-local consumption is only authoritative for routes whose shard group is led by the **same node** that runs the detector — i.e. the default-group leader (where the scheduler must run, §6.1). In a multi-group cluster (`--raftGroups` with G1, G2, …), the default-group leader is **not necessarily** the leader of G1/G2/…: each node runs a single `ShardedCoordinator` and `MemSampler` and only `Observe`s traffic it *receives* (`kv/sharded_coordinator.go:1795-1824`). Concretely, in a 3-node cluster where node A leads the default group and node B leads G1, **node A's sampler has no data for routes in G1 served through node B**, so the detector on A cannot score or split them. This is **broader than "heavy follower-forwarded reads"**: it is a structural gap for any route whose group leader differs from the default-group leader, not just an edge case of read forwarding.

Expand Down
4 changes: 2 additions & 2 deletions docs/design/2026_06_23_proposed_scaling_roadmap.md
Original file line number Diff line number Diff line change
Expand Up @@ -259,8 +259,8 @@ memory each group's private cache/memtable pins.

- **keyviz** — the per-route load sampler is wired and allocation-free on the
write hot path (`observeMutation`, `kv/sharded_coordinator.go:1841-1846`),
with proposed extensions for cluster fan-out
(`2026_04_27_proposed_keyviz_cluster_fanout.md`),
with implemented extensions for cluster fan-out
(`2026_04_27_implemented_keyviz_cluster_fanout.md`),
subrange sampling (`2026_05_25_implemented_keyviz_subrange_sampling.md`),
hot-key top-K (`2026_05_28_implemented_keyviz_hot_key_topk.md`), and per-cell
conflict (implemented). It is the detection signal M3 reuses. Current
Expand Down
2 changes: 1 addition & 1 deletion internal/admin/keyviz_fanout.go
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,7 @@ const keyVizMergeBucketHint = 64
// a stable row order; Responded counts ok=true entries; Expected is
// the configured peer count plus self.
//
// See docs/design/2026_04_27_proposed_keyviz_cluster_fanout.md 5.
// See docs/design/2026_04_27_implemented_keyviz_cluster_fanout.md 5.
type FanoutResult struct {
Nodes []FanoutNodeStatus `json:"nodes"`
Responded int `json:"responded"`
Expand Down
2 changes: 1 addition & 1 deletion internal/admin/keyviz_handler.go
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ const keyVizRowBudgetCap = 1024
//
// Fanout is non-nil when the handler is configured for cluster-wide
// fan-out (Phase 2-C): it carries per-node status so the SPA can
// surface degraded responses inline (see design 2026_04_27_proposed_keyviz_cluster_fanout.md).
// surface degraded responses inline (see design 2026_04_27_implemented_keyviz_cluster_fanout.md).
// The field is omitted from the wire form when fan-out is disabled
// so old clients keep working unchanged.
type KeyVizMatrix struct {
Expand Down
2 changes: 1 addition & 1 deletion main.go
Original file line number Diff line number Diff line change
Expand Up @@ -236,7 +236,7 @@ var (
// HTTP endpoints (host:port or scheme://host:port). When set,
// the admin keyviz handler aggregates the local matrix with
// peer responses; when empty, behaviour is unchanged
// (single-node view). See docs/design/2026_04_27_proposed_keyviz_cluster_fanout.md.
// (single-node view). See docs/design/2026_04_27_implemented_keyviz_cluster_fanout.md.
keyvizFanoutNodes = flag.String("keyvizFanoutNodes", "", "Comma-separated peer admin endpoints (host:port) for keyviz cluster-wide fan-out; empty disables")
keyvizFanoutTimeout = flag.Duration("keyvizFanoutTimeout", keyvizFanoutDefaultTimeout, "Per-peer timeout for keyviz fan-out HTTP calls")
)
Expand Down
15 changes: 8 additions & 7 deletions scripts/rolling-update.env.example
Original file line number Diff line number Diff line change
Expand Up @@ -116,13 +116,14 @@ ADMIN_ENABLED="false"
# merges matrices from each peer so the dashboard renders a cluster-
# wide heatmap regardless of which node served the request. The
# aggregator forwards the operator's session cookie to each peer
# (PR #692), so peers running with --adminEnabled accept the fan-out
# call as long as the cookie is valid on every node — i.e. the same
# admin signing key (ADMIN_SESSION_SIGNING_KEY_FILE) and matching
# role allow-lists must be configured cluster-wide. Peers without
# --adminEnabled expose an unauthenticated keyviz endpoint and
# respond unconditionally.
# See docs/design/2026_04_27_proposed_keyviz_cluster_fanout.md for the
# (PR #692), so every peer in the list must run with --adminEnabled;
# when admin is disabled the KeyViz handler is not mounted. The cookie
# must be valid on every node — i.e. the same admin signing key
# (ADMIN_SESSION_SIGNING_KEY_FILE) and matching role allow-lists must
# be configured cluster-wide. TLS admin listeners must be listed as
# explicit https://host:port entries; host:port entries are treated as
# http://host:port.
# See docs/design/2026_04_27_implemented_keyviz_cluster_fanout.md for the
# full design.
KEYVIZ_ENABLED="false"
# KEYVIZ_FANOUT_NODES="10.0.0.1:8080,10.0.0.2:8080,10.0.0.3:8080"
Expand Down
Loading