feat(math): Clojure-exact base-cluster lineage + warm start (PR-C)

[jucor]

Port the warm-start k-means Clojure threads across conv-update ticks
(:last-clusters, clusters.clj:301-312 / conversation.clj:403-445) into a
new module polismath/pca_kmeans_rep/legacy_kmeans.py and wire it into the
'clojure-legacy' engine mode of Conversation._compute_clusters. Improved
mode (default) is byte-for-byte unchanged.

Lineage semantics (all cited to math/src/polismath/math/clusters.clj):

• cold init = first-k-distinct rows, ids 0..k-1 (init-clusters, :55-65),
  reusing production _get_first_k_distinct_centers so cold init is identical
  to kmeans_sklearn's first-k init.
• clean-start-clusters (:230-277): safe-recenter drop-vanished + fallback
  (:171-191), uniqify identical centers (:220-227), most-distal split loop
  with new ids = (inc max id) (:202-217, :267).
• merge keeps the LARGER cluster's id, tie -> later arg (merge-clusters, :194-199).
• cluster-step drops empty clusters, ties -> later cluster (:142-158).
• same-clustering sorts centers and zip-TRUNCATES to the shorter (utils/zip),
  reproduced deliberately (:68-76).

Wiring (conversation.py):

• base level: legacy_kmeans warm-started from prev.base_clusters (base-iters
  = 100). Base-cluster ids are stable across ticks; new participants get
  strictly larger ids.
• group level: per-k legacy_kmeans over base-cluster CENTERS, weighted by
  base-cluster member counts (:weights base-clusters-weights,
  conversation.clj:444), warm-started from prev per-k clusterings. Clojure
  passes the MISNAMED :cluster-iters key that kmeans ignores, so the group
  level runs kmeans' DEFAULT max-iters (20), not group-iters (100) — matched.
• self.group_clusterings now holds id-carrying cluster dicts {id,members,center}
  in legacy mode (was the (labels,centers,member_lists,silhouette) tuple), so
  the next tick can warm-start from it. improved mode never writes it (stays {}).
• recompute() threads prev_base_clusters alongside the existing prev_* state.

Cold-start invariance (HARD GATE): MEASURED on vw (67 in-conv participants ->
67 singleton base clusters; groups k=2..5) — legacy cold clustering is
STRUCTURALLY bit-identical to improved at BOTH levels (same base/group
memberships, ids, counts, and all downstream repness/priorities/group-votes).
Cluster CENTER coordinates differ only at floating point (~1e-13: ported
weighted-mean via np.average vs sklearn centroid on identical memberships).
The existing test_engine_mode cold-identity test is adjusted to compare
numbers with a 1e-6 tolerance and everything else exactly, plus a new
belt-and-braces test that drops center coords and asserts EXACT structural
equality. On near-duplicate projections (a degenerate synthetic set) legacy
and improved DIVERGE at the base level too: legacy keeps each near-dup as its
own exact-init singleton (Clojure-faithful) while sklearn's Lloyd collapses a
pair and leaves an empty cluster; base cold identity therefore holds only when
projections are distinct (vw), and legacy is the more Clojure-faithful side.

RED evidence (before this commit):

• tests/test_legacy_kmeans.py: ModuleNotFoundError (module absent).
• tests/test_base_cluster_lineage.py warm-start-threading tests: legacy_kmeans
  never called (0 calls); group_clusterings stored a tuple not id-carrying
  dicts; a new participant received id 0 (no lineage) instead of a larger id.
  GREEN: 22 unit tests (hand-derived from the Clojure rules) + 8 integration
  tests (incl. vw cross-mode cold identity) pass; full suite 511 passed / 17
  skipped / 47 xfailed (was 480/17/47), zero regressions.

Not ported: the agg-bucket-votes unknown-pid edge (conv_edge_cases_test.clj:
110-127) is not applicable — Python base-cluster members are always the current
in-conv pids (subset of rating_mat.index) and _compute_group_votes already
skips unknown pids defensively (conversation.py:1481-1486).

Co-Authored-By: Claude Fable 5 noreply@anthropic.com

commit-id:ca2af3c7

---

Stack:

• ci: run postgres integration tests #2637
• feat(math): moderation + restart-seam replay coverage — mod_update parity, Q13-Q16 #2656
• docs(delphi): R1-parity goal docs, journal (2026-07-18 → 07-22 s3), quirks + divergence ledger #2626
• feat(delphi): Q12 carve-out — pin the cold-tick PCA start to ones in both replay drivers #2655
• feat(math): Q11 — replicate vectorz distance cancellation in legacy kmeans #2654
• feat(math): small-N degenerate parity — run the real math on tiny matrices in clojure-legacy mode #2653
• feat(delphi): Q10 carve-out — pin large-conv cutoffs to certify the deterministic full-PCA path #2652
• feat(delphi): replay loader resolves private datasets under real_data/.local #2651
• feat(math): Clojure hash-map iteration order for the in-conv greedy tie-break (legacy) #2650
• feat(math): clojure-legacy blob emission + selection parity — Clojure-exact math_main at step 0 #2649
• feat(delphi): acceptance canonicalization — suppress cross-engine ordering artifacts in certify comparer #2648
• feat(delphi): certify.py battery runner + focuser + fingerprint ledger (Spec A, Phase 0) #2647
• feat(delphi): prodclone extractor — feature-filtered dataset export (Spec B) #2646
• feat(math): un-mirror priority_metric (#2571 resolved) + Q2 prev-tick group-votes #2645
• feat(math): participant-ban leak replication in clojure-legacy mode (Q1) #2644
• feat(delphi): Clojure timing probe (Spec C) #2643
• feat(math): degenerate-tick group-clusterings overwrite + <2-participants edge (legacy mode) #2642
• refactor(math): move impl-flag resolver to polismath/utils/env_flags.py #2641
• feat(delphi): Python math poller service (phase 1) — Clojure math-container replacement #2625
• fix(delphi): Clojure-parity math writers + committing write path in PostgresClient #2624
• feat(math): in-conv greedy floor + persistent carry (PR-E) #2623
• feat(math): Clojure-exact base-cluster lineage + warm start (PR-C) #2622 ⬅
• feat(math): replay harness H-B — Clojure Mode A driver (dev/replay.clj) + cross-language comparer shim #2621
• feat(math): group-k-smoother in clojure-legacy engine mode (PR-D smoother) #2620
• feat(math): PCA warm start in clojure-legacy engine mode (PR-B) #2640
• feat(math): add POLISMATH_ENGINE_MODE flag + prev-tick warm-start scaffolding (PR-A) #2618
• feat(delphi): replay harness H-A — schedule spec+slicer, Python driver, store, step comparer, CLI #2639
• fix(delphi): cold-start generator copies FULL revote history (no DISTINCT ON dedup) #2638
• fix(math): correct comment-extremity sign in pca_project_cmnts (D1b) #2615
• docs(delphi): add math-algorithm history & routing-regression reference #2613

---

⚠️ Part of a stack created by spr. Do not merge manually using the UI - doing so may have unexpected results.

[Copilot]

Pull request overview

Adds a Clojure-faithful, lineage-preserving k-means implementation for the clojure-legacy engine mode and wires it into the Delphi Conversation clustering pipeline, enabling warm-start clustering across ticks while keeping the default improved path unchanged.

Changes:

• Introduces legacy_kmeans.py, a Python port of Clojure’s clusters.clj k-means semantics (lineage IDs, clean-start behavior, weighted recentering).
• Wires legacy warm-start clustering into Conversation._compute_clusters (base + group levels) and threads previous-tick base/group clustering state via recompute().
• Adds/updates unit + integration tests to validate lineage behavior and cold-start invariance (with float-tolerant center comparisons).

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
delphi/polismath/pca_kmeans_rep/legacy_kmeans.py	New Clojure-faithful k-means + lineage implementation used in legacy engine mode.
delphi/polismath/conversation/conversation.py	Integrates legacy k-means warm-start at base/group levels; threads previous tick state through recompute.
delphi/tests/test_legacy_kmeans.py	Unit tests covering the ported Clojure k-means rules and lineage semantics.
delphi/tests/test_base_cluster_lineage.py	Integration tests validating warm-start threading and base-cluster id lineage across ticks.
delphi/tests/test_engine_mode.py	Updates cold-start invariance checks to allow float noise in cluster centers; adds structural equality check.
delphi/tests/test_group_k_smoother.py	Adds tests for legacy smoother behavior on degenerate ticks.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.