feat(delphi): Clojure-parity math fixes + storage-v2/replay design (s…

[jucor]

…quash of 15 PRs)

SQUASH MERGE of the 15-PR Clojure-parity + design stack into edge
(spr stack, PRs #2564–#2597). GitHub's default squash message keeps only
the top PR's description; this message preserves every squashed PR's commit
message so no history is lost.

What it does:
• Clojure-parity math fixes D10–D15 and supporting changes (rep-comment
selection, consensus selection, comment priorities, group-id order,
ns/PASS-vote counting, repness cleanup) so the Python math pipeline
matches the legacy Clojure output.
• powerit-pca legacy-mode PCA port (Clojure-parity power iteration).
• DynamoDB consensus serialization/robustness fixes (Decimal conversion,
dict-shape normalization incl. present-but-None guard).
• Test-infra: require_dynamodb skips locally / fails loudly in CI.
• Docs: parity journal + PLAN refresh, replay-harness design (R1/R2),
storage-v2 / job-id reproducibility design, and archival of stale docs.

───────────────────────────────────────────────────────────────────────
Squashed PRs (bottom → top of stack):
───────────────────────────────────────────────────────────────────────

▁▁▁ #2564 ▁▁▁
refactor(delphi): delete dead scalar paths in repness.py (PR 14a)

The scalar implementations in delphi/polismath/pca_kmeans_rep/repness.py
were test-only — production calls only compute_group_comment_stats_df,
select_rep_comments_df, and select_consensus_comments_df (via
conv_repness). Maintaining two parallel implementations created
"where do I put this helper?" ambiguity for the upcoming D10/D11/D12 fixes
(all of which add new helpers to repness.py) and obscured the structure
of the production path.

Foundation pass before D10/D11/D12 + PR 14b/14c. No behavioural change —
production path untouched.

Production code (delphi/polismath/pca_kmeans_rep/repness.py)

DELETED (445 lines):

• Primitives: prop_test, two_prop_test (no production callers).
• Orchestration: comment_stats, add_comparative_stats, repness_metric,
  finalize_cmt_stats, passes_by_test, best_agree, best_disagree,
  select_rep_comments, select_consensus_comments.
• Unused helper: calculate_kl_divergence (no callers anywhere in repo).

KEPT:

• z_score_sig_90, z_score_sig_95 — trivial threshold checks consumed
  scalar-side; vectorizing would not save lines.

ENRICHED:

• prop_test_vectorized and two_prop_test_vectorized docstrings now embed
  the scalar-equivalent closed-form algebra. The formulas stay readable even
  though the scalar functions are gone.

Tests

DELETED entirely:

• tests/test_old_format_repness.py (557 lines, scalar-only sibling of
  test_repness_unit.py).

DELETED classes/methods in tests/test_repness_unit.py:

• TestCommentStats, TestSelectionFunctions, TestConsensusAndGroupRepness
  (all scalar-only).
• TestStatisticalFunctions::test_prop_test, ::test_two_prop_test.

MIGRATED to single vectorized DataFrame calls in tests/test_discrepancy_fixes.py:

• D4/D5/D6 BlobInjection classes — build a DataFrame from the blob's repness
  entries, run one prop_test_vectorized / two_prop_test_vectorized call,
  compare element-wise. Tests the actual production code path, produces
  cleaner diagnostics via .to_string().
• TestD5ProportionTest::test_prop_test_matches_clojure_formula (consolidated
  with the n=0 boundary case).
• TestD6TwoPropTest::test_two_prop_test_matches_clojure_formula (+ edge cases
  consolidated, + pi_hat=1 boundary cases).

CONSOLIDATED in tests/test_discrepancy_fixes.py:

• TestD8FinalizeStats's 7 scalar boundary tests collapse into a single
  parametrized DataFrame test test_repful_classification_boundary that
  exercises the production np.where(rat > rdt, 'agree', 'disagree') logic.
  All boundary cases preserved (rat<rdt, rat>rdt, rat==rdt non-zero,
  rat==rdt==0, negative z-scores).

DELETED redundant tests:

• TestSyntheticEdgeCases::test_prop_test_matches_clojure_formula_synthetic
  (duplicated by migrated TestD5ProportionTest).
• TestSyntheticEdgeCases::test_clojure_repness_metric_product (duplicated
  by migrated TestD7RepnessMetric).
• TestSyntheticEdgeCases::test_clojure_repful_uses_rat_vs_rdt (purely
  tautological).

Cross-checks in tests/test_repness_unit.py::TestVectorizedFunctions now
use _prop_test_reference and _two_prop_test_reference closed-form
staticmethods in place of scalar calls.

Suite delta

• Pre (edge @ 2dce738): 330 passed, 12 skipped, 58 xfailed.
• Post: 295 passed, 12 skipped, 58 xfailed.
• Delta: -35 passed, 0 failed, 0 new xfailed. Matches deleted scalar-test
  count exactly.

For PR 14c (readability refactor — runs later)

The deleted scalar code is the readability reference for PR 14c. Retrieve via:

git show <this-commit>~1:delphi/polismath/pca_kmeans_rep/repness.py \
    | sed -n '161,302p'

Specifically (pre-deletion line numbers): comment_stats 161-201,
add_comparative_stats 203-235, repness_metric 237-271,
finalize_cmt_stats 273-301. Clojure originals at
math/src/polismath/math/repness.clj:78-100,173-188,191-200.

Documentation

• delphi/docs/PLAN_DISCREPANCY_FIXES.md: added PR 14a row to the stack
  cross-reference table.
• delphi/docs/CLJ-PARITY-FIXES-JOURNAL.md: appended "Session: PR 14a —
  Scalar deletion (2026-06-11)" entry with the full scope, suite delta,
  and the git show recipe for PR 14c.

Out of scope (handed off separately)

• Pyright pandas-stubs noise (10+ false positives on pd.DataFrame(columns=...)
  and df['col'] = value in compute_group_comment_stats_df) is pre-existing
  on edge HEAD; PR chore(delphi): standardize venv as .venv and add Pyright config #2560's pyright config didn't set rule overrides.
  Handoff: ~/polis/HANDOFF_PYRIGHT_PANDAS_STUBS.md. Tracked separately.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com

▁▁▁ #2570 ▁▁▁
fix(delphi): ns includes PASS votes (Clojure parity, pre-D10)

Bug

compute_group_comment_stats_df in
delphi/polismath/pca_kmeans_rep/repness.py computed ns = na + nd and
total_votes = total_agree + total_disagree, silently dropping PASS (0)
votes from every vote count.

Clojure reference: math/src/polismath/math/repness.clj:56-61, :70:

(defn- count-votes [votes & [vote]]
  (let [filt-fn (if vote #(= vote %) identity)]
    (count (filter filt-fn votes))))
...
:ns (fnk [votes] (count-votes votes))

count-votes with no vote argument uses identity as the filter
predicate. In Clojure 0 is truthy, so (filter identity ...) keeps every
non-nil entry — including PASS. Therefore Clojure
ns = na + nd + np (PASS count).

Every downstream metric that consumes ns — pa, pd, pat, pdt,
ra, rd, rat, rdt, agree_metric, disagree_metric, and the
upcoming D11 consensus_stats_df — was off whenever PASS votes existed.

Fix

Both total_counts and group_counts now compute the count column via
('vote', 'size') directly in the pandas groupby aggregation. The frame
is dropna(subset=['vote'])-filtered upstream, so size() counts
exactly the non-NaN rows — including PASS (0). This is the Clojure
(count (filter identity votes)) recipe verbatim. Both sites carry a
comment citing the Clojure reference.

Why D5 BlobInjection didn't catch this

D5's blob-injection tests pull (n-success, n-trials) directly from the
Clojure blob's repness entries and feed them to prop_test_vectorized.
They bypass compute_group_comment_stats_df entirely, so anything
downstream of ns = na + nd was invisible. The same gap will recur for
every fix whose stage-inputs are built by Python code. Pure-formula
tests over a tiny vote matrix are the only RED path.

Tests added

TestNsIncludesPassVotes in tests/test_repness_unit.py:

• test_ns_includes_pass_votes — single comment, 2 agree / 1 disagree /
  2 pass → na=2, nd=1, ns=5.
• test_ns_all_pass_column — all-PASS column → na=0, nd=0, ns=3.
• test_ns_mixed_with_nan_only_explicit_votes_count — NaN never counts;
  PASS always does.
• test_other_votes_includes_other_group_pass — other_votes (the
  complement of group ns) also includes out-of-group PASS.

Suite delta

Pre: 295 passed, 12 skipped, 58 xfailed (PR 14a baseline).
Post: 299 passed, 12 skipped, 58 xfailed. Delta = +4 (the new tests).
No pre-existing test broke — the existing TestVectorizedFunctions
fixtures used only AGREE/DISAGREE/NaN and were never sensitive.

Cascade to D11

D11's consensus_stats_df(vote_matrix_df) (whole-conversation
counterpart) will mirror the same recipe at the conversation level. This
fix lands BEFORE D10 in the stack so D11's implementation, when it
arrives, starts from the corrected ns semantics. D11 should follow the
same pure-formula test pattern.

Goldens

DEFERRED. Re-recording is gated on sklearn-KMeans-seeding consensus; no
golden values shift at the goldens commit until D10/D11/D12 land.

▁▁▁ #2566 ▁▁▁
feat(delphi): D10 — Clojure-parity rep comment selection (PR 8)

Replaces the pre-D10 botched-port select_rep_comments_df with a
single-pass reduce that mirrors Clojure select-rep-comments
(math/src/polismath/math/repness.clj:212-281).

Sits on top of PR 14a in the spr stack.

Helpers added (top-level in repness.py)

• passes_by_test(s) — Clojure passes-by-test? (repness.clj:165-170).
  OR'd on (rat, pat) and (rdt, pdt) z-sig-90. NO pa >= 0.5 gate; the
  pre-D10 Python gate was an over-restriction with no Clojure analog.

• beats_best_by_test(s, current_best_z) — Clojure beats-best-by-test?
  (repness.clj:133-139). Strict > on max(rat, rdt).

• beats_best_agr(s, current_best) — Clojure beats-best-agr?
  (repness.clj:142-162). Four-branch agree-priority logic:

  1. na == 0 AND nd == 0 → reject.
  2. current_best AND current_best.ra > 1.0 → compare 4-way signed product
     ra * rat * pa * pat.
  3. current_best (else, ra <= 1.0) → compare pa * pat only.
  4. No current_best → accept if z90(pat) OR (ra > 1.0 AND pa > 0.5).
     current_best stores the RAW row (Clojure repness.clj:250) so the
     ra/rat/pa/pat surface stays available across iterations.

• _finalize_row_for_output(row, *, is_best_agree=False) — Clojure
  finalize-cmt-stats (repness.clj:173-188) + best-agree flagging
  (repness.clj:262-264). Emits best_agree=True and n_agree=na for the
  best-agree slot.

select_rep_comments_df rewrite

Signature now: (stats_df, mod_out=None) -> List[Dict[str, Any]]. Drops
the agree_count / disagree_count kwargs (Clojure has only a cap of 5).

Per-row state {sufficient, best, best_agree} updated by the helpers.
Final assembly: dedup best_agree from sufficient → sort by metric
(agree_metric for repful=='agree', disagree_metric for 'disagree')
→ prepend finalized+flagged best_agree → take 5 → agrees-before-disagrees.

The caller in conv_repness drops the _stats_row_to_dict wrapping step
(the new function returns finalized dicts directly).

Two pre-D10 bugs fixed alongside the rewrite

• pa >= 0.5 / pd >= 0.5 over-gate in the passing filter — removed (no
  Clojure analog).
• "Fill from other category" + "first row" fallback blocks — deleted. The
  :best / :best_agree mechanism IS the Clojure fallback.

Tests (18 new in tests/test_discrepancy_fixes.py)

• TestD10PassesByTest (4): agree-side, disagree-side, neither, no pa-gate.
• TestD10BeatsBestByTest (3): None-best, max(rat,rdt), strict >.
• TestD10BeatsBestAgr (6): one per Clojure branch + boundary.
• TestD10SelectRepCommentsBoundary (5): empty input, single unvoted row
  → best fallback, sufficient-empty-best-agree-only, take-5 cap +
  agrees-before-disagrees, the eviction edge case (best_agree outside
  sufficient evicting 5th-highest-metric).

Eviction edge case — flagged

take(5) runs AFTER prepending best_agree. If best_agree was kept by
beats_best_agr as a non-significant agree-priority fallback (failed
passes_by_test, qualified via Branch 4) AND :sufficient already has 5
entries, the prepend pushes total to 6 and take(5) evicts the
5th-highest-metric sufficient entry — possibly a strong dissenting view.

Mirrors Clojure exactly for blob parity. # TODO(parity-eviction) comment
at the take(5) site in the production code; entry added under
"Pending — needs team discussion" in PLAN.md; pinned by the synthetic
test above.

Re-xfailed with updated reasons (D14 / D1 upstream divergence)

Six per-shared-(gid, tid) blob-comparison tests were previously xfailed as
"D5/D6/D7/D8/D10: no shared comments to compare". After D10 there ARE
shared comments (overlap ~20% on vw cold_start), but the per-(gid, tid)
stats still mismatch because Python and Clojure place different
participants in the "same" group ID. That's upstream PCA/KMeans
group-membership divergence (D14 / D1), not D10. Updated xfail reasons
point at D14 / D1. D10 itself is verified by the 18 synthetic tests.

Affected: TestD9ZScoreThresholds::test_z_values_match_clojure,
TestD5ProportionTest::test_pat_values_match_clojure_blob,
TestD6TwoPropTest::test_rat_values_match_clojure_blob,
TestD7RepnessMetric::test_repness_metric_matches_clojure_blob,
TestD8FinalizeStats::test_repful_matches_clojure_blob,
TestD10RepCommentSelection::test_rep_comments_match_clojure.

Suite delta

• Pre (post-14a baseline): 295 passed, 12 skipped, 58 xfailed.
• Post: 313 passed, 12 skipped, 58 xfailed.
• Delta: +18 passed (the 18 new D10 synthetic tests), 0 failed, no new
  xfailed.

Documentation

• delphi/docs/PLAN_DISCREPANCY_FIXES.md: PR 14a row marked landed
  (refactor(delphi): delete dead scalar paths in repness.py (PR 14a) #2564), PR 8 (D10) marked in-flight, eviction concern added to
  "Pending — needs team discussion".
• delphi/docs/CLJ-PARITY-FIXES-JOURNAL.md: "Session: PR 8 — D10 rep
  comment selection (2026-06-11)" entry with full scope, suite delta,
  and decisions log pointer.

/goal mode

This PR is part of an autonomous run (/goal) targeting D10 + D11 + D12 +
golden snapshots as a stacked PR series. Decisions made autonomously
(key naming, return type, etc.) are documented in
~/polis/D10_D11_D12_GOLDENS_DECISIONS.md for batch user review at the
end of the run.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com

▁▁▁ #2567 ▁▁▁
feat(delphi): D11 — Clojure-parity consensus comment selection (PR 9)

Replaces the pre-D11 consensus logic (per-group pa > 0.6 for all filter,
top 2 by avg_pa) with a whole-conversation per-comment-stats stage and
two independent top-5 lists (agree / disagree), matching Clojure
consensus-stats + select-consensus-comments
(math/src/polismath/math/repness.clj:284-323).

Sits on top of PR 8 (D10) in the spr stack.

Production changes (repness.py)

• consensus_stats_df(vote_matrix_df, mod_out=None) -> pd.DataFrame:
  new helper. Whole-conversation per-comment stats (no group split). Output
  DataFrame indexed by tid with cols [na, nd, ns, pa, pd, pat, pdt].
  Vectorized port of Clojure consensus-stats (repness.clj:284-290).

  ns includes PASS (Clojure parity, repness.clj:56-61): computed as
  vote_matrix_df.notna().sum(axis=0) rather than na + nd. Matches
  Clojure (count (filter identity ...)) where 0 (PASS) is truthy.

• select_consensus_comments_df rewrite: new signature
  (cons_stats) -> Dict[str, List[Dict]]. Filters and ordering:

  • agree: pa > 0.5 AND z-sig-90(pat), sorted desc by am = pa * pat.
  • disagree: pd > 0.5 AND z-sig-90(pdt), sorted desc by dm = pd * pdt.
    Cap: top 5 each side. Output: {'agree': [...], 'disagree': [...]}.
    With PSEUDO_COUNT=2, pa + pd = 1 exact → pa > 0.5 ⟺ pd < 0.5, so
    the same tid cannot appear in both lists.

• conv_repness grows mod_out kwarg, forwarded to both
  select_rep_comments_df and consensus_stats_df. Consensus is now run
  unconditionally — Clojure has no len(groups) > 1 guard.

• _stats_row_to_dict deleted — orphan after D11.

Caller (conversation.py)

_compute_repness passes mod_out=self.mod_out_tids to conv_repness,
matching Clojure's mod-out propagation (repness.clj:222 and :296).

Downstream output shape change

conv.repness['consensus_comments'] was a flat list with
{repful: 'consensus', comment_id, avg_agree, stats} entries. After D11
it's {'agree': [entries], 'disagree': [entries]} matching Clojure's
math-blob shape. Each entry has Python-convention keys (decision S1):
{comment_id, n_success, n_trials, p_success, p_test}.

Updated consumers in the test suite:

• tests/test_repness_smoke.py::test_repness_structure — iterates the
  new dict shape.
• tests/test_pipeline_integrity.py::test_full_pipeline — same.

External downstream consumers (client-report/normalizeConsensus.js) may
need a parallel update; flagged in the decisions log for batch review.

Tests (13 new in tests/test_discrepancy_fixes.py)

• TestD11ConsensusStatsDf (5): basic counts, pseudocount pa/pd
  smoothing, ns=0 uninformative fallback, mod_out tid filter,
  ns-includes-PASS Clojure parity.
• TestD11SelectConsensusBoundary (8): empty input, clear agree
  consensus, clear disagree consensus, divisive (no consensus), top-5
  cap, entry-key shape, disagree-side key mapping (n_success ← nd,
  p_success ← pd, p_test ← pdt), mutually-exclusive agree/disagree lists.

ns-PASS fix (Clojure parity)

After D11 was first landed (PR #2567), the real-data test
test_consensus_matches_clojure showed 3-5/5 overlap on cold_start.
Investigation revealed that Clojure's :ns (via count-votes with
filter identity — repness.clj:56-61) INCLUDES PASS votes (0 is truthy
in Clojure), while Python's ns = na + nd excluded them.

Fixed here by switching consensus_stats_df to count via
vote_matrix_df.notna().sum(axis=0). After the fix, 3 of 4 dataset
variants (vw-incremental, vw-cold_start, biodiversity-cold_start) match
Clojure exactly. The biodiversity-incremental variant still mismatches
on the disagree side (likely residual upstream PCA/KMeans group-membership
divergence) — remains xfail(strict=False) and tracked in the journal.

(The companion fix for compute_group_comment_stats_df ships in a
separate pre-D10 commit qyskkqkovtmn.)

B1 + B2 sub-agent fixes (relocated from D12 per batch review 2026-06-11)

These two fixes were originally landed in PR #2568 (D12) because the D11
sub-agent review happened AFTER D11 had been pushed. They belong in D11,
so they are squashed into this commit:

• B1 (polismath/conversation/conversation.py:830-837): the
  no-groups branch of _compute_repness returned
  'consensus_comments': [] (list). After D11, the public shape is the
  dict {'agree': [...], 'disagree': [...]}. Downstream consumers
  (test_repness_smoke, test_pipeline_integrity) iterate the dict shape
  and would crash on the legacy list. Fixed to always return the dict
  shape, even with no groups.

• B2 (tests/test_legacy_repness_comparison.py:197-205): the
  legacy-comparison test extracted py_consensus = py_results.get( 'consensus_comments', []) and treated it as a flat list. Post-D11
  this is a dict, so the ID extraction silently produced an empty set.
  Fixed to flatten the dict (agree + disagree) for the legacy comparison,
  with a legacy fallback branch in case the value is still a list.

Suite delta

• Pre (post-D10): 313 passed, 12 skipped, 58 xfailed.
• Post (this PR): 330 passed, 12 skipped, 55 xfailed, 3 xpassed.
• Delta: +17 passed, -3 xfailed (D11 real-data test now passes on 3 of
  4 dataset variants; biodiversity-incremental remains
  xfail(strict=False)). Zero regressions.

/goal mode

Part of an autonomous stacked PR series (D10 + D11 + D12 + goldens) per
user request. Decisions are documented for batch review in
~/polis/D10_D11_D12_GOLDENS_DECISIONS.md.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com

▁▁▁ #2568 ▁▁▁
feat(delphi): D12 — comment priorities (PR 11)

Implements comment-priorities matching Clojure
(math/src/polismath/math/conversation.clj:648-679). Pre-D12 Python emitted
nothing for comment_priorities, which forced the TypeScript server's
getNextPrioritizedComment to fall back to uniform random comment routing.

Sits on top of PR 9 (D11) in the spr stack.

New helpers in pca.py

• pca_project_cmnts(center, comps) -> np.ndarray (shape (n_cmnts, n_components)):
  Vectorized projection. Closed-form derived from Clojure's sparsity-aware
  projection (pca.clj:134-178) collapsing to a single non-nil column per
  comment:
  proj[i] = -sqrt(n_cmnts) * (1 + center[i]) * [pc1[i], pc2[i]]

• compute_comment_extremity(cmnt_proj) -> np.ndarray: L2 norm per row.
  Clojure with-proj-and-extremtiy (conversation.clj:341-352).

New module-level functions in conversation.py

• META_PRIORITY = 7 (Clojure conversation.clj:319).
• importance_metric(A, P, S, E) -> float (Clojure conversation.clj:311-315).
• priority_metric(is_meta, A, P, S, E) -> float (Clojure conversation.clj:321-330).
  Squared output. Meta: META_PRIORITY^2 = 49. Non-meta:
  (importance * (1 + 8*2^(-S/5)))^2 — the decay factor lets new comments
  bubble up; importance falls as votes accumulate.

New Conversation._compute_comment_priorities() method

Wired into recompute() after _compute_repness(). For each tid:

• Compute extremity from pca_project_cmnts + compute_comment_extremity.
• Aggregate A/D/S across all groups via _compute_group_votes().
• Derive P = S - (A + D) (Clojure conversation.clj:661).
• Check tid in self.meta_tids for the meta branch.
• Call priority_metric and store under self.comment_priorities[int(tid)].

The serialization infrastructure (to_dict, to_dynamo_dict,
underscore→hyphen conversion in _convert_inner) already existed but was
emitting empty. Now populated.

B1 + B2 fixes folded in from D11 sub-agent review

• B1: conversation.py:834 no-groups early-return now emits
  consensus_comments: {'agree': [], 'disagree': []} (dict) instead of
  [] (list) — restores shape consistency with the new D11 shape.
• B2: test_legacy_repness_comparison.py:197 flattens the new
  consensus dict before iterating, instead of crashing on
  'agree'.get('comment_id', '').

Tests (11 new in tests/test_discrepancy_fixes.py)

• TestD12PCAProjectComments (5): output shape, formula verification per
  row, empty inputs, L2 extremity, empty extremity.
• TestD12PriorityMetrics (6): importance_metric matches Clojure
  reference value 4/9 (from conversation.clj:335 comment), extremity
  boost behavior, meta priority constant = 49, non-meta squared formula,
  decay factor monotonicity (low-S boosts, high-S fades), META_PRIORITY == 7.

Real-data test xfailed: Clojure blob has constant priorities

vw and biodiversity blobs both have EVERY tid set to priority = 49.0
(= META_PRIORITY^2). Likely caused by Clojure's (if 0 ...) truthiness
quirk — 0 is truthy in Clojure (only nil/false are falsy), so any value
returned by (get meta-tids tid 0) triggers the meta branch.

Python correctly distinguishes meta from non-meta via Boolean set
membership, producing varied priorities 0.18-31.46. Spearman comparison
meaningless when Clojure side has zero variance (returns nan).

Test xfailed with full reason. D12 logic verified by the 11 synthetic
tests. Logged for batch review — Python may be MORE correct than Clojure
here.

Suite delta

• Pre (post-D11): 325 passed, 12 skipped, 58 xfailed.
• Post (this PR): 336 passed, 12 skipped, 56 xfailed, 2 xpassed.
• Delta: +11 (the 11 new D12 synthetic tests), 0 failed, 2 xfailed →
  xpassed (the cold_start D12 tests now run cleanly; the new xfail is
  on a different test).

/goal mode

Autonomous stack PR (D10 + D11 + D12 + goldens). Decisions documented in
~/polis/D10_D11_D12_GOLDENS_DECISIONS.md for batch user review.

D11 sub-agent flagged additional concerns (B3: math-blob plumbing
hardcodes empty consensus in to_dict/to_dynamo_dict; D11 data
computed-but-unused at serialization layer). NOT addressed here —
larger scope than D11/D12, deliberate per S1. Documented for batch
review.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com

▁▁▁ #2572 ▁▁▁
feat(delphi): plumb D11 consensus + D12 priorities through to_dict / to_dynamo_dict

▁▁▁ #2586 ▁▁▁
fix(delphi): address Copilot review on D10-D12 stack (consensus blob keys, priority serialization, defensive fixes)

Verified triage of all 83 Copilot review threads on PRs #2564-#2573:
1 real blocker + 4 escalations confirmed by direct code verification;
the rest nitpicks / already-fixed / intentional bug-mirrors. All fixes
TDD RED->GREEN.

• Consensus entry keys -> Clojure blob shape {tid, n-success, n-trials,
  p-success, p-test} (was Python-convention comment_id/n_success/...).
  Narrows the S1 deferral: these entries flow raw into
  result['consensus'], where server-helpers.ts:298-313 and client-report
  majorityStrict.jsx:23-27 pluck tid - the Python keys broke both
  consumers. Rep-comment entries keep comment_id until the deferred
  math-blob alignment PR.
• DynamoDB writer read D11 consensus from a key to_dynamo_dict never
  emits (repness.consensus_comments) -> always wrote the empty default;
  D11 data never reached Delphi_PCAResults. Now reads top-level
  result['consensus']. The round-trip test had stubbed to_dynamo_dict
  with the writer's wrong nested shape - stub corrected to the real
  producer shape.
• to_dynamo_dict comment priorities: int(value) -> Decimal-preserving.
  int() floors sub-1 priorities to 0 (real formula spans ~0.18-31.46 per
  D12.6), which the TS server's weighted routing reads as "no priority
  data". Harmless today (bug-mirror pins 49.0), landmine once Clojure math.repness: priority-metric meta detection treats 0 as truthy, assigning META_PRIORITY^2 to every comment #2571
  resolves. Legacy writer path Decimal-wrapped too.
• DynamoDB reader normalizes legacy list-shaped consensus to the dict
  shape (pre-D11 blobs).
• mod_out truthiness -> is not None x2 in repness.py (numpy
  array/Index-safe).
• _compute_comment_priorities fails closed (error log + empty dict) on
  PCA/columns desync instead of silently zip-truncating extremities.
• bench_repness.py imported the 14a-deleted comment_stats (ImportError
  on any benchmark run); benchmark import test added.
• Per-variant xfails replace blanket xfail(strict=False) on D11/D12
  real-data tests, so the variants that match Clojure gate again.
  DISCOVERY: scoping the blanket unmasked two previously-invisible
  incremental divergences (bg2018-incremental, pakistan-incremental
  consensus vs Clojure) that the blanket had silently absorbed -
  documented as known-bad incremental xfails, same family as
  biodiversity-incremental, deferred to the sequential-parity work.
  3 pre-existing CCR failures (verified identical on edge 722640e)
  marked with precise per-variant reasons. PGR regression tests skip
  with the 2026-06-11 goldens-deferral reason - the mark S3-5 claimed
  to add but never committed.
• test_repness_smoke consensus-entry structure assertion updated to the
  Clojure key shape (exact key-set check).
• ns docstrings corrected (ns includes PASS post ns-PASS fix; pa+pd <= 1
  reasoning in select_consensus_comments_df).
• PERF deferral comment on the _compute_group_votes scan in
  _compute_comment_priorities (follow-up issue to be filed).

Baseline (2026-07-04, stack top, --include-local): 13 failed / 476
passed / 18 skipped / 143 xfailed. All 13 accounted for: 10 stale-PGR
comparisons (now skipped per deferral), 3 pre-existing CCR (now precise
xfails).

Co-Authored-By: Claude Fable 5 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_013CiESAwcrZ6BcVfdkumnxa

▁▁▁ #2593 ▁▁▁
fix(delphi): Decimal-convert D11 consensus for DynamoDB write

Caught by CI's test_math_pipeline_runs_e2e on the stack branch
(2026-07-05): "Float types are not supported. Use Decimal types
instead." at the Delphi_PCAResults put_item.

Once the writer read top-level consensus (the key to_dynamo_dict
actually emits — fixed in the PR below this one), REAL D11 consensus
data flowed to DynamoDB for the first time, carrying float
p-success/p-test values. Only the LEGACY writer branch Decimal-converted
its payload; the pre-formatted branch wrote dynamo_data['consensus']
raw into the Item, and boto3's TypeSerializer rejects raw floats.

Invisible to every local gate, three ways: the e2e test needs DynamoDB
(local skip list), the consensus round-trip tests use MagicMock (no
TypeSerializer runs), and the float-serialization pins covered
priorities but not consensus.

Fix, TDD RED->GREEN with boto3's REAL TypeSerializer:

• to_dynamo_dict: consensus surface now passes through float_to_decimal
  (same treatment as pca and comment_priorities in the same function).
• DynamoDB writer Site 1: belt-and-braces _replace_floats_with_decimals
  at the boto3 boundary, mirroring the legacy branch (idempotent on
  already-converted data).
• New TestToDynamoDictConsensusSerialization (Layer 2c) pins the exact
  failure through the real serializer; two mock-era expectations updated
  from float-identity to value-preserving comparisons.

Co-Authored-By: Claude Fable 5 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_013CiESAwcrZ6BcVfdkumnxa

▁▁▁ #2594 ▁▁▁
test(delphi): require_dynamodb skips locally, fails loudly in CI

DynamoDB-gated tests (test_math_pipeline_runs_e2e, test_batch_id) needed
a blanket --ignore in local runs because require_dynamodb hard-failed
when the service was absent. Now: pytest.skip locally with a how-to-run
hint (docker run --rm -d -p 8002:8000 amazon/dynamodb-local +
DYNAMODB_ENDPOINT), pytest.fail in CI — where DynamoDB is provisioned
and its absence is an infrastructure failure that must not silently
disable the only end-to-end gate (the 2026-07-05 consensus-float crash
was caught precisely because CI runs it).

Discriminator is GITHUB_ACTIONS, deliberately NOT the generic CI:
local supply-chain wrappers (SafeDep pmg) inject CI=true into wrapped
package-manager invocations, which would force the loud-fail path on
developer machines (observed 2026-07-05 on uv run via the pmg alias).

Verified matrix: local+down 5 skipped / local+up(8002) e2e PASSES
against real DynamoDB (the consensus-Decimal fix validated end-to-end)
/ GITHUB_ACTIONS+down errors loudly.

Co-Authored-By: Claude Fable 5 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_013CiESAwcrZ6BcVfdkumnxa

▁▁▁ #2573 ▁▁▁
docs(delphi): archive 33 stale leftover docs, fix stale references

Audit of delphi/docs/ against the current code (2026-06-11). Per review
feedback (Colin): the stale docs are MOVED to delphi/docs/archive/ rather
than deleted — they capture the original research and design intent of the
2025 build-out, which is an independent deliverable worth keeping as raw
material for future models to mine. archive/CLAUDE.md marks the folder as
historical, instructs agents not to treat it as current documentation, and
indexes each file's original purpose and the reason it was archived.

• Move 33 stale docs to docs/archive/ byte-identical (git records renames):
  completed one-off fix memos and session logs, unimplemented design
  proposals (IGAS topic-consensus metrics, job-id migration, DAG job
  system, spatial topic prioritization, UMAP viz plan, 16-week
  topic-agenda migration), and docs describing deleted architecture
  (legacy poller, run_tests.py, simplified tests, eda_notebooks, 600/802
  scripts, custom power-iteration PCA).
• Amend 15 surviving docs: status headers on handoffs and
  deep-analysis-for-julien/07 (which D-fixes are merged vs open),
  deleted-script references (start_poller.sh, 600_*.py), table name
  DelphiJobQueue -> Delphi_JobQueue, uv instead of pip/venv,
  TopicAgenda.jsx -> .tsx, versioned-key delimiter correction.
• Rewrite DOCUMENTATION_DIRECTORY.md to index surviving docs + archive.
• Fix references to moved docs and deleted scripts in delphi/CLAUDE.md and
  DELPHI_JOB_SYSTEM_TROUBLESHOOTING.md (run_delphi.sh -> run_delphi.py,
  start_poller.sh -> start_poller.py, nonexistent reset_database.sh,
  dead absolute-path link to DATABASE_NAMING_PROPOSAL.md).

Kept in place deliberately, as they document still-unfixed bugs:
ZID_EXPOSURE_AUDIT.md (zid still exposed in delphi API responses) and
TOPIC_LABEL_MISALIGNMENT_ANALYSIS.md (700_datamapplot label sorting).

Co-Authored-By: Claude Fable 5 noreply@anthropic.com

▁▁▁ #2589 ▁▁▁
fix(delphi): keep Clojure group-id order — resolves the gid 0-1 label swap

Root cause of the label swap confirmed by the S3-4 trace (2026-06-11:
Python g0 = Clojure g1 with 50/50 identical membership on vw-cold_start):
_compute_clusters re-sorted group clusters by size (descending) and
reassigned ids. Clojure assigns group ids by first-k-distinct encounter
order over base-cluster centers (init-clusters, clusters.clj:55-64),
keeps them through merge lineage, and orders output by sort-by :id
(conversation.clj:437) - it never re-orders by size. The base-cluster
level already preserved k-means id order for exactly this reason (K-inv);
the group level now follows the same rule.

Fix: remove the size re-sort + id reassignment; keep k-means label order
(sklearn preserves init-index-to-label correspondence, and the base-center
row order is already Clojure-parity per K-inv). Pinned by a synthetic
test: with the smaller group's center encountered first, group id 0 must
be the smaller group (fails under any size re-sort).

Test-gate harvest, verified against a full --include-local run:

• TestD8FinalizeStats::test_repful_matches_clojure_blob: xfail LIFTED on
  9/11 variants (all but vw-incremental / pakistan-incremental, which
  keep per-variant xfails for residual incremental trajectory divergence).
• D9 significance-sets + D10 rep-selection: biodiversity-cold_start now
  matches Clojure exactly and gates; other variants keep per-variant
  xfails (residual per-(gid,tid) membership/stat divergence).
• z-values / rat-values xfail reasons corrected: the label swap is now
  FALSIFIED as their cause (fix did not flip them); residual cause is
  group-membership divergence.
• D12 priorities known-bad incremental lists refined: FLI-incremental and
  bg2050-incremental Clojure blobs carry the all-49 truthy-0 signature,
  match Python's Clojure math.repness: priority-metric meta detection treats 0 as truthy, assigning META_PRIORITY^2 to every comment #2571 mirror, and now gate.

Co-Authored-By: Claude Fable 5 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_013CiESAwcrZ6BcVfdkumnxa

▁▁▁ #2590 ▁▁▁
chore(delphi): remove vestigial np.random.seed(42); carry Clojure seeding note

The global np.random.seed(42) in cluster_dataframe was the only random
reference in the module and seeded an RNG nothing ever draws from:
k-means init is first-k-distinct (deterministic by construction), and
cluster_dataframe is not on the production path anyway (production uses
kmeans_sklearn exclusively; the sole caller is tests/test_clusters.py).
Also drops the module's dead import random.

Evidence (2026-07-05, scratch/determinism_check.py + journal): 5
consecutive full-pipeline runs on vw + biodiversity are bit-for-bit
identical except math_tick (a wall-clock version counter, varies by
design). The determinism comes from first-k-distinct + n_init=1 +
explicit random_state, not from this global seed.

Per Julien: the original Clojure author's verbatim note on seeding
(pca.clj:80-81 — "Should really throw a parallelizable random number
generator in the equation here... With seeds fed in and persisted...
XXX") now lives in pca.py next to the random_state setting, with the
seeding-history context (Clojure never fixes a seed; k-means
deterministic; PCA cold-start rand unseeded, warm-started after).

Co-Authored-By: Claude Fable 5 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_013CiESAwcrZ6BcVfdkumnxa

▁▁▁ #2591 ▁▁▁
feat(delphi): powerit-pca port (Clojure-parity PCA) behind legacy-mode flag

Numpy port of Clojure's power-iteration PCA (pca.clj:38-105): per-component
power iteration with fixed iteration count (iters=100, the Clojure default),
Gram-Schmidt deflation, and a start_vectors parameter for warm-start pinning
(the replay harness's mechanism for collapsing cold-start jitter, and a
prerequisite for the pure-Python R2 replayer — see
docs/REPLAY_HARNESS_DESIGN.md section 9/10).

Flag: POLISMATH_PCA_IMPL env var, read at call time in
pca_project_dataframe. 'powerit' (DEFAULT - legacy/parity mode) or
'sklearn' (the improved path, previous behavior). First instance of the
permanent legacy-vs-improved dual-path pattern (Julien, 2026-07-05).
Imputation and sparsity scaling are untouched; only the eigen-solver
branches. Data-prep parity verified: Clojure imputes nil -> column average
(conversation.clj:360-380) = Python's nanmean imputation.

Documented decision - deterministic cold start: Clojure draws an UNSEEDED
random start vector (pca.clj:79-82); Python defaults to a fixed
deterministic start so the pipeline stays bit-for-bit reproducible (the
2026-07-05 determinism verification is a project invariant). Power
iteration converges to the same dominant eigenvector for almost any start,
so the fixed start is one specific draw of Clojure's random one;
start_vectors overrides for pinning. Carries the requested TODO: switch to
a proper convergence criterion once we move to improving the Python
implementation.

Silhouette guard (companion fix — required by making powerit the default):

• calculate_silhouette_sklearn (clusters.py) now treats any
  n_labels >= n_samples clustering as undefined and returns the neutral 0.0
  sentinel instead of letting sklearn raise. sklearn's silhouette_score
  requires 2 <= n_labels <= n_samples - 1; the old guard only covered
  n_labels <= 1 / n_samples <= 1.
• Why it surfaced here: making powerit the default projects some small
  conversations onto exactly two base clusters, which feeds a 2-point /
  2-label silhouette call in the group-cluster k-selection loop
  (conversation.py). That raised "Number of labels is 2. Valid values are 2
  to n_samples - 1", crashing TestConversation.test_recompute and erroring 8
  test_serialization_unfolding cases. All green now under BOTH powerit and
  sklearn.
• The new guard is a strict superset of the old one — valid clusterings
  (n_labels < n_samples) are unchanged, and with only two base clusters the
  k-selection loop is forced to k=2 regardless, so the selected clustering is
  identical; the fix only removes the crash.
• 3 unit tests added: tests/test_clusters.py::TestCalculateSilhouetteSklearn
  (2-samples/2-labels returns 0.0 not raise; single-label stays 0.0; a valid
  3-sample/2-label clustering still returns a genuine score).

Results:

• 17 new tests (tests/test_powerit_pca.py): correctness vs numpy eigh,
  deflation orthogonality, determinism, start-vector honoring, flag
  behavior, solver agreement (PC1 0 deg, PC2 2.1e-3 deg on the reference
  fixture).
• CCR angle deltas vs sklearn baseline: <= 0.02 deg across all datasets
  and variants; 28 passed / 27 xfailed unchanged.
• Evidence: the bg2050-incremental PC2 miss (10.7086 deg > 10 deg) is
  BIT-IDENTICAL under powerit - upstream incremental-state divergence,
  NOT solver difference. Its xfail annotation stands.
• Benchmark (vw, 69x125): powerit 0.765 ms avg vs sklearn 1.007 ms -
  ~1.3x faster at this size. bench_pca.py --compare-impls added.
• Final gate: 304 passed / 33 skipped / 116 xfailed / 0 failed
  (baseline 283/33/116 + 17 new + 4 benchmark-import), plus the 3
  silhouette-guard unit tests above.

Also rewords the determinism-evidence pointer to cite the journal entry
(tracked) instead of the gitignored scratch script (Copilot on #2590,
convergent with internal review).

Co-Authored-By: Claude Fable 5 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_013CiESAwcrZ6BcVfdkumnxa

▁▁▁ #2592 ▁▁▁
docs(delphi): parity journal 2026-07-04/07, PLAN refresh, replay-harness design

Three docs deliverables from the 2026-07-04/05 host sessions:

• CLJ-PARITY-FIXES-JOURNAL.md: full session entries — spr-squash
  reconciliation (closed+mergedAt:null = landed; squash titles lie),
  Copilot triage of all 83 threads (1 real blocker + 5 verified
  escalations incl. the DynamoDB consensus data-loss), review-fix PR
  fix(delphi): address Copilot review on D10-D12 stack (consensus blob keys, priority serialization, defensive fixes) #2586, per-variant xfail scoping (which unmasked bg2018/pakistan
  incremental consensus divergences the blanket had absorbed), gid
  label-swap root cause + fix (fix(delphi): keep Clojure group-id order — resolves the gid 0-1 label… #2589), seed removal (chore(delphi): remove vestigial np.random.seed(42); carry Clojure see… #2590),
  determinism verification (5 runs bit-identical except math_tick),
  Clojure randomness facts (never seeds; fixed-iteration power
  iteration; unseeded :twister sampling), R2 python-only constraint,
  powerit-pca GO.
• PLAN_DISCREPANCY_FIXES.md: D10/D11/D12 status rows corrected (were
  still "VM draft — NEEDS REWORK"; actually code-complete open PRs).
• REPLAY_HARNESS_DESIGN.md (NEW): design for the replay harness (H) —
  schedule spec as first-class input, Python driver (the future R2
  forward model, python-only per Julien 2026-07-05), Clojure driver
  narrowed to R1 certification (Mode A pure conv-update reduce; blob
  capture sufficient, EDN on demand), nondeterminism policy
  (tolerance classes, warm-start pinning, self-jitter measurement),
  storage/provenance, phased build plan H-0..H-D.

Also appends the 2026-07-06/07 session entry: the silhouette-guard fix
for the powerit-PCA default (#2591) — powerit collapsing a small
conversation to two base clusters fed a 2-point/2-label silhouette call
that sklearn rejects; calculate_silhouette_sklearn now returns 0.0 when
n_labels >= n_samples (strict superset of the old guard, chosen
clustering unchanged). Verified green; details in the journal.

The journal's "Determinism verification" entry is the tracked evidence
pointer cited from pca.py (Copilot on #2590).

Co-Authored-By: Claude Fable 5 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_013CiESAwcrZ6BcVfdkumnxa

▁▁▁ #2597 ▁▁▁
docs(delphi): storage v2 design — job-id provenance, replayable runs, dual-backend storage

Study + design from the 2026-07-06 storage/reproducibility audit:

• current-state audit: 18 Delphi_ DynamoDB tables, job_id lifecycle gaps
  (env-var-only propagation reaching 2 of 18 tables), pseudo-random
  math_tick, unconditional reset-per-run, live-Postgres re-reads per
  stage, Clojure math_main dependency, nondeterminism inventory,
  server/client consumption map
• target schema: immutable runs + run_inputs + append-only artifacts +
  first-class latest pointer (6 logical entities)
• backend-neutral repository (DynamoDB or PostgreSQL via
  DELPHI_STORAGE_BACKEND), Python + TypeScript implementations kept
  honest by a shared JSON conformance spec
• input-level reproducibility: per-run input snapshots incl. the Clojure
  math_main blob, seeds, config_effective, LLM prompt/response recording
• replay CLI (show/run/diff)
• zero-downtime migration (expand/backfill/verify/flip/contract):
  dual-write new data in both formats, progressively backfill old data
  as IMPORTED runs, serve old-format-only until 100% coverage + zero
  shadow-read divergence, then a single reversible flip with old writes
  kept on for instant rollback; contract after a trust period. Three
  flip-back invariants: importer no-clobber, bidirectional writers,
  single queue master.
• cross-referenced with REPLAY_HARNESS_DESIGN.md (schedule replay vs
  recorded-job replay)

Co-Authored-By: Claude Fable 5 noreply@anthropic.com

Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com

commit-id:30219b34

---

Stack:

• feat(delphi): Clojure-parity math fixes + storage-v2/replay design (s… #2597 ⬅
• docs(delphi): parity journal 2026-07-04/07, PLAN refresh, replay-harness design #2592
• test(delphi): require_dynamodb skips locally, fails loudly in CI #2594

---

⚠️ Part of a stack created by spr. Do not merge manually using the UI - doing so may have unexpected results.

[Copilot]

Pull request overview

Adds a new Delphi design document that captures the findings from the 2026-07-06 storage/reproducibility audit and proposes a “Storage V2” architecture aimed at reproducible, replayable runs with a unified schema and dual DynamoDB/PostgreSQL backend support.

Changes:

• Introduces a Storage V2 design covering run manifests, immutable input snapshots, append-only artifacts, and a first-class “latest” pointer.
• Documents current-state gaps (job_id provenance, nondeterminism, table inventory/ownership, live PG re-reads) and proposes a migration plan (expand/backfill/verify/flip/contract).
• Outlines a cross-language storage abstraction with a shared conformance spec and replay CLI concept.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.