feat(delphi): plumb D11 consensus + D12 priorities through to_dict / …#2572
Closed
jucor wants to merge 6 commits into
Closed
feat(delphi): plumb D11 consensus + D12 priorities through to_dict / …#2572jucor wants to merge 6 commits into
jucor wants to merge 6 commits into
Conversation
This was referenced Jun 11, 2026
Contributor
There was a problem hiding this comment.
Pull request overview
This PR aims to plumb Delphi’s D11 consensus comment selection and D12 comment priorities through Conversation.to_dict() and Conversation.to_dynamo_dict(), and adds regression tests to prevent these fields from silently being dropped during serialization.
Changes:
- Update
to_dict()to surfaceself.repness['consensus_comments']viaresult['consensus'](previously hardcoded empty). - Update
to_dynamo_dict()to surfaceself.repness['consensus_comments']viaresult['consensus'](previously hardcoded empty). - Add serialization-focused tests covering consensus + comment priorities.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 5 comments.
| File | Description |
|---|---|
delphi/polismath/conversation/conversation.py |
Changes serializer outputs to include D11 consensus in result['consensus'] for both JSON and Dynamo formats. |
delphi/tests/test_discrepancy_fixes.py |
Adds tests intended to lock in D11/D12 serializer plumb-through behavior. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
jucor
changed the title
The scalar implementations in `delphi/polismath/pca_kmeans_rep/repness.py` were test-only — production calls only `compute_group_comment_stats_df`, `select_rep_comments_df`, and `select_consensus_comments_df` (via `conv_repness`). Maintaining two parallel implementations created "where do I put this helper?" ambiguity for the upcoming D10/D11/D12 fixes (all of which add new helpers to `repness.py`) and obscured the structure of the production path. Foundation pass before D10/D11/D12 + PR 14b/14c. No behavioural change — production path untouched. ## Production code (`delphi/polismath/pca_kmeans_rep/repness.py`) DELETED (445 lines): - Primitives: `prop_test`, `two_prop_test` (no production callers). - Orchestration: `comment_stats`, `add_comparative_stats`, `repness_metric`, `finalize_cmt_stats`, `passes_by_test`, `best_agree`, `best_disagree`, `select_rep_comments`, `select_consensus_comments`. - Unused helper: `calculate_kl_divergence` (no callers anywhere in repo). KEPT: - `z_score_sig_90`, `z_score_sig_95` — trivial threshold checks consumed scalar-side; vectorizing would not save lines. ENRICHED: - `prop_test_vectorized` and `two_prop_test_vectorized` docstrings now embed the scalar-equivalent closed-form algebra. The formulas stay readable even though the scalar functions are gone. ## Tests DELETED entirely: - `tests/test_old_format_repness.py` (557 lines, scalar-only sibling of `test_repness_unit.py`). DELETED classes/methods in `tests/test_repness_unit.py`: - `TestCommentStats`, `TestSelectionFunctions`, `TestConsensusAndGroupRepness` (all scalar-only). - `TestStatisticalFunctions::test_prop_test`, `::test_two_prop_test`. MIGRATED to single vectorized DataFrame calls in `tests/test_discrepancy_fixes.py`: - D4/D5/D6 BlobInjection classes — build a DataFrame from the blob's `repness` entries, run one `prop_test_vectorized` / `two_prop_test_vectorized` call, compare element-wise. Tests the actual production code path, produces cleaner diagnostics via `.to_string()`. - `TestD5ProportionTest::test_prop_test_matches_clojure_formula` (consolidated with the n=0 boundary case). - `TestD6TwoPropTest::test_two_prop_test_matches_clojure_formula` (+ edge cases consolidated, + pi_hat=1 boundary cases). CONSOLIDATED in `tests/test_discrepancy_fixes.py`: - `TestD8FinalizeStats`'s 7 scalar boundary tests collapse into a single parametrized DataFrame test `test_repful_classification_boundary` that exercises the production `np.where(rat > rdt, 'agree', 'disagree')` logic. All boundary cases preserved (rat<rdt, rat>rdt, rat==rdt non-zero, rat==rdt==0, negative z-scores). DELETED redundant tests: - `TestSyntheticEdgeCases::test_prop_test_matches_clojure_formula_synthetic` (duplicated by migrated TestD5ProportionTest). - `TestSyntheticEdgeCases::test_clojure_repness_metric_product` (duplicated by migrated TestD7RepnessMetric). - `TestSyntheticEdgeCases::test_clojure_repful_uses_rat_vs_rdt` (purely tautological). Cross-checks in `tests/test_repness_unit.py::TestVectorizedFunctions` now use `_prop_test_reference` and `_two_prop_test_reference` closed-form staticmethods in place of scalar calls. ## Suite delta - Pre (edge @ 2dce738): 330 passed, 12 skipped, 58 xfailed. - Post: 295 passed, 12 skipped, 58 xfailed. - Delta: -35 passed, 0 failed, 0 new xfailed. Matches deleted scalar-test count exactly. ## For PR 14c (readability refactor — runs later) The deleted scalar code is the readability reference for PR 14c. Retrieve via: git show <this-commit>~1:delphi/polismath/pca_kmeans_rep/repness.py \ | sed -n '161,302p' Specifically (pre-deletion line numbers): `comment_stats` 161-201, `add_comparative_stats` 203-235, `repness_metric` 237-271, `finalize_cmt_stats` 273-301. Clojure originals at `math/src/polismath/math/repness.clj:78-100,173-188,191-200`. ## Documentation - `delphi/docs/PLAN_DISCREPANCY_FIXES.md`: added PR 14a row to the stack cross-reference table. - `delphi/docs/CLJ-PARITY-FIXES-JOURNAL.md`: appended "Session: PR 14a — Scalar deletion (2026-06-11)" entry with the full scope, suite delta, and the `git show` recipe for PR 14c. ## Out of scope (handed off separately) - Pyright pandas-stubs noise (10+ false positives on `pd.DataFrame(columns=...)` and `df['col'] = value` in `compute_group_comment_stats_df`) is pre-existing on edge HEAD; PR #2560's pyright config didn't set rule overrides. Handoff: `~/polis/HANDOFF_PYRIGHT_PANDAS_STUBS.md`. Tracked separately. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> commit-id:694a6768
Bug
---
`compute_group_comment_stats_df` in
`delphi/polismath/pca_kmeans_rep/repness.py` computed `ns = na + nd` and
`total_votes = total_agree + total_disagree`, silently dropping PASS (0)
votes from every vote count.
Clojure reference: `math/src/polismath/math/repness.clj:56-61, :70`:
(defn- count-votes [votes & [vote]]
(let [filt-fn (if vote #(= vote %) identity)]
(count (filter filt-fn votes))))
...
:ns (fnk [votes] (count-votes votes))
`count-votes` with no `vote` argument uses `identity` as the filter
predicate. In Clojure 0 is truthy, so `(filter identity ...)` keeps every
non-nil entry — including PASS. Therefore Clojure
`ns = na + nd + np` (PASS count).
Every downstream metric that consumes `ns` — `pa`, `pd`, `pat`, `pdt`,
`ra`, `rd`, `rat`, `rdt`, `agree_metric`, `disagree_metric`, and the
upcoming D11 `consensus_stats_df` — was off whenever PASS votes existed.
Fix
---
Both `total_counts` and `group_counts` now compute the count column via
`('vote', 'size')` directly in the pandas groupby aggregation. The frame
is `dropna(subset=['vote'])`-filtered upstream, so `size()` counts
exactly the non-NaN rows — including PASS (0). This is the Clojure
`(count (filter identity votes))` recipe verbatim. Both sites carry a
comment citing the Clojure reference.
Why D5 BlobInjection didn't catch this
--------------------------------------
D5's blob-injection tests pull `(n-success, n-trials)` directly from the
Clojure blob's `repness` entries and feed them to `prop_test_vectorized`.
They bypass `compute_group_comment_stats_df` entirely, so anything
downstream of `ns = na + nd` was invisible. The same gap will recur for
every fix whose stage-inputs are built by Python code. Pure-formula
tests over a tiny vote matrix are the only RED path.
Tests added
-----------
`TestNsIncludesPassVotes` in `tests/test_repness_unit.py`:
- `test_ns_includes_pass_votes` — single comment, 2 agree / 1 disagree /
2 pass → `na=2, nd=1, ns=5`.
- `test_ns_all_pass_column` — all-PASS column → `na=0, nd=0, ns=3`.
- `test_ns_mixed_with_nan_only_explicit_votes_count` — NaN never counts;
PASS always does.
- `test_other_votes_includes_other_group_pass` — `other_votes` (the
complement of group `ns`) also includes out-of-group PASS.
Suite delta
-----------
Pre: 295 passed, 12 skipped, 58 xfailed (PR 14a baseline).
Post: 299 passed, 12 skipped, 58 xfailed. Delta = +4 (the new tests).
No pre-existing test broke — the existing `TestVectorizedFunctions`
fixtures used only AGREE/DISAGREE/NaN and were never sensitive.
Cascade to D11
--------------
D11's `consensus_stats_df(vote_matrix_df)` (whole-conversation
counterpart) will mirror the same recipe at the conversation level. This
fix lands BEFORE D10 in the stack so D11's implementation, when it
arrives, starts from the corrected ns semantics. D11 should follow the
same pure-formula test pattern.
Goldens
-------
DEFERRED. Re-recording is gated on sklearn-KMeans-seeding consensus; no
golden values shift at the goldens commit until D10/D11/D12 land.
commit-id:0761e6c3
Replaces the pre-D10 botched-port `select_rep_comments_df` with a
single-pass reduce that mirrors Clojure `select-rep-comments`
(math/src/polismath/math/repness.clj:212-281).
Sits on top of PR 14a in the spr stack.
## Helpers added (top-level in `repness.py`)
- `passes_by_test(s)` — Clojure `passes-by-test?` (repness.clj:165-170).
OR'd on (rat, pat) and (rdt, pdt) z-sig-90. NO `pa >= 0.5` gate; the
pre-D10 Python gate was an over-restriction with no Clojure analog.
- `beats_best_by_test(s, current_best_z)` — Clojure `beats-best-by-test?`
(repness.clj:133-139). Strict `>` on `max(rat, rdt)`.
- `beats_best_agr(s, current_best)` — Clojure `beats-best-agr?`
(repness.clj:142-162). Four-branch agree-priority logic:
1. na == 0 AND nd == 0 → reject.
2. current_best AND current_best.ra > 1.0 → compare 4-way signed product
`ra * rat * pa * pat`.
3. current_best (else, ra <= 1.0) → compare `pa * pat` only.
4. No current_best → accept if `z90(pat)` OR `(ra > 1.0 AND pa > 0.5)`.
`current_best` stores the RAW row (Clojure repness.clj:250) so the
ra/rat/pa/pat surface stays available across iterations.
- `_finalize_row_for_output(row, *, is_best_agree=False)` — Clojure
`finalize-cmt-stats` (repness.clj:173-188) + best-agree flagging
(repness.clj:262-264). Emits `best_agree=True` and `n_agree=na` for the
best-agree slot.
## `select_rep_comments_df` rewrite
Signature now: `(stats_df, mod_out=None) -> List[Dict[str, Any]]`. Drops
the `agree_count` / `disagree_count` kwargs (Clojure has only a cap of 5).
Per-row state `{sufficient, best, best_agree}` updated by the helpers.
Final assembly: dedup best_agree from sufficient → sort by metric
(agree_metric for repful=='agree', disagree_metric for 'disagree')
→ prepend finalized+flagged best_agree → take 5 → agrees-before-disagrees.
The caller in `conv_repness` drops the `_stats_row_to_dict` wrapping step
(the new function returns finalized dicts directly).
## Two pre-D10 bugs fixed alongside the rewrite
- `pa >= 0.5 / pd >= 0.5` over-gate in the passing filter — removed (no
Clojure analog).
- "Fill from other category" + "first row" fallback blocks — deleted. The
`:best` / `:best_agree` mechanism IS the Clojure fallback.
## Tests (18 new in `tests/test_discrepancy_fixes.py`)
- TestD10PassesByTest (4): agree-side, disagree-side, neither, no pa-gate.
- TestD10BeatsBestByTest (3): None-best, max(rat,rdt), strict `>`.
- TestD10BeatsBestAgr (6): one per Clojure branch + boundary.
- TestD10SelectRepCommentsBoundary (5): empty input, single unvoted row
→ best fallback, sufficient-empty-best-agree-only, take-5 cap +
agrees-before-disagrees, **the eviction edge case** (best_agree outside
sufficient evicting 5th-highest-metric).
## Eviction edge case — flagged
`take(5)` runs AFTER prepending best_agree. If `best_agree` was kept by
`beats_best_agr` as a non-significant agree-priority fallback (failed
`passes_by_test`, qualified via Branch 4) AND `:sufficient` already has 5
entries, the prepend pushes total to 6 and `take(5)` evicts the
5th-highest-metric sufficient entry — possibly a strong dissenting view.
Mirrors Clojure exactly for blob parity. `# TODO(parity-eviction)` comment
at the take(5) site in the production code; entry added under
"Pending — needs team discussion" in PLAN.md; pinned by the synthetic
test above.
## Re-xfailed with updated reasons (D14 / D1 upstream divergence)
Six per-shared-(gid, tid) blob-comparison tests were previously xfailed as
"D5/D6/D7/D8/D10: no shared comments to compare". After D10 there ARE
shared comments (overlap ~20% on vw cold_start), but the per-(gid, tid)
stats still mismatch because Python and Clojure place different
participants in the "same" group ID. That's upstream PCA/KMeans
group-membership divergence (D14 / D1), not D10. Updated xfail reasons
point at D14 / D1. D10 itself is verified by the 18 synthetic tests.
Affected: TestD9ZScoreThresholds::test_z_values_match_clojure,
TestD5ProportionTest::test_pat_values_match_clojure_blob,
TestD6TwoPropTest::test_rat_values_match_clojure_blob,
TestD7RepnessMetric::test_repness_metric_matches_clojure_blob,
TestD8FinalizeStats::test_repful_matches_clojure_blob,
TestD10RepCommentSelection::test_rep_comments_match_clojure.
## Suite delta
- Pre (post-14a baseline): 295 passed, 12 skipped, 58 xfailed.
- Post: 313 passed, 12 skipped, 58 xfailed.
- Delta: +18 passed (the 18 new D10 synthetic tests), 0 failed, no new
xfailed.
## Documentation
- `delphi/docs/PLAN_DISCREPANCY_FIXES.md`: PR 14a row marked landed
(#2564), PR 8 (D10) marked in-flight, eviction concern added to
"Pending — needs team discussion".
- `delphi/docs/CLJ-PARITY-FIXES-JOURNAL.md`: "Session: PR 8 — D10 rep
comment selection (2026-06-11)" entry with full scope, suite delta,
and decisions log pointer.
## /goal mode
This PR is part of an autonomous run (`/goal`) targeting D10 + D11 + D12 +
golden snapshots as a stacked PR series. Decisions made autonomously
(key naming, return type, etc.) are documented in
`~/polis/D10_D11_D12_GOLDENS_DECISIONS.md` for batch user review at the
end of the run.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
commit-id:1b681fef
Replaces the pre-D11 consensus logic (per-group `pa > 0.6 for all` filter,
top 2 by `avg_pa`) with a whole-conversation per-comment-stats stage and
two independent top-5 lists (agree / disagree), matching Clojure
`consensus-stats` + `select-consensus-comments`
(math/src/polismath/math/repness.clj:284-323).
Sits on top of PR 8 (D10) in the spr stack.
## Production changes (`repness.py`)
- **`consensus_stats_df(vote_matrix_df, mod_out=None) -> pd.DataFrame`**:
new helper. Whole-conversation per-comment stats (no group split). Output
DataFrame indexed by tid with cols [na, nd, ns, pa, pd, pat, pdt].
Vectorized port of Clojure `consensus-stats` (repness.clj:284-290).
**`ns` includes PASS** (Clojure parity, repness.clj:56-61): computed as
`vote_matrix_df.notna().sum(axis=0)` rather than `na + nd`. Matches
Clojure `(count (filter identity ...))` where `0` (PASS) is truthy.
- **`select_consensus_comments_df` rewrite**: new signature
`(cons_stats) -> Dict[str, List[Dict]]`. Filters and ordering:
- agree: `pa > 0.5 AND z-sig-90(pat)`, sorted desc by `am = pa * pat`.
- disagree: `pd > 0.5 AND z-sig-90(pdt)`, sorted desc by `dm = pd * pdt`.
Cap: top 5 each side. Output: `{'agree': [...], 'disagree': [...]}`.
With PSEUDO_COUNT=2, `pa + pd = 1` exact → `pa > 0.5 ⟺ pd < 0.5`, so
the same tid cannot appear in both lists.
- **`conv_repness` grows `mod_out` kwarg**, forwarded to both
`select_rep_comments_df` and `consensus_stats_df`. Consensus is now run
unconditionally — Clojure has no `len(groups) > 1` guard.
- **`_stats_row_to_dict` deleted** — orphan after D11.
## Caller (`conversation.py`)
`_compute_repness` passes `mod_out=self.mod_out_tids` to `conv_repness`,
matching Clojure's mod-out propagation (repness.clj:222 and :296).
## Downstream output shape change
`conv.repness['consensus_comments']` was a flat list with
`{repful: 'consensus', comment_id, avg_agree, stats}` entries. After D11
it's `{'agree': [entries], 'disagree': [entries]}` matching Clojure's
math-blob shape. Each entry has Python-convention keys (decision S1):
`{comment_id, n_success, n_trials, p_success, p_test}`.
Updated consumers in the test suite:
- `tests/test_repness_smoke.py::test_repness_structure` — iterates the
new dict shape.
- `tests/test_pipeline_integrity.py::test_full_pipeline` — same.
External downstream consumers (`client-report/normalizeConsensus.js`) may
need a parallel update; flagged in the decisions log for batch review.
## Tests (13 new in `tests/test_discrepancy_fixes.py`)
- `TestD11ConsensusStatsDf` (5): basic counts, pseudocount pa/pd
smoothing, ns=0 uninformative fallback, mod_out tid filter,
**ns-includes-PASS Clojure parity**.
- `TestD11SelectConsensusBoundary` (8): empty input, clear agree
consensus, clear disagree consensus, divisive (no consensus), top-5
cap, entry-key shape, disagree-side key mapping (n_success ← nd,
p_success ← pd, p_test ← pdt), mutually-exclusive agree/disagree lists.
## `ns`-PASS fix (Clojure parity)
After D11 was first landed (PR #2567), the real-data test
`test_consensus_matches_clojure` showed 3-5/5 overlap on cold_start.
Investigation revealed that Clojure's `:ns` (via `count-votes` with
`filter identity` — repness.clj:56-61) INCLUDES PASS votes (`0` is truthy
in Clojure), while Python's `ns = na + nd` excluded them.
Fixed here by switching `consensus_stats_df` to count via
`vote_matrix_df.notna().sum(axis=0)`. After the fix, 3 of 4 dataset
variants (vw-incremental, vw-cold_start, biodiversity-cold_start) match
Clojure exactly. The `biodiversity-incremental` variant still mismatches
on the disagree side (likely residual upstream PCA/KMeans group-membership
divergence) — remains `xfail(strict=False)` and tracked in the journal.
(The companion fix for `compute_group_comment_stats_df` ships in a
separate pre-D10 commit `qyskkqkovtmn`.)
## B1 + B2 sub-agent fixes (relocated from D12 per batch review 2026-06-11)
These two fixes were originally landed in PR #2568 (D12) because the D11
sub-agent review happened AFTER D11 had been pushed. They belong in D11,
so they are squashed into this commit:
- **B1** (`polismath/conversation/conversation.py:830-837`): the
no-groups branch of `_compute_repness` returned
`'consensus_comments': []` (list). After D11, the public shape is the
dict `{'agree': [...], 'disagree': [...]}`. Downstream consumers
(test_repness_smoke, test_pipeline_integrity) iterate the dict shape
and would crash on the legacy list. Fixed to always return the dict
shape, even with no groups.
- **B2** (`tests/test_legacy_repness_comparison.py:197-205`): the
legacy-comparison test extracted `py_consensus = py_results.get(
'consensus_comments', [])` and treated it as a flat list. Post-D11
this is a dict, so the ID extraction silently produced an empty set.
Fixed to flatten the dict (agree + disagree) for the legacy comparison,
with a `legacy fallback` branch in case the value is still a list.
## Suite delta
- Pre (post-D10): 313 passed, 12 skipped, 58 xfailed.
- Post (this PR): 330 passed, 12 skipped, 55 xfailed, 3 xpassed.
- Delta: +17 passed, -3 xfailed (D11 real-data test now passes on 3 of
4 dataset variants; biodiversity-incremental remains
xfail(strict=False)). Zero regressions.
## /goal mode
Part of an autonomous stacked PR series (D10 + D11 + D12 + goldens) per
user request. Decisions are documented for batch review in
`~/polis/D10_D11_D12_GOLDENS_DECISIONS.md`.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
commit-id:bf7eabfe
Implements `comment-priorities` matching Clojure
(math/src/polismath/math/conversation.clj:648-679). Pre-D12 Python emitted
nothing for `comment_priorities`, which forced the TypeScript server's
`getNextPrioritizedComment` to fall back to uniform random comment routing.
Sits on top of PR 9 (D11) in the spr stack.
## New helpers in `pca.py`
- `pca_project_cmnts(center, comps) -> np.ndarray` (shape (n_cmnts, n_components)):
Vectorized projection. Closed-form derived from Clojure's sparsity-aware
projection (pca.clj:134-178) collapsing to a single non-nil column per
comment:
proj[i] = -sqrt(n_cmnts) * (1 + center[i]) * [pc1[i], pc2[i]]
- `compute_comment_extremity(cmnt_proj) -> np.ndarray`: L2 norm per row.
Clojure `with-proj-and-extremtiy` (conversation.clj:341-352).
## New module-level functions in `conversation.py`
- `META_PRIORITY = 7` (Clojure conversation.clj:319).
- `importance_metric(A, P, S, E) -> float` (Clojure conversation.clj:311-315).
- `priority_metric(is_meta, A, P, S, E) -> float` (Clojure conversation.clj:321-330).
Squared output. Meta: `META_PRIORITY^2 = 49`. Non-meta:
`(importance * (1 + 8*2^(-S/5)))^2` — the decay factor lets new comments
bubble up; importance falls as votes accumulate.
## New `Conversation._compute_comment_priorities()` method
Wired into `recompute()` after `_compute_repness()`. For each tid:
- Compute extremity from `pca_project_cmnts` + `compute_comment_extremity`.
- Aggregate A/D/S across all groups via `_compute_group_votes()`.
- Derive P = S - (A + D) (Clojure conversation.clj:661).
- Check `tid in self.meta_tids` for the meta branch.
- Call `priority_metric` and store under `self.comment_priorities[int(tid)]`.
The serialization infrastructure (`to_dict`, `to_dynamo_dict`,
underscore→hyphen conversion in `_convert_inner`) already existed but was
emitting empty. Now populated.
## B1 + B2 fixes folded in from D11 sub-agent review
- **B1**: `conversation.py:834` no-groups early-return now emits
`consensus_comments: {'agree': [], 'disagree': []}` (dict) instead of
`[]` (list) — restores shape consistency with the new D11 shape.
- **B2**: `test_legacy_repness_comparison.py:197` flattens the new
consensus dict before iterating, instead of crashing on
`'agree'.get('comment_id', '')`.
## Tests (11 new in `tests/test_discrepancy_fixes.py`)
- `TestD12PCAProjectComments` (5): output shape, formula verification per
row, empty inputs, L2 extremity, empty extremity.
- `TestD12PriorityMetrics` (6): `importance_metric` matches Clojure
reference value `4/9` (from conversation.clj:335 comment), extremity
boost behavior, meta priority constant = 49, non-meta squared formula,
decay factor monotonicity (low-S boosts, high-S fades), `META_PRIORITY == 7`.
## Real-data test xfailed: Clojure blob has constant priorities
vw and biodiversity blobs both have EVERY tid set to priority = 49.0
(= META_PRIORITY^2). Likely caused by Clojure's `(if 0 ...)` truthiness
quirk — 0 is truthy in Clojure (only nil/false are falsy), so any value
returned by `(get meta-tids tid 0)` triggers the meta branch.
Python correctly distinguishes meta from non-meta via Boolean set
membership, producing varied priorities 0.18-31.46. Spearman comparison
meaningless when Clojure side has zero variance (returns nan).
Test xfailed with full reason. D12 logic verified by the 11 synthetic
tests. Logged for batch review — Python may be MORE correct than Clojure
here.
## Suite delta
- Pre (post-D11): 325 passed, 12 skipped, 58 xfailed.
- Post (this PR): 336 passed, 12 skipped, 56 xfailed, 2 xpassed.
- Delta: +11 (the 11 new D12 synthetic tests), 0 failed, 2 xfailed →
xpassed (the cold_start D12 tests now run cleanly; the new xfail is
on a different test).
## /goal mode
Autonomous stack PR (D10 + D11 + D12 + goldens). Decisions documented in
`~/polis/D10_D11_D12_GOLDENS_DECISIONS.md` for batch user review.
D11 sub-agent flagged additional concerns (B3: math-blob plumbing
hardcodes empty `consensus` in to_dict/to_dynamo_dict; D11 data
computed-but-unused at serialization layer). NOT addressed here —
larger scope than D11/D12, deliberate per S1. Documented for batch
review.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
commit-id:d02440bf
…to_dynamo_dict commit-id:ff4764e4
Delphi Coverage Report
|
jucor
added a commit
that referenced
this pull request
Jul 10, 2026
…quash of 15 PRs) SQUASH MERGE of the 15-PR Clojure-parity + design stack into edge (spr stack, PRs #2564–#2597). GitHub's default squash message keeps only the top PR's description; this message preserves every squashed PR's commit message so no history is lost. What it does: • Clojure-parity math fixes D10–D15 and supporting changes (rep-comment selection, consensus selection, comment priorities, group-id order, ns/PASS-vote counting, repness cleanup) so the Python math pipeline matches the legacy Clojure output. • powerit-pca legacy-mode PCA port (Clojure-parity power iteration). • DynamoDB consensus serialization/robustness fixes (Decimal conversion, dict-shape normalization incl. present-but-None guard). • Test-infra: require_dynamodb skips locally / fails loudly in CI. • Docs: parity journal + PLAN refresh, replay-harness design (R1/R2), storage-v2 / job-id reproducibility design, and archival of stale docs. ─────────────────────────────────────────────────────────────────────── Squashed PRs (bottom → top of stack): ─────────────────────────────────────────────────────────────────────── ▁▁▁ #2564 ▁▁▁ refactor(delphi): delete dead scalar paths in repness.py (PR 14a) The scalar implementations in `delphi/polismath/pca_kmeans_rep/repness.py` were test-only — production calls only `compute_group_comment_stats_df`, `select_rep_comments_df`, and `select_consensus_comments_df` (via `conv_repness`). Maintaining two parallel implementations created "where do I put this helper?" ambiguity for the upcoming D10/D11/D12 fixes (all of which add new helpers to `repness.py`) and obscured the structure of the production path. Foundation pass before D10/D11/D12 + PR 14b/14c. No behavioural change — production path untouched. ## Production code (`delphi/polismath/pca_kmeans_rep/repness.py`) DELETED (445 lines): - Primitives: `prop_test`, `two_prop_test` (no production callers). - Orchestration: `comment_stats`, `add_comparative_stats`, `repness_metric`, `finalize_cmt_stats`, `passes_by_test`, `best_agree`, `best_disagree`, `select_rep_comments`, `select_consensus_comments`. - Unused helper: `calculate_kl_divergence` (no callers anywhere in repo). KEPT: - `z_score_sig_90`, `z_score_sig_95` — trivial threshold checks consumed scalar-side; vectorizing would not save lines. ENRICHED: - `prop_test_vectorized` and `two_prop_test_vectorized` docstrings now embed the scalar-equivalent closed-form algebra. The formulas stay readable even though the scalar functions are gone. ## Tests DELETED entirely: - `tests/test_old_format_repness.py` (557 lines, scalar-only sibling of `test_repness_unit.py`). DELETED classes/methods in `tests/test_repness_unit.py`: - `TestCommentStats`, `TestSelectionFunctions`, `TestConsensusAndGroupRepness` (all scalar-only). - `TestStatisticalFunctions::test_prop_test`, `::test_two_prop_test`. MIGRATED to single vectorized DataFrame calls in `tests/test_discrepancy_fixes.py`: - D4/D5/D6 BlobInjection classes — build a DataFrame from the blob's `repness` entries, run one `prop_test_vectorized` / `two_prop_test_vectorized` call, compare element-wise. Tests the actual production code path, produces cleaner diagnostics via `.to_string()`. - `TestD5ProportionTest::test_prop_test_matches_clojure_formula` (consolidated with the n=0 boundary case). - `TestD6TwoPropTest::test_two_prop_test_matches_clojure_formula` (+ edge cases consolidated, + pi_hat=1 boundary cases). CONSOLIDATED in `tests/test_discrepancy_fixes.py`: - `TestD8FinalizeStats`'s 7 scalar boundary tests collapse into a single parametrized DataFrame test `test_repful_classification_boundary` that exercises the production `np.where(rat > rdt, 'agree', 'disagree')` logic. All boundary cases preserved (rat<rdt, rat>rdt, rat==rdt non-zero, rat==rdt==0, negative z-scores). DELETED redundant tests: - `TestSyntheticEdgeCases::test_prop_test_matches_clojure_formula_synthetic` (duplicated by migrated TestD5ProportionTest). - `TestSyntheticEdgeCases::test_clojure_repness_metric_product` (duplicated by migrated TestD7RepnessMetric). - `TestSyntheticEdgeCases::test_clojure_repful_uses_rat_vs_rdt` (purely tautological). Cross-checks in `tests/test_repness_unit.py::TestVectorizedFunctions` now use `_prop_test_reference` and `_two_prop_test_reference` closed-form staticmethods in place of scalar calls. ## Suite delta - Pre (edge @ 2dce738): 330 passed, 12 skipped, 58 xfailed. - Post: 295 passed, 12 skipped, 58 xfailed. - Delta: -35 passed, 0 failed, 0 new xfailed. Matches deleted scalar-test count exactly. ## For PR 14c (readability refactor — runs later) The deleted scalar code is the readability reference for PR 14c. Retrieve via: git show <this-commit>~1:delphi/polismath/pca_kmeans_rep/repness.py \ | sed -n '161,302p' Specifically (pre-deletion line numbers): `comment_stats` 161-201, `add_comparative_stats` 203-235, `repness_metric` 237-271, `finalize_cmt_stats` 273-301. Clojure originals at `math/src/polismath/math/repness.clj:78-100,173-188,191-200`. ## Documentation - `delphi/docs/PLAN_DISCREPANCY_FIXES.md`: added PR 14a row to the stack cross-reference table. - `delphi/docs/CLJ-PARITY-FIXES-JOURNAL.md`: appended "Session: PR 14a — Scalar deletion (2026-06-11)" entry with the full scope, suite delta, and the `git show` recipe for PR 14c. ## Out of scope (handed off separately) - Pyright pandas-stubs noise (10+ false positives on `pd.DataFrame(columns=...)` and `df['col'] = value` in `compute_group_comment_stats_df`) is pre-existing on edge HEAD; PR #2560's pyright config didn't set rule overrides. Handoff: `~/polis/HANDOFF_PYRIGHT_PANDAS_STUBS.md`. Tracked separately. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> ▁▁▁ #2570 ▁▁▁ fix(delphi): ns includes PASS votes (Clojure parity, pre-D10) Bug --- `compute_group_comment_stats_df` in `delphi/polismath/pca_kmeans_rep/repness.py` computed `ns = na + nd` and `total_votes = total_agree + total_disagree`, silently dropping PASS (0) votes from every vote count. Clojure reference: `math/src/polismath/math/repness.clj:56-61, :70`: (defn- count-votes [votes & [vote]] (let [filt-fn (if vote #(= vote %) identity)] (count (filter filt-fn votes)))) ... :ns (fnk [votes] (count-votes votes)) `count-votes` with no `vote` argument uses `identity` as the filter predicate. In Clojure 0 is truthy, so `(filter identity ...)` keeps every non-nil entry — including PASS. Therefore Clojure `ns = na + nd + np` (PASS count). Every downstream metric that consumes `ns` — `pa`, `pd`, `pat`, `pdt`, `ra`, `rd`, `rat`, `rdt`, `agree_metric`, `disagree_metric`, and the upcoming D11 `consensus_stats_df` — was off whenever PASS votes existed. Fix --- Both `total_counts` and `group_counts` now compute the count column via `('vote', 'size')` directly in the pandas groupby aggregation. The frame is `dropna(subset=['vote'])`-filtered upstream, so `size()` counts exactly the non-NaN rows — including PASS (0). This is the Clojure `(count (filter identity votes))` recipe verbatim. Both sites carry a comment citing the Clojure reference. Why D5 BlobInjection didn't catch this -------------------------------------- D5's blob-injection tests pull `(n-success, n-trials)` directly from the Clojure blob's `repness` entries and feed them to `prop_test_vectorized`. They bypass `compute_group_comment_stats_df` entirely, so anything downstream of `ns = na + nd` was invisible. The same gap will recur for every fix whose stage-inputs are built by Python code. Pure-formula tests over a tiny vote matrix are the only RED path. Tests added ----------- `TestNsIncludesPassVotes` in `tests/test_repness_unit.py`: - `test_ns_includes_pass_votes` — single comment, 2 agree / 1 disagree / 2 pass → `na=2, nd=1, ns=5`. - `test_ns_all_pass_column` — all-PASS column → `na=0, nd=0, ns=3`. - `test_ns_mixed_with_nan_only_explicit_votes_count` — NaN never counts; PASS always does. - `test_other_votes_includes_other_group_pass` — `other_votes` (the complement of group `ns`) also includes out-of-group PASS. Suite delta ----------- Pre: 295 passed, 12 skipped, 58 xfailed (PR 14a baseline). Post: 299 passed, 12 skipped, 58 xfailed. Delta = +4 (the new tests). No pre-existing test broke — the existing `TestVectorizedFunctions` fixtures used only AGREE/DISAGREE/NaN and were never sensitive. Cascade to D11 -------------- D11's `consensus_stats_df(vote_matrix_df)` (whole-conversation counterpart) will mirror the same recipe at the conversation level. This fix lands BEFORE D10 in the stack so D11's implementation, when it arrives, starts from the corrected ns semantics. D11 should follow the same pure-formula test pattern. Goldens ------- DEFERRED. Re-recording is gated on sklearn-KMeans-seeding consensus; no golden values shift at the goldens commit until D10/D11/D12 land. ▁▁▁ #2566 ▁▁▁ feat(delphi): D10 — Clojure-parity rep comment selection (PR 8) Replaces the pre-D10 botched-port `select_rep_comments_df` with a single-pass reduce that mirrors Clojure `select-rep-comments` (math/src/polismath/math/repness.clj:212-281). Sits on top of PR 14a in the spr stack. ## Helpers added (top-level in `repness.py`) - `passes_by_test(s)` — Clojure `passes-by-test?` (repness.clj:165-170). OR'd on (rat, pat) and (rdt, pdt) z-sig-90. NO `pa >= 0.5` gate; the pre-D10 Python gate was an over-restriction with no Clojure analog. - `beats_best_by_test(s, current_best_z)` — Clojure `beats-best-by-test?` (repness.clj:133-139). Strict `>` on `max(rat, rdt)`. - `beats_best_agr(s, current_best)` — Clojure `beats-best-agr?` (repness.clj:142-162). Four-branch agree-priority logic: 1. na == 0 AND nd == 0 → reject. 2. current_best AND current_best.ra > 1.0 → compare 4-way signed product `ra * rat * pa * pat`. 3. current_best (else, ra <= 1.0) → compare `pa * pat` only. 4. No current_best → accept if `z90(pat)` OR `(ra > 1.0 AND pa > 0.5)`. `current_best` stores the RAW row (Clojure repness.clj:250) so the ra/rat/pa/pat surface stays available across iterations. - `_finalize_row_for_output(row, *, is_best_agree=False)` — Clojure `finalize-cmt-stats` (repness.clj:173-188) + best-agree flagging (repness.clj:262-264). Emits `best_agree=True` and `n_agree=na` for the best-agree slot. ## `select_rep_comments_df` rewrite Signature now: `(stats_df, mod_out=None) -> List[Dict[str, Any]]`. Drops the `agree_count` / `disagree_count` kwargs (Clojure has only a cap of 5). Per-row state `{sufficient, best, best_agree}` updated by the helpers. Final assembly: dedup best_agree from sufficient → sort by metric (agree_metric for repful=='agree', disagree_metric for 'disagree') → prepend finalized+flagged best_agree → take 5 → agrees-before-disagrees. The caller in `conv_repness` drops the `_stats_row_to_dict` wrapping step (the new function returns finalized dicts directly). ## Two pre-D10 bugs fixed alongside the rewrite - `pa >= 0.5 / pd >= 0.5` over-gate in the passing filter — removed (no Clojure analog). - "Fill from other category" + "first row" fallback blocks — deleted. The `:best` / `:best_agree` mechanism IS the Clojure fallback. ## Tests (18 new in `tests/test_discrepancy_fixes.py`) - TestD10PassesByTest (4): agree-side, disagree-side, neither, no pa-gate. - TestD10BeatsBestByTest (3): None-best, max(rat,rdt), strict `>`. - TestD10BeatsBestAgr (6): one per Clojure branch + boundary. - TestD10SelectRepCommentsBoundary (5): empty input, single unvoted row → best fallback, sufficient-empty-best-agree-only, take-5 cap + agrees-before-disagrees, **the eviction edge case** (best_agree outside sufficient evicting 5th-highest-metric). ## Eviction edge case — flagged `take(5)` runs AFTER prepending best_agree. If `best_agree` was kept by `beats_best_agr` as a non-significant agree-priority fallback (failed `passes_by_test`, qualified via Branch 4) AND `:sufficient` already has 5 entries, the prepend pushes total to 6 and `take(5)` evicts the 5th-highest-metric sufficient entry — possibly a strong dissenting view. Mirrors Clojure exactly for blob parity. `# TODO(parity-eviction)` comment at the take(5) site in the production code; entry added under "Pending — needs team discussion" in PLAN.md; pinned by the synthetic test above. ## Re-xfailed with updated reasons (D14 / D1 upstream divergence) Six per-shared-(gid, tid) blob-comparison tests were previously xfailed as "D5/D6/D7/D8/D10: no shared comments to compare". After D10 there ARE shared comments (overlap ~20% on vw cold_start), but the per-(gid, tid) stats still mismatch because Python and Clojure place different participants in the "same" group ID. That's upstream PCA/KMeans group-membership divergence (D14 / D1), not D10. Updated xfail reasons point at D14 / D1. D10 itself is verified by the 18 synthetic tests. Affected: TestD9ZScoreThresholds::test_z_values_match_clojure, TestD5ProportionTest::test_pat_values_match_clojure_blob, TestD6TwoPropTest::test_rat_values_match_clojure_blob, TestD7RepnessMetric::test_repness_metric_matches_clojure_blob, TestD8FinalizeStats::test_repful_matches_clojure_blob, TestD10RepCommentSelection::test_rep_comments_match_clojure. ## Suite delta - Pre (post-14a baseline): 295 passed, 12 skipped, 58 xfailed. - Post: 313 passed, 12 skipped, 58 xfailed. - Delta: +18 passed (the 18 new D10 synthetic tests), 0 failed, no new xfailed. ## Documentation - `delphi/docs/PLAN_DISCREPANCY_FIXES.md`: PR 14a row marked landed (#2564), PR 8 (D10) marked in-flight, eviction concern added to "Pending — needs team discussion". - `delphi/docs/CLJ-PARITY-FIXES-JOURNAL.md`: "Session: PR 8 — D10 rep comment selection (2026-06-11)" entry with full scope, suite delta, and decisions log pointer. ## /goal mode This PR is part of an autonomous run (`/goal`) targeting D10 + D11 + D12 + golden snapshots as a stacked PR series. Decisions made autonomously (key naming, return type, etc.) are documented in `~/polis/D10_D11_D12_GOLDENS_DECISIONS.md` for batch user review at the end of the run. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> ▁▁▁ #2567 ▁▁▁ feat(delphi): D11 — Clojure-parity consensus comment selection (PR 9) Replaces the pre-D11 consensus logic (per-group `pa > 0.6 for all` filter, top 2 by `avg_pa`) with a whole-conversation per-comment-stats stage and two independent top-5 lists (agree / disagree), matching Clojure `consensus-stats` + `select-consensus-comments` (math/src/polismath/math/repness.clj:284-323). Sits on top of PR 8 (D10) in the spr stack. ## Production changes (`repness.py`) - **`consensus_stats_df(vote_matrix_df, mod_out=None) -> pd.DataFrame`**: new helper. Whole-conversation per-comment stats (no group split). Output DataFrame indexed by tid with cols [na, nd, ns, pa, pd, pat, pdt]. Vectorized port of Clojure `consensus-stats` (repness.clj:284-290). **`ns` includes PASS** (Clojure parity, repness.clj:56-61): computed as `vote_matrix_df.notna().sum(axis=0)` rather than `na + nd`. Matches Clojure `(count (filter identity ...))` where `0` (PASS) is truthy. - **`select_consensus_comments_df` rewrite**: new signature `(cons_stats) -> Dict[str, List[Dict]]`. Filters and ordering: - agree: `pa > 0.5 AND z-sig-90(pat)`, sorted desc by `am = pa * pat`. - disagree: `pd > 0.5 AND z-sig-90(pdt)`, sorted desc by `dm = pd * pdt`. Cap: top 5 each side. Output: `{'agree': [...], 'disagree': [...]}`. With PSEUDO_COUNT=2, `pa + pd = 1` exact → `pa > 0.5 ⟺ pd < 0.5`, so the same tid cannot appear in both lists. - **`conv_repness` grows `mod_out` kwarg**, forwarded to both `select_rep_comments_df` and `consensus_stats_df`. Consensus is now run unconditionally — Clojure has no `len(groups) > 1` guard. - **`_stats_row_to_dict` deleted** — orphan after D11. ## Caller (`conversation.py`) `_compute_repness` passes `mod_out=self.mod_out_tids` to `conv_repness`, matching Clojure's mod-out propagation (repness.clj:222 and :296). ## Downstream output shape change `conv.repness['consensus_comments']` was a flat list with `{repful: 'consensus', comment_id, avg_agree, stats}` entries. After D11 it's `{'agree': [entries], 'disagree': [entries]}` matching Clojure's math-blob shape. Each entry has Python-convention keys (decision S1): `{comment_id, n_success, n_trials, p_success, p_test}`. Updated consumers in the test suite: - `tests/test_repness_smoke.py::test_repness_structure` — iterates the new dict shape. - `tests/test_pipeline_integrity.py::test_full_pipeline` — same. External downstream consumers (`client-report/normalizeConsensus.js`) may need a parallel update; flagged in the decisions log for batch review. ## Tests (13 new in `tests/test_discrepancy_fixes.py`) - `TestD11ConsensusStatsDf` (5): basic counts, pseudocount pa/pd smoothing, ns=0 uninformative fallback, mod_out tid filter, **ns-includes-PASS Clojure parity**. - `TestD11SelectConsensusBoundary` (8): empty input, clear agree consensus, clear disagree consensus, divisive (no consensus), top-5 cap, entry-key shape, disagree-side key mapping (n_success ← nd, p_success ← pd, p_test ← pdt), mutually-exclusive agree/disagree lists. ## `ns`-PASS fix (Clojure parity) After D11 was first landed (PR #2567), the real-data test `test_consensus_matches_clojure` showed 3-5/5 overlap on cold_start. Investigation revealed that Clojure's `:ns` (via `count-votes` with `filter identity` — repness.clj:56-61) INCLUDES PASS votes (`0` is truthy in Clojure), while Python's `ns = na + nd` excluded them. Fixed here by switching `consensus_stats_df` to count via `vote_matrix_df.notna().sum(axis=0)`. After the fix, 3 of 4 dataset variants (vw-incremental, vw-cold_start, biodiversity-cold_start) match Clojure exactly. The `biodiversity-incremental` variant still mismatches on the disagree side (likely residual upstream PCA/KMeans group-membership divergence) — remains `xfail(strict=False)` and tracked in the journal. (The companion fix for `compute_group_comment_stats_df` ships in a separate pre-D10 commit `qyskkqkovtmn`.) ## B1 + B2 sub-agent fixes (relocated from D12 per batch review 2026-06-11) These two fixes were originally landed in PR #2568 (D12) because the D11 sub-agent review happened AFTER D11 had been pushed. They belong in D11, so they are squashed into this commit: - **B1** (`polismath/conversation/conversation.py:830-837`): the no-groups branch of `_compute_repness` returned `'consensus_comments': []` (list). After D11, the public shape is the dict `{'agree': [...], 'disagree': [...]}`. Downstream consumers (test_repness_smoke, test_pipeline_integrity) iterate the dict shape and would crash on the legacy list. Fixed to always return the dict shape, even with no groups. - **B2** (`tests/test_legacy_repness_comparison.py:197-205`): the legacy-comparison test extracted `py_consensus = py_results.get( 'consensus_comments', [])` and treated it as a flat list. Post-D11 this is a dict, so the ID extraction silently produced an empty set. Fixed to flatten the dict (agree + disagree) for the legacy comparison, with a `legacy fallback` branch in case the value is still a list. ## Suite delta - Pre (post-D10): 313 passed, 12 skipped, 58 xfailed. - Post (this PR): 330 passed, 12 skipped, 55 xfailed, 3 xpassed. - Delta: +17 passed, -3 xfailed (D11 real-data test now passes on 3 of 4 dataset variants; biodiversity-incremental remains xfail(strict=False)). Zero regressions. ## /goal mode Part of an autonomous stacked PR series (D10 + D11 + D12 + goldens) per user request. Decisions are documented for batch review in `~/polis/D10_D11_D12_GOLDENS_DECISIONS.md`. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> ▁▁▁ #2568 ▁▁▁ feat(delphi): D12 — comment priorities (PR 11) Implements `comment-priorities` matching Clojure (math/src/polismath/math/conversation.clj:648-679). Pre-D12 Python emitted nothing for `comment_priorities`, which forced the TypeScript server's `getNextPrioritizedComment` to fall back to uniform random comment routing. Sits on top of PR 9 (D11) in the spr stack. ## New helpers in `pca.py` - `pca_project_cmnts(center, comps) -> np.ndarray` (shape (n_cmnts, n_components)): Vectorized projection. Closed-form derived from Clojure's sparsity-aware projection (pca.clj:134-178) collapsing to a single non-nil column per comment: proj[i] = -sqrt(n_cmnts) * (1 + center[i]) * [pc1[i], pc2[i]] - `compute_comment_extremity(cmnt_proj) -> np.ndarray`: L2 norm per row. Clojure `with-proj-and-extremtiy` (conversation.clj:341-352). ## New module-level functions in `conversation.py` - `META_PRIORITY = 7` (Clojure conversation.clj:319). - `importance_metric(A, P, S, E) -> float` (Clojure conversation.clj:311-315). - `priority_metric(is_meta, A, P, S, E) -> float` (Clojure conversation.clj:321-330). Squared output. Meta: `META_PRIORITY^2 = 49`. Non-meta: `(importance * (1 + 8*2^(-S/5)))^2` — the decay factor lets new comments bubble up; importance falls as votes accumulate. ## New `Conversation._compute_comment_priorities()` method Wired into `recompute()` after `_compute_repness()`. For each tid: - Compute extremity from `pca_project_cmnts` + `compute_comment_extremity`. - Aggregate A/D/S across all groups via `_compute_group_votes()`. - Derive P = S - (A + D) (Clojure conversation.clj:661). - Check `tid in self.meta_tids` for the meta branch. - Call `priority_metric` and store under `self.comment_priorities[int(tid)]`. The serialization infrastructure (`to_dict`, `to_dynamo_dict`, underscore→hyphen conversion in `_convert_inner`) already existed but was emitting empty. Now populated. ## B1 + B2 fixes folded in from D11 sub-agent review - **B1**: `conversation.py:834` no-groups early-return now emits `consensus_comments: {'agree': [], 'disagree': []}` (dict) instead of `[]` (list) — restores shape consistency with the new D11 shape. - **B2**: `test_legacy_repness_comparison.py:197` flattens the new consensus dict before iterating, instead of crashing on `'agree'.get('comment_id', '')`. ## Tests (11 new in `tests/test_discrepancy_fixes.py`) - `TestD12PCAProjectComments` (5): output shape, formula verification per row, empty inputs, L2 extremity, empty extremity. - `TestD12PriorityMetrics` (6): `importance_metric` matches Clojure reference value `4/9` (from conversation.clj:335 comment), extremity boost behavior, meta priority constant = 49, non-meta squared formula, decay factor monotonicity (low-S boosts, high-S fades), `META_PRIORITY == 7`. ## Real-data test xfailed: Clojure blob has constant priorities vw and biodiversity blobs both have EVERY tid set to priority = 49.0 (= META_PRIORITY^2). Likely caused by Clojure's `(if 0 ...)` truthiness quirk — 0 is truthy in Clojure (only nil/false are falsy), so any value returned by `(get meta-tids tid 0)` triggers the meta branch. Python correctly distinguishes meta from non-meta via Boolean set membership, producing varied priorities 0.18-31.46. Spearman comparison meaningless when Clojure side has zero variance (returns nan). Test xfailed with full reason. D12 logic verified by the 11 synthetic tests. Logged for batch review — Python may be MORE correct than Clojure here. ## Suite delta - Pre (post-D11): 325 passed, 12 skipped, 58 xfailed. - Post (this PR): 336 passed, 12 skipped, 56 xfailed, 2 xpassed. - Delta: +11 (the 11 new D12 synthetic tests), 0 failed, 2 xfailed → xpassed (the cold_start D12 tests now run cleanly; the new xfail is on a different test). ## /goal mode Autonomous stack PR (D10 + D11 + D12 + goldens). Decisions documented in `~/polis/D10_D11_D12_GOLDENS_DECISIONS.md` for batch user review. D11 sub-agent flagged additional concerns (B3: math-blob plumbing hardcodes empty `consensus` in to_dict/to_dynamo_dict; D11 data computed-but-unused at serialization layer). NOT addressed here — larger scope than D11/D12, deliberate per S1. Documented for batch review. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> ▁▁▁ #2572 ▁▁▁ feat(delphi): plumb D11 consensus + D12 priorities through to_dict / to_dynamo_dict ▁▁▁ #2586 ▁▁▁ fix(delphi): address Copilot review on D10-D12 stack (consensus blob keys, priority serialization, defensive fixes) Verified triage of all 83 Copilot review threads on PRs #2564-#2573: 1 real blocker + 4 escalations confirmed by direct code verification; the rest nitpicks / already-fixed / intentional bug-mirrors. All fixes TDD RED->GREEN. - Consensus entry keys -> Clojure blob shape {tid, n-success, n-trials, p-success, p-test} (was Python-convention comment_id/n_success/...). Narrows the S1 deferral: these entries flow raw into result['consensus'], where server-helpers.ts:298-313 and client-report majorityStrict.jsx:23-27 pluck `tid` - the Python keys broke both consumers. Rep-comment entries keep comment_id until the deferred math-blob alignment PR. - DynamoDB writer read D11 consensus from a key to_dynamo_dict never emits (repness.consensus_comments) -> always wrote the empty default; D11 data never reached Delphi_PCAResults. Now reads top-level result['consensus']. The round-trip test had stubbed to_dynamo_dict with the writer's wrong nested shape - stub corrected to the real producer shape. - to_dynamo_dict comment priorities: int(value) -> Decimal-preserving. int() floors sub-1 priorities to 0 (real formula spans ~0.18-31.46 per D12.6), which the TS server's weighted routing reads as "no priority data". Harmless today (bug-mirror pins 49.0), landmine once #2571 resolves. Legacy writer path Decimal-wrapped too. - DynamoDB reader normalizes legacy list-shaped consensus to the dict shape (pre-D11 blobs). - mod_out truthiness -> `is not None` x2 in repness.py (numpy array/Index-safe). - _compute_comment_priorities fails closed (error log + empty dict) on PCA/columns desync instead of silently zip-truncating extremities. - bench_repness.py imported the 14a-deleted comment_stats (ImportError on any benchmark run); benchmark import test added. - Per-variant xfails replace blanket xfail(strict=False) on D11/D12 real-data tests, so the variants that match Clojure gate again. DISCOVERY: scoping the blanket unmasked two previously-invisible incremental divergences (bg2018-incremental, pakistan-incremental consensus vs Clojure) that the blanket had silently absorbed - documented as known-bad incremental xfails, same family as biodiversity-incremental, deferred to the sequential-parity work. 3 pre-existing CCR failures (verified identical on edge 722640e) marked with precise per-variant reasons. PGR regression tests skip with the 2026-06-11 goldens-deferral reason - the mark S3-5 claimed to add but never committed. - test_repness_smoke consensus-entry structure assertion updated to the Clojure key shape (exact key-set check). - ns docstrings corrected (ns includes PASS post ns-PASS fix; pa+pd <= 1 reasoning in select_consensus_comments_df). - PERF deferral comment on the _compute_group_votes scan in _compute_comment_priorities (follow-up issue to be filed). Baseline (2026-07-04, stack top, --include-local): 13 failed / 476 passed / 18 skipped / 143 xfailed. All 13 accounted for: 10 stale-PGR comparisons (now skipped per deferral), 3 pre-existing CCR (now precise xfails). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013CiESAwcrZ6BcVfdkumnxa ▁▁▁ #2593 ▁▁▁ fix(delphi): Decimal-convert D11 consensus for DynamoDB write Caught by CI's test_math_pipeline_runs_e2e on the stack branch (2026-07-05): "Float types are not supported. Use Decimal types instead." at the Delphi_PCAResults put_item. Once the writer read top-level `consensus` (the key to_dynamo_dict actually emits — fixed in the PR below this one), REAL D11 consensus data flowed to DynamoDB for the first time, carrying float p-success/p-test values. Only the LEGACY writer branch Decimal-converted its payload; the pre-formatted branch wrote `dynamo_data['consensus']` raw into the Item, and boto3's TypeSerializer rejects raw floats. Invisible to every local gate, three ways: the e2e test needs DynamoDB (local skip list), the consensus round-trip tests use MagicMock (no TypeSerializer runs), and the float-serialization pins covered priorities but not consensus. Fix, TDD RED->GREEN with boto3's REAL TypeSerializer: - to_dynamo_dict: consensus surface now passes through float_to_decimal (same treatment as pca and comment_priorities in the same function). - DynamoDB writer Site 1: belt-and-braces _replace_floats_with_decimals at the boto3 boundary, mirroring the legacy branch (idempotent on already-converted data). - New TestToDynamoDictConsensusSerialization (Layer 2c) pins the exact failure through the real serializer; two mock-era expectations updated from float-identity to value-preserving comparisons. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013CiESAwcrZ6BcVfdkumnxa ▁▁▁ #2594 ▁▁▁ test(delphi): require_dynamodb skips locally, fails loudly in CI DynamoDB-gated tests (test_math_pipeline_runs_e2e, test_batch_id) needed a blanket --ignore in local runs because require_dynamodb hard-failed when the service was absent. Now: pytest.skip locally with a how-to-run hint (`docker run --rm -d -p 8002:8000 amazon/dynamodb-local` + DYNAMODB_ENDPOINT), pytest.fail in CI — where DynamoDB is provisioned and its absence is an infrastructure failure that must not silently disable the only end-to-end gate (the 2026-07-05 consensus-float crash was caught precisely because CI runs it). Discriminator is GITHUB_ACTIONS, deliberately NOT the generic CI: local supply-chain wrappers (SafeDep pmg) inject CI=true into wrapped package-manager invocations, which would force the loud-fail path on developer machines (observed 2026-07-05 on `uv run` via the pmg alias). Verified matrix: local+down 5 skipped / local+up(8002) e2e PASSES against real DynamoDB (the consensus-Decimal fix validated end-to-end) / GITHUB_ACTIONS+down errors loudly. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013CiESAwcrZ6BcVfdkumnxa ▁▁▁ #2573 ▁▁▁ docs(delphi): archive 33 stale leftover docs, fix stale references Audit of delphi/docs/ against the current code (2026-06-11). Per review feedback (Colin): the stale docs are MOVED to delphi/docs/archive/ rather than deleted — they capture the original research and design intent of the 2025 build-out, which is an independent deliverable worth keeping as raw material for future models to mine. archive/CLAUDE.md marks the folder as historical, instructs agents not to treat it as current documentation, and indexes each file's original purpose and the reason it was archived. - Move 33 stale docs to docs/archive/ byte-identical (git records renames): completed one-off fix memos and session logs, unimplemented design proposals (IGAS topic-consensus metrics, job-id migration, DAG job system, spatial topic prioritization, UMAP viz plan, 16-week topic-agenda migration), and docs describing deleted architecture (legacy poller, run_tests.py, simplified tests, eda_notebooks, 600/802 scripts, custom power-iteration PCA). - Amend 15 surviving docs: status headers on handoffs and deep-analysis-for-julien/07 (which D-fixes are merged vs open), deleted-script references (start_poller.sh, 600_*.py), table name DelphiJobQueue -> Delphi_JobQueue, uv instead of pip/venv, TopicAgenda.jsx -> .tsx, versioned-key delimiter correction. - Rewrite DOCUMENTATION_DIRECTORY.md to index surviving docs + archive. - Fix references to moved docs and deleted scripts in delphi/CLAUDE.md and DELPHI_JOB_SYSTEM_TROUBLESHOOTING.md (run_delphi.sh -> run_delphi.py, start_poller.sh -> start_poller.py, nonexistent reset_database.sh, dead absolute-path link to DATABASE_NAMING_PROPOSAL.md). Kept in place deliberately, as they document still-unfixed bugs: ZID_EXPOSURE_AUDIT.md (zid still exposed in delphi API responses) and TOPIC_LABEL_MISALIGNMENT_ANALYSIS.md (700_datamapplot label sorting). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> ▁▁▁ #2589 ▁▁▁ fix(delphi): keep Clojure group-id order — resolves the gid 0-1 label swap Root cause of the label swap confirmed by the S3-4 trace (2026-06-11: Python g0 = Clojure g1 with 50/50 identical membership on vw-cold_start): _compute_clusters re-sorted group clusters by size (descending) and reassigned ids. Clojure assigns group ids by first-k-distinct encounter order over base-cluster centers (init-clusters, clusters.clj:55-64), keeps them through merge lineage, and orders output by sort-by :id (conversation.clj:437) - it never re-orders by size. The base-cluster level already preserved k-means id order for exactly this reason (K-inv); the group level now follows the same rule. Fix: remove the size re-sort + id reassignment; keep k-means label order (sklearn preserves init-index-to-label correspondence, and the base-center row order is already Clojure-parity per K-inv). Pinned by a synthetic test: with the smaller group's center encountered first, group id 0 must be the smaller group (fails under any size re-sort). Test-gate harvest, verified against a full --include-local run: - TestD8FinalizeStats::test_repful_matches_clojure_blob: xfail LIFTED on 9/11 variants (all but vw-incremental / pakistan-incremental, which keep per-variant xfails for residual incremental trajectory divergence). - D9 significance-sets + D10 rep-selection: biodiversity-cold_start now matches Clojure exactly and gates; other variants keep per-variant xfails (residual per-(gid,tid) membership/stat divergence). - z-values / rat-values xfail reasons corrected: the label swap is now FALSIFIED as their cause (fix did not flip them); residual cause is group-membership divergence. - D12 priorities known-bad incremental lists refined: FLI-incremental and bg2050-incremental Clojure blobs carry the all-49 truthy-0 signature, match Python's #2571 mirror, and now gate. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013CiESAwcrZ6BcVfdkumnxa ▁▁▁ #2590 ▁▁▁ chore(delphi): remove vestigial np.random.seed(42); carry Clojure seeding note The global np.random.seed(42) in cluster_dataframe was the only `random` reference in the module and seeded an RNG nothing ever draws from: k-means init is first-k-distinct (deterministic by construction), and cluster_dataframe is not on the production path anyway (production uses kmeans_sklearn exclusively; the sole caller is tests/test_clusters.py). Also drops the module's dead `import random`. Evidence (2026-07-05, scratch/determinism_check.py + journal): 5 consecutive full-pipeline runs on vw + biodiversity are bit-for-bit identical except math_tick (a wall-clock version counter, varies by design). The determinism comes from first-k-distinct + n_init=1 + explicit random_state, not from this global seed. Per Julien: the original Clojure author's verbatim note on seeding (pca.clj:80-81 — "Should really throw a parallelizable random number generator in the equation here... With seeds fed in and persisted... XXX") now lives in pca.py next to the random_state setting, with the seeding-history context (Clojure never fixes a seed; k-means deterministic; PCA cold-start rand unseeded, warm-started after). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013CiESAwcrZ6BcVfdkumnxa ▁▁▁ #2591 ▁▁▁ feat(delphi): powerit-pca port (Clojure-parity PCA) behind legacy-mode flag Numpy port of Clojure's power-iteration PCA (pca.clj:38-105): per-component power iteration with fixed iteration count (iters=100, the Clojure default), Gram-Schmidt deflation, and a start_vectors parameter for warm-start pinning (the replay harness's mechanism for collapsing cold-start jitter, and a prerequisite for the pure-Python R2 replayer — see docs/REPLAY_HARNESS_DESIGN.md section 9/10). Flag: POLISMATH_PCA_IMPL env var, read at call time in pca_project_dataframe. 'powerit' (DEFAULT - legacy/parity mode) or 'sklearn' (the improved path, previous behavior). First instance of the permanent legacy-vs-improved dual-path pattern (Julien, 2026-07-05). Imputation and sparsity scaling are untouched; only the eigen-solver branches. Data-prep parity verified: Clojure imputes nil -> column average (conversation.clj:360-380) = Python's nanmean imputation. Documented decision - deterministic cold start: Clojure draws an UNSEEDED random start vector (pca.clj:79-82); Python defaults to a fixed deterministic start so the pipeline stays bit-for-bit reproducible (the 2026-07-05 determinism verification is a project invariant). Power iteration converges to the same dominant eigenvector for almost any start, so the fixed start is one specific draw of Clojure's random one; start_vectors overrides for pinning. Carries the requested TODO: switch to a proper convergence criterion once we move to improving the Python implementation. Silhouette guard (companion fix — required by making powerit the default): - calculate_silhouette_sklearn (clusters.py) now treats any n_labels >= n_samples clustering as undefined and returns the neutral 0.0 sentinel instead of letting sklearn raise. sklearn's silhouette_score requires 2 <= n_labels <= n_samples - 1; the old guard only covered n_labels <= 1 / n_samples <= 1. - Why it surfaced here: making powerit the default projects some small conversations onto exactly two base clusters, which feeds a 2-point / 2-label silhouette call in the group-cluster k-selection loop (conversation.py). That raised "Number of labels is 2. Valid values are 2 to n_samples - 1", crashing TestConversation.test_recompute and erroring 8 test_serialization_unfolding cases. All green now under BOTH powerit and sklearn. - The new guard is a strict superset of the old one — valid clusterings (n_labels < n_samples) are unchanged, and with only two base clusters the k-selection loop is forced to k=2 regardless, so the selected clustering is identical; the fix only removes the crash. - 3 unit tests added: tests/test_clusters.py::TestCalculateSilhouetteSklearn (2-samples/2-labels returns 0.0 not raise; single-label stays 0.0; a valid 3-sample/2-label clustering still returns a genuine score). Results: - 17 new tests (tests/test_powerit_pca.py): correctness vs numpy eigh, deflation orthogonality, determinism, start-vector honoring, flag behavior, solver agreement (PC1 0 deg, PC2 2.1e-3 deg on the reference fixture). - CCR angle deltas vs sklearn baseline: <= 0.02 deg across all datasets and variants; 28 passed / 27 xfailed unchanged. - Evidence: the bg2050-incremental PC2 miss (10.7086 deg > 10 deg) is BIT-IDENTICAL under powerit - upstream incremental-state divergence, NOT solver difference. Its xfail annotation stands. - Benchmark (vw, 69x125): powerit 0.765 ms avg vs sklearn 1.007 ms - ~1.3x faster at this size. bench_pca.py --compare-impls added. - Final gate: 304 passed / 33 skipped / 116 xfailed / 0 failed (baseline 283/33/116 + 17 new + 4 benchmark-import), plus the 3 silhouette-guard unit tests above. Also rewords the determinism-evidence pointer to cite the journal entry (tracked) instead of the gitignored scratch script (Copilot on #2590, convergent with internal review). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013CiESAwcrZ6BcVfdkumnxa ▁▁▁ #2592 ▁▁▁ docs(delphi): parity journal 2026-07-04/07, PLAN refresh, replay-harness design Three docs deliverables from the 2026-07-04/05 host sessions: - CLJ-PARITY-FIXES-JOURNAL.md: full session entries — spr-squash reconciliation (closed+mergedAt:null = landed; squash titles lie), Copilot triage of all 83 threads (1 real blocker + 5 verified escalations incl. the DynamoDB consensus data-loss), review-fix PR #2586, per-variant xfail scoping (which unmasked bg2018/pakistan incremental consensus divergences the blanket had absorbed), gid label-swap root cause + fix (#2589), seed removal (#2590), determinism verification (5 runs bit-identical except math_tick), Clojure randomness facts (never seeds; fixed-iteration power iteration; unseeded :twister sampling), R2 python-only constraint, powerit-pca GO. - PLAN_DISCREPANCY_FIXES.md: D10/D11/D12 status rows corrected (were still "VM draft — NEEDS REWORK"; actually code-complete open PRs). - REPLAY_HARNESS_DESIGN.md (NEW): design for the replay harness (H) — schedule spec as first-class input, Python driver (the future R2 forward model, python-only per Julien 2026-07-05), Clojure driver narrowed to R1 certification (Mode A pure conv-update reduce; blob capture sufficient, EDN on demand), nondeterminism policy (tolerance classes, warm-start pinning, self-jitter measurement), storage/provenance, phased build plan H-0..H-D. Also appends the 2026-07-06/07 session entry: the silhouette-guard fix for the powerit-PCA default (#2591) — powerit collapsing a small conversation to two base clusters fed a 2-point/2-label silhouette call that sklearn rejects; calculate_silhouette_sklearn now returns 0.0 when n_labels >= n_samples (strict superset of the old guard, chosen clustering unchanged). Verified green; details in the journal. The journal's "Determinism verification" entry is the tracked evidence pointer cited from pca.py (Copilot on #2590). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013CiESAwcrZ6BcVfdkumnxa ▁▁▁ #2597 ▁▁▁ docs(delphi): storage v2 design — job-id provenance, replayable runs, dual-backend storage Study + design from the 2026-07-06 storage/reproducibility audit: - current-state audit: 18 Delphi_ DynamoDB tables, job_id lifecycle gaps (env-var-only propagation reaching 2 of 18 tables), pseudo-random math_tick, unconditional reset-per-run, live-Postgres re-reads per stage, Clojure math_main dependency, nondeterminism inventory, server/client consumption map - target schema: immutable runs + run_inputs + append-only artifacts + first-class latest pointer (6 logical entities) - backend-neutral repository (DynamoDB or PostgreSQL via DELPHI_STORAGE_BACKEND), Python + TypeScript implementations kept honest by a shared JSON conformance spec - input-level reproducibility: per-run input snapshots incl. the Clojure math_main blob, seeds, config_effective, LLM prompt/response recording - replay CLI (show/run/diff) - zero-downtime migration (expand/backfill/verify/flip/contract): dual-write new data in both formats, progressively backfill old data as IMPORTED runs, serve old-format-only until 100% coverage + zero shadow-read divergence, then a single reversible flip with old writes kept on for instant rollback; contract after a trust period. Three flip-back invariants: importer no-clobber, bidirectional writers, single queue master. - cross-referenced with REPLAY_HARNESS_DESIGN.md (schedule replay vs recorded-job replay) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> commit-id:30219b34
Collaborator
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
…to_dynamo_dict
commit-id:ff4764e4
Stack: