diff --git a/delphi/CLAUDE.md b/delphi/CLAUDE.md index 3f0e39738..41f4a3f16 100644 --- a/delphi/CLAUDE.md +++ b/delphi/CLAUDE.md @@ -16,7 +16,7 @@ this avoids the confusion of having anything called a "cid", the joke was "conve ## helpful background -this was built in two parts, the pca/kmenas/repness and the umap/narrative, and these are combined in the run_delphi.sh script. +this was built in two parts, the pca/kmenas/repness and the umap/narrative, and these are combined in the run_delphi.py script. ## Local Python Environment @@ -175,7 +175,7 @@ AWS_SECRET_ACCESS_KEY=dummy AWS_REGION=us-east-1 ``` -These are configured in run_delphi.sh for all DynamoDB operations. +These are configured in run_delphi.py for all DynamoDB operations. ### DynamoDB Job Queue System @@ -191,13 +191,13 @@ Delphi now includes a distributed job queue system built on DynamoDB: 2. **Processing Jobs**: Start the job poller service: ```bash - ./start_poller.sh + python start_poller.py ``` 3. **Table Management**: To reset the job queue: ```bash - aws dynamodb delete-table --table-name DelphiJobQueue --endpoint-url http://localhost:8000 && \ + aws dynamodb delete-table --table-name Delphi_JobQueue --endpoint-url http://localhost:8000 && \ docker exec -e PYTHONPATH=/app polis-dev-delphi-1 python /app/create_dynamodb_tables.py --endpoint-url http://host.docker.internal:8000 ``` @@ -210,7 +210,7 @@ Delphi now includes a distributed job queue system built on DynamoDB: ### Table Creation - Primary script: `/create_dynamodb_tables.py` - Creates BOTH Polis math and EVōC tables -- This script is used in `run_delphi.sh` and now integrated into `umap_narrative/run_pipeline.py` +- This script is used in `run_delphi.py` and now integrated into `umap_narrative/run_pipeline.py` ### Schema Definitions @@ -242,7 +242,7 @@ Delphi now includes a distributed job queue system built on DynamoDB: - `Delphi_CollectiveStatement` - Collective statements generated for topics > **Note:** All table names now use the `Delphi_` prefix for consistency. -> For complete documentation on the table renaming, see `/Users/colinmegill/polis/delphi/docs/DATABASE_NAMING_PROPOSAL.md` +> Table definitions in `create_dynamodb_tables.py` are the canonical reference for names and schemas. ## Reset Single Conversation @@ -281,7 +281,7 @@ See [RESET_SINGLE_CONVERSATION.md](docs/RESET_SINGLE_CONVERSATION.md) for detail After identifying the correct conversation ZID, run the Delphi pipeline directly with: ```bash -./run_delphi.sh --zid=[ZID] +python run_delphi.py --zid [ZID] ``` Additional options include: @@ -297,7 +297,7 @@ For production environments, use the job queue system: 1. Start the poller service on your worker machine: ```bash - ./start_poller.sh + python start_poller.py ``` 2. Submit a job from any machine with access to DynamoDB: @@ -323,13 +323,6 @@ For production environments, use the job queue system: docker exec -e PYTHONPATH=/app polis-dev-delphi-1 python /app/create_dynamodb_tables.py --endpoint-url http://host.docker.internal:8000 ``` - Or use the reset_database.sh script to recreate all tables: - - ```bash - # Reset all tables (both Polis math and EVōC tables) - ./reset_database.sh - ``` - 2. **Testing specific pipeline stages**: ```bash diff --git a/delphi/docs/CLJ-PARITY-FIXES-JOURNAL.md b/delphi/docs/CLJ-PARITY-FIXES-JOURNAL.md index 629179770..5ca54f840 100644 --- a/delphi/docs/CLJ-PARITY-FIXES-JOURNAL.md +++ b/delphi/docs/CLJ-PARITY-FIXES-JOURNAL.md @@ -1216,3 +1216,700 @@ ever appears in real Polis data. It does. out of this committed doc; the unredacted findings are in Claude's per-project memory store (`~/.claude/projects/...`). Open a follow-up discussion with the team before any user-facing action. + +## Session: PR 14a — Scalar deletion (2026-06-11) + +Foundation pass before D10/D11/D12. The scalar implementations in +`repness.py` were test-only (production calls only `compute_group_comment_stats_df` ++ `select_rep_comments_df` + `select_consensus_comments_df` via +`conv_repness`). Deleting them removes the "where do I put this helper?" +ambiguity for D10/D11/D12 (which all add new helpers to `repness.py`) and +shrinks the test surface by ~35 obsolete unit tests. + +### What landed + +**Production code (`delphi/polismath/pca_kmeans_rep/repness.py`)** — 445 lines deleted: +- DELETE primitives: `prop_test`, `two_prop_test` (both also dead in production — + only test consumers). +- DELETE orchestration: `comment_stats`, `add_comparative_stats`, `repness_metric`, + `finalize_cmt_stats`, `passes_by_test`, `best_agree`, `best_disagree`, + `select_rep_comments`, `select_consensus_comments`. +- DELETE unused: `calculate_kl_divergence` (no callers anywhere). +- KEEP: `z_score_sig_90`, `z_score_sig_95` (trivial threshold checks used scalar-side + in selection logic; vectorizing them would not save lines). +- ENRICH docstrings: `prop_test_vectorized` and `two_prop_test_vectorized` now + embed the scalar-equivalent closed-form algebra (so the formula stays readable + even though the scalar functions are gone). Pattern from the user during the + PR 14a discussion: "where we cannot achieve readability on the vectorized, + put in comments showing the non-vectorized equivalent." + +**Tests** — net -35 passed tests (296 → 295... actually delta is 330 → 295): +- DELETE entirely: `tests/test_old_format_repness.py` (557 lines, mirror of + scalar-only tests in `test_repness_unit.py`; the "old format" was the scalar + dict-in/dict-out API). +- DELETE classes: `TestCommentStats`, `TestSelectionFunctions`, + `TestConsensusAndGroupRepness` in `test_repness_unit.py`. Also + `TestStatisticalFunctions::test_prop_test` and `test_two_prop_test`. +- MIGRATE D4/D5/D6 BlobInjection (`test_discrepancy_fixes.py:1602+`) from + per-(gid, tid) scalar loop calls to a single vectorized call on a DataFrame + built from the blob's `repness` entries. This pattern (1) tests the actual + production code path, (2) produces a `.to_string()` diagnostic that beats + hand-formatted f-strings, (3) drops loop overhead. +- MIGRATE `TestD5ProportionTest::test_prop_test_matches_clojure_formula`, + `TestD6TwoPropTest::test_two_prop_test_matches_clojure_formula` + edge cases + to single vectorized calls on N-row DataFrames. +- CONSOLIDATE `TestD8FinalizeStats`'s 7 scalar boundary tests into one + parametrized DataFrame test (`test_repful_classification_boundary`) that + exercises the production `np.where(rat > rdt, 'agree', 'disagree')` logic. + Boundary cases preserved: `rat < rdt`, `rat > rdt`, `rat == rdt` (non-zero, + zero), negative z-scores. +- MIGRATE `TestD7RepnessMetric::test_metric_formula_is_product` to hand-computed + reference values (`1.3*1.8*0.8*2.5 = 4.68` for agree, `0.7*-0.9*0.2*-1.5 = 0.189` + for disagree — signed product). +- DELETE redundant scalar-formula tests in `TestSyntheticEdgeCases` + (test_prop_test_matches_clojure_formula_synthetic — duplicated by migrated + TestD5ProportionTest; test_clojure_repness_metric_product — duplicated by + TestD7RepnessMetric; test_clojure_repful_uses_rat_vs_rdt — purely tautological). +- Rename misleading `test_compute_group_comment_stats_matches_scalar` → + `test_compute_group_comment_stats_consistency_with_conv_repness`. +- Cross-checks in `TestVectorizedFunctions` (test_repness_unit.py) replaced + inline scalar calls with `_prop_test_reference` / `_two_prop_test_reference` + closed-form staticmethods. + +### Suite delta (pre/post PR 14a) + +- Pre-baseline (edge @ 2dce7385f): **330 passed, 12 skipped, 58 xfailed**. +- Post (@ this PR): **295 passed, 12 skipped, 58 xfailed**. +- Delta: -35 passed, 0 failed, 0 new xfailed. The -35 matches the deleted + scalar-only test count (test_old_format_repness ~20 + scalar classes in + test_repness_unit ~9 + scalar test methods in test_discrepancy_fixes ~6 + consolidated/removed). + +### For PR 14c (readability refactor) + +PR 14c will refactor `compute_group_comment_stats_df` for readability and +needs to mirror the scalar recipe. **The deleted scalar code is the +reference.** Retrieve via: + +```bash +git show ~1:delphi/polismath/pca_kmeans_rep/repness.py \ + | sed -n '161,302p' +``` + +Specifically (using the pre-deletion line numbers — file was 1008 lines at +edge HEAD 2dce7385f): + +- `comment_stats` lines 161-201 — the per-(group, comment) recipe. +- `add_comparative_stats` lines 203-235 — in-vs-out comparison. +- `repness_metric` lines 237-271 — the `r*rt*p*pt` product. +- `finalize_cmt_stats` lines 273-301 — agree-vs-disagree branch. + +The pre-PR-14a commit hash will be the parent of PR 14a's commit. Clojure +originals: `math/src/polismath/math/repness.clj:78-100,173-188,191-200`. + +### Pyright noise (unrelated to PR 14a, raised same session) + +Discovered during PR 14a that pyright produces ~10 errors on `repness.py` from +pandas-stubs false positives (`pd.DataFrame(columns=...)`, `df['col'] = value`, +Series-vs-DataFrame narrowing). PR #2560 added a pyright config that points +at `delphi/.venv` but did NOT set any rule overrides, so default-mode +`reportArgumentType` / `reportIndexIssue` errors surface for valid pandas code. +Verified pre-existing on edge HEAD (not introduced by PR 14a). + +Handoff written: `~/polis/HANDOFF_PYRIGHT_PANDAS_STUBS.md`. Do NOT just turn +off rules globally — investigate pandas-stubs version, community patterns, +targeted ignores. Tracked as Claude task #8. + +### What's Next + +PR 14a unblocks (in stack order): +1. **D10** — Rep comment selection. Research agent already produced a fix + proposal (this session). Helpers `passes_by_test`, `beats_best_by_test`, + `beats_best_agr` go top-level in `repness.py`; reduce structure uses + `df.to_dict('records')` iteration with mutable `{sufficient, best, best_agree}` + state. Boundary cases identified for synthetic test fixtures. +2. **D11** — Consensus selection. Research agent produced a fix proposal: + needs new `consensus_stats_df(vote_matrix_df) -> pd.DataFrame` (whole-data, + not per-group), plus rewrite of `select_consensus_comments_df` with the + `{'agree': [...], 'disagree': [...]}` output shape (top 5 each). + `conv_repness` must grow `mod_out` kwarg. +3. **D12** — Comment priorities. Research agent produced a fix proposal: + Clojure source at `conversation.clj:311-330,341-352,648-679`; + `pca.clj:167-178`. Needs new `pca_project_cmnts`, `comment_extremity`, + `importance_metric`, `priority_metric` in Python. `meta_tids` shape mismatch + (Python set vs Clojure map) flagged. +4. **PR 14b** — Backfill missing blob injection tests (D7 metric, D8 finalize, + full stats-stage injection). +5. **Goldens** — Re-record `vw` and `biodiversity` (sklearn KMeans seeding + decision pending — see `delphi/scratch/COPILOT_MATH_QUESTIONS.md`). +6. **PR 14c** — Readability refactor of `compute_group_comment_stats_df`, + using the deleted scalar code (retrievable via `git show`) as the + readability reference. Research agent produced a clean split proposal: + `_build_group_comment_index` (plumbing) + `_compute_per_group_stats` + (math) + 5-line orchestrator. + +## Session: ns-PASS fix (2026-06-11) + +**Context.** While preparing the D10/D11/D12 rework on top of PR 14a, a +Clojure re-read surfaced a latent bug in `compute_group_comment_stats_df`: +`ns` (and `total_votes`) were computed as `na + nd`, silently dropping PASS +votes. Clojure's `:ns` is `(count-votes votes)` (math/repness.clj:56-61, +:70), which calls `(filter identity votes)`. In Clojure 0 is truthy, so +PASS (0) counts; only `nil` is filtered out. Therefore Clojure +`ns = na + nd + np`, and every downstream metric (`pa`, `pd`, `pat`, +`pdt`, `ra`, `rd`, `rat`, `rdt`, `agree_metric`, `disagree_metric`, plus +D11's `consensus_stats_df` which will mirror the same recipe at the +whole-conversation level) was off whenever PASS votes existed. + +**Why D5 BlobInjection didn't catch it.** D5's blob-injection tests pull +`(n-success, n-trials)` straight from the Clojure blob's `repness` +entries and feed them to `prop_test_vectorized`. They bypass +`compute_group_comment_stats_df` entirely, so the bug downstream of +`ns = na + nd` was invisible. The same gap will recur for D11 / D12 until +we ship pure-formula tests that build a tiny vote matrix and assert on the +counts. Lesson: blob-injection is necessary but not sufficient — every +formula whose inputs are themselves computed by Python needs at least one +pure-formula unit test that exercises the input-building code. + +**TDD cycle.** +- **BASELINE** — full suite at PR 14a parent: 295 passed, 12 skipped, + 58 xfailed. +- **RED** — added `TestNsIncludesPassVotes` in `tests/test_repness_unit.py` + with four pure-formula tests: single-comment mixed AGREE/DISAGREE/PASS, + all-PASS column, NaN-vs-PASS distinction, two-group `other_votes` + including out-group PASS. All four failed on the buggy code (4 fails). +- **GREEN** — in `polismath/pca_kmeans_rep/repness.py`: + - `total_counts` now computes `total_votes=('vote', 'size')` directly in + the groupby agg, instead of `total_agree + total_disagree`. + - `group_counts` now computes `ns=('vote', 'size')` directly, instead of + `na + nd`. + - `'size'` on the already-`dropna(subset=['vote'])`-filtered frame counts + exactly the non-NaN entries — including PASS (0). This is the + Clojure `(count (filter identity votes))` recipe verbatim. + - Both sites carry a comment citing repness.clj:56-61, :70 and the + truthy-0 reasoning. + - Docstring updated: `ns` now documented as `agree + disagree + PASS`. +- **FULL SUITE** — 299 passed, 12 skipped, 58 xfailed. Delta = +4 + (exactly the new ns-PASS tests). No existing pre-PR-14a or PR-14a test + broke. The pre-existing `TestVectorizedFunctions` fixtures use only + AGREE/DISAGREE/NaN (no PASS), so they were never sensitive to the bug. + +**Cascade to D11.** D11's plan introduces a `consensus_stats_df(vote_matrix_df)` +function computing whole-conversation stats (the `:mod-out` Clojure path). +That function will inevitably mirror the same `na + nd + np` recipe — so +the ns-PASS fix lands BEFORE D10 in the stack to keep D11's implementation +clean. D11 should follow the same pure-formula test pattern: build a vote +matrix with mixed PASS and assert `ns == count of non-NaN cells`. + +**Goldens.** Stays DEFERRED. Re-recording is gated on +sklearn-KMeans-seeding consensus (see scratch/COPILOT_MATH_QUESTIONS.md); +no values shift at the goldens commit until D10/D11/D12 land. + +**Stack position.** New commit inserted between PR 14a (#2564) and D10 +(#2566). D10, D11, D12, goldens rebased cleanly on top; 2-sided docs +conflicts in PLAN/JOURNAL at each downstream commit resolved manually +to merge both edits (downstream docs additions kept; ns-PASS row and +session entry preserved). + +## Session: PR 8 — D10 rep comment selection (2026-06-11) + +Landed in `/goal` mode (autonomous run targeting D10 + D11 + D12 + goldens +as a stacked PR series). Decisions made without inline user check are +documented in `~/polis/D10_D11_D12_GOLDENS_DECISIONS.md` for batch review. + +### What landed + +**Production code (`delphi/polismath/pca_kmeans_rep/repness.py`)** — added +3 top-level helpers + `_finalize_row_for_output` + rewrote `select_rep_comments_df`: + +- `passes_by_test(s) -> bool` — Clojure `passes-by-test?` (repness.clj:165-170). + OR'd on `(rat, pat)` and `(rdt, pdt)` z-sig-90. **NO `pa >= 0.5` gate** — + the pre-D10 Python gate was a botched-port over-restriction with no + Clojure analog. +- `beats_best_by_test(s, current_best_z) -> bool` — Clojure `beats-best-by-test?` + (repness.clj:133-139). Strict `>` on `max(rat, rdt)` vs current best z. +- `beats_best_agr(s, current_best) -> bool` — Clojure `beats-best-agr?` + (repness.clj:142-162). Four-branch agree-priority logic: + 1. `na == 0 and nd == 0` → reject. + 2. Current best AND `current_best['ra'] > 1.0` → compare 4-way signed + product `ra * rat * pa * pat`. + 3. Current best (else, `ra <= 1.0`) → compare `pa * pat`. + 4. No current best → accept if `z90(pat)` OR `(ra > 1.0 AND pa > 0.5)`. +- `_finalize_row_for_output(row, *, is_best_agree=False)` — Clojure + `finalize-cmt-stats` (repness.clj:173-188) + best-agree flagging + (repness.clj:262-264). Adds `best_agree=True` and `n_agree=na` keys for + the best-agree slot. +- `select_rep_comments_df(stats_df, mod_out=None) -> List[Dict[str, Any]]` — + single-pass reduce over `stats_df.to_dict('records')` mirroring Clojure + `select-rep-comments` (repness.clj:212-281). Per-row state + `{sufficient, best, best_agree}` updated by the three helpers; final + assembly is dedup-best-agree-from-sufficient → sort by metric → + prepend best-agree → take 5 → agrees-before-disagrees. + +**Caller (`conv_repness`)** — dropped the `_stats_row_to_dict` wrapping +step since `select_rep_comments_df` now returns finalized dicts directly. + +**Two pre-D10 bugs fixed alongside the rewrite** (research-agent flagged): +- `pa >= 0.5 / pd >= 0.5` over-gate in the passing filter — removed. No + Clojure analog; was dropping legitimate candidates. +- "Fill-from-other-category" + "first-row" fallback blocks — deleted. The + `:best` / `:best_agree` mechanism IS the Clojure fallback. + +**Tests** — 18 new tests (in `tests/test_discrepancy_fixes.py`): +- `TestD10PassesByTest` (4 tests): agree-side significant, disagree-side + significant, neither significant, no `pa >= 0.5` gate. +- `TestD10BeatsBestByTest` (3 tests): None-best, max-rat-rdt, strict `>`. +- `TestD10BeatsBestAgr` (6 tests): Branch 1 (na=nd=0), Branch 2 (ra>1), + Branch 3 (ra<=1), Branch 4 z90(pat), Branch 4 (ra>1 AND pa>0.5), Branch 4 + rejection. +- `TestD10SelectRepCommentsBoundary` (5 tests): empty input, single unvoted + row → best fallback, sufficient-empty-best-agree-only, take-5 cap with + agrees-before-disagrees ordering, **the eviction edge case** (best_agree + outside sufficient evicting 5th-highest-metric). + +**Re-xfailed with updated reasons** (D14 / D1 upstream divergence): +- `TestD9ZScoreThresholds::test_z_values_match_clojure` +- `TestD5ProportionTest::test_pat_values_match_clojure_blob` +- `TestD6TwoPropTest::test_rat_values_match_clojure_blob` +- `TestD7RepnessMetric::test_repness_metric_matches_clojure_blob` +- `TestD8FinalizeStats::test_repful_matches_clojure_blob` +- `TestD10RepCommentSelection::test_rep_comments_match_clojure` + +Why xfailed despite D10 landing: D10 enables shared comments in the +selection (overlap rises from 0% to ~20% on vw cold_start), but +per-(gid, tid) stats still mismatch because Python and Clojure put +different participants in the "same" group ID. That's upstream +PCA/KMeans group-membership divergence (D14 / D1), not D10. D10 is +verified via the 18 synthetic helper + boundary tests above. + +### Suite delta (pre/post D10) + +- Pre (post-14a): 295 passed, 12 skipped, 58 xfailed. +- Post (this PR): 313 passed, 12 skipped, 58 xfailed. +- Delta: +18 passed, 0 failed, 0 new xfailed. The +18 matches the 18 new + D10 synthetic tests exactly. + +### Decisions made autonomously (under `/goal` mode) + +See `~/polis/D10_D11_D12_GOLDENS_DECISIONS.md`. Highlights: +- **S1**: Python convention key names (`repful`, `best_agree`, `n_agree`) + instead of Clojure hyphens. Math blob alignment is a future PR. +- **S2**: `select_rep_comments_df` returns `List[Dict[str, Any]]` instead + of `pd.DataFrame` — variable extra keys (best_agree flag) make list-of- + dicts cleaner than DF-with-NaN-columns. +- **D10.1**: Two pre-D10 bugs (pa>=0.5 gate, fill-from-other fallback) + folded into D10 rather than separate PRs — the rewrite replaces the + function so a surgical fix would be more noise than value. +- **D10.7**: Real-data blob-comparison tests re-xfailed with reasons + pointing at D14/D1, not softened to overlap-thresholds — more honest. + +### What's Next + +PR 9 (D11) on top of D10 in the same spr stack. + +## Session: PR 9 — D11 consensus comment selection (2026-06-11) + +Landed in `/goal` mode. Decisions documented in +`~/polis/D10_D11_D12_GOLDENS_DECISIONS.md` (D11.x section). + +### What landed + +**Production (`delphi/polismath/pca_kmeans_rep/repness.py`):** +- New `consensus_stats_df(vote_matrix_df, mod_out=None) -> pd.DataFrame`: + whole-conversation per-comment stats (no group split, no `ra/rd/rat/rdt`). + Vectorized port of Clojure `consensus-stats` (repness.clj:284-290). +- Rewrite `select_consensus_comments_df(cons_stats) -> Dict[str, List[Dict]]`: + matches Clojure `select-consensus-comments` (repness.clj:293-323). + Filters: agree `pa > 0.5 AND z-sig-90(pat)`, disagree + `pd > 0.5 AND z-sig-90(pdt)`. Ordering: descending `pa*pat` / `pd*pdt`. + Cap: top 5 each side. Output: `{'agree': [...], 'disagree': [...]}`. +- `conv_repness` grows `mod_out: Optional[Iterable[int]] = None` kwarg. + Forwarded to both `select_rep_comments_df` and `consensus_stats_df`. +- Consensus is now computed unconditionally (Clojure parity — pre-D11 + Python had a `len(group_clusters) > 1` guard with no Clojure analog). +- `_stats_row_to_dict` deleted (orphan after D11). + +**Caller (`conversation.py`):** +- `_compute_repness` passes `mod_out=self.mod_out_tids` to `conv_repness`. + +**Downstream consumers updated** for the new dict shape: +- `tests/test_repness_smoke.py::test_repness_structure` — iterates + `consensus['agree']` and `consensus['disagree']`. +- `tests/test_pipeline_integrity.py::test_full_pipeline` — same. + +**Tests (12 new in `tests/test_discrepancy_fixes.py`):** +- `TestD11ConsensusStatsDf` (4): basic counts, pseudocount pa/pd, + ns=0 fallback, mod_out filter. +- `TestD11SelectConsensusBoundary` (8): empty input, clear agree + consensus, clear disagree consensus, divisive (no consensus), top-5 + cap, entry keys (Python convention per S1), disagree entry key + mapping (n_success ← nd, p_success ← pd, p_test ← pdt), + mutually-exclusive agree/disagree lists. + +### Suite delta + +- Pre (post-D10): 313 passed, 12 skipped, 58 xfailed. +- Post (this PR): 325 passed, 12 skipped, 58 xfailed. +- Delta: +12 (the 12 new D11 synthetic tests). Zero regressions. + +### DISCOVERY: ns-PASS divergence + +The D11 real-data test (`test_consensus_matches_clojure`) showed 3-5/5 +overlap on cold_start — close but not exact. Investigation revealed a +deeper bug: + +**Clojure's `:ns`** (via `count-votes` with `filter identity` — +repness.clj:56-61) INCLUDES PASS votes (`0` is truthy in Clojure). + +**Python's `ns`** in BOTH `compute_group_comment_stats_df` and the new +`consensus_stats_df` computes `ns = na + nd`, EXCLUDING PASS. + +This means every downstream metric (pa, pd, pat, pdt, ra, rd, rat, rdt, +agree_metric, disagree_metric) is computed with the wrong denominator +when PASS votes are present. The D5 PR #2519 journal claim that "PASS NOT +included, matching Clojure" was a misreading of `count-votes`. + +**Impact:** +- D5/D6/D7/D8 blob-comparison tests' "mismatches" were not (only) + upstream PCA/KMeans divergence — the ns-PASS divergence is at least + a contributing cause. +- D11 consensus partial overlap is consistent with this divergence. +- Fixing requires a separate PR affecting two production functions and + re-recording goldens. + +D11 real-data test xfailed with the right reason. Logic pinned by the +12 synthetic tests (which never exercise PASS, so they don't show the +divergence). + +This is now the top item under "Pending — needs team discussion" in +PLAN.md, with a sketch of the fix. + +### What's Next + +PR 11 (D12) on top of D11 in the same spr stack. + +## Session: PR 11 — D12 comment priorities (2026-06-11) + +Landed in `/goal` mode. Decisions documented in +`~/polis/D10_D11_D12_GOLDENS_DECISIONS.md` (D12.x section). + +### What landed + +**`pca.py`:** +- `pca_project_cmnts(center, comps) -> np.ndarray`: vectorized projection + of each comment into 2D PCA space. Closed-form derivation: + `proj[i] = -sqrt(n_cmnts) * (1 + center[i]) * [pc1[i], pc2[i]]`. +- `compute_comment_extremity(cmnt_proj) -> np.ndarray`: L2 norm per row. + +Clojure parity: `pca-project-cmnts` (pca.clj:167-178) + +`with-proj-and-extremtiy` (conversation.clj:341-352). + +**`conversation.py` module-level:** +- `META_PRIORITY = 7` constant (Clojure conversation.clj:319). +- `importance_metric(A, P, S, E) -> float`: Clojure conversation.clj:311-315. +- `priority_metric(is_meta, A, P, S, E) -> float`: Clojure conversation.clj:321-330. + Squared formula. For meta: `META_PRIORITY^2 = 49`. For non-meta: + `(importance * (1 + 8*2^(-S/5)))^2` where the decay factor lets new + (low-S) comments bubble up. + +**`Conversation._compute_comment_priorities()`:** +- Computes comment projection + extremity from PCA. +- Aggregates A/D/S across all groups per tid; derives P = S - (A + D). +- Looks up extremity per tid (via `self.rating_mat.columns` column order). +- Checks `tid in self.meta_tids` for the meta branch. +- Stores `{tid: priority_float}` on `self.comment_priorities`. + +Wired into `recompute()` after `_compute_repness()`. The serialization +infrastructure (`to_dict`, `to_dynamo_dict`, underscore→hyphen conversion) +already existed but was emitting empty. + +**B1 + B2 fixes from D11 sub-agent review folded in:** +- B1: `conversation.py:834` no-groups early-return now emits + `consensus_comments: {'agree': [], 'disagree': []}` (dict) instead of `[]`. +- B2: `test_legacy_repness_comparison.py:197` updated to read the dict + shape + flatten for ID extraction. + +### Tests (11 new + 1 xfail flipped + 2 xpassed = 14 new+repurposed) + +- `TestD12PCAProjectComments` (5): output shape, formula verification, empty + input, L2 extremity, empty extremity. +- `TestD12PriorityMetrics` (6): importance formula vs Clojure ref values + (conversation.clj:335), high-extremity boosts, meta constant=49, non-meta + squared formula, decay-factor lets-new-bubble-up, META_PRIORITY=7. +- `TestD12CommentPriorities::test_comment_priorities_exist` xfail dropped + (existed pre-PR), then re-xfailed for a different reason: Clojure blob + has constant priorities (all 49.0 = META_PRIORITY^2) on vw/biodiversity + → Spearman comparison meaningless. + +### DISCOVERY: Clojure blob has all-meta priorities + +`vw-cold_start`: ALL 125 tids have priority = 49.0 in Clojure blob. +`biodiversity-cold_start`: ALL 314 tids have priority = 49.0. + +Either: +- (a) Every tid was meta-tagged in those Clojure runs. +- (b) Clojure's `(if 0 ...)` truthiness quirk: 0 is truthy in Clojure, so + ANY value (even `0`) returned by `(get meta-tids tid 0)` triggers the + meta branch. + +Python correctly distinguishes meta from non-meta via Boolean set membership, +producing varied priorities 0.18-31.46. + +Logged for batch review. Python may be more correct than Clojure here. + +### Suite delta + +- Pre (post-D11): 325 passed, 12 skipped, 58 xfailed. +- Post (this PR): 336 passed, 12 skipped, 56 xfailed, 2 xpassed. +- Delta: +11 (the 11 new D12 synthetic tests), 0 failed, -2 xfailed + (those became xpassed — the 2 cold_start `test_comment_priorities_exist` + variants run cleanly now; the new xfail is on a different basis). + +### What's Next + +Re-record vw + biodiversity Python golden snapshots (PR-stack tip). + + +## Session: Copilot triage, review-fix PR #2586, merge prep (2026-07-04/05) + +Host session ("Fable-polis-merge-then-replay"). Goal: assess and execute the +merge of the open 7-PR stack. Outcome: stack is code-complete, gate-green, +review-resolved, and pushed — **merge deliberately NOT executed** (edge +frozen for a prod issue; Julien: push PRs, merge nothing). + +### Reconciliation findings (recon, 3 parallel agents + verification) + +- spr squash-merges auto-close per-commit PRs with `mergedAt: null` — + "closed" ≠ dead. All of D2/D4/D5–D9/D15/K-inv landed via TWO squash + commits: #2515 ("Speed up regression tests") and #2561 (titled "Docs: + plan + journal updates" but carrying ALL the D5–D15 math). Verified via + `git log -S` for `rat > rdt`, `PSEUDO_COUNT = 2.0`, signed-product + repness_metric. **Squash titles lie; reconcile by commit-id trailers.** +- The "golden re-record + seed decision" merge blockers had dissolved: + vw/bio goldens are PGRs deliberately deleted at #2516 (tests skip; + `SKIP_GOLDEN=1` in CI), and the seed decision was de facto made by K-inv + (first-k-distinct + n_init=1 + random_state=42). +- Dormant `review` jj workspace (empty commit inside the stack chain) + forgotten + abandoned before rebase (user-approved). Stack rebased onto + edge 722640eb0 (+#2581 gid-coercion, +#2579 node pin) — zero conflicts. + +### Copilot triage (all 83 threads, 7 PRs — 0 were resolved before this) + +Verified against the stack TREE (not the working copy — an early audit +agent read edge by mistake and produced garbage classifications): +- 1 real blocker: consensus entries used Python keys + (comment_id/n_success/…) while Clojure/server-helpers.ts/ + majorityStrict.jsx expect tid/n-success/… . +- Copilot-WRONG: "priority_metric always returns 49" is the DELIBERATE + D12.6 bug-mirror (#2571). +- 5 escalations verified REAL: (g1) DynamoDB writer read consensus from + `repness.consensus_comments`, a key `to_dynamo_dict` never emits → + always wrote the empty default (round-trip test had stubbed the WRONG + nested shape, masking it); (g2) bench_repness imported 14a-deleted + `comment_stats` (ImportError); (g3) reader passed legacy list-shaped + consensus through; (g4) silent zip truncation in + `_compute_comment_priorities`; (g5) blanket `xfail(strict=False)` + masking variants that pass. + +### Review-fix commit → PR #2586 (inserted below the docs commit) + +TDD RED→GREEN (14 RED failures with exact predicted signatures → 38/38 +GREEN): consensus entries → Clojure blob shape (narrowed S1: consensus +only; rep-comment entries keep comment_id until the math-blob alignment +PR); writer reads top-level `consensus`; Decimal-preserving priorities +(int() floored sub-1 priorities to 0 = "no priority data" to the TS +router; latent until #2571 resolves); legacy-list normalization on read; +`mod_out is not None` ×2; fail-closed PCA/columns desync guard; benchmark +import fix + import tests; ns docstrings corrected. + +Test-gate honesty work: per-variant xfails replace the blankets. +**DISCOVERY: scoping unmasked bg2018-incremental and pakistan-incremental +consensus divergences** the blanket had silently absorbed (same +incremental family as biodiversity-incremental; deferred to +sequential-parity work). PGR regression tests now SKIP with the +2026-06-11 goldens-deferral reason (S3-5 claimed this mark but never +committed it — docs-vs-diff lesson again). 3 pre-existing CCR failures +(verified identical on edge): bg2050-incremental PC2 angle 10.71°>10°, +pakistan-incremental shape (2,9030)≠(2,194), bg2018-cold_start +clustering — precise per-variant xfails. + +### Gates + +- Baseline (stack top, --include-local): 13 failed / 476 passed / 18 + skipped / 143 xfailed — all 13 accounted for (10 stale-PGR, 3 CCR). +- Final: **0 failed / 502 passed / 28 skipped / 146 xfailed / 7 xpassed**. +- xdist note: `get_or_compute_conversation` recomputes per worker under + `-n auto` (xdist_group markers were removed as "dead") — BLAS + oversubscription + duplicated fixture work melted the host. Throttled + (`-n 4`, OMP/OPENBLAS threads=1) the suite runs in ~11 min. Test-infra + improvement candidate: restore dataset-based xdist_group. + +### Determinism verification (COPILOT_MATH_QUESTIONS.md:283 checklist) + +5 consecutive full-pipeline runs on vw + biodiversity: **bit-for-bit +identical except `math_tick`** (wall-clock version counter, varies by +design; per-stage hashing localized it; scratch/determinism_check.py). +Seed question CLOSED: pipeline is deterministic. Proposal pending +Julien's go: delete the vestigial `np.random.seed(42)` at +clusters.py:766 — the only `random` reference in the module, seeds a +global RNG nothing draws from, and `cluster_dataframe` isn't on the +production path (only tests/test_clusters.py; production uses +kmeans_sklearn exclusively). Candidate follow-up (separate decision): +delete the dead manual-kmeans path 14a-style. + +### Process + +- All 83 Copilot threads replied-to + resolved (classification-specific + replies citing #2586 / #2571 / #2587). +- Perf deferral filed: issue #2587 (_compute_comment_priorities re-scans + group votes every tick). +- PLAN status table corrected (D10/D11/D12 rows were still "VM draft — + NEEDS REWORK"). + +### What's Next + +1. **Merge when edge reopens** (user hold, prod issue): `jj spr merge + --count 8` → #2564, #2570, #2566, #2567, #2568, #2572, #2586, #2573. + Verify the squash title reflects real content (#2561 mis-title + lesson). Then post-merge jj hygiene (fetch, rebase survivors, bookmark + check). +2. Seed cleanup PR on Julien's go (clusters.py:766, evidence above). +3. NO PGR re-record until the Python-vs-Python phase (label-swap fix + first — S3-4: Python g0 = Clojure g1 EXACTLY on vw-cold_start; fix is + canonical group-id ordering or permutation-invariant comparison). +4. Track-1 frontier after merge: label-swap fix → sequential bits (D) → + replay harness (H, design doc) → R1 → R2. Track 2 (EVOC research) can + launch any time — independent surface. +## Session addendum: gid label-swap fix + seed removal + replay design (2026-07-05) + +### gid 0↔1 label swap — FIXED (root cause found) + +Root cause: `_compute_clusters` re-sorted group clusters by size +(descending) and reassigned ids — while Clojure assigns group ids by +first-k-distinct encounter order over base-cluster centers +(init-clusters, clusters.clj:55-64), keeps them through merge lineage, +and only ever `sort-by :id`. The base level already preserved k-means id +order (K-inv) with a comment warning against exactly this; the group +level did the forbidden thing three steps later. Fix: remove the re-sort ++ reassignment; pin with a synthetic first-encountered-is-id-0 test +(RED under any size sort). + +Harvest (verified on a full --include-local run, then re-validated — +232 passed / 138 xfailed / 0 xpassed / 0 failed): +- D8 repful blob comparison: xfail LIFTED on 9/11 variants (residual: + vw-incremental, pakistan-incremental — incremental trajectory). +- D9 significance-sets + D10 rep-selection: biodiversity-cold_start now + matches Clojure EXACTLY and gates. +- z-values / rat-values: label swap FALSIFIED as their cause (no variant + flipped) — reasons corrected to residual membership divergence. +- D12 priorities: FLI + bg2050 incremental blobs carry the all-49 + truthy-0 signature → match the #2571 mirror → now gate (known-bad + list shrunk to vw/biodiversity/bg2018/engage/pakistan incrementals). + +### Seed removal (Julien go, 2026-07-05) + +`np.random.seed(42)` (cluster_dataframe) removed + dead `import random`: +only `random` reference in the module, seeded an RNG nothing draws from, +not on the production path. The Clojure author's verbatim seeding note +(pca.clj:80-81) now lives in pca.py next to random_state, with the +seeding-history context and the 5-run determinism evidence. + +### Clojure randomness — verified facts (for the record) + +Clojure never fixes a seed: k-means deterministic by construction; PCA +power iteration uses UNSEEDED `(rand)` start on cold start only +(warm-started from previous eigenvectors after; conversation.clj:759 +uses unseeded :twister sampling for large convs). Fixed ITERATION COUNT +(not convergence threshold) → even Clojure-vs-Clojure cold starts are +not bit-identical. Consequences: tolerance-based comparison is the only +well-posed target for cold-start PCA; warm-start pinning collapses the +jitter (replay design §9). + +### EDN dumps: NO as-were history exists (R2 confirmed as inference) + +`conv-update-dump` has exactly one call site — conv_man.clj:321, the +update-ERROR handler — writing errorconv..edn to worker-local +(ephemeral) disk. Production never dumped healthy states; prodclone +holds votes + latest math_main only. R2's evidence: final blob + +math_tick counter (bounds #recomputes) + last_vote_timestamp. + +### Replay harness design doc + +`docs/REPLAY_HARNESS_DESIGN.md` (this commit): architecture, schedule +spec (first-class input — R2 = search over schedules with H as forward +model), Clojure driver Mode A (pure conv-update reduce + conv-update-dump +per step) / Mode B (Dockerized poller checkpointing), Python driver +(chained update_votes), nondeterminism policy (tolerance classes, +warm-start pinning, self-jitter measurement), storage/provenance, phased +build plan H-A..H-D. Review copy at scratch/REPLAY_HARNESS_DESIGN.md. + +### Proposed next math-core PR (awaiting go): powerit-pca port + +sklearn has NO equivalent of Clojure's per-component fixed-iteration +power iteration with deflation and start vectors (randomized SVD is +block+QR, no start-vector injection; scipy svds is Lanczos). Proposal: +~25-line numpy port of powerit-pca (same deflation, same fixed iters, +start_vectors param — feeds replay warm-start pinning), used in place of +sklearn SVD for parity; sklearn retained as the designated +post-parity implementation ("switch to a proper convergence criterion +once we move to improving the Python implementation" — per Julien). + +### R2 constraint + powerit-pca GO (Julien, 2026-07-05) + +- **R2 replayer must be PYTHON-ONLY** — no Clojure server; works purely + from Postgres data; candidate trajectories regenerated by the Python + engine in legacy-reproduction mode (which must therefore be an exact + AND much faster reproduction). Design doc updated (§1.3, §5, §10): + Clojure driver narrowed to R1 certification only. +- R1 comparison: per-step BLOB capture from a regular Clojure run + suffices for pass/fail; EDN dumps stay Clojure-only on-demand + (divergence localization + warm-start pinning). Open Q5 resolved. +- **powerit-pca port: GO** (sklearn has no equivalent — randomized SVD + is block+QR without start-vector injection). Two PERMANENT code paths + behind a flag: `clojure-legacy` (powerit fixed-iters + start_vectors, + "switch to a proper convergence criterion once we improve the Python + implementation") and `improved` (sklearn PCA). Benchmark + sklearn-vs-powerit from scratch as part of the PR. Future note: + scipy LOBPCG/ARPACK (`svds(v0=…)`) as library replacement for our + powerit once Clojure-exact fidelity is no longer required. +- test_participant_info golden comparisons (4 private datasets) joined + the PGR-deferral skips: their goldens embed per-gid correlations and + predate the gid re-ordering — stale by design, not regression. + +--- + +## Session 2026-07-06/07 — CI green-up of the powerit-PCA + storage-v2 stacks + +### Silhouette guard for the powerit-PCA default (#2591) + +Making `POLISMATH_PCA_IMPL=powerit` the default (#2591) surfaced a latent +crash — a robustness gap, not a parity defect. On small/synthetic +conversations the powerit projection collapses to exactly **two base +clusters**, and group-cluster k-selection (`conversation.py`) then calls +`calculate_silhouette_sklearn` on 2 points / 2 labels. sklearn requires +`2 <= n_labels <= n_samples - 1`, so it raised +`ValueError: Number of labels is 2. Valid values are 2 to n_samples - 1`. +This crashed `TestConversation.test_recompute` and errored 8 +`test_serialization_unfolding` cases in CI. Every one of them **passes under +`POLISMATH_PCA_IMPL=sklearn`**, which pinned the powerit default as the +trigger (the guard gap was always latent; sklearn's projection just never +collapsed this data to two base clusters). + +**Fix (squashed into #2591):** `calculate_silhouette_sklearn` +(`polismath/pca_kmeans_rep/clusters.py`) now returns the neutral `0.0` +sentinel whenever `n_labels >= n_samples` (silhouette is undefined there), +instead of letting sklearn raise. It is a strict **superset** of the old +`n_labels <= 1 || n_samples <= 1` guard, so valid clusterings are unchanged; +and with only two base clusters, k-selection is forced to `k=2` regardless, +so the chosen clustering is identical — the fix only removes the crash. Added +3 unit tests (`tests/test_clusters.py::TestCalculateSilhouetteSklearn`: +2-samples/2-labels → 0.0 not raise; single-label → 0.0; valid 3-sample/2-label +→ genuine score). + +Verified: local full suite **403 passed / 0 failed** (baseline was 1 failed + +8 errors); CI #2591 `test` job green. Follow-on cleanup for the improved +(sklearn) path: none needed — the guard is impl-agnostic. + +_(Storage-v2 CI green-up — delphi_storage Dockerfile COPY, the +postgres://→postgresql:// backend hardening, and the PG-conformance CI wiring +— is tracked in `STORAGE_V2_IMPLEMENTATION_NOTES.md`, not here.)_ diff --git a/delphi/docs/CLOJURE_COMPARISON.md b/delphi/docs/CLOJURE_COMPARISON.md index bed71f06c..72333c164 100644 --- a/delphi/docs/CLOJURE_COMPARISON.md +++ b/delphi/docs/CLOJURE_COMPARISON.md @@ -49,10 +49,13 @@ The Clojure reference implementation is in: **`math/src/polismath/math/clusters. This is the **primary reason** clustering results differ between Python and Clojure: -**Python** (Single-level clustering): -- `group_clusters`: Direct clustering of participants into k groups -- Member IDs: Participant IDs -- Example: {id: 0, members: [ptpt1, ptpt2, ...]} +**Python** (Two-level clustering, matching Clojure since PR #2431): +- `base_clusters`: First-level clustering (~100 small clusters of participants) + - Member IDs: Participant IDs + - Example: 100 base clusters with 3-7 participants each +- `group_clusters`: Second-level clustering of base clusters into k groups + - Members stored as base-cluster IDs internally, unfolded to participant IDs for serialization + - Example: {id: 0, members: [0, 1, 5, 8, ...]} where numbers are base cluster IDs (internally) **Clojure** (Two-level clustering): 1. `base-clusters`: First-level clustering of participants into ~100 small clusters @@ -70,7 +73,7 @@ Beyond the architecture, there's also an initialization difference: | Aspect | Python | Clojure | |--------|--------|---------| -| **Algorithm** | K-means++ (seed 42) | First k distinct points | +| **Algorithm** | First k distinct points (matching Clojure) | First k distinct points | | **Rationale** | Better convergence, industry standard | Simpler implementation | | **Result** | Different local optima | Different local optima | | **Quality** | Both are valid clustering algorithms | Both are valid clustering algorithms | @@ -96,12 +99,9 @@ Beyond the architecture, there's also an initialization difference: ### Why Tests Fail -The clustering test **intentionally fails** because: -1. Python uses K-means++ initialization → different initial cluster centers -2. K-means converges to nearest local optimum → different final clusters -3. Tests use very tight thresholds (95% Jaccard, 5% L1) to detect any difference - -This is **expected behavior** until we implement Option A (match Clojure initialization). +The clustering test **xfails conditionally** on some dataset variants due to incremental-blob +in-conv divergence / cold-start PCA landscape flatness — NOT initialization mismatch. +Python now uses first-k-distinct initialization, matching Clojure (since PR #2431). ## Running Tests diff --git a/delphi/docs/DELPHI_DOCKER.md b/delphi/docs/DELPHI_DOCKER.md index 140a60b0f..83b8f886c 100644 --- a/delphi/docs/DELPHI_DOCKER.md +++ b/delphi/docs/DELPHI_DOCKER.md @@ -7,7 +7,7 @@ This document provides information about the Delphi Docker container setup and o When the Delphi container starts, it performs the following steps: 1. Initializes DynamoDB tables using `create_dynamodb_tables.py` -2. Starts the job poller service using `start_poller.sh` +2. Starts the job poller by running `python scripts/job_poller.py` directly (the Dockerfile CMD invokes the script) ## Environment Variables @@ -32,8 +32,7 @@ The Delphi container runs the following services: If the container exits with code 127, check that: 1. The scripts directory is correctly copied into the container -2. The `start_poller.sh` script is executable -3. The DynamoDB endpoint is correct and accessible +2. The DynamoDB endpoint is correct and accessible ## Maintaining State diff --git a/delphi/docs/DELPHI_JOB_SYSTEM_TROUBLESHOOTING.md b/delphi/docs/DELPHI_JOB_SYSTEM_TROUBLESHOOTING.md index 8a714c5b1..36faa5561 100644 --- a/delphi/docs/DELPHI_JOB_SYSTEM_TROUBLESHOOTING.md +++ b/delphi/docs/DELPHI_JOB_SYSTEM_TROUBLESHOOTING.md @@ -325,5 +325,5 @@ print(f"Reset {count} stuck jobs") ## Related Documentation - [JOB_QUEUE_SCHEMA.md](JOB_QUEUE_SCHEMA.md) - Details about the job queue schema -- [ANTHROPIC_BATCH_API_GUIDE.md](ANTHROPIC_BATCH_API_GUIDE.md) - Guide for working with Anthropic's Batch API -- [DATABASE_NAMING_PROPOSAL.md](DATABASE_NAMING_PROPOSAL.md) - Information about database naming conventions \ No newline at end of file +- [DATA_FORMAT_STANDARDS.md](DATA_FORMAT_STANDARDS.md) - DynamoDB key formats and reserved-keyword handling +- [JOB_STATE_MACHINE_DESIGN.md](JOB_STATE_MACHINE_DESIGN.md) - Job types and state transitions \ No newline at end of file diff --git a/delphi/docs/DOCKER_BUILD_OPTIMIZATION.md b/delphi/docs/DOCKER_BUILD_OPTIMIZATION.md index 6b51dba96..7facd68cf 100644 --- a/delphi/docs/DOCKER_BUILD_OPTIMIZATION.md +++ b/delphi/docs/DOCKER_BUILD_OPTIMIZATION.md @@ -88,6 +88,8 @@ vim pyproject.toml # 2. Regenerate lock file make generate-requirements +# Note: the Makefile target calls pip-compile directly; project tooling policy is uv +# (equivalent: uv pip compile pyproject.toml -o requirements.lock) # 3. Rebuild Docker image make docker-build @@ -145,7 +147,7 @@ make generate-requirements-upgrade ```txt # requirements.lock (generated by pip-compile) # -# This file is autogenerated by pip-compile with Python 3.13 +# This file is autogenerated by pip-compile with Python 3.12 # by the following command: # # pip-compile --output-file=requirements.lock pyproject.toml diff --git a/delphi/docs/DOCUMENTATION_DIRECTORY.md b/delphi/docs/DOCUMENTATION_DIRECTORY.md index f3952928d..6d02022b4 100644 --- a/delphi/docs/DOCUMENTATION_DIRECTORY.md +++ b/delphi/docs/DOCUMENTATION_DIRECTORY.md @@ -1,91 +1,71 @@ # Delphi Documentation Directory -This document provides an overview of key documentation files in the Delphi system, organized by topic for easy reference. +Index of the documentation in `delphi/docs/`, organized by topic. -## Core System Documentation +> **Last cleaned: 2026-06-11.** 33 stale leftover docs (completed fix memos, session +> logs, unimplemented design proposals, docs describing deleted architecture) were +> moved to [archive/](archive/) — kept as raw material capturing the original design +> intent of the 2025 build-out, but **not documentation of the current system** (see +> [archive/CLAUDE.md](archive/CLAUDE.md) for the per-file index). Surviving docs were +> spot-checked against the code on that date; still, when a doc and the code disagree, +> trust the code. -| Document | Description | -|----------|-------------| -| [CLAUDE.md](../CLAUDE.md) | Main reference guide with configuration details, database interactions, and system operation | -| [README.md](../README.md) | Project overview and basic setup instructions | -| [QUICK_START.md](QUICK_START.md) | Get started quickly with the Delphi system | -| [RUNNING_THE_SYSTEM.md](RUNNING_THE_SYSTEM.md) | Step-by-step instructions for operating the Delphi system | -| [architecture_overview.md](architecture_overview.md) | High-level overview of the system architecture | -| [project_structure.md](project_structure.md) | Explanation of the project's directory and file organization | - -## Database and Data Format Documentation +## Canonical / living documents | Document | Description | |----------|-------------| -| [DATABASE_NAMING_PROPOSAL.md](DATABASE_NAMING_PROPOSAL.md) | Explanation of table naming conventions and migration plan | -| [DATA_FORMAT_STANDARDS.md](DATA_FORMAT_STANDARDS.md) | **Critical standards for data formats throughout the system, including DynamoDB key formats** | -| [JOB_QUEUE_SCHEMA.md](JOB_QUEUE_SCHEMA.md) | Schema documentation for the job queue system | -| [S3_STORAGE.md](S3_STORAGE.md) | Information about S3 storage configuration and access | +| [PLAN_DISCREPANCY_FIXES.md](PLAN_DISCREPANCY_FIXES.md) | **Canonical plan** for the Clojure-parity fix campaign (D-fixes), statuses, ordering | +| [CLJ-PARITY-FIXES-JOURNAL.md](CLJ-PARITY-FIXES-JOURNAL.md) | **Append-only session journal** of the parity work — findings, decisions, test results | +| [deep-analysis-for-julien/](deep-analysis-for-julien/) | Deep Clojure-vs-Python analysis (architecture, PCA, clustering, repness, routing, discrepancy catalog). Historical reference; see status note in `07-discrepancies.md` | -## Job System Documentation +## Getting started & operations | Document | Description | |----------|-------------| -| [JOB_SYSTEM_DESIGN.md](JOB_SYSTEM_DESIGN.md) | Overall job system architecture and design principles | -| [JOB_STATE_MACHINE_DESIGN.md](JOB_STATE_MACHINE_DESIGN.md) | **Detailed explanation of the job state machine and workflow design** | -| [DELPHI_JOB_SYSTEM_TROUBLESHOOTING.md](DELPHI_JOB_SYSTEM_TROUBLESHOOTING.md) | **Comprehensive guide to troubleshooting common job system issues** | -| [JOB_ID_MIGRATION_PLAN.md](JOB_ID_MIGRATION_PLAN.md) | Plan for migrating to the new job ID system | - -## API Integration Documentation +| [QUICK_START.md](QUICK_START.md) | Environment setup (uv/.venv) and standard test invocation | +| [RUNNING_THE_SYSTEM.md](RUNNING_THE_SYSTEM.md) | Operating the pipeline: run_delphi, CLI, job submission | +| [DELPHI_DOCKER.md](DELPHI_DOCKER.md) | Delphi container overview | +| [DOCKER_BUILD_OPTIMIZATION.md](DOCKER_BUILD_OPTIMIZATION.md) | Layered Docker builds, requirements.lock workflow | +| [DELPHI_AUTOSCALING_SETUP.md](DELPHI_AUTOSCALING_SETUP.md) | Instance-size based worker configuration (`configure_instance.py`, `INSTANCE_SIZE`) | +| [RESET_SINGLE_CONVERSATION.md](RESET_SINGLE_CONVERSATION.md) | Removing all Delphi data for one conversation | +| [S3_STORAGE.md](S3_STORAGE.md) | S3/MinIO storage for visualizations | +| [OLLAMA_MODEL_CONFIG.md](OLLAMA_MODEL_CONFIG.md) | Ollama model configuration for topic naming | +| [CLI_STATUS_COMMAND.md](CLI_STATUS_COMMAND.md) | `./delphi status ` CLI command | + +## Job system | Document | Description | |----------|-------------| -| [ANTHROPIC_BATCH_API_GUIDE.md](ANTHROPIC_BATCH_API_GUIDE.md) | **Complete guide for working with Anthropic's Batch API, including common issues and solutions** | -| [OLLAMA_MODEL_CONFIG.md](OLLAMA_MODEL_CONFIG.md) | Configuration guide for Ollama models | -| [CLI_STATUS_COMMAND.md](CLI_STATUS_COMMAND.md) | Documentation for the CLI status command | +| [JOB_STATE_MACHINE_DESIGN.md](JOB_STATE_MACHINE_DESIGN.md) | Job types (FULL_PIPELINE, CREATE_NARRATIVE_BATCH, AWAITING_NARRATIVE_BATCH) and transitions | +| [JOB_QUEUE_SCHEMA.md](JOB_QUEUE_SCHEMA.md) | `Delphi_JobQueue` schema, GSIs, locking patterns | +| [DELPHI_JOB_SYSTEM_TROUBLESHOOTING.md](DELPHI_JOB_SYSTEM_TROUBLESHOOTING.md) | Diagnosing stuck jobs, DynamoDB gotchas, log locations | +| [DATA_FORMAT_STANDARDS.md](DATA_FORMAT_STANDARDS.md) | DynamoDB key formats (`#` delimiters), reserved keywords, type conversions | -## Deployment and Infrastructure +## Math & Clojure parity reference | Document | Description | |----------|-------------| -| [DELPHI_AUTOSCALING_SETUP.md](DELPHI_AUTOSCALING_SETUP.md) | Configuration for auto-scaling the system | -| [DISTRIBUTED_SYSTEM_ROADMAP.md](DISTRIBUTED_SYSTEM_ROADMAP.md) | Roadmap for distributed system improvements | +| [CLOJURE_COMPARISON.md](CLOJURE_COMPARISON.md) | Clojure-vs-Python comparison test infrastructure and known differences | +| [CLOJURE_TWO_LEVEL_CLUSTERING.md](CLOJURE_TWO_LEVEL_CLUSTERING.md) | Two-level (base→group) clustering architecture, as implemented | +| [SUBGROUP_CLUSTERING_THIRD_LEVEL.md](SUBGROUP_CLUSTERING_THIRD_LEVEL.md) | Clojure's third clustering level (subgroups) — unported, unused by consumers | +| [regression_testing.md](regression_testing.md) | Golden-snapshot regression testing: recorder, comparer, datasets | +| [INVESTIGATION_K_DIVERGENCE.md](INVESTIGATION_K_DIVERGENCE.md) | K-means k divergence investigation (RESOLVED — kept as record) | +| [SESSION_HANDOFF_KMEANS.md](SESSION_HANDOFF_KMEANS.md) | K-means parity session background (partially historical — see status note) | +| [HANDOFF_PR14_VECTORIZED_REFACTOR.md](HANDOFF_PR14_VECTORIZED_REFACTOR.md) | Repness refactor handoff (14a in open stack; 14b/14c open) | +| [HANDOFF_REGRESSION_TEST_PERF.md](HANDOFF_REGRESSION_TEST_PERF.md) | Regression-test performance (mostly resolved — see status note) | -## Algorithm and Analysis Documentation +## Topic pipeline (umap_narrative) | Document | Description | |----------|-------------| -| [algorithm_analysis.md](algorithm_analysis.md) | Analysis of the core algorithms used in Delphi | -| [TOPIC_NAMING.md](TOPIC_NAMING.md) | Topic naming pipeline: exact prompt, deterministic 5‑comment sampling, logging, storage | -| [usage_examples.md](usage_examples.md) | Examples of system usage and output interpretations | +| [TOPIC_NAMING.md](TOPIC_NAMING.md) | Topic naming: prompt, sampling, storage in `Delphi_CommentClustersLLMTopicNames` | +| [topic-moderation-system.md](topic-moderation-system.md) | Topic moderation endpoints and `Delphi_TopicModerationStatus` | +| [TOPIC_AGENDA_STORAGE_DESIGN.md](TOPIC_AGENDA_STORAGE_DESIGN.md) | Topic agenda storage (`topic_agenda_selections`); Phase 3 never implemented | +| [VERSIONED_TOPIC_KEYS_IMPLEMENTATION.md](VERSIONED_TOPIC_KEYS_IMPLEMENTATION.md) | Versioned topic/section keys (`report_id#section#model`) | -## Testing and Development +## Open issues & audits (still unresolved — do not delete until fixed) | Document | Description | |----------|-------------| -| [SIMPLIFIED_TESTS.md](SIMPLIFIED_TESTS.md) | Simplified testing procedures | -| [TESTING_LOG.md](TESTING_LOG.md) | Log of testing activities and results | -| [TEST_RESULTS_SUMMARY.md](TEST_RESULTS_SUMMARY.md) | Summary of test results | - -## Recently Added Documentation - -The following documentation was recently added to address specific system challenges: - -1. **[ANTHROPIC_BATCH_API_GUIDE.md](ANTHROPIC_BATCH_API_GUIDE.md)** - Comprehensive guide for working with Anthropic's Batch API in the Delphi system, including: - - Handling JSON Lines (JSONL) responses from the API - - Proper error handling for API interactions - - Key format requirements for storing results in DynamoDB - - Debugging strategies for batch processing issues - -2. **[DELPHI_JOB_SYSTEM_TROUBLESHOOTING.md](DELPHI_JOB_SYSTEM_TROUBLESHOOTING.md)** - Detailed guide for troubleshooting job system issues, including: - - Strategies for diagnosing stuck jobs - - Solutions for common DynamoDB reserved keyword issues - - Techniques for tracing end-to-end job execution - - Database verification processes - -3. **[DATA_FORMAT_STANDARDS.md](DATA_FORMAT_STANDARDS.md)** - Critical standards document focusing on: - - Required format for DynamoDB keys (using # as delimiters) - - JSON structure standards for reports - - Handling of reserved keywords in DynamoDB - - Conversion between PostgreSQL and DynamoDB data types - -4. **[JOB_STATE_MACHINE_DESIGN.md](JOB_STATE_MACHINE_DESIGN.md)** - Documentation of the state machine design for job processing: - - Explicit job types for different processing stages - - Clear script mapping between job types and processing scripts - - Clean state transition patterns - - Error handling best practices \ No newline at end of file +| [ZID_EXPOSURE_AUDIT.md](ZID_EXPOSURE_AUDIT.md) | **Open security issue**: zid/conversation_id exposed in delphi API responses | +| TOPIC_LABEL_MISALIGNMENT_ANALYSIS.md | **Open bug**: label/cluster sorting misalignment in `700_datamapplot_for_layer.py` (local-only file, in `.git/info/exclude` — not present on fresh clones) | diff --git a/delphi/docs/HANDOFF_PR14_VECTORIZED_REFACTOR.md b/delphi/docs/HANDOFF_PR14_VECTORIZED_REFACTOR.md index f58691454..882cf0da3 100644 --- a/delphi/docs/HANDOFF_PR14_VECTORIZED_REFACTOR.md +++ b/delphi/docs/HANDOFF_PR14_VECTORIZED_REFACTOR.md @@ -1,5 +1,7 @@ # Handoff: PR 14 — Make Vectorized Code Readable + Blob Injection Tests +> **Status (2026-06-11):** PR 14a (delete dead scalar paths in repness.py) is in the open spr stack as PR #2564. Tasks 14b (vectorized blob-injection tests) and 14c (readability refactor) remain open and are tracked in `PLAN_DISCREPANCY_FIXES.md`. The branch names and stack listing below are from the 2026-03/06 sessions and are stale — do not branch from them. + ## Goal The scalar functions (`comment_stats`, `add_comparative_stats`, `repness_metric`, diff --git a/delphi/docs/HANDOFF_REGRESSION_TEST_PERF.md b/delphi/docs/HANDOFF_REGRESSION_TEST_PERF.md index 2692b3c53..87cd74327 100644 --- a/delphi/docs/HANDOFF_REGRESSION_TEST_PERF.md +++ b/delphi/docs/HANDOFF_REGRESSION_TEST_PERF.md @@ -1,5 +1,7 @@ # Handoff: Regression Test Performance Investigation +> **Status (2026-06-11):** Bottleneck 1 (`_compute_participant_info_optimized`) was vectorized (conversation.py group-correlation matrix ops). Bottleneck 2 (benchmark 3× runs) is resolved — `benchmark=False` is now the default in `comparer.py` and `test_regression.py` never enables it. `SKIP_GOLDEN` landed via #2515. Only the intermediate-stage redundancy question (Bottleneck 3) remains open. + ## Problem The `test_regression.py` tests are slow for large private datasets, particularly diff --git a/delphi/docs/JOB_QUEUE_SCHEMA.md b/delphi/docs/JOB_QUEUE_SCHEMA.md index 8a3aaba55..ebc1fb00f 100644 --- a/delphi/docs/JOB_QUEUE_SCHEMA.md +++ b/delphi/docs/JOB_QUEUE_SCHEMA.md @@ -7,7 +7,7 @@ This document defines the schema for the Delphi job queue system. The job queue ## Table Design ### Table Name -`DelphiJobQueue` +`Delphi_JobQueue` ### Primary Key Structure - **Partition Key**: `job_id` (String) - Unique identifier for each job (UUID v4) @@ -287,12 +287,14 @@ To manage the growth of the job queue table: ## Implementation Code +> **Caution:** The sample below predates the final schema — the actual table (see `create_dynamodb_tables.py`) uses `job_id` as the sole hash key, not `status`+`created_at`. + Here's a sample Python code for creating the job queue table: ```python import boto3 -def create_job_queue_table(dynamodb=None, table_name='DelphiJobQueue'): +def create_job_queue_table(dynamodb=None, table_name='Delphi_JobQueue'): if not dynamodb: dynamodb = boto3.resource('dynamodb') diff --git a/delphi/docs/PLAN_DISCREPANCY_FIXES.md b/delphi/docs/PLAN_DISCREPANCY_FIXES.md index d2df7b48d..b6dd5ff61 100644 --- a/delphi/docs/PLAN_DISCREPANCY_FIXES.md +++ b/delphi/docs/PLAN_DISCREPANCY_FIXES.md @@ -28,10 +28,11 @@ This plan's "PR N" labels map to actual GitHub PRs as follows: | PR 7 (D8) | #2522 | Stack 15/17 | Fix D8: finalize comment stats | | PR 12 (D15) | #2523 | Stack 16/17 | Fix D15: moderation handling | | (K-inv) | #2524 | Stack 17/17 | Fix K-means k divergence: preserve row order | -| PR 8 (D10) | — (WIP) | — | Fix D10: rep comment selection — **NEEDS REWORK** | -| PR 9 (D11) | — (WIP) | — | Fix D11: consensus selection — **NEEDS REWORK** | +| PR 14a (scalar deletion) | #2564 | — | Delete dead scalar paths in `repness.py`; migrate blob injection tests to vectorized | +| PR 8 (D10) | #2566 | — | Fix D10: rep comment selection — single-pass reduce matching Clojure | +| PR 9 (D11) | #2567 | — | Fix D11: consensus selection — whole-conv stats + per-side top-5 matching Clojure | | PR 10 (D3) | — (WIP) | — | Fix D3: k-smoother buffer — **NEEDS REWORK** | -| PR 11 (D12) | — (WIP) | — | Fix D12: comment priorities — **NEEDS REWORK** | +| PR 11 (D12) | — (in flight) | — | Fix D12: comment priorities — Clojure-parity importance/priority metrics + PCA comment projection | | PR 13 (D1) | — (WIP) | — | Fix D1: PCA sign flip prevention — **NEEDS REWORK** | | PR 15 | — (WIP) | — | Fix load_votes timestamp ordering — **NEEDS REWORK** | @@ -516,9 +517,9 @@ See `delphi/docs/INVESTIGATION_K_DIVERGENCE.md` for the full investigation. | D7 | Repness metric | PR 6 | **#2521** | **DONE** ✓ (formula change landed scalar + vectorized 2026-06-09; original PR diff was docs-only — recovered) | | D8 | Finalize cmt stats | PR 7 | **#2522** | **DONE** ✓ (rat > rdt classification landed scalar + vectorized 2026-06-09; original PR diff was docs-only — recovered) | | D9 | Z-score thresholds | **PR 3** | **#2518** | **DONE** ✓ | -| D10 | Rep comment selection | PR 8 | — (WIP) | VM draft — **NEEDS REWORK** (no blob injection tests) | -| D11 | Consensus selection | PR 9 | — (WIP) | VM draft — **NEEDS REWORK** (no blob injection tests) | -| D12 | Comment priorities | PR 11 | — (WIP) | VM draft — **NEEDS REWORK** (no blob injection tests) | +| D10 | Rep comment selection | PR 8 | **#2566** | Code-complete + 18 synthetic tests (was mislabeled "VM draft" until 2026-07-04); Copilot-review fixes in **#2586**; open in stack, merge pending edge freeze | +| D11 | Consensus selection | PR 9 | **#2567** | Code-complete + 12 synthetic tests; consensus entries now Clojure blob shape (tid/n-success/… — #2586); open in stack, merge pending edge freeze | +| D12 | Comment priorities | PR 11 | **#2568** | Code-complete + 11 synthetic tests (bug-mirror per #2571); Decimal-preserving serialization (#2586); open in stack, merge pending edge freeze | | D13 | Subgroup clustering | — | — | **Deferred** (unused) | | D14 | Large conv optimization | — | — | **Deferred** (Python fast enough) | | D15 | Moderation handling | PR 12 | **#2523** | **DONE** ✓ (zero-out-columns + downstream `to_math_blob` / `_compute_vote_stats` regressions fixed 2026-06-09 — `to_dict` now routes through `_compute_user_vote_counts()` / `_compute_votes_base()`; `_compute_vote_stats` uses `_get_clean_matrix(raw=True)`) | @@ -914,3 +915,30 @@ Tagging this as a follow-up. No code changes until we discuss. - **`to_dynamo_dict` parallel inline implementations** were refactored to route through the same helpers as `to_dict` in PR #2523 follow-up. No further action needed. +- **`ns` includes-PASS-divergence** (DISCOVERED 2026-06-11 during D11). Clojure's + `:ns` (via `count-votes` with `filter identity` — repness.clj:56-61) INCLUDES + PASS votes. Python's `compute_group_comment_stats_df` and `consensus_stats_df` + both compute `ns = na + nd`, excluding PASS. This is a real divergence that + affects `pa, pd, pat, pdt, ra, rd, rat, rdt` everywhere — every downstream + metric and selection. The D5 PR #2519 journal claim ("PASS NOT included, + matching Clojure") was based on a misreading of `count-votes`. Currently + causing 3-5% divergence in pat values for tids with non-zero PASS counts; + visible at the consensus-selection margins (4/6 overlap on vw cold_start + agree, 1/3 on disagree). Needs a dedicated PR — affects: + - `compute_group_comment_stats_df` (line ~283: `ns = na + nd`). + - `consensus_stats_df` (line ~435: `ns = na + nd`). + Fix: `ns = (vote_matrix_df != 0).sum(axis=0)` no, actually we want to + count non-NaN: `ns = vote_matrix_df.notna().sum(axis=0)` for wide format; + for long-format `votes_long.groupby('comment').size()` after dropna. + Re-record goldens afterward. +- **D10 take-5 eviction edge case** (2026-06-11). The Clojure-parity + `select_rep_comments_df` introduced in PR 8 mirrors Clojure exactly: + `take(5)` runs AFTER prepending the `best_agree` slot. When `best_agree` + was kept by `beats_best_agr?` as a non-significant agree-priority fallback + (i.e. it failed `passes_by_test?` but qualified via Branch 4) AND + `:sufficient` already has 5 entries, the prepend pushes the total to 6 + and `take(5)` silently evicts the 5th-highest-metric `sufficient` entry + — possibly a strong dissenting view. Mirrored for blob parity; flag for + future product review. See `# TODO(parity-eviction)` in + `delphi/polismath/pca_kmeans_rep/repness.py::select_rep_comments_df` and + the synthetic test `TestD10SelectRepCommentsBoundary::test_take_5_eviction_when_best_agree_outside_sufficient`. diff --git a/delphi/docs/QUICK_START.md b/delphi/docs/QUICK_START.md index be70d2a47..a9d01ac6c 100644 --- a/delphi/docs/QUICK_START.md +++ b/delphi/docs/QUICK_START.md @@ -45,71 +45,14 @@ is almost always the cause. ## Running Tests -### Using the Test Runner - -The most reliable way to test the system is using the simplified tests: - -```bash -# With the virtual environment activated -python run_tests.py --simplified -``` - -These tests run the core algorithms with minimal dependencies and are known to work correctly. - -You can also run other test types: - -```bash -# Run only unit tests (Note: some may fail due to implementation differences) -python run_tests.py --unit - -# Run demo scripts -python run_tests.py --demo -``` - -### System Test - -To run a comprehensive system test with real data: - -```bash -# Test with the biodiversity dataset (default) -python run_system_test.py - -# Test with the VW dataset -python run_system_test.py --dataset vw -``` - -Note: The system test is more prone to issues as it relies on specific attribute names and data structures. Check the `TESTING_LOG.md` file for known issues and their fixes. - -## Running Analysis Notebooks - -To run the biodiversity analysis directly without Jupyter: - -```bash -# Navigate to the eda_notebooks directory -cd eda_notebooks - -# Run the analysis script -python run_analysis.py -``` - -This will: -1. Load data from the biodiversity dataset -2. Process votes and comments -3. Run PCA and clustering -4. Calculate representativeness -5. Save results to the `output` directory - -To verify that the environment is set up correctly: - -```bash -python run_analysis.py --check -``` - -To launch the notebook server (if you prefer interactive analysis): - ```bash -# If you have Jupyter installed -jupyter notebook biodiversity_analysis.ipynb +cd delphi && uv run pytest tests/ -v --tb=short \ + --ignore=tests/test_batch_id.py \ + --ignore=tests/simplified_repness_test.py \ + --ignore=tests/test_pakistan_conversation.py \ + --ignore=tests/test_postgres_real_data.py \ + --ignore=tests/test_minio_access.py \ + --ignore=tests/test_math_pipeline_runs_e2e.py ``` ## Core Files to Understand @@ -118,30 +61,17 @@ Here are the key files to understand the system: 1. **Package Structure:** - `polismath/` - The main package directory - - `polismath/math/` - Core mathematical components + - `polismath/pca_kmeans_rep/` - Core mathematical components - `polismath/conversation/` - Conversation state management 2. **Core Math Components:** - - `polismath/math/named_matrix.py` - Data structure for matrices with named rows and columns - - `polismath/math/pca.py` - PCA implementation using power iteration - - `polismath/math/clusters.py` - K-means clustering implementation - - `polismath/math/repness.py` - Representativeness calculation + - `polismath/pca_kmeans_rep/pca.py` - PCA implementation + - `polismath/pca_kmeans_rep/clusters.py` - K-means clustering implementation + - `polismath/pca_kmeans_rep/repness.py` - Representativeness calculation + - `polismath/pca_kmeans_rep/corr.py` - Correlation utilities -3. **Simplified Implementations:** - - `simplified_test.py` - Standalone PCA and clustering implementation (more reliable) - - `simplified_repness_test.py` - Standalone representativeness calculation (more reliable) - - These files provide the clearest examples of how the algorithms work - -4. **Test Files:** +3. **Test Files:** - `tests/` - Unit and integration tests - - `run_tests.py` - Test runner script - - `run_system_test.py` - End-to-end system test with real data - -5. **End-to-End Examples:** - - `eda_notebooks/biodiversity_analysis.ipynb` - Complete analysis of a real conversation - - `eda_notebooks/run_analysis.py` - Script version of the notebook analysis - - `simple_demo.py` - Simple demonstration of core functionality - - `final_demo.py` - More comprehensive demonstration ## Documentation @@ -149,7 +79,7 @@ For more detailed documentation, refer to: - `README.md` - Main project documentation - `RUNNING_THE_SYSTEM.md` - Comprehensive guide on running the system -- `TESTING_LOG.md` - Log of testing process, issues, and fixes +- `regression_testing.md` - Regression testing approach and golden snapshots - `tests/TEST_MAP.md` - Map of all test files and their purposes - `tests/TESTING_RESULTS.md` - Current testing status and improvements @@ -195,8 +125,6 @@ To work with your own data: If you encounter issues: -1. Check `TESTING_LOG.md` for known issues and their solutions -2. Look at the simplified test scripts (`simplified_test.py` and `simplified_repness_test.py`) for reliable examples -3. Try running `run_analysis.py --check` to verify your environment -4. Examine error messages and try to isolate the problem -5. The `run_system_test.py` script provides a good template for loading and processing real data \ No newline at end of file +1. Check `regression_testing.md` for regression testing guidance and golden snapshot usage +2. See `RUNNING_THE_SYSTEM.md` for full pipeline documentation +3. Examine error messages and try to isolate the problem \ No newline at end of file diff --git a/delphi/docs/REPLAY_HARNESS_DESIGN.md b/delphi/docs/REPLAY_HARNESS_DESIGN.md new file mode 100644 index 000000000..4244f29af --- /dev/null +++ b/delphi/docs/REPLAY_HARNESS_DESIGN.md @@ -0,0 +1,318 @@ +# Replay Harness (H) — Design + +**Status:** DRAFT for review — 2026-07-05 +**Author:** Claude (host session "Fable-polis-merge-then-replay"), for Julien +**Recon basis:** file:line pointers verified 2026-07-05 against `math/` and `delphi/`. + +## 1. Purpose + +Replay a conversation's vote history through BOTH math implementations at +arbitrary, explicitly-chosen recompute points ("schedules"), recording full +intermediate state at every point, so we can compare them step-by-step. + +The harness serves four consumers, in order: + +1. **Gap measurement** — quantify Python(full-recompute) vs + Clojure(sequential) divergence per pipeline stage, to size the + sequential-bits work (warm-start `last-clusters`, k-smoothers, dispatcher). +2. **R1 — sequential parity validation** on schedules WE define (synthetic + and historic-with-chosen-cut-points): record schedule CCRs from Clojure, + run Python on the same schedule, compare. +3. **R2 — the inverse problem**: prodclone keeps NO history of math states + (`math_main` is latest-only UPSERT, `UNIQUE(zid, math_env)`; + `math_ticks` is a bare counter — see §10). The schedule Clojure actually + used in production is LATENT. R2 = infer it: search candidate + schedules, score against the recorded final blob. A harness whose + schedule is a first-class INPUT makes R2 a loop around H; a hardcoded + one makes R2 a rebuild. + **HARD CONSTRAINT (Julien, 2026-07-05): the R2 replayer is + PYTHON-ONLY** — no Clojure server, working purely from the data in + Postgres. Candidate trajectories are regenerated by OUR Python engine + running in legacy-reproduction mode. The Clojure driver (§5) exists + solely to CERTIFY, via R1, that Python-legacy-mode ≡ Clojure on + defined schedules; once certified, Clojure exits the loop. This makes + two things prerequisites for R2, not nice-to-haves: (a) the + sequential-bits port (warm-start last-clusters, k-smoothers, in-conv + carry-over), and (b) the powerit-pca port with start_vectors (§9) — + and it demands the Python engine be much FASTER than Clojure, since + R2 search runs many candidate replays. +4. **Eval framework (later)** — off-policy replay with a different + routing/recompute policy plugged into the same driver; plus the + idea-space-over-time visualization, which falls out of the per-step + records for free. + +## 2. Vocabulary + +- **Schedule**: ordered list of cut points partitioning a conversation's + event stream (votes + moderation events) into batches; recompute fires at + each cut point. +- **Step**: one (batch-ingest → recompute → record) cycle. +- **Schedule CCR** (type ① golden): Clojure's recorded state at each step of + a defined schedule. Cross-implementation TRUTH. +- **PGR** (type ②): Python regression freeze — out of scope here; PGRs stay + deferred to the Python-vs-Python phase. + +## 3. Architecture + +``` + votes CSV (order+timing, incl. revotes) ──┐ + prodclone votes table (created ms) ───────┤ + ▼ + ┌────────────────────────┐ + schedule spec (JSON) ───────────────────►│ SCHEDULE SLICER │ + (by count / time / fraction / explicit) │ sort → batch → events │ + └───────┬────────────────┘ + ┌─────────────────────┴──────────────────────┐ + ▼ ▼ + ┌─────────────────────────┐ ┌─────────────────────────┐ + │ CLOJURE DRIVER │ │ PYTHON DRIVER │ + │ new-conv → reduce over │ │ Conversation() → chain │ + │ conv-update / mod-update│ │ update_votes / │ + │ (pure, no Postgres) │ │ update_moderation │ + └──────┬──────────────────┘ └──────┬──────────────────┘ + │ per step │ per step + ▼ ▼ + step-NNN.edn (FULL state, step-NNN.json (to_dict blob + conv-update-dump) + full-state extras) + step-NNN.blob.json (prep-main │ + whitelist = math_main view) │ + └────────────────┬──────────────────────────┘ + ▼ + ┌──────────────────────┐ + │ STEP COMPARER │ + │ ConversationComparer │ + │ per-step, per-field, │ + │ tolerance classes │ + └──────────┬───────────┘ + ▼ + per-step / per-stage divergence report +``` + +## 4. Schedule spec (first-class input — R2 depends on this) + +JSON, one file per (dataset, schedule): + +```json +{ + "dataset": "vw", + "schedule_id": "front-loaded-01", + "source": "votes-csv", // or "prodclone" + "cuts": {"mode": "vote-count", "at": [50, 100, 200, 400, 800, "end"]}, + // modes: vote-count | timestamp | fraction | explicit-event-index + "moderation": "interleave-by-timestamp", // or "none" | explicit list + "clojure": {"warm_start": "chain"}, // chain | none | from-blob: + "notes": "front-loads recomputes in the early conversation" +} +``` + +Presets to ship: `uniform-N`, `front-loaded`, `back-loaded`, `every-vote` +(small datasets only), `single-cut` (≡ today's cold-start), `per-day` (from +real timestamps). Multiple schedules per conversation is the point. + +## 5. Clojure driver — R1 certification ONLY + +Role (narrowed 2026-07-05): produce schedule CCRs so R1 can certify that +Python-legacy-mode reproduces Clojure step-for-step. It is NOT part of the +R2 loop (§1.3) and can be retired once certification holds. + +For the R1 pass/fail comparison, capturing the per-step BLOB view +(prep-main JSON — the `math_main` shape Python's `to_dict` targets) from +a regular Clojure run over the schedule is SUFFICIENT. The EDN full-state +dump is optional depth: (a) divergence LOCALIZATION when a step +mismatches (smoothers, last-clusters, in-conv trajectory are not in the +blob), and (b) warm-start pinning material (§9). Default: record blobs +always, EDN on demand. + +**Mode A (primary): pure in-process.** No Postgres, no poller, no Docker. + +- Seed: `conv/new-conv` + `:zid`/`:meta-tids`, exactly as + `export.clj:624-632` (`get-export-data-at-time`) already does. +- Step: `(reduce conv/conv-update conv batches)` threading the returned + conv — the pattern already demonstrated at `dev/user.clj:416-428` and + `test/conversation_test.clj:40-168` (which runs conv-update on in-memory + matrices with zero DB). Interleave `conv/mod-update` + (conversation.clj:838) for moderation cut points. +- Record per step: + - `conv-update-dump` (conversation.clj:920) — serializes the ENTIRE conv + to EDN incl. core.matrix values (custom print-methods :882-899); reload + via `load-conv-update:932`. This is the full-fidelity CCR. + - `prep-main` (conv_man.clj:43-74) — the key-whitelisted production view + → JSON. This is the CROSS-LANGUAGE comparison surface (it is exactly + the `math_main` blob shape Python's `to_dict` targets). +- Packaging: a small `dev/replay.clj` (or `-M` alias) in `math/`, driven by + the schedule JSON. Resurrection risk is LOW: tools-deps + JDK17, live + docker-compose service, active commits. Watch items: core.matrix 0.63 / + vectorz 0.48 pins (the EDN print-methods depend on `mikera.*` classes). + +**Mode B (fidelity fallback): Dockerized poller.** Reuse +`generate_cold_start_clojure.py`'s fake-zid + `MATH_ZID_ALLOWLIST` + +throwaway-container pattern, but inject votes batch-by-batch and checkpoint +`math_main` between injections. Slower, blob-only (lossy), but exercises +the REAL production path (poller batching, conv-man actor). Use to +cross-validate Mode A once, then rely on Mode A. + +**Vote sourcing correctness (both modes):** +- Feed UNCORRECTED vote signs — the sign flip is export-only + (export.clj:106-113); the math consumes raw DB signs. +- Sort by timestamp before slicing: the votes CSVs are NOT pre-sorted + (vw: 2136 out-of-order rows) and `prepare_votes_data` (utils.py:267-274) + currently loads file order unsorted — a latent quirk the harness must NOT + inherit. +- Do NOT dedup revotes (vw: 87 revoted pairs): both engines implement + later-vote-wins merging internally; dedup-at-source (as the cold-start + SQL does via `DISTINCT ON`) erases the revote dynamics we specifically + want to replay. + +## 6. Python driver + +Nearly free: `Conversation.update_votes(votes, recompute=…)` is +pure-functional (deepcopy → new object, conversation.py:177-187), so the +driver is a chain over batches with `recompute=True` at cut points, plus +`update_moderation` interleaving. Record per step: `to_dict()` (blob view) +plus internal extras (base_clusters pre-fold, silhouettes, in-conv set) for +diagnosis. Lives in `delphi/polismath/replay/` with a thin +`scripts/replay_driver.py` CLI. + +## 7. Recording format & provenance + +``` +real_data/.local/replays/// + schedule.json # the input, verbatim + provenance.json # git commits (math/ and delphi/), dataset + # file hashes, timestamps, mode A/B, JVM/py versions + clj/step-000.edn # full Clojure state (CCR, Mode A only) + clj/step-000.blob.json # prep-main view (cross-language surface) + py/step-000.json # Python to_dict + extras + report/step-000.diff.json # comparer output +``` + +Under `.local/` (private-data footprint — replays of private datasets must +not leak into the public repo; public-dataset replays could later move). +Provenance satisfies the reproducible-traces requirement: any replay is +re-derivable from (schedule, dataset, two commits). + +## 8. Comparison + +Reuse `ConversationComparer` (comparer.py:25): it already does recursive +tolerant diff, PCA sign-flip detection (:949), scaling-factor detection +(:1028), and outlier fractions — and it accepts arbitrary nested +`{key: blob}` maps, so repointing from the fixed 6 stages to +`{step_i: blob}` is mechanical. + +Additions needed: +- **Tolerance classes per field family** (exact: counts, in-conv, mod sets, + selections/ids; tolerant: PCA comps/proj (angle-based), silhouettes, + repness stats; see §9). +- **Group-label permutation guard**: until the gid label-swap fix lands + (proposed 2026-07-05: remove the size-sort at conversation.py:794-798), + compare group-keyed structures under best-permutation matching, and + REPORT when the identity permutation wasn't the best one. + +## 9. Nondeterminism policy (Clojure-side, verified in source) + +Clojure never fixes a seed. Three consequences shape what "match" can mean: + +| Source | Where | Impact | +|---|---|---| +| Cold-start PCA start vector | `rand-starting-vec` pca.clj:79-82, bare `(rand)` | Power iteration runs a FIXED iteration count, so cold-start PCA output carries run-to-run jitter (and sign ambiguity). Even Clojure-vs-Clojure cold starts are not bit-identical. | +| Warm-start ticks | `powerit-pca` pca.clj:98 takes `start-vectors` from the previous tick | After step 0, PCA initialization is deterministic given the chain — sequential replays are MORE reproducible than cold starts. | +| Large-conv sampling | conversation.clj:759, unseeded `:twister` | Conversations crossing the large-conv threshold (>10k ptpts / >5k cmts) have ongoing sampling noise. None of the current test datasets cross it; flag if one does. | + +Policy: +- Comparisons are tolerance-based by field class; "bit-for-bit" is only + demanded where the algorithm is deterministic (counts, k-means given + fixed input order, selections given fixed stats). +- **Warm-start pinning**: for R1 debugging and all of R2, seed each Clojure + step's PCA from the recorded previous step (`start-vectors`), or from a + prodclone `math_main` blob — collapsing cold-start jitter. The schedule + spec's `clojure.warm_start` field selects this. +- Record N≥2 Clojure runs for at least one schedule to EMPIRICALLY measure + self-jitter per field; those envelopes become the tolerance floors + (a tolerance below Clojure's own jitter is unfalsifiable). + +## 10. R2 compatibility (why the design looks like this) + +Prodclone facts (verified): `votes` is append-only with full revote history +and ms timestamps (initial.sql:737-755); `math_main` is latest-only +(UPSERT, postgres.clj:324-338); `math_ticks` is a counter, not history. So +historic intermediate states are UNRECOVERABLE by reading — only +re-derivable by replay. This confirms the "direct replay isn't feasible" +experience: without the schedule, you can't regenerate the recorded blob +exactly. + +Also confirmed: `conv-update-dump` has exactly ONE call site — +conv_man.clj:321, the update-ERROR handler — writing to worker-local +ephemeral disk. Production never dumped healthy states anywhere. There is +no as-were intermediate history to recover, full stop. + +R2 therefore = schedule inference, **executed entirely in Python** (§1.3 +constraint): propose candidate schedules (priors from the production +poller's behavior: 1s time-window batching, poller.clj:12-36 — i.e. cuts +≈ "every gap > poll interval in the vote timeline", modulated by worker +downtime/restarts), run the PYTHON driver in legacy-reproduction mode as +the forward model, score candidates against the recorded final blob +(+ `last_vote_timestamp`, `math_tick` count as side information — the +tick counter bounds HOW MANY recomputes happened, a strong constraint). +Statistical inference enters in the scoring (which blob fields are +schedule-sensitive: in-conv trajectory, k-smoother state, base-cluster +geometry) and in handling the PCA-jitter noise floor. All of this is +possible ONLY because the schedule is an input file, the Python driver is +deterministic-given-schedule, and R1 has certified Python ≡ Clojure. + +**Legacy-reproduction mode (flag):** the Python engine grows two code +paths behind a single switch — `clojure-legacy` (faithful reproduction: +powerit-pca fixed-iteration with start_vectors, first-k-distinct k-means, +warm-start carry-over, truthy-0 priorities mirror) and `improved` +(sklearn PCA with proper convergence, and whatever we improve next). +Both paths are permanent, useful assets: legacy mode powers R2 and +parity work; improved mode is the product's future. R2 uses legacy mode +exclusively. + +## 11. Build plan + +- **Phase H-0 (parity prerequisites, math-core):** powerit-pca numpy port + (same fixed-iteration per-component power iteration + Gram-Schmidt + deflation, `start_vectors` param; GO 2026-07-05) behind the + legacy-reproduction flag (§10); benchmark vs sklearn from scratch. + Sequential-bits port (warm-start last-clusters, k-smoothers) follows, + sized by H-C. +- **Phase H-A (Python side + spine, pure Python):** schedule spec + slicer + (sorting/revote handling per §5), Python driver, store layout, + comparer repointing. Small; unblocks synthetic-schedule R1 for + Python-vs-Python-expectation tests immediately. This driver IS the + future R2 forward model (§1.3). +- **Phase H-B (Clojure Mode A):** `dev/replay.clj` + blob-per-step + recording (EDN on demand) + provenance. First real schedule CCRs. Then + the **self-jitter measurement** (§9) before any cross-language claims. +- **Phase H-C (first science):** gap-measurement report on vw + + biodiversity across 3 schedules (uniform / front / back): per-step, + per-stage divergence table. This sizes the sequential-bits (D) work and + is the go/no-go input for warm-start porting. +- **Phase H-D (later):** Mode B cross-validation; R1 certification runs; + R2 prototype on one small conversation (pure Python, legacy mode); + eval-framework policy hook. + +Ordering note: H-A/H-B can start now — they touch no math-core formulas +and don't depend on the stack merge. The GAP MEASUREMENT (H-C) should run +on post-merge code (else D10-D12 deltas confound sequencing deltas). + +## 12. Open questions (for review) + +1. Storage: `.local/replays/` OK? Public-dataset replays public later? +2. Mode B: is one-time cross-validation of Mode A enough, or do we want + the poller path exercised routinely (slower, but catches conv-man + batching semantics)? +3. Schedule presets: which historic datasets first, beyond vw/biodiversity? +4. `math_tick` as R2 side-info: prodclone's tick counter per zid — worth + pulling into the dataset exports now so R2 has it later? +5. ~~EDN full-state dumps~~ RESOLVED 2026-07-05 (Julien): per-step BLOB + capture from a regular Clojure run suffices for R1 pass/fail; EDN + stays Clojure-only, on demand, for divergence localization and + warm-start pinning. +6. powerit-pca future: once the improved path matures, evaluate replacing + our numpy powerit with a library iterative eigensolver — sklearn has + no per-component power iteration (randomized SVD is block+QR, no + start-vector injection), so candidates are scipy LOBPCG/ARPACK + (`svds(v0=…)`, Lanczos — different trajectory, fine once we no longer + need Clojure-exact fidelity). Until then: our port for legacy mode, + sklearn PCA for improved mode. diff --git a/delphi/docs/RUNNING_THE_SYSTEM.md b/delphi/docs/RUNNING_THE_SYSTEM.md index 10b318601..78264d479 100644 --- a/delphi/docs/RUNNING_THE_SYSTEM.md +++ b/delphi/docs/RUNNING_THE_SYSTEM.md @@ -26,76 +26,26 @@ This document provides a comprehensive guide on how to set up, run, and test the # Navigate to the delphi directory cd delphi -# Create a virtual environment -python -m venv .venv - -# Activate the virtual environment -# On Linux/macOS -source .venv/bin/activate -# On Windows -.venv\Scripts\activate +# Install all dependencies (creates delphi/.venv) +uv sync ``` -## Package Installation - -Once your environment is set up, install the package in development mode: - -```bash -# Make sure you're in the delphi directory -pip install -e . -``` - -This will install all the required dependencies and make the `polismath` package available in your environment. +Alternatively, `make venv` creates the venv and sets up the editor-discovery symlink at the repo root in one step. ## Running Tests -### Using the Test Runner Script - -The most straightforward way to run tests is using the provided `run_tests.py` script: +Use the standard pytest invocation (see `QUICK_START.md` for the full command with required `--ignore` flags): ```bash -# Run all tests -python run_tests.py - -# Run only unit tests -python run_tests.py --unit - -# Run only real data tests -python run_tests.py --real - -# Run only demo scripts -python run_tests.py --demo - -# Run only simplified test scripts -python run_tests.py --simplified +cd delphi && uv run pytest tests/ -v --tb=short \ + --ignore=tests/test_batch_id.py \ + --ignore=tests/simplified_repness_test.py \ + --ignore=tests/test_pakistan_conversation.py \ + --ignore=tests/test_postgres_real_data.py \ + --ignore=tests/test_minio_access.py \ + --ignore=tests/test_math_pipeline_runs_e2e.py ``` -### Using pytest Directly - -For more control over test execution, you can use pytest directly: - -```bash -# Run all tests -python -m pytest tests/ - -# Run a specific test file -python -m pytest tests/test_pca.py - -# Run tests with coverage -python -m pytest --cov=polismath tests/ -``` - -### Understanding Test Output - -Test output will indicate whether each component passes its tests. The real data tests will provide additional information: - -- Number of participants and comments processed -- Number of groups found -- Top representative comments for each group -- Comparison with Clojure output (where available) - -Test results for real data are saved to the `python_output` directory within each dataset's folder for manual inspection. - ## Using the System ### Running the Full Pipeline @@ -183,43 +133,6 @@ clusters = conv.group_clusters repness = conv.repness ``` -## Working with Notebooks - -The `eda_notebooks` directory contains Jupyter notebooks for exploratory data analysis and demonstrating system capabilities. - -### Running the Biodiversity Analysis Notebook - -1. Make sure your environment is set up and the package is installed -2. Navigate to the `eda_notebooks` directory -3. Start Jupyter Notebook or Jupyter Lab: - -```bash -cd delphi/eda_notebooks -jupyter notebook -# or -jupyter lab -``` - -4. Open `biodiversity_analysis.ipynb` -5. Run all cells to see the complete analysis - -### Creating Your Own Analysis - -To create your own analysis: - -1. Copy one of the existing notebooks as a template -2. Update the data paths to your own dataset -3. Customize the analysis as needed - -### Helper Script - -You can use the included helper script to launch a notebook server: - -```bash -cd delphi/eda_notebooks -./launch_notebook.sh -``` - ## Command-line Interface The package provides several CLI entry points: @@ -242,38 +155,12 @@ delphi list See `pyproject.toml` for the full list of CLI entry points. -## Running the Simplified Test Scripts - -The repository includes simplified versions of the core algorithms that can be run independently: - -```bash -# Run the simplified PCA and clustering test -python simplified_test.py - -# Run the simplified representativeness test -python simplified_repness_test.py -``` - -These scripts demonstrate the core algorithms without depending on the full package structure and can be useful for understanding the underlying mathematics. - -## Running the Demo Scripts - -The repository includes demo scripts that demonstrate the system's capabilities: - -```bash -# Run the simple demo -python simple_demo.py - -# Run the final demo -python final_demo.py -``` - ## Troubleshooting ### Common Issues 1. **ImportError or ModuleNotFoundError** - - Make sure you've installed the package with `pip install -e .` + - Make sure you've installed the package with `uv sync` - Check if your virtual environment is activated 2. **File Not Found Errors** @@ -296,4 +183,4 @@ If you encounter issues, check: This guide covers the basics of setting up, running, and testing the Pol.is math Python implementation. For more details on the implementation, refer to the README.md and the source code documentation. -If you're new to the system, we recommend starting with the notebooks in the `eda_notebooks` directory, particularly `biodiversity_analysis.ipynb`, which provides a comprehensive demonstration of the system's capabilities. \ No newline at end of file +If you're new to the system, see `QUICK_START.md` for environment setup and the standard test invocation. \ No newline at end of file diff --git a/delphi/docs/SESSION_HANDOFF_KMEANS.md b/delphi/docs/SESSION_HANDOFF_KMEANS.md index a600f910d..59347f64b 100644 --- a/delphi/docs/SESSION_HANDOFF_KMEANS.md +++ b/delphi/docs/SESSION_HANDOFF_KMEANS.md @@ -1,5 +1,7 @@ # K-means Two-Level Clustering - Session Handoff +> **Status (2026-06-11):** Two-level clustering and cold-start blob generation described below are merged (#2431, #2485). The k-divergence investigation concluded — see `INVESTIGATION_K_DIVERGENCE.md` (RESOLVED). Still open: incremental clustering warm-start (`:last-clusters`) and the D3 k-smoother, tracked in `PLAN_DISCREPANCY_FIXES.md`. Kept as background reference; do not treat its TODO lists as current. + ## Goal **Modify Python to match Clojure's EXACT two-level clustering architecture**, including: diff --git a/delphi/docs/STORAGE_V2_DESIGN.md b/delphi/docs/STORAGE_V2_DESIGN.md new file mode 100644 index 000000000..491f1b4d1 --- /dev/null +++ b/delphi/docs/STORAGE_V2_DESIGN.md @@ -0,0 +1,569 @@ +# Delphi Storage V2 — Reproducible Runs, Unified Schema, Dual-Backend Storage + +**Status:** DRAFT for review — 2026-07-06 +**Author:** Claude (host session "Fable JobID"), for Julien +**Recon basis:** file:line pointers verified 2026-07-06 against `delphi/`, `server/`, and clients. + +## 1. Problem + +The Delphi pipeline (Python ML in `delphi/`, TypeScript server endpoints, reporting +clients) is **not reproducible from code**: + +- No `job_id` flows through the operations pipeline — computations and their stored + results cannot be traced back to the job that produced them. +- Input sets (votes, comments, moderation state, config) are not recorded per run. +- State is keyed by `zid` and **overwritten** on each re-run — no history, no replay. +- 18 entangled DynamoDB tables (`Delphi_*` prefix) with unclear ownership and duplication. + +Goal: every computation replayable; the full state of a conversation reconstructible at +any point in time; a simpler schema; and a storage abstraction that can host the data in +**either DynamoDB or PostgreSQL** (config-selected). + +## 2. Decisions (Julien, 2026-07-06) + +| Question | Decision | +|---|---| +| Deliverable | Thorough study/audit + target design + phased implementation plan | +| Schema migration | **Dual-run transition**: new schema written alongside old tables; old readers keep working until explicitly switched over | +| Replay strictness | **Input-level reproducibility**: record exact inputs (vote/comment set, moderation state, config, seeds) + all outputs per job; LLM steps store prompt+response but re-runs may differ in phrasing | +| Backend abstraction | **Strictly neutral** repository interface; neither DynamoDB nor Postgres privileged | +| Narrative stores | **Unify** `Delphi_NarrativeReports` (Python) and `report_narrative_store` (server generator) into one entity | +| Clojure `math_main` dependency | **Snapshot as run input** (copy the blob); switching consumers to Python PCA results stays in the parity effort | +| PG backend rollout | **Both sides from the start** — Python AND TypeScript repository implementations land together; PG-only deployment viable as soon as the new schema exists | + +## 3. Current-state audit + +### 3.1 DynamoDB table inventory (18 tables) + +Two schema sources **disagree**: `create_dynamodb_tables.py` (canonical, PAY_PER_REQUEST, +GSIs) vs `polismath/database/dynamodb.py::_ensure_tables_exist` (provisioned, no GSIs, +duplicate for the 6 math tables). Whichever runs first wins. + +**Math tables** (written by `polismath/database/dynamodb.py::DynamoDBClient`, raw dicts, no Pydantic): + +- `Delphi_PCAConversationConfig` — PK zid, **overwrite**; holds `latest_math_tick` pointer. +- `Delphi_PCAResults` — PK zid + SK math_tick; `Delphi_KMeansClusters`, + `Delphi_CommentRouting`, `Delphi_RepresentativeComments`, + `Delphi_PCAParticipantProjections` — keyed by `"{zid}:{math_tick}"` composites. *Look* + versioned, but **`math_tick = 25000 + (time.time() % 10000)`** — computed at TWO + serializer sites (`conversation.py:1932` and `:2479`; any fix must hit both) — + pseudo-random, non-monotonic, collides within 10000s windows. And `run_delphi.py:54-69` + calls `reset_conversation.py` **unconditionally at the start of every run**, wiping all + 16 tables → effective semantics is single-version replace-everything. + +**UMAP tables** (written by `DynamoDBStorage` in +`umap_narrative/polismath_commentgraph/utils/storage.py`, Pydantic-based): + +- `Delphi_UMAPConversationConfig`, `Delphi_CommentEmbeddings`, + `Delphi_CommentHierarchicalClusterAssignments`, `Delphi_CommentClustersStructureKeywords`, + `Delphi_UMAPGraph`, `Delphi_CommentClustersFeatures`, `Delphi_CommentExtremity` — all + keyed by `conversation_id` (+item SK), **overwrite per item**, no job_id. +- `Delphi_CommentClustersLLMTopicNames` — the ONE UMAP table versioned by job_id + (`topic_key = "{job_id}#{layer}#{cluster}"`). + +**Narrative/job/server tables:** + +- `Delphi_NarrativeReports` — PK `"{report_id}#{section}#{model}"` + SK timestamp — + append; embeds job_id in section keys. +- `Delphi_JobQueue` — PK job_id, 4 GSIs, optimistic-lock mutation. +- `Delphi_CollectiveStatement` (PK `zid_topic_jobid`), `Delphi_TopicAgendaSelections` — + **server-owned** (written by Node/TS, only created/reset from Python). + +**In-place mutation hotspots:** `Delphi_CommentRouting.priority` (stage 502), +`Delphi_JobQueue` status transitions. + +**Pydantic coverage:** only the 7 UMAP tables; math tables, JobQueue, NarrativeReports, +and the actual extremity writer bypass models. Latent bugs: `LLMTopicName` model silently +drops job_id (only the key carries it); `EnhancedTopicName` path references a nonexistent +table key (dead). + +### 3.2 job_id lifecycle and gaps + +- Created at submission: `scripts/delphi_cli.py:81` `uuid.uuid4()` (server submits + similarly, but `batchReports.ts` uses a **different format**: + `batch_report_{rid}_{ts}_{rand}`). +- Propagated **only as env var** `DELPHI_JOB_ID` set by `scripts/job_poller.py:727` — + `run_delphi.py` doesn't read it; subprocesses just inherit env. +- Lands only in: `umap_narrative/run_pipeline.py:1378` (→ LLM topic-name keys) and + `801_narrative_report_batch.py` (→ report section keys + JobQueue updates). +- **Dropped everywhere else**: the entire polismath/math side (zero job_id awareness; + uses pseudo-random math_tick), all UMAP embedding/graph/cluster/keyword/features + writes, stages 501/502. + +### 3.3 Pipeline orchestration & input flow + +Two historically-separate pipelines stitched by `run_delphi.py` (no `run_delphi.sh` +anymore), communicating via **live Postgres re-reads** and DynamoDB tables, not in-memory +hand-off: + +| # | Stage | Entry | Reads | Writes | +|---|---|---|---|---| +| 0 | Reset | `umap_narrative/reset_conversation.py` | — | **deletes** all DynamoDB rows for zid | +| 1 | Math (PCA/kmeans/repness) | `polismath/run_math_pipeline.py` (raw psycopg2) | **live PG** votes (ALL rows, `ORDER BY created`, LIMIT/OFFSET batches), comments, moderation | DynamoDB PCA/KMeans/Repness/Routing/Projections/Config tables | +| 2 | UMAP narrative | `umap_narrative/run_pipeline.py` | **live PG** comments + `report_comment_selections` (NOT votes) | DynamoDB meta/embeddings/graph/cluster-assignments/topics | +| 3 | Comment extremity | `501_calculate_comment_extremity.py` | **live PG `math_main` (CLOJURE blob!)**, fallback placeholder heuristic from raw votes | DynamoDB `Delphi_CommentExtremity` | +| 4 | Priorities | `502_calculate_priorities.py` | DynamoDB CommentRouting + CommentExtremity | DynamoDB CommentRouting.priority | +| 5 | Visualizations | `700_datamapplot_for_layer.py` | DynamoDB cluster assignments | HTML/PNG/SVG + S3/MinIO | +| N | Narrative (separate job type `CREATE_NARRATIVE_BATCH`) | `801_narrative_report_batch.py` → Anthropic Batch API → `803_check_batch_status.py` | **live PG `math_main` (Clojure)** + live PG comments + DynamoDB clusters/topics | DynamoDB `Delphi_NarrativeReports` | + +**Job queue** (`scripts/job_poller.py`, `scripts/delphi_cli.py`, table `Delphi_JobQueue` +keyed by `job_id`, 4 GSIs): optimistic-locking claim via conditional update + `version`; +zombie-lock re-queue; job types actually dispatched are only `FULL_PIPELINE`, +`CREATE_NARRATIVE_BATCH`, `AWAITING_NARRATIVE_BATCH` (PCA/UMAP "types" exist only as +unused `job_config.stages`). Priority stored+indexed but **not used in ordering**. Retry +fields exist but no retry is implemented. Logs stored in the job item, truncated to last +50 entries. Job size routing queries live PG comment count. 3–4 separate PostgresClient +implementations exist (`polismath/database/postgres.py` SQLAlchemy; +`umap_narrative/.../utils/storage.py`; `job_poller.py`'s own; raw psycopg2 in +`run_math_pipeline.py`). + +**Input provenance recorded: NONE on the Python path.** No last-vote-timestamp, vote +count, tick, or hash stored with outputs. (Clojure's `math_main` has +`last_vote_timestamp`/`math_tick` but Python only reads, never writes them.) Moderation +filtered in Python not SQL; stage 1 uses raw `votes` (incl. superseded votes) while other +paths use `votes_latest_unique` — inconsistent. Vote-sign flip (PG AGREE=-1 → Delphi +AGREE=+1) duplicated in 3 places. + +**For faithful replay, must snapshot:** PG `votes`, `comments`, `participants`, +`conversations`, `report_comment_selections`, **and the Clojure `math_main` blob** +(extremity/consensus/narrative depend on it). Stages 4–5 are replayable from stages 1–3 +outputs if those are captured. + +**Postgres writes from delphi: none live.** `polismath/database/postgres.py` has +dead-code write methods mirroring the Clojure worker (`write_math_main`, +`increment_math_tick`, `math_ptptstats`, `worker_tasks`) — zero callers. Prior art for a +PG results backend. + +### 3.4 Nondeterminism inventory + +- Seeded: sklearn PCA (`random_state=42`), math KMeans (`random_state=42` — but init + depends on vote-encounter row order, deliberate for Clojure parity; the vestigial + `np.random.seed(42)` was removed lower in this stack), UMAP (`random_state=42`), + KMeans fallback (42). +- **Unseeded: EVōC clustering** (`run_pipeline.py:184`, `evoc.EVoC(min_samples=5)`) — + primary nondeterminism source. +- Embeddings deterministic given model, but model version only partially recorded. +- LLMs: no temperature/seed set anywhere; Ollama topic-name responses NOT persisted; + Anthropic narrative **responses persisted** (`{report_id}#{section}#{model}` key) but + **prompts NOT persisted** (assembled at runtime from XML templates + live comments). + +### 3.5 Config handling + +Config passed via env vars + CLI args + hardcoded values; `polismath/components/config.py` +(layered Config with save/load) exists but is NOT wired into the production run path. +`job_config` JSON on job items is **not a faithful record** (records UMAP params the code +hardcodes differently). No record of seeds, library versions, `MATH_ENV`, git SHA, prompts. + +### 3.6 Server/TypeScript + client consumption + +**Routes** (registered in `server/app.ts` monolith, handlers in `server/src/routes/**`; +clients always speak `report_id`, server resolves rid→zid then queries Dynamo by +`conversation_id=String(zid)`): + +- **Job enqueue** (the ONLY Delphi triggers — explicit admin/user action, no cron, gated + on `delphiEnabled`): `POST /api/v3/delphi/jobs` (uuid job_id, FULL_PIPELINE → + `Delphi_JobQueue`); `POST /api/v3/delphi/batchReports` (different job_id format, + CREATE_NARRATIVE_BATCH). +- **Result reads**: `GET /delphi` (LLMTopicNames + NarrativeReports), `GET /delphi/reports` + (NarrativeReports via GSI, groups by job_id, returns `current_job_id` + + `available_runs[]` — **the server/clients already have a run-pinning notion**), + `GET /delphi/visualizations` (JobQueue GSI + S3), `topicMod/*` (LLMTopicNames, + CommentClusters, TopicModerationStatus, UMAPGraph, ClusterAssignments, + StructureKeywords), `topicStats`, `collectiveStatement` (writes + `Delphi_CollectiveStatement`), `topicAgenda/selections` (Postgres table + `topic_agenda_selections`, stamps `delphi_job_id` from latest COMPLETED job), RSS + `feeds`, and `nextComment.ts` reads Delphi cluster tables directly for comment routing. +- `GET /reportNarrative` — a **second, server-side narrative generator** writing a + PARALLEL store `report_narrative_store` via `DynamoStorageService` + (`server/src/utils/storage.ts`) — the only existing TS storage abstraction; everything + else uses ad-hoc per-file Dynamo clients (~10 files re-deriving creds/endpoint). +- **Server reads tables that no creation script defines**: `Delphi_CommentClusters`, + `Delphi_TopicModerationStatus`. +- "Latest run" is inferred by sorting timestamps — no first-class latest pointer. + +**Legacy Clojure math prior art for a PG backend** +(`server/postgres/migrations/000000_initial.sql`): JSONB blobs keyed `(zid, math_env)` +(`math_main`, `math_ptptstats`, `math_bidtopid`, `math_cache`, ...; `(rid, math_env)` for +`math_report_correlationmatrix`), monotonic version in `math_ticks` +(`UNIQUE(zid, math_env)`), `caching_tick` for server cache polling +(`server/src/utils/pca.ts`), and `worker_tasks` as the job queue. This is the model to port. + +**Clients**: `client-report` is the main consumer (`/delphi`, `/delphi/reports` with +`available_runs`/`current_job_id`, `/delphi/visualizations`, `topicStats`, +`collectiveStatement`, `topicAgenda`); `client-admin` topic moderation uses `topicMod/*`; +`client-participation-alpha` has typed wrappers (`src/api/delphi.ts`, `topicAgenda.ts`). +Clients never see zid. + +**Implication: the abstraction layer must be two-sided.** Both the Python pipeline AND +the TypeScript server touch the store directly. A neutral repository interface needs a +Python implementation (pipeline read/write) and a TypeScript implementation (server: job +enqueue + result reads + topicMod/collectiveStatement writes). + +### 3.7 Existing docs / prior proposals + +- `docs/DATABASE_NAMING_PROPOSAL.md` — authoritative table→key→purpose catalog. +- `docs/JOB_QUEUE_SCHEMA.md` — job table design; documents unimplemented features + (CANCELLED, dependencies, retention). +- `docs/deep-analysis-for-julien/` — cleanest dataflow map; PG = source of truth. +- `docs/DATA_FORMAT_STANDARDS.md` — composite-key formats (`#` delimiter). +- `docs/REPLAY_HARNESS_DESIGN.md` — the schedule-replay harness (see §8; complementary, + different kind of replay). +- No existing doc addresses run manifests / input snapshots — a genuinely new axis. + Closest prior art: golden-snapshot regression harness (snapshots outputs, not inputs). + +## 4. Design + +### 4.1 Core move + +From "18 tables keyed by zid, overwritten on every run" to **"immutable runs + +append-only artifacts + a latest pointer"**. Every computed result becomes an artifact of +a run; "current state of a conversation" is a pointer, not a table. This single change +eliminates: the unconditional `reset_conversation.py` wipe, the pseudo-random +`math_tick`, the in-place `CommentRouting.priority` mutation, and timestamp-sorting to +find "latest". + +### 4.2 Six logical entities (replacing 18+ tables) + +1. **`runs`** — manifest AND job queue in one (the queue row becomes the manifest as the + job executes). Key: `job_id` (uuid for ALL job types; the `batch_report_...` format + retires). Holds: zid, rid, job_type (`FULL_PIPELINE`|`NARRATIVE_BATCH`| + `SERVER_NARRATIVE`|`IMPORTED` — §6.3), status, priority, optimistic-lock `version`, + worker/lease fields, + `config_requested` vs **`config_effective`** (what the code actually used — assembled + by stages registering their real params at execution time), `code_version` (git SHA + + library versions), `seeds`, `input_fingerprints` (sha256 + row counts + max vote + `created`), per-stage status/timings, `replay_of`, `math_tick_legacy` (transition only). +2. **`run_inputs`** — immutable snapshots written once at job start. Key + `(job_id, kind#part)`. Kinds: `votes` (RAW stream, order preserved — see §4.4), + `comments` (full rows incl. mutable `mod`), `participants`, + `report_comment_selections`, `conversation_meta`, **`clojure_math_main`** (full JSONB + copy — locked decision), `config_effective`. +3. **`artifacts`** — every stage output. Key `(job_id, artifact_key)` using the existing + `#` composite convention: `math#pca`, `math#kmeans`, `math#repness`, + `math#routing#`, `math#projections#`, `umap#meta`, + `umap#embeddings#`, `umap#assignments#`, `umap#graph#`, + `umap#keywords#`, `umap#features#`, `umap#extremity`, + `umap#topic##`, **`priorities`** (own artifact — no more in-place + mutation), `viz##` (S3 keys + hashes), **`llm##`** + (prompt+response+model+params — fixes unpersisted Ollama responses and Anthropic + prompts), **`narrative#
#`** (the unified narrative entity), + `log#` (append-only, replaces truncate-to-50). Large numeric payloads stored + zstd-compressed packed float64 (base64 in Dynamo, `bytea` in PG) — bit-exact + round-trip, ~5-10x smaller; chunking >300KB is a Dynamo-backend concern hidden behind + the interface. +4. **`latest`** — first-class "latest successful run" pointer. Key `scope` + (`zid##FULL_PIPELINE`, `rid##NARRATIVE_BATCH`, `rid##SERVER_NARRATIVE`) + → `job_id` + monotonic `seq` (the legitimate heir of `math_ticks`/`caching_tick`, + supports server cache polling) + the run's `job_type` (so conditional writes can + discriminate real vs IMPORTED runs in a single atomic operation — §6.2 invariant 1). + **Written last, after manifest flips COMPLETED — it is the commit point.** Crashed + half-written runs are simply never referenced; a janitor (generalizing today's + zombie-lock re-queue, lands with P7) marks lease-expired runs FAILED while + **retaining their artifacts and `log#` chunks for diagnosis** — deletion happens only + via the retention policy (§9). +5. **`topic_moderation`** and 6. **`collective_statements`** — server-owned USER state + (not computed), separate entities but behind the same repository so PG-only + deployments include them. PG `topic_agenda_selections` stays a native PG table as today. + +Physical naming: Dynamo `Delphi2_*` (prefix configurable); PG schema `delphi` with typed +key columns + JSONB/bytea payload, following the legacy `math_main` pattern +(`server/postgres/migrations/000000_initial.sql:658`). DDL: new migration +`server/postgres/migrations/000019_create_delphi_storage.sql` + additions to +`create_dynamodb_tables.py`. + +**Disposition of all 18 existing tables** maps 1:1 into these entities. Notable: +`Delphi_TopicAgendaSelections` (Dynamo) is vestigial → drop after verifying zero readers; +phantom `Delphi_CommentClusters` reads redirect to `umap#assignments`; +`report_narrative_store` → `narrative#` artifacts under `SERVER_NARRATIVE` runs with +one-time backfill as `IMPORTED` runs. + +### 4.3 Storage abstraction (both languages, one conformance spec) + +- Python package **`delphi/delphi_storage/`**: `interface.py` (protocol), `keys.py`, + `models.py` (Pydantic), `codec.py` (canonical JSON, zstd, packed floats, chunking, + Decimal handling), `backends/dynamodb.py`, `backends/postgres.py`, `factory.py`, + `inputs.py` (snapshot capture), `manifest.py`, `llm_recorder.py`, `replay.py`, + `conformance/cases/*.json`. +- TS package **`server/src/storage/delphi/`**: `interface.ts`, `keys.ts`, `codec.ts`, + `dynamoStore.ts`, `postgresStore.ts`, `factory.ts`. +- **Shared conformance spec**: JSON operation-scripts in `delphi_storage/conformance/cases/` + executed by BOTH pytest (parametrized over backends) and jest — round-trips (unicode, + floats, chunk-spanning payloads), prefix queries + ordering, latest-pointer monotonic + seq, queue claim incl. two-claimants-one-wins race, idempotent completion. This keeps + the two implementations honest. +- Operations: generic `get/put/put_batch/query(prefix|between)/delete_partition` + + semantic queue ops `enqueue_run / claim_next_run / extend_lease / update_run_status / + complete_run / append_log / get_latest / list_runs`. Claim: Dynamo = conditional update + on `version` (lift of working `job_poller.py:490-537` logic); PG = `UPDATE ... WHERE + job_id = (SELECT ... FOR UPDATE SKIP LOCKED) RETURNING *`. Priority finally + participates in claim ordering (both backends). +- Config: `DELPHI_STORAGE_BACKEND=dynamodb|postgres` (+ `DELPHI_STORAGE_TABLE_PREFIX`, + `DELPHI_STORAGE_PG_SCHEMA`, `DELPHI_STORAGE_PG_URL`). Migration flags (§6): + `DELPHI_WRITE_MODE=old|both|v2` (all writers, pipeline AND server-side; `both` for the + whole M1–M5 window; an explicit tri-state rather than a boolean so its meaning never + inverts mid-migration, and **fail-loud if unset** while old tables still exist — + a silently-defaulted writer would break §6.2 invariant 2 undetected), + `DELPHI_READ_V2=|all|none` (serving flip + canary), + `DELPHI_ENQUEUE_V2` (single queue-master flag), + `DELPHI_SHADOW_READS=` (divergence logging, both directions). +- Consistency, honestly: never require cross-item atomicity — write order is + inputs (once, at job start) → artifacts (as stages complete) → manifest COMPLETED → + latest flip. Dynamo uses ConsistentRead for + runs/latest; claim conditions on the base table (as today). Float fidelity solved at + codec level so fingerprints/diffs are backend-independent. +- Consolidation: the 3-4 duplicate PostgresClients collapse into `inputs.py` (source PG + read ONCE per run — architectural fix: today stages 1-3 each re-read live PG and can + see different data within one run); ~12 direct-boto3 sites and ~10 ad-hoc server Dynamo + clients funnel through the factories. + +### 4.4 job_id threading + snapshots + +- Explicit `--job-id` args: poller passes on the command line; `run_delphi.py` gains + `--job-id` (auto `local-` for dev) and threads to all 8 stage entry points; env + fallback kept one transition phase, then removed. math_tick keeps feeding legacy tables + during dual-write only. +- **Votes snapshot = full compressed copy of the RAW stream** (all rows, + `ORDER BY created`, order preserved — exactly what stage 1 reads; preserves the + vote-encounter order the math KMeans init deliberately depends on). + `votes_latest_unique` derived deterministically in-pipeline (tested against the PG + view). ~0.3-0.7MB zstd per 100k votes. Full copy chosen over cutoff-pointer + reconstruction because upstream mutation (GDPR deletions, `comments.mod` changes) + silently breaks pointer-based replay; hashes still recorded as fingerprints. +- EVōC gets seeded (`run_pipeline.py:184`, currently no seed); if numba parallelism keeps + residual nondeterminism, record that in the manifest rather than pretend. Seeding + changes UMAP-side outputs (documented); math goldens unaffected. + +### 4.5 Unified narrative store + +One narrative entity: `artifacts` rows of kind `narrative#
#`. Two producers: + +- **Python batch pipeline** (801/803): sections become artifacts of the batch job; + Anthropic Batch request bodies and results recorded as `llm#narrative#`. +- **Server `/reportNarrative` generator**: on generation it creates a lightweight run + (`job_type=SERVER_NARRATIVE`) with a slim manifest (model, params, prompts+responses as + `llm#` artifacts, input snapshots of exactly what it read) and writes sections as + `narrative#` artifacts, then flips `latest` for `rid##SERVER_NARRATIVE`. Its + current cache lookup becomes: `latest(...)` → get `narrative#
#`. + `DynamoStorageService` is superseded and deleted at decommission. + +## 5. Replay tooling + +New CLI `delphi/scripts/delphi_replay.py` backed by `delphi_storage/replay.py`: + +- `replay show ` — manifest, fingerprints, artifact inventory. +- `replay run [--stages ...]` — materializes the stored snapshot, runs the + pipeline with `--input-source=store://` (the same seam the snapshot-first phase + introduces), writes results under a fresh job_id with `replay_of` set. No live PG, no + reset, no old-table involvement. +- `replay diff ` — artifact-by-artifact comparison reusing the tolerance machinery + from `polismath/regression/comparer.py` (extract the numeric-dict-diff core into + `polismath/regression/artifact_diff.py` so the golden harness and replay share it). + Deterministic artifacts diff numerically; LLM-derived artifacts diff structurally + (presence, keys, models, token counts) per the input-level-reproducibility decision. +- CI-friendly exit codes. + +## 6. Zero-downtime migration (expand → backfill → verify → flip → contract) + +**Constraint (Julien, 2026-07-06):** this is a production service with many active +conversations. No big-bang migration, no downtime. New data is recorded in BOTH formats +from the start; old data is progressively imported into the new format; **serving uses +only the old format until the import has fully caught up**; then a single reversible +switch moves serving to the new format (while still recording to the old format, just in +case); flip back instantly if anything looks wrong; decommission the old format only +after a trust period. + +**Feasibility: yes.** This is the standard expand/backfill/verify/flip/contract +migration pattern, and §4's design already assumed dual-write and flag-gated readers. +The additions are: a backfill importer (one new component), catch-up/parity verification +tooling, and three invariants (§6.2) that make the flip-back guarantee real. Cost: +roughly 4–5 extra PRs plus a longer dual-write window (double writes and double storage +for the transition period — acceptable at Delphi's scale). + +### 6.1 Phases + +- **M0 — Expand.** New schema exists (Dynamo tables + PG migration), nothing writes. + Zero behavior change. +- **M1 — Dual-write new data.** `DELPHI_WRITE_MODE=both`: every pipeline run writes old + tables exactly as today (math outputs unchanged per the golden suite; row-level + old-table parity proven by `scripts/verify_dual_write.py`) AND v2 + runs/inputs/artifacts/latest. Server-side writers (topicMod, collectiveStatement, + reportNarrative) dual-write their entities too. Serving: 100% old format. + **Reproducibility guarantees begin here** — from M1 on, every run has a manifest and + input snapshot. +- **M2 — Progressive backfill.** `scripts/backfill_v2.py` walks all conversations (and + report narrative histories) with old-format data and synthesizes **`IMPORTED` runs** + (§6.3). Rate-limited, idempotent, resumable, safe to run continuously alongside M1. + Serving: still 100% old format. +- **M3 — Catch-up + verify.** Coverage check: every zid/rid that has old-format data has + a v2 `latest` pointer (from a dual-written run or an IMPORTED run). Then **shadow + reads**: serving endpoints keep answering from the old format but (sampled) also read + v2 and log any divergence (`verify_dual_write.py` for writes; shadow-read middleware + for reads). Flip is gated on: coverage = 100%, shadow divergence = 0 over an agreed + observation window. +- **M4 — Flip.** `DELPHI_READ_V2=all` (or group-by-group for a canary day: + visualizations first, `nextComment` last). Serving: 100% new format. **Old-format + writes CONTINUE** (both pipeline and server-side user state) — that is the rollback + insurance. Rollback = set `DELPHI_READ_V2=none`; instant, config-only, loses nothing + because the old format never stopped being complete (§6.2, invariant 2). +- **M5 — Trust period.** Weeks on v2 serving with old writes still on. Monitoring: + shadow-compare now runs in the OTHER direction (serve new, sample-compare old) to + confirm the old path would still be a safe landing zone. +- **M6 — Contract.** Stop old writes (`DELPHI_WRITE_MODE=v2`), retention window, then + delete old writers (`DynamoDBClient`, + `DynamoDBStorage`, `DynamoStorageService`), the old read paths, `reset_conversation.py` + old-table logic, math_tick generation, the `DELPHI_JOB_ID` env fallback; drop the 18 + old tables + `report_narrative_store`. + +### 6.2 Invariants that make flip-back safe (the load-bearing details) + +1. **Importer never clobbers real runs.** The backfill only sets a `latest` pointer via + a conditional write (set-if-absent, or only over another IMPORTED run with a lower + `seq`) — atomic in one operation because the `latest` item carries the run's + `job_type` (§4.2); a read-then-write check would reintroduce the race this invariant + exists to close. If a conversation got a dual-written (real) v2 run since M1, the importer + skips it — real provenance always beats imported. +2. **Every writer is bidirectional for the whole M1–M5 window.** Not just the pipeline: + topic moderation, collective statements, and the server narrative generator must + write old AND new on every mutation, in both serving modes. If v2-only writes were + allowed after the flip, flipping back would silently lose user actions taken while on + v2. This invariant is what makes M4's rollback lossless, and it is tested (a + write-through-both assertion in the route tests for every mutating endpoint). +3. **The queue never has two masters.** The poller claims from BOTH queues during the + whole transition (old `Delphi_JobQueue` first, then v2 `runs`); the ENQUEUE target is + a single flag flipped with M4 (and back on rollback). One job exists in exactly one + queue, both queues drain naturally, no drain-and-wait step, no double execution. + +### 6.3 `IMPORTED` runs (backfill semantics) + +The old format holds only the LATEST state per conversation (overwritten per run), so +the importer synthesizes **one v2 run per conversation** capturing current old-format +state: `job_type=IMPORTED`, `provenance=legacy`, artifacts mapped 1:1 from the old +tables (math tables at `latest_math_tick`, UMAP tables, extremity, priorities), plus +per-narrative-run IMPORTED runs reconstructed from `Delphi_NarrativeReports` / +`report_narrative_store` history (those two are append-keyed, so historical narrative +runs ARE recoverable). IMPORTED runs have **partial manifests**: no input snapshots, no +seeds, no config_effective — they are servable but **not replayable**, and are marked so +(`replayable=false`). This is not a design compromise to fix later: the old format never +recorded its inputs, so pre-M1 history is unrecoverable in principle (independently +confirmed in `REPLAY_HARNESS_DESIGN.md` §10). The importer is a scan-cursor checkpointed +job (resumable), rate-limited, and re-runnable at any time — re-running refreshes +IMPORTED runs for conversations whose old-format state changed and which have no real v2 +run yet. + +### 6.4 Fresh deployments skip the migration + +M0–M6 exist for the running production install. A fresh deployment (or a dev +environment) has no old data: it starts directly on v2 +(`DELPHI_READ_V2=all`, `DELPHI_ENQUEUE_V2=true`, `DELPHI_WRITE_MODE=v2`) with either +backend — including PG-only — as soon as Stacks 1–3 code exists. + +### 6.5 What this changes vs. a naive gradual reader switch + +The per-endpoint-group `DELPHI_READ_V2` flags remain, but their role changes: they are a +**canary mechanism at flip time**, not a months-long progressive migration. Readers stay +on the old format wholesale until M3's gate passes, because serving a half-backfilled v2 +would show older data than the old format for not-yet-imported conversations — the +user-visible regression this strategy exists to prevent. + +## 7. Phased implementation (PR-sized, TDD, spr-stacked) + +~22-28 PRs total — plan as **4 milestone stacks**: + +**Stack 1 — Foundation** (P1-P4): +- P1: this design doc + conformance case schema + first cases. +- P2: Python `delphi_storage/` interface+codec+**both backends** + pytest conformance + (dynamo-needing tests follow the standard skip convention; PG tests use dockerized test + PG). +- P3 (∥ P2): TS `server/src/storage/delphi/` both backends + jest conformance reading the + SAME case files. +- P4: DDL (Dynamo table additions + PG migration 000019); conformance green against real + stores. + +**Stack 2 — Provenance plumbing** (P5-P8): +- P5 (∥ anything): `--job-id` explicit threading through all 8 entry points. +- P6: input snapshots — 6a capture-only at job start; 6b stages read from snapshot + (`--input-source` seam). **Golden suite must pass unchanged for both** (riskiest step, + deliberately isolated; vote-order + votes_latest_unique-derivation tests are the RED + phase). +- P7: manifest + dual-write per stage group (math; umap 500s; 501/502+700s; 801/803), + each PR with `verify_dual_write` parity test. +- P8: LLM recorder (Ollama + Anthropic), EVōC seeding, `config_effective` registration. + +**Stack 3 — Backfill + consumers** (P9-P11e): +- P9: **backfill importer** `scripts/backfill_v2.py` (IMPORTED runs per §6.3, no-clobber + invariant, checkpointed cursor, rate limit) + coverage report + (`scripts/backfill_coverage.py`). RED phase: importer-idempotence and + no-clobber-vs-real-run tests. +- P10 (∥ P9): server v2 READ code behind `DELPHI_READ_V2` (default `none`), one endpoint + group per PR — lands dark, exercised by tests and shadow reads only until M4. +- P11: server-owned writes become **bidirectional** (topicMod, collectiveStatement, + reportNarrative dual-write old + new; write-through-both assertions per §6.2 + invariant 2); dual-queue poller + single enqueue-target flag (§6.2 invariant 3); + shadow-read sampling middleware + divergence logging. + +**Stack 4 — Flip, replay, contract** (P12-P14): +- P12: replay CLI + `artifact_diff.py` extraction + replay-of-recorded-fixture e2e test + (∥ Stack 3 — depends only on Stack 2). +- P13: M3/M4 operational gate — coverage=100% + shadow-divergence=0 dashboards, then the + flip (`DELPHI_READ_V2=all`), canary order visualizations-first/`nextComment`-last, + reverse-direction shadow compare during M5. +- P14: contract (M6) — stop old writes, retention window, delete old + writers/read-paths/tables/reset/math_tick/env-fallback. + +Critical path: P1 → P2/P3 → P4 → P6 → P7 → P9/P10/P11 → P13 → P14. The flip itself (M4) is +an ops action gated on M3 evidence, not a code change. + +**Hard constraints honored throughout:** output-invariance on `polismath/` math results +(golden suite runs in every phase touching input plumbing; NO numerical changes); `uv` +for Python; standard pytest `--ignore` set locally; propose-then-wait applies to anything +touching math-core logic (the redesign deliberately avoids it — `conversation.py` changes +are surfacing-only). + +## 8. Relationship to REPLAY_HARNESS_DESIGN.md (H) + +Two different meanings of "replay", deliberately kept separate: + +- **H** replays a conversation's *vote history* through both math engines at chosen + recompute schedules — a research harness for Clojure-parity science (gap measurement, + R1 certification, R2 schedule inference). Its §10 documents that historic math states + are unrecoverable today (`math_main` latest-only) — the very gap Storage V2 closes + going forward. +- **Storage V2** replays a *recorded production job* from its input snapshot — an + operations capability. + +Convergences to exploit: both reuse `ConversationComparer` tolerance machinery (the +`artifact_diff.py` extraction in §5 serves both); H's per-step recording could later +write into `run_inputs`/`artifacts` under synthetic job_ids; and once V2 manifests exist, +R2-style schedule inference becomes unnecessary for post-V2 data (the schedule is +recorded, not latent). + +## 9. Open considerations + +- **Data retention/GDPR**: input snapshots copy votes/comments outside the source PG. + Needs a purge path (`delete_partition` per job + a per-zid purge tool replacing today's + `reset_conversation.py` role) and a stated retention policy for old runs. Include purge + tool in P6. +- Residual EVōC/numba nondeterminism possible even seeded — manifest records determinism + status. +- `viz#` artifacts reference S3 objects; S3 lifecycle must match run retention. + +## 10. Verification + +- **Conformance suite** (pytest + jest, same JSON cases) green on both backends — the + contract. +- **Golden-snapshot invariance**: standard test suite + `scripts/regression_comparer.py` + — unchanged math outputs at every phase; any diff = regression. +- **Dual-write parity**: `scripts/verify_dual_write.py` compares old-table rows vs v2 + artifacts per job on real runs. +- **Migration gates (§6)**: backfill coverage = 100% of conversations with old-format + data; shadow-read divergence = 0 over the observation window before M4; bidirectional + write-through assertions on every mutating endpoint; a rehearsed flip-back + (`DELPHI_READ_V2=all` → `none` → `all`) on staging with writes flowing, proving zero + loss. +- **End-to-end replay proof** (the point of it all): run a job on a dev conversation → + mutate the source data (add votes, moderate a comment) → `replay run ` → + `replay diff` shows numeric-identical math/umap artifacts vs the original run. +- **PG-only smoke test**: `DELPHI_STORAGE_BACKEND=postgres DELPHI_READ_V2=all` full + pipeline + server reads with the DynamoDB container stopped. +- Server: jest integration tests per switched endpoint group against seeded v2 fixtures; + client-report manual check of `/delphi/reports` `available_runs`/`current_job_id`. diff --git a/delphi/docs/TOPIC_AGENDA_STORAGE_DESIGN.md b/delphi/docs/TOPIC_AGENDA_STORAGE_DESIGN.md index 920a19f1a..ae29dfc2e 100644 --- a/delphi/docs/TOPIC_AGENDA_STORAGE_DESIGN.md +++ b/delphi/docs/TOPIC_AGENDA_STORAGE_DESIGN.md @@ -151,12 +151,14 @@ Same structure as POST, but replaces existing selections entirely. ### Phase 2: Frontend Integration -1. Update `TopicAgenda.jsx` to call save API on "Done" click +1. Update `TopicAgenda.tsx` to call save API on "Done" click 2. Add loading states and error handling 3. Implement retrieval on component mount 4. Add confirmation UI for overwrites -### Phase 3: Cross-Run Persistence +### Phase 3: Cross-Run Persistence (designed, never implemented) + +The current system stores selections by comment ID and does not handle cluster drift across Delphi re-runs. 1. Implement comment matching algorithm for new Delphi runs 2. Create migration logic for when clusters change diff --git a/delphi/docs/VERSIONED_TOPIC_KEYS_IMPLEMENTATION.md b/delphi/docs/VERSIONED_TOPIC_KEYS_IMPLEMENTATION.md index 19a63dd5e..eeff8f244 100644 --- a/delphi/docs/VERSIONED_TOPIC_KEYS_IMPLEMENTATION.md +++ b/delphi/docs/VERSIONED_TOPIC_KEYS_IMPLEMENTATION.md @@ -84,6 +84,15 @@ const globalSections = [ **Current State**: TopicReport uses dynamic construction **Target State**: Shared utility function for key construction +> **Correction**: The actual key format used in `801_narrative_report_batch.py` is +> `{report_id}#{section}#{model}` with `#` as the delimiter (not underscore). +> Example: `9c867bbb-1616-44e3-947c-1406bc56e4d2#0#42`. +> The `constructSectionKey` example below uses underscore delimiters and does NOT +> reflect the current production format. +> +> Note: `CommentsReport.jsx` updates and `sectionKeyUtils.js` creation (marked 🚧 below) +> were never implemented. + **Implementation**: ```javascript // Shared utility function @@ -98,9 +107,9 @@ const constructSectionKey = (sectionName, jobUuid = null) => { ``` **Files to Modify**: -- `/client-report/src/util/sectionKeyUtils.js` (new file) +- `/client-report/src/util/sectionKeyUtils.js` (new file) 🚧 never implemented - `/client-report/src/components/topicReport/TopicReport.jsx` -- `/client-report/src/components/commentsReport/CommentsReport.jsx` +- `/client-report/src/components/commentsReport/CommentsReport.jsx` 🚧 never implemented ## Testing Requirements diff --git a/delphi/docs/ZID_EXPOSURE_AUDIT.md b/delphi/docs/ZID_EXPOSURE_AUDIT.md index 346cbb1b3..aab8c3cf8 100644 --- a/delphi/docs/ZID_EXPOSURE_AUDIT.md +++ b/delphi/docs/ZID_EXPOSURE_AUDIT.md @@ -1,5 +1,7 @@ # ZID Exposure Audit - Delphi Routes +> **Status (2026-06-11):** Still open — `conversation_id` (zid) is still exposed in delphi API responses (e.g., `server/src/routes/delphi.ts` response assembly). The remediation steps below were never executed. + ## 🚨 **CRITICAL WARNING - FIELD NAME AMBIGUITY** **The term "conversation_id" is DANGEROUSLY AMBIGUOUS and could mean:** @@ -292,6 +294,6 @@ href={`${urlPrefix + conversation.conversation_id}`} --- -**Document Created**: $(date) -**Last Updated**: $(date) +**Document Created**: 2025-06-07 +**Last Updated**: 2025-06-07 **Status**: 🔴 Active remediation required \ No newline at end of file diff --git a/delphi/docs/702_CONSENSUS_DIVISIVE_README.md b/delphi/docs/archive/702_CONSENSUS_DIVISIVE_README.md similarity index 100% rename from delphi/docs/702_CONSENSUS_DIVISIVE_README.md rename to delphi/docs/archive/702_CONSENSUS_DIVISIVE_README.md diff --git a/delphi/docs/ANTHROPIC_BATCH_API_GUIDE.md b/delphi/docs/archive/ANTHROPIC_BATCH_API_GUIDE.md similarity index 100% rename from delphi/docs/ANTHROPIC_BATCH_API_GUIDE.md rename to delphi/docs/archive/ANTHROPIC_BATCH_API_GUIDE.md diff --git a/delphi/docs/BATCH_API_BUGFIX.md b/delphi/docs/archive/BATCH_API_BUGFIX.md similarity index 100% rename from delphi/docs/BATCH_API_BUGFIX.md rename to delphi/docs/archive/BATCH_API_BUGFIX.md diff --git a/delphi/docs/BATCH_NARRATIVE_README.md b/delphi/docs/archive/BATCH_NARRATIVE_README.md similarity index 100% rename from delphi/docs/BATCH_NARRATIVE_README.md rename to delphi/docs/archive/BATCH_NARRATIVE_README.md diff --git a/delphi/docs/archive/CLAUDE.md b/delphi/docs/archive/CLAUDE.md new file mode 100644 index 000000000..f538ed6c6 --- /dev/null +++ b/delphi/docs/archive/CLAUDE.md @@ -0,0 +1,58 @@ +# Archived Delphi documentation — read this first + +Nothing in this folder describes the current system. These documents are +**historical raw material** from the initial build-out of Delphi (2025, +largely written with LLM assistance in the Claude Sonnet 3.5/3.7 era). They +were moved here on 2026-06-11 (PR #2573) after an audit verified that each +one no longer matches the code. + +**If you are an AI agent working on this codebase: do not use these files as +documentation.** Do not follow their instructions; do not trust their table +names, script names, formulas, file paths, or architecture descriptions. +Current documentation lives one level up in `delphi/docs/` (start with +`DOCUMENTATION_DIRECTORY.md`); the canonical references are the code itself, +`docs/PLAN_DISCREPANCY_FIXES.md`, and `docs/CLJ-PARITY-FIXES-JOURNAL.md`. + +**Why these files are kept:** they capture the *original research and design +intent* behind the system — goals, abandoned directions, and the reasoning of +the first build — which is an independent deliverable in its own right. +Extracting intent, requirements, or design history from them is the +legitimate use of this folder. + +## Index — what each file was, and why it was archived + +| File | What it was | Why archived | +|------|-------------|--------------| +| `702_CONSENSUS_DIVISIVE_README.md` | Usage notes for the standalone 702 consensus/divisive visualization script | The 702 step is disabled in the pipeline (its invocation is commented out) | +| `algorithm_analysis.md` | Pre-port analysis of the Clojure algorithms and Python porting choices | Describes custom power-iteration PCA and hand-rolled k-means since replaced by sklearn (#2416) | +| `ANTHROPIC_BATCH_API_GUIDE.md` | Guide to the narrative batch-API flow | Documents the dead `802_process_batch_results.py` / `Delphi_BatchJobs` path; the live flow is 801/803 | +| `architecture_overview.md` | Overview of the *Clojure* math service internals | Superseded by `deep-analysis-for-julien/01-overview-and-architecture.md` | +| `BATCH_API_BUGFIX.md` | Session memo for a job_poller batch-routing bug | Fix applied long ago | +| `BATCH_NARRATIVE_README.md` | README for the 801/802/803 batch workflow | 802 is dead code; references a DynamoDB table that is never created | +| `conversion_plan.md` | Original Clojure→Python conversion plan with status ticks | Statuses are wrong: the poller/server it marks "Completed" were deleted (#2423); PCA is now sklearn | +| `DATABASE_NAMING_PROPOSAL.md` | Migration plan to the `Delphi_` table-name prefix | Migration completed; `create_dynamodb_tables.py` is the canonical reference | +| `DEAD_CODE_CLEANUP_REPORT.md` | Session report of the Jan-2026 dead-code cleanup | Work merged (#2423); the "archived to docs/archive/" it claims never happened at the time | +| `DISTRIBUTED_SYSTEM_ROADMAP.md` | Multi-phase distributed-system roadmap | Unimplemented aspirations; references scripts that don't exist | +| `DOCKER.md` | Eight-line Docker stub | Hardcoded to a developer's machine; superseded by `DELPHI_DOCKER.md` | +| `EVOC_LAYER_HIERARCHY_DEBUG.md` | Debug log of the EVoC layer-hierarchy direction issue | Root cause identified; session closed | +| `GLOBAL_SECTION_TEMPLATE_MAPPING_FIX.md` | Fix memo for narrative global-section template mapping | Fix applied in `801_narrative_report_batch.py` | +| `JOB_ID_MIGRATION_PLAN.md` | Plan to re-key DynamoDB tables on `job_id` | Never implemented; tables remain keyed by zid/conversation_id | +| `JOB_SYSTEM_DESIGN.md` | DAG-based job-stage dependency design | Superseded by the simpler FULL_PIPELINE / narrative-batch job types that shipped | +| `NARRATIVE_DROPDOWN_DESIGN_ANALYSIS.md` | Options analysis for unifying two report dropdown components | Decision deferred and abandoned; only the immediate sorting fix shipped | +| `NARRATIVE_INVERSION_INVESTIGATION.md` | Investigation of the agree/disagree sign inversion in narratives | Fix applied (`postgres_vote_to_delphi`, #2330) | +| `NEXT_STEPS.md` | Next-steps list from the early port | Every item refers to the pre-#2423/#2416/#2282 architecture | +| `project_structure.md` | Proposed package layout | The layout described was never what got built | +| `SIMPLIFIED_TESTS.md` | Guide to the root-level simplified test scripts | Those scripts were deleted (#2126) | +| `SMART_COMMENT_FILTERING_PLAN.md` | Multi-week comment-filtering implementation plan | Never executed; superseded by the simpler 501 extremity step | +| `SPATIAL_TOPIC_PRIORITIZATION_SYSTEM.md` | Spatial topic-prioritization (STPS) design | The endpoints and DynamoDB tables it specifies were never built | +| `summary.md` | Early system summary | Describes a FastAPI server / background-poller architecture that no longer exists | +| `TEST_RESULTS_SUMMARY.md` | Point-in-time test-pass snapshot (2025-06) | Counts and referenced files long stale | +| `TESTING_LOG.md` | Early testing session log | Predates the Clojure-parity campaign entirely | +| `TOPIC_AGENDA_IMPLEMENTATION_SUMMARY.md` | Pre-implementation topic-agenda design memo | Feature shipped (migration 000012, `topicAgenda.ts`, `TopicAgenda.tsx`) | +| `TOPIC_AGENDA_MIGRATION_PLAN.md` | 16-week GraphQL/WASM/CDN topic-agenda migration plan | None of it was adopted; the shipped system is REST + Postgres JSONB + Astro | +| `TOPIC_GROUP_CONSENSUS_METRIC.md` | IGAS topic-consensus metric design (cosine variant) | Never adopted; production uses the group-aware consensus product | +| `TOPIC_GROUP_CONSENSUS_METRIC_REVISED.md` | Revised IGAS design (JSD, bootstrap CIs, calibration) | Never adopted | +| `TOPIC_GROUP_CONSENSUS_o3_stub.MD` | Raw o3-model output behind the REVISED doc | Raw LLM stub; the ~700-line TypeScript it contains was never committed | +| `UMAP_VISUALIZATION_PLAN.md` | Plan for a D3 UMAP scatter card in the topic hierarchy UI | The visualization was never built | +| `usage_examples.md` | API usage examples for `ConversationManager` + a FastAPI wrapper | Several referenced methods don't exist; not a production code path | +| `vulture_analysis_output.txt` | Raw vulture dead-code scan output (March 2026) | Acted on by #2423 and the repness dead-path removal; line numbers stale | diff --git a/delphi/docs/DATABASE_NAMING_PROPOSAL.md b/delphi/docs/archive/DATABASE_NAMING_PROPOSAL.md similarity index 100% rename from delphi/docs/DATABASE_NAMING_PROPOSAL.md rename to delphi/docs/archive/DATABASE_NAMING_PROPOSAL.md diff --git a/delphi/docs/DEAD_CODE_CLEANUP_REPORT.md b/delphi/docs/archive/DEAD_CODE_CLEANUP_REPORT.md similarity index 100% rename from delphi/docs/DEAD_CODE_CLEANUP_REPORT.md rename to delphi/docs/archive/DEAD_CODE_CLEANUP_REPORT.md diff --git a/delphi/docs/DISTRIBUTED_SYSTEM_ROADMAP.md b/delphi/docs/archive/DISTRIBUTED_SYSTEM_ROADMAP.md similarity index 100% rename from delphi/docs/DISTRIBUTED_SYSTEM_ROADMAP.md rename to delphi/docs/archive/DISTRIBUTED_SYSTEM_ROADMAP.md diff --git a/delphi/docs/DOCKER.md b/delphi/docs/archive/DOCKER.md similarity index 100% rename from delphi/docs/DOCKER.md rename to delphi/docs/archive/DOCKER.md diff --git a/delphi/docs/EVOC_LAYER_HIERARCHY_DEBUG.md b/delphi/docs/archive/EVOC_LAYER_HIERARCHY_DEBUG.md similarity index 100% rename from delphi/docs/EVOC_LAYER_HIERARCHY_DEBUG.md rename to delphi/docs/archive/EVOC_LAYER_HIERARCHY_DEBUG.md diff --git a/delphi/docs/GLOBAL_SECTION_TEMPLATE_MAPPING_FIX.md b/delphi/docs/archive/GLOBAL_SECTION_TEMPLATE_MAPPING_FIX.md similarity index 100% rename from delphi/docs/GLOBAL_SECTION_TEMPLATE_MAPPING_FIX.md rename to delphi/docs/archive/GLOBAL_SECTION_TEMPLATE_MAPPING_FIX.md diff --git a/delphi/docs/JOB_ID_MIGRATION_PLAN.md b/delphi/docs/archive/JOB_ID_MIGRATION_PLAN.md similarity index 100% rename from delphi/docs/JOB_ID_MIGRATION_PLAN.md rename to delphi/docs/archive/JOB_ID_MIGRATION_PLAN.md diff --git a/delphi/docs/JOB_SYSTEM_DESIGN.md b/delphi/docs/archive/JOB_SYSTEM_DESIGN.md similarity index 100% rename from delphi/docs/JOB_SYSTEM_DESIGN.md rename to delphi/docs/archive/JOB_SYSTEM_DESIGN.md diff --git a/delphi/docs/NARRATIVE_DROPDOWN_DESIGN_ANALYSIS.md b/delphi/docs/archive/NARRATIVE_DROPDOWN_DESIGN_ANALYSIS.md similarity index 100% rename from delphi/docs/NARRATIVE_DROPDOWN_DESIGN_ANALYSIS.md rename to delphi/docs/archive/NARRATIVE_DROPDOWN_DESIGN_ANALYSIS.md diff --git a/delphi/docs/NARRATIVE_INVERSION_INVESTIGATION.md b/delphi/docs/archive/NARRATIVE_INVERSION_INVESTIGATION.md similarity index 100% rename from delphi/docs/NARRATIVE_INVERSION_INVESTIGATION.md rename to delphi/docs/archive/NARRATIVE_INVERSION_INVESTIGATION.md diff --git a/delphi/docs/NEXT_STEPS.md b/delphi/docs/archive/NEXT_STEPS.md similarity index 100% rename from delphi/docs/NEXT_STEPS.md rename to delphi/docs/archive/NEXT_STEPS.md diff --git a/delphi/docs/SIMPLIFIED_TESTS.md b/delphi/docs/archive/SIMPLIFIED_TESTS.md similarity index 100% rename from delphi/docs/SIMPLIFIED_TESTS.md rename to delphi/docs/archive/SIMPLIFIED_TESTS.md diff --git a/delphi/docs/SMART_COMMENT_FILTERING_PLAN.md b/delphi/docs/archive/SMART_COMMENT_FILTERING_PLAN.md similarity index 100% rename from delphi/docs/SMART_COMMENT_FILTERING_PLAN.md rename to delphi/docs/archive/SMART_COMMENT_FILTERING_PLAN.md diff --git a/delphi/docs/SPATIAL_TOPIC_PRIORITIZATION_SYSTEM.md b/delphi/docs/archive/SPATIAL_TOPIC_PRIORITIZATION_SYSTEM.md similarity index 100% rename from delphi/docs/SPATIAL_TOPIC_PRIORITIZATION_SYSTEM.md rename to delphi/docs/archive/SPATIAL_TOPIC_PRIORITIZATION_SYSTEM.md diff --git a/delphi/docs/TESTING_LOG.md b/delphi/docs/archive/TESTING_LOG.md similarity index 100% rename from delphi/docs/TESTING_LOG.md rename to delphi/docs/archive/TESTING_LOG.md diff --git a/delphi/docs/TEST_RESULTS_SUMMARY.md b/delphi/docs/archive/TEST_RESULTS_SUMMARY.md similarity index 100% rename from delphi/docs/TEST_RESULTS_SUMMARY.md rename to delphi/docs/archive/TEST_RESULTS_SUMMARY.md diff --git a/delphi/docs/TOPIC_AGENDA_IMPLEMENTATION_SUMMARY.md b/delphi/docs/archive/TOPIC_AGENDA_IMPLEMENTATION_SUMMARY.md similarity index 100% rename from delphi/docs/TOPIC_AGENDA_IMPLEMENTATION_SUMMARY.md rename to delphi/docs/archive/TOPIC_AGENDA_IMPLEMENTATION_SUMMARY.md diff --git a/delphi/docs/TOPIC_AGENDA_MIGRATION_PLAN.md b/delphi/docs/archive/TOPIC_AGENDA_MIGRATION_PLAN.md similarity index 100% rename from delphi/docs/TOPIC_AGENDA_MIGRATION_PLAN.md rename to delphi/docs/archive/TOPIC_AGENDA_MIGRATION_PLAN.md diff --git a/delphi/docs/TOPIC_GROUP_CONSENSUS_METRIC.md b/delphi/docs/archive/TOPIC_GROUP_CONSENSUS_METRIC.md similarity index 100% rename from delphi/docs/TOPIC_GROUP_CONSENSUS_METRIC.md rename to delphi/docs/archive/TOPIC_GROUP_CONSENSUS_METRIC.md diff --git a/delphi/docs/TOPIC_GROUP_CONSENSUS_METRIC_REVISED.md b/delphi/docs/archive/TOPIC_GROUP_CONSENSUS_METRIC_REVISED.md similarity index 100% rename from delphi/docs/TOPIC_GROUP_CONSENSUS_METRIC_REVISED.md rename to delphi/docs/archive/TOPIC_GROUP_CONSENSUS_METRIC_REVISED.md diff --git a/delphi/docs/TOPIC_GROUP_CONSENSUS_o3_stub.MD b/delphi/docs/archive/TOPIC_GROUP_CONSENSUS_o3_stub.MD similarity index 100% rename from delphi/docs/TOPIC_GROUP_CONSENSUS_o3_stub.MD rename to delphi/docs/archive/TOPIC_GROUP_CONSENSUS_o3_stub.MD diff --git a/delphi/docs/UMAP_VISUALIZATION_PLAN.md b/delphi/docs/archive/UMAP_VISUALIZATION_PLAN.md similarity index 100% rename from delphi/docs/UMAP_VISUALIZATION_PLAN.md rename to delphi/docs/archive/UMAP_VISUALIZATION_PLAN.md diff --git a/delphi/docs/algorithm_analysis.md b/delphi/docs/archive/algorithm_analysis.md similarity index 100% rename from delphi/docs/algorithm_analysis.md rename to delphi/docs/archive/algorithm_analysis.md diff --git a/delphi/docs/architecture_overview.md b/delphi/docs/archive/architecture_overview.md similarity index 100% rename from delphi/docs/architecture_overview.md rename to delphi/docs/archive/architecture_overview.md diff --git a/delphi/docs/conversion_plan.md b/delphi/docs/archive/conversion_plan.md similarity index 100% rename from delphi/docs/conversion_plan.md rename to delphi/docs/archive/conversion_plan.md diff --git a/delphi/docs/project_structure.md b/delphi/docs/archive/project_structure.md similarity index 100% rename from delphi/docs/project_structure.md rename to delphi/docs/archive/project_structure.md diff --git a/delphi/docs/summary.md b/delphi/docs/archive/summary.md similarity index 100% rename from delphi/docs/summary.md rename to delphi/docs/archive/summary.md diff --git a/delphi/docs/usage_examples.md b/delphi/docs/archive/usage_examples.md similarity index 100% rename from delphi/docs/usage_examples.md rename to delphi/docs/archive/usage_examples.md diff --git a/delphi/docs/vulture_analysis_output.txt b/delphi/docs/archive/vulture_analysis_output.txt similarity index 100% rename from delphi/docs/vulture_analysis_output.txt rename to delphi/docs/archive/vulture_analysis_output.txt diff --git a/delphi/docs/deep-analysis-for-julien/01-overview-and-architecture.md b/delphi/docs/deep-analysis-for-julien/01-overview-and-architecture.md index 35da75cf7..42a997401 100644 --- a/delphi/docs/deep-analysis-for-julien/01-overview-and-architecture.md +++ b/delphi/docs/deep-analysis-for-julien/01-overview-and-architecture.md @@ -136,7 +136,7 @@ Clojure serializes its entire conversation state (including all computed fields) ### 3.2 Python (Delphi) -- **`poller.py`**: Polls Postgres for new votes/moderation/tasks on separate threads +- **`polismath/poller.py`**: DELETED in #2423 (commit 0ff8e3e52). The current entry point is `scripts/job_poller.py`, which polls the DynamoDB `Delphi_JobQueue` and dispatches the math/UMAP/narrative scripts. - **`run_math_pipeline.py`**: CLI tool for one-shot processing - **`manager.py`**: Thread-safe management of multiple `Conversation` objects - **`conversation.py:Conversation.update_votes()`**: Main entry, calls `recompute()` diff --git a/delphi/docs/deep-analysis-for-julien/07-discrepancies.md b/delphi/docs/deep-analysis-for-julien/07-discrepancies.md index 288e6d3ea..b07f607ce 100644 --- a/delphi/docs/deep-analysis-for-julien/07-discrepancies.md +++ b/delphi/docs/deep-analysis-for-julien/07-discrepancies.md @@ -1,5 +1,7 @@ # ALL Discrepancies: Clojure (CORRECT) vs Python (Delphi) +> **Status as of 2026-06-11**: D2, D4, D5, D6, D7, D8, D9 are merged on `edge`; D10, D11, D12 are in the open spr stack (PRs #2566–#2568). D3 (k-smoother) and D1/D1b (PCA sign flip / projection source) remain open. See `PLAN_DISCREPANCY_FIXES.md` (canonical) — the analysis below is kept as historical reference and per-discrepancy detail. + This is the critical reference document. Every discrepancy is rated by severity and lists the exact code locations. --- diff --git a/delphi/docs/topic-moderation-system.md b/delphi/docs/topic-moderation-system.md index 0ca8fe51f..251b2574a 100644 --- a/delphi/docs/topic-moderation-system.md +++ b/delphi/docs/topic-moderation-system.md @@ -141,8 +141,9 @@ Before using TopicMod, ensure the Delphi pipeline has been run: # Generate embeddings and clusters python 500_generate_embedding_umap_cluster.py -# Generate topic names using LLM -python 600_generate_llm_topic_names.py +# Topic naming now runs inline in umap_narrative/run_pipeline.py (Ollama) +# as part of the full pipeline job — no separate script. +# (600_generate_llm_topic_names.py was deleted) # Create visualizations python 700_datamapplot_for_layer.py diff --git a/delphi/polismath/benchmarks/bench_pca.py b/delphi/polismath/benchmarks/bench_pca.py index 196a00872..85e8cd35c 100755 --- a/delphi/polismath/benchmarks/bench_pca.py +++ b/delphi/polismath/benchmarks/bench_pca.py @@ -6,16 +6,20 @@ cd delphi python -m polismath.benchmarks.bench_pca [--runs N] python -m polismath.benchmarks.bench_pca --profile + python -m polismath.benchmarks.bench_pca --compare-impls Example: python -m polismath.benchmarks.bench_pca real_data/.local/r7wehfsmutrwndviddnii-bg2050/2025-11-25-1909-r7wehfsmutrwndviddnii-votes.csv --runs 3 python -m polismath.benchmarks.bench_pca real_data/.local/r7wehfsmutrwndviddnii-bg2050/2025-11-25-1909-r7wehfsmutrwndviddnii-votes.csv --profile + python -m polismath.benchmarks.bench_pca real_data/r6vbnhffkxbd7ifmfbdrd-vw/2025-11-11-1704-r6vbnhffkxbd7ifmfbdrd-votes.csv --compare-impls """ +import os import time from pathlib import Path import click +import numpy as np from polismath.benchmarks.benchmark_utils import ( load_votes_from_csv, @@ -24,7 +28,11 @@ runs_option, ) from polismath.conversation import Conversation -from polismath.pca_kmeans_rep.pca import pca_project_dataframe +from polismath.pca_kmeans_rep.pca import ( + PCA_IMPL_CHOICES, + PCA_IMPL_ENV_VAR, + pca_project_dataframe, +) profile_option = click.option( @@ -33,6 +41,13 @@ help='Run with line profiler on PCA functions', ) +compare_impls_option = click.option( + '--compare-impls', '-c', + is_flag=True, + help='Cold-start comparison of PCA solvers (POLISMATH_PCA_IMPL values: ' + 'powerit = legacy/Clojure-parity, sklearn = improved)', +) + def setup_conversation(votes_csv: Path) -> tuple[Conversation, str, int, float]: """ @@ -128,6 +143,82 @@ def benchmark_pca(votes_csv: Path, runs: int = 3) -> dict: } +def benchmark_impl_comparison(votes_csv: Path, runs: int = 3) -> dict: + """ + Cold-start wall-time + component-angle comparison of the PCA solvers. + + Times pca_project_dataframe under each POLISMATH_PCA_IMPL value on the + same clean (NaN-imputed identically inside) matrix, then reports the + angle between the components the two solvers produce. + + Args: + votes_csv: Path to votes CSV file + runs: Number of runs to average per solver + + Returns: + Dictionary with per-solver timings and per-component angles. + """ + conv, dataset_name, n_votes, _ = setup_conversation(votes_csv) + clean_matrix = conv._get_clean_matrix() + + results: dict = {'dataset': dataset_name, 'n_votes': n_votes, + 'shape': clean_matrix.shape, 'impls': {}} + saved_env = os.environ.get(PCA_IMPL_ENV_VAR) + try: + for impl in PCA_IMPL_CHOICES: + os.environ[PCA_IMPL_ENV_VAR] = impl + print(f"Benchmarking {PCA_IMPL_ENV_VAR}={impl} ({runs} runs)...") + times = [] + pca_results = None + for i in range(runs): + start = time.perf_counter() + pca_results, _ = pca_project_dataframe(clean_matrix, 2) + elapsed = time.perf_counter() - start + times.append(elapsed) + print(f" Run {i+1}: {elapsed:.3f}s") + results['impls'][impl] = { + 'times': times, + 'avg': sum(times) / len(times), + 'min': min(times), + 'max': max(times), + 'comps': pca_results['comps'] if pca_results is not None else None, + } + finally: + # Belt-and-braces: restore whatever the caller had set. + if saved_env is None: + os.environ.pop(PCA_IMPL_ENV_VAR, None) + else: + os.environ[PCA_IMPL_ENV_VAR] = saved_env + + print() + print("=" * 50) + print(f"Dataset: {dataset_name}") + print(f"Votes: {n_votes:,}") + print(f"Matrix shape: {clean_matrix.shape}") + for impl, r in results['impls'].items(): + print(f"{impl:>8}: avg {r['avg']:.3f}s (min {r['min']:.3f}s / max {r['max']:.3f}s)") + + impl_names = list(results['impls'].keys()) + if len(impl_names) == 2: + comps_a = results['impls'][impl_names[0]]['comps'] + comps_b = results['impls'][impl_names[1]]['comps'] + if comps_a is not None and comps_b is not None and comps_a.shape == comps_b.shape: + angles = [] + for i in range(comps_a.shape[0]): + norm_a = np.linalg.norm(comps_a[i]) + norm_b = np.linalg.norm(comps_b[i]) + if norm_a == 0.0 or norm_b == 0.0: + angles.append(float('nan')) + continue + cos = abs(float(np.dot(comps_a[i], comps_b[i]))) / (norm_a * norm_b) + angles.append(float(np.degrees(np.arccos(np.clip(cos, -1.0, 1.0))))) + results['angles_deg'] = angles + for i, angle in enumerate(angles): + print(f"PC{i+1} angle {impl_names[0]} vs {impl_names[1]}: {angle:.3e}°") + + return results + + def profile_pca(votes_csv: Path) -> None: """ Run line profiler on PCA functions. @@ -162,10 +253,13 @@ def profile_pca(votes_csv: Path) -> None: @votes_csv_argument @runs_option @profile_option -def main(votes_csv: Path, runs: int, profile: bool): +@compare_impls_option +def main(votes_csv: Path, runs: int, profile: bool, compare_impls: bool): """Benchmark PCA computation performance.""" if profile: profile_pca(votes_csv) + elif compare_impls: + benchmark_impl_comparison(votes_csv, runs) else: benchmark_pca(votes_csv, runs) diff --git a/delphi/polismath/benchmarks/bench_repness.py b/delphi/polismath/benchmarks/bench_repness.py index 669435768..c0a5fe105 100755 --- a/delphi/polismath/benchmarks/bench_repness.py +++ b/delphi/polismath/benchmarks/bench_repness.py @@ -28,7 +28,6 @@ from polismath.conversation import Conversation from polismath.pca_kmeans_rep.repness import ( conv_repness, - comment_stats, compute_group_comment_stats_df, select_rep_comments_df, select_consensus_comments_df, diff --git a/delphi/polismath/conversation/conversation.py b/delphi/polismath/conversation/conversation.py index b6f2178a8..478229d3b 100644 --- a/delphi/polismath/conversation/conversation.py +++ b/delphi/polismath/conversation/conversation.py @@ -15,7 +15,11 @@ from datetime import datetime from natsort import natsorted -from polismath.pca_kmeans_rep.pca import pca_project_dataframe +from polismath.pca_kmeans_rep.pca import ( + pca_project_dataframe, + pca_project_cmnts, + compute_comment_extremity, +) from polismath.pca_kmeans_rep.clusters import ( kmeans_sklearn, calculate_silhouette_sklearn @@ -37,6 +41,98 @@ logger.setLevel(logging.INFO) +# ============================================================================= +# D12: Comment-priority metrics (Clojure parity) +# ============================================================================= +# +# Ports of `importance-metric` and `priority-metric` from Clojure +# (math/src/polismath/math/conversation.clj:311-330). Public so they can be +# unit-tested in isolation. + +META_PRIORITY = 7 # Clojure: meta-priority (conversation.clj:319). "TODO TUNE." + + +def importance_metric(A: float, P: float, S: float, E: float) -> float: + """ + Clojure importance-metric (conversation.clj:311-315). + + (defn importance-metric + [A P S E] + (let [p (/ (+ P 1) (+ S 2)) + a (/ (+ A 1) (+ S 2))] + (* (- 1 p) (+ E 1) a))) + + Smoothed (Beta(2,2)) probability of pass `p`, smoothed agree `a`, with + extremity boost `(E + 1)`. Higher when fewer passes, more agrees, more + extreme (higher PCA extremity). + + Args: + A: agree count (across all groups). + P: pass count = S - (A + D) across all groups. + S: seen count (total votes seen — agree + disagree + pass). + E: comment extremity (L2 norm of PCA projection). + """ + p = (P + 1) / (S + 2) + a = (A + 1) / (S + 2) + return (1 - p) * (E + 1) * a + + +def priority_metric(is_meta: bool, + A: float, P: float, S: float, E: float) -> float: + """ + Clojure priority-metric (conversation.clj:321-330). + + (defn priority-metric + [is-meta A P S E] + (matrix/pow + (if is-meta + meta-priority + (* (importance-metric A P S E) + (+ 1 (* 8 (matrix/pow 2 (/ S -5)))))) + 2)) + + Squared to deepen bias (toward extremes). Meta comments get a constant + `META_PRIORITY^2 = 49`. Non-meta comments get `importance * decay`, where + the decay factor `1 + 8 * 2^(-S/5)` lets new (low-S) comments bubble up + and fades as more votes accumulate. + + Args: + is_meta: True for meta comments (treated as constant priority). + A, P, S, E: see `importance_metric`. + + Returns: + Squared priority value. + + .. warning:: + **Current behavior (parity-bug mirror):** this function ALWAYS + returns ``META_PRIORITY ** 2`` and ignores ``is_meta`` and + ``A, P, S, E``. It deliberately mirrors a Clojure bug — Clojure + treats meta-tid value 0 as truthy, so every tid takes the meta + branch — for byte-for-byte parity. The branching formula described + above is the *intended* semantics, restored once + https://github.com/compdemocracy/polis/issues/2571 is fixed. See the + ``TODO(clojure-parity-bug)`` in the body below. + """ + # TODO(clojure-parity-bug): Clojure (conversation.clj:325) treats meta-tid + # value 0 as TRUTHY in (if is-meta ...), so every tid takes the meta branch. + # We mirror this bug for byte-for-byte Clojure parity. Switch back to + # honoring `is_meta` once the GitHub issue resolves: + # https://github.com/compdemocracy/polis/issues/2571 + # Original semantic-correct code preserved below for reference and future + # restoration. + # + # Clojure-parity-bug-mirror: ALWAYS take the meta branch, ignoring is_meta. + return META_PRIORITY ** 2 + + # Original semantically-correct logic, restore when Clojure bug is fixed: + # if is_meta: + # inner = META_PRIORITY + # else: + # decay_factor = 1 + 8 * (2 ** (-S / 5)) + # inner = importance_metric(A, P, S, E) * decay_factor + # return inner ** 2 + + class Conversation: """ Manages the state and computation for a Pol.is conversation. @@ -82,6 +178,7 @@ def __init__(self, self.participant_info = {} self.vote_stats = {} self.group_votes = {} # Initialize group_votes to avoid attribute errors + self.comment_priorities: Dict[Any, float] = {} # D12 (PR 11) # Initialize with votes if provided if votes: @@ -704,11 +801,16 @@ def _compute_clusters(self) -> None: 'members': member_base_cluster_ids }) - # Sort group clusters by size (number of base clusters) for consistency - group_clusters.sort(key=lambda c: len(c['members']), reverse=True) - # Reassign IDs based on sorted order - for i, cluster in enumerate(group_clusters): - cluster['id'] = i + # Keep group clusters in k-means ID order (matching Clojure's + # sort-by :id, conversation.clj:437). Do NOT sort by size or + # reassign IDs: Clojure assigns group ids by first-k-distinct + # encounter order over base-cluster centers (init-clusters, + # clusters.clj:55-64) and never re-orders by size. The former + # size-descending re-sort here was the root cause of the gid 0↔1 + # label swap vs Clojure blobs (S3-4 trace, 2026-06-11: identical + # memberships modulo label permutation on vw-cold_start). Mirrors + # the identical rule at the base-cluster level above. + group_clusters.sort(key=lambda c: c['id']) logger.info(f"Created {len(group_clusters)} group clusters") @@ -753,16 +855,22 @@ def _compute_repness(self) -> None: # Check if we have groups if not self.group_clusters: + # B1 fix (D11 sub-agent review): consensus_comments must always be + # `{'agree': [], 'disagree': []}` (dict) post-D11, never `[]` (list). self.repness = { 'comment_ids': list(self.rating_mat.columns), 'group_repness': {}, - 'consensus_comments': [] + 'consensus_comments': {'agree': [], 'disagree': []} } logger.info(f"Representativeness completed in {time.time() - start_time:.2f}s (no groups)") return - # Compute representativeness (needs participant IDs, not base-cluster IDs) - self.repness = conv_repness(self.rating_mat, self._unfolded_group_clusters()) + # Compute representativeness (needs participant IDs, not base-cluster IDs). + # `mod_out=self.mod_out_tids` forwards moderated-out tids to the rep + consensus + # selectors (Clojure parity per D11 / PR 9; matches repness.clj:222 and :296). + self.repness = conv_repness(self.rating_mat, + self._unfolded_group_clusters(), + mod_out=self.mod_out_tids) logger.info(f"Representativeness completed in {time.time() - start_time:.2f}s") def _compute_participant_info_optimized(self, vote_matrix: pd.DataFrame, group_clusters: List[Dict[str, Any]]) -> Dict[str, Any]: @@ -1001,11 +1109,99 @@ def recompute(self) -> 'Conversation': # Compute representativeness result._compute_repness() - + + # Compute comment priorities (D12 / PR 11). Needs PCA + group_votes. + result._compute_comment_priorities() + # Compute participant info result._compute_participant_info() - + return result + + def _compute_comment_priorities(self) -> Dict[Any, float]: + """ + Compute per-tid comment priorities matching Clojure + `:comment-priorities` (conversation.clj:648-679). + + Per-tid: sum A/D/S across all groups → P = S - (A + D) → call + `priority_metric(is_meta, A, P, S, E)` where E is the comment + extremity computed from PCA. + + Stores the result on `self.comment_priorities` and also returns it. + TS server `nextComment.ts::getNextPrioritizedComment` consumes this + for weighted comment routing — pre-D12 Python emitted nothing, so + the server fell back to uniform random selection. + """ + if self.pca is None or self.rating_mat is None or self.rating_mat.empty: + self.comment_priorities = {} + return self.comment_priorities + + center = np.asarray(self.pca.get('center')) + comps = np.asarray(self.pca.get('comps')) + if center.size == 0 or comps.size == 0: + self.comment_priorities = {} + return self.comment_priorities + + # Comment projection + extremity (Clojure with-proj-and-extremtiy, + # conversation.clj:341-352). + cmnt_proj = pca_project_cmnts(center, comps) + extremity_arr = compute_comment_extremity(cmnt_proj) + + # Fail closed on desync: if the PCA vectors were computed on a + # different column set than the current rating_mat (e.g. moderation + # changed between recomputes), zip() would silently truncate and + # assign E=0 to the overflow tids — wrong priorities with no + # signal. Empty priorities degrade the TS server to uniform + # routing, which is honest; silently wrong extremities are not. + # (Copilot review 2026-07-04, g4.) + n_cols = len(self.rating_mat.columns) + if len(extremity_arr) != n_cols: + logger.error( + f"comment_priorities: extremity length {len(extremity_arr)} " + f"!= rating_mat column count {n_cols} (stale PCA?); " + f"skipping priorities for this tick") + self.comment_priorities = {} + return self.comment_priorities + + # Column order of `center`/`comps`/`extremity_arr` matches + # `self.rating_mat.columns` (PCA is computed on rating_mat). + tid_extremity = dict(zip(self.rating_mat.columns, extremity_arr)) + + # Per-group A/D/S aggregation. `_compute_group_votes` returns + # {str(gid): {'n-members': N, 'votes': {tid: {A, D, S}}}}. S includes + # PASS (line ~1222: `np.sum(~np.isnan(votes))`), matching Clojure. + # PERF (deferred, Copilot on PR #2568): this is an O(groups × + # comments × members) scan on every recompute; vectorize or reuse + # the repness-stage aggregation — tracked in the follow-up issue + # "delphi: _compute_comment_priorities recomputes group votes on + # every tick". + group_votes = self._compute_group_votes() + + priorities: Dict[Any, float] = {} + for tid in self.rating_mat.columns: + A_total = 0 + D_total = 0 + S_total = 0 + for gv_data in group_votes.values(): + votes_for_tid = gv_data.get('votes', {}).get( + tid, {'A': 0, 'D': 0, 'S': 0}) + A_total += votes_for_tid.get('A', 0) + D_total += votes_for_tid.get('D', 0) + S_total += votes_for_tid.get('S', 0) + # Clojure: P = S - (A + D) (conversation.clj:661). + P_total = S_total - (A_total + D_total) + E = float(tid_extremity.get(tid, 0)) + is_meta = tid in self.meta_tids + # Match key type with the rest of the codebase (int when possible). + try: + tid_key = int(tid) + except (ValueError, TypeError): + tid_key = tid + priorities[tid_key] = float(priority_metric( + is_meta, A_total, P_total, S_total, E)) + + self.comment_priorities = priorities + return priorities def get_summary(self) -> Dict[str, Any]: """ @@ -1731,12 +1927,15 @@ def numpy_to_list(arr): # a list-of-dicts format that would break server/src/report.ts, # server/src/utils/pca.ts, and client-participation-alpha consumers. - # Add empty consensus structure for compatibility - result['consensus'] = { - 'agree': [], - 'disagree': [], - 'comment-stats': {} - } + # Surface D11 consensus comments (Clojure parity: client-report's Majority + # view consumes result['consensus']). Pre-Investigation-B this block was + # hardcoded empty, which silently zeroed the Majority view regardless of + # the D11 selection. Falls back to the empty shape when repness is missing + # or did not produce a consensus_comments dict (older blobs, no-group convs). + result['consensus'] = ( + self.repness.get('consensus_comments', {'agree': [], 'disagree': []}) + if self.repness else {'agree': [], 'disagree': []} + ) # Add math_tick value current_time = int(time.time()) @@ -2272,12 +2471,18 @@ def float_to_decimal(obj): } result['pca'] = float_to_decimal(pca_data) - # Add consensus structure - result['consensus'] = { - 'agree': [], - 'disagree': [], - 'comment_stats': {} - } + # Surface D11 consensus comments (Clojure parity). Pre-Investigation-B + # this block was hardcoded empty, so the DynamoDB blob never carried the + # D11 dict even when repness produced one. Falls back to the empty shape + # when repness is missing or didn't produce consensus_comments. + # float_to_decimal is REQUIRED: entries carry float p-success/p-test and + # writer Site 1 puts this dict straight into the Delphi_PCAResults Item — + # boto3 rejects raw floats (caught by CI's e2e run, 2026-07-05; the + # legacy writer branch converts, the pre-formatted branch did not). + result['consensus'] = float_to_decimal( + self.repness.get('consensus_comments', {'agree': [], 'disagree': []}) + if self.repness else {'agree': [], 'disagree': []} + ) # Add math_tick value current_time = int(time.time()) @@ -2289,10 +2494,18 @@ def float_to_decimal(obj): logger.info(f"[{time.time() - start_time:.2f}s] Processing comment priorities...") priorities = {} for cid, priority in self.comment_priorities.items(): + # Preserve the float VALUE as Decimal (boto3 rejects raw + # floats). The previous int() truncation was harmless while + # the D12.6 bug-mirror pins every priority to 49.0, but the + # real formula (restored when issue #2571 resolves) spans + # ~0.18–31.46 on real data: int() floors sub-1 priorities + # to 0, which the TS server's weighted routing treats as + # "no priority data" — those comments would never be routed. + value = float_to_decimal(float(priority)) try: - priorities[int(cid)] = int(priority) + priorities[int(cid)] = value except (ValueError, TypeError): - priorities[cid] = int(priority) + priorities[cid] = value result['comment_priorities'] = priorities # Process repness data efficiently diff --git a/delphi/polismath/database/dynamodb.py b/delphi/polismath/database/dynamodb.py index 3d74433de..5bde39d64 100644 --- a/delphi/polismath/database/dynamodb.py +++ b/delphi/polismath/database/dynamodb.py @@ -300,6 +300,23 @@ def write_conversation(self, conv) -> bool: if analysis_table: if dynamo_data: # Use pre-formatted data + # D11 cascade fix (Investigation B, Site 1), corrected + # 2026-07-04: `to_dynamo_dict()` surfaces consensus at + # TOP-LEVEL `result['consensus']` — its `repness` dict + # carries only `comment_repness`. The previous read of + # `repness.consensus_comments` matched a key that never + # exists, so the writer always stored the empty default + # (the round-trip test masked this by stubbing + # to_dynamo_dict with the wrong nested shape). + consensus_comments = dynamo_data.get( + 'consensus', {'agree': [], 'disagree': []} + ) + # Belt-and-braces: to_dynamo_dict already emits Decimals, + # but this Item write is the boto3 boundary — convert + # defensively like the legacy branch below does + # (idempotent on already-converted data). + consensus_comments = self._replace_floats_with_decimals( + self._numpy_to_list(consensus_comments)) analysis_table.put_item(Item={ 'zid': zid, 'math_tick': math_tick, @@ -308,7 +325,7 @@ def write_conversation(self, conv) -> bool: 'comment_count': dynamo_data.get('comment_count', 0), 'group_count': dynamo_data.get('group_count', 0), 'pca': dynamo_data.get('pca', {}), - 'consensus_comments': dynamo_data.get('consensus', {}).get('agree', []) + 'consensus_comments': consensus_comments }) else: # Legacy format @@ -321,11 +338,21 @@ def write_conversation(self, conv) -> bool: } # Replace floats with Decimal for DynamoDB pca_data = self._replace_floats_with_decimals(pca_data) - - # Create the analysis record with Decimal conversion - consensus_comments = self._numpy_to_list(conv.consensus) if hasattr(conv, 'consensus') else [] + + # D11 cascade fix (Investigation B, Site 2): the old code + # sourced from `conv.consensus`, which is always `[]` post-D11 + # (the attribute was deprecated). Source from + # `conv.repness['consensus_comments']` instead — the new shape + # is `{'agree': [...], 'disagree': [...]}`. + if hasattr(conv, 'repness') and conv.repness: + consensus_comments = conv.repness.get( + 'consensus_comments', {'agree': [], 'disagree': []} + ) + else: + consensus_comments = {'agree': [], 'disagree': []} + consensus_comments = self._numpy_to_list(consensus_comments) consensus_comments = self._replace_floats_with_decimals(consensus_comments) - + analysis_table.put_item(Item={ 'zid': zid, 'math_tick': math_tick, @@ -433,7 +460,13 @@ def write_conversation(self, conv) -> bool: batch.put_item(Item={ 'zid_tick': zid_tick, 'comment_id': str(comment_id), - 'priority': comment_priorities.get(comment_id, 0), + # Legacy branch reads conv.comment_priorities + # directly (raw floats) — convert like the + # stats/consensus_score fields above, or + # boto3 rejects the write (Copilot + # 2026-07-04, e). + 'priority': self._replace_floats_with_decimals( + comment_priorities.get(comment_id, 0)), 'stats': stats, 'consensus_score': consensus_score, 'zid': zid, @@ -840,7 +873,24 @@ def read_math_by_tick(self, zid: str, math_tick: int) -> Dict[str, Any]: } # Set consensus - result['consensus'] = analysis.get('consensus_comments', []) + # D11 cascade fix (Investigation B, Site 3): default to the + # new dict shape `{'agree': [], 'disagree': []}` rather than + # the obsolete empty list `[]`, so downstream consumers + # always receive a uniformly-shaped value. + stored_consensus = analysis.get( + 'consensus_comments', {'agree': [], 'disagree': []} + ) + # Normalize legacy/degenerate blobs (Copilot 2026-07-04 + # g3, and #2591): pre-D11 writers stored consensus as a + # (hardcoded-empty) LIST, and a present-but-`None` + # attribute makes `.get(..., default)` return None rather + # than the default. Guard on "not a dict" so any + # non-dict (list, None, str, ...) maps to the empty dict + # shape — downstream consumers always receive + # `{'agree': [], 'disagree': []}` with both keys present. + if not isinstance(stored_consensus, dict): + stored_consensus = {'agree': [], 'disagree': []} + result['consensus'] = stored_consensus # 2. Get groups data groups_table = self.tables.get('Delphi_KMeansClusters') diff --git a/delphi/polismath/pca_kmeans_rep/clusters.py b/delphi/polismath/pca_kmeans_rep/clusters.py index bc4df35de..cc976d85c 100644 --- a/delphi/polismath/pca_kmeans_rep/clusters.py +++ b/delphi/polismath/pca_kmeans_rep/clusters.py @@ -8,7 +8,6 @@ import numpy as np import pandas as pd from typing import Dict, List, Optional, Tuple, Union, Any -import random from copy import deepcopy from sklearn.cluster import KMeans from sklearn.metrics import silhouette_score @@ -674,8 +673,15 @@ def calculate_silhouette_sklearn(data: np.ndarray, Returns: Silhouette coefficient (between -1 and 1, higher is better) """ - # sklearn requires at least 2 clusters and 2 samples - if len(np.unique(labels)) <= 1 or data.shape[0] <= 1: + # sklearn's silhouette_score requires 2 <= n_labels <= n_samples - 1. + # When there are as many (or more) distinct labels as samples — e.g. only + # two base clusters fed into a k=2 group clustering (2 points / 2 labels) — + # the coefficient is undefined; return the neutral 0.0 sentinel instead of + # letting sklearn raise ValueError. (powerit PCA can collapse a small + # conversation to two base clusters; see #2591.) + n_labels = len(np.unique(labels)) + n_samples = data.shape[0] + if n_labels <= 1 or n_labels >= n_samples: return 0.0 return silhouette_score(data, labels, metric=metric) @@ -762,9 +768,6 @@ def cluster_dataframe(df: pd.DataFrame, row_to_idx = {name: i for i, name in enumerate(df.index)} last_clusters_internal = clusters_from_dict(last_clusters, row_to_idx) - # Use fixed random seed for initialization to be more consistent - np.random.seed(42) - # Perform clustering clusters_result = kmeans( matrix_data, @@ -774,10 +777,15 @@ def cluster_dataframe(df: pd.DataFrame, weights_array ) - # Sort clusters by size (descending) to match Clojure behavior + # NOTE: this size-descending sort + id reassignment does NOT match + # Clojure (the old comment here claimed it did). Clojure keeps + # first-k-distinct encounter-order ids and only ever sorts by :id — + # see the 2026-07-05 gid label-swap fix in conversation.py, which + # removed the same pattern from the LIVE path. This function is not + # on the production path (kmeans_sklearn is; sole caller is + # tests/test_clusters.py, whose expectations pin this ordering), so + # the behavior is kept as-is here rather than silently changed. clusters_result.sort(key=lambda x: len(x.members), reverse=True) - - # Reassign IDs based on sorted order to match Clojure behavior for i, cluster in enumerate(clusters_result): cluster.id = i diff --git a/delphi/polismath/pca_kmeans_rep/pca.py b/delphi/polismath/pca_kmeans_rep/pca.py index cf6291975..6ba273ad0 100644 --- a/delphi/polismath/pca_kmeans_rep/pca.py +++ b/delphi/polismath/pca_kmeans_rep/pca.py @@ -6,12 +6,235 @@ """ import logging +import os import numpy as np import pandas as pd -from typing import Dict, List, Optional, Tuple, Union, Any +from typing import Dict, List, Optional, Sequence, Tuple, Union, Any logger = logging.getLogger(__name__) + +# ============================================================================= +# Implementation switch: legacy/Clojure-parity vs improved +# ============================================================================= +# +# Pattern for legacy-vs-improved switches (reuse this idiom for future ones, +# e.g. a k-means solver switch): a module-level env var name + default + +# allowed values, resolved by `_resolve_impl_flag` AT CALL TIME (never at +# import time), so tests and operators can flip the env var without +# re-importing. Unknown values fall back to the default with a warning +# (defensive: a typo in a deployment env must not crash the math worker). + +PCA_IMPL_ENV_VAR = 'POLISMATH_PCA_IMPL' +PCA_IMPL_POWERIT = 'powerit' # legacy/Clojure-parity solver (default) +PCA_IMPL_SKLEARN = 'sklearn' # improved solver (exact SVD) +PCA_IMPL_DEFAULT = PCA_IMPL_POWERIT +PCA_IMPL_CHOICES = (PCA_IMPL_POWERIT, PCA_IMPL_SKLEARN) + + +def _resolve_impl_flag(env_var: str, default: str, choices: Sequence[str]) -> str: + """ + Resolve a legacy-vs-improved implementation switch from the environment. + + Args: + env_var: Environment variable name to read (at call time). + default: Value to use when the variable is unset or invalid. + choices: Allowed values (lowercase). + + Returns: + One of `choices`. + """ + raw = os.environ.get(env_var) + if raw is None: + return default + value = raw.strip().lower() + if value not in choices: + logger.warning("%s=%r is not one of %s; falling back to %r", + env_var, raw, tuple(choices), default) + return default + return value + +# ============================================================================= +# Clojure-parity power-iteration PCA +# ============================================================================= +# +# Port of math/src/polismath/math/pca.clj: +# power-iteration (l.38-56), proj-vec (l.59-63), factor-matrix (l.66-76), +# rand-starting-vec (l.79-82), powerit-pca (l.86-105). +# +# The production Clojure pipeline (conversation.clj:381-386) calls this with +# :n-comps 2 and :pca-iters 100 (conversation.clj:145-146), warm-starting +# :start-vectors from the previous tick's comps. +# +# START-VECTOR POLICY — DOCUMENTED DECISION: +# Clojure draws an UNSEEDED random start vector on cold start +# (rand-starting-vec, pca.clj:79-82 — the author's own comment there says +# "Should really throw a parallelizable random number generator in the +# equation here... With seeds fed in and persisted... XXX"). For Python we +# instead default to a DETERMINISTIC start (fixed-seed generator below) so +# the pipeline stays bit-for-bit reproducible — the 2026-07-05 determinism +# verification (5 identical consecutive runs on vw + biodiversity) is a +# project invariant we must not break. Power iteration converges to the same +# dominant eigenvector for almost any start vector (any start not exactly +# orthogonal to it), so a fixed start is simply one specific draw of +# Clojure's random one. `start_vectors` overrides the default for warm-start +# pinning (e.g. the R2 replayer pinning Clojure's previous-tick comps). +# +# TODO(julien): switch to a proper convergence criterion once we move to +# improving the Python implementation. + +# Fixed seed for the deterministic cold-start vector draw (see policy above). +_POWERIT_START_SEED = 42 + + +def _power_iteration(data: np.ndarray, + iters: int = 100, + start_vector: Optional[np.ndarray] = None) -> np.ndarray: + """ + First eigenvector of data.T @ data via power iteration. + + Port of Clojure `power-iteration` (pca.clj:38-56): runs a FIXED number of + multiplications by XᵀX (iters + 1 in total, matching the Clojure loop + structure), with an early exit only when the eigenvalue estimate is + EXACTLY equal to the previous one (float equality, as in Clojure). + + Args: + data: 2D array (rows are observations), typically already centered. + iters: Iteration budget (Clojure default 100, pca.clj:43). + start_vector: Starting vector. Defaults to all-ones (pca.clj:45). + If shorter than the column count it is padded with 1s, matching + Clojure's handling of new comments adding columns (pca.clj:46-49). + + Returns: + Unit-norm dominant eigenvector of data.T @ data, or a zero vector if + the data has no variance left in any direction (defensive: Clojure + would call normalise on a zero vector there). + """ + n_cols = data.shape[1] + if start_vector is None: + vec = np.ones(n_cols, dtype=np.float64) + else: + vec = np.asarray(start_vector, dtype=np.float64).ravel().copy() + if vec.shape[0] < n_cols: + # Clojure parity (pca.clj:46-49): pad with 1s when new comments + # have added columns since the start vector was recorded. + vec = np.concatenate([vec, np.ones(n_cols - vec.shape[0])]) + elif vec.shape[0] > n_cols: + # Defensive divergence: Clojure would error on a longer start + # vector (shape mismatch in inner-product); we truncate instead. + vec = vec[:n_cols] + + remaining = int(iters) + last_eigval = 0.0 + while True: + # xtxr (pca.clj:25-35): product = Xᵀ (X v), i.e. one power step. + product = data.T @ (data @ vec) + eigval = float(np.linalg.norm(product)) + if eigval == 0.0: + # No variance in the remaining subspace. Return the zero vector + # rather than normalising it (belt-and-braces; see docstring). + return product + normed = product / eigval + if remaining <= 0 or eigval == last_eigval: + return normed + remaining -= 1 + vec = normed + last_eigval = eigval + + +def _factor_matrix(data: np.ndarray, xs: np.ndarray) -> np.ndarray: + """ + Gram-Schmidt deflation: remove the direction `xs` from every row of data. + + Port of Clojure `factor-matrix` + `proj-vec` (pca.clj:59-76): each row + becomes row - ((xs·row)/(xs·xs)) * xs, leaving no variance along xs. + + Args: + data: 2D array. + xs: Direction to factor out (the principal component just found). + + Returns: + Deflated copy of data (data itself if xs is the zero vector, matching + the Clojure zero-eigenvector guard at pca.clj:71). + """ + denom = float(np.dot(xs, xs)) + if denom == 0.0: + return data + coeffs = (data @ xs) / denom + return data - np.outer(coeffs, xs) + + +def powerit_pca(matrix: np.ndarray, + n_comps: int = 2, + iters: int = 100, + start_vectors: Optional[Sequence[np.ndarray]] = None + ) -> Dict[str, np.ndarray]: + """ + Clojure-parity PCA via per-component power iteration with deflation. + + Port of Clojure `powerit-pca` (pca.clj:86-105): center on column means, + then for each component run `_power_iteration` on the (deflated) centered + data and factor the found component out (`_factor_matrix`) before finding + the next one. The number of components is clamped to + min(n_comps, min(n_rows, n_cols)) exactly as in Clojure (pca.clj:93,96). + + Start vectors: `start_vectors[i]` seeds component i (warm start, as fed + from the previous tick's comps at conversation.clj:385). Missing or + all-zero entries (wrapped-pca maps all-zero to nil, pca.clj:122-123) fall + back to a DETERMINISTIC uniform[0,1) draw — see the START-VECTOR POLICY + comment above for why this deliberately differs from Clojure's unseeded + (rand). + + Args: + matrix: 2D array-like, observations in rows. NaNs must already be + imputed by the caller (the Clojure pipeline feeds a matrix whose + nils were replaced by column averages, conversation.clj:360-380 — + identical to `pca_project_dataframe`'s nanmean imputation). + n_comps: Number of principal components to compute. + iters: Power-iteration budget per component (Clojure default 100). + start_vectors: Optional per-component starting vectors. + + Returns: + Dict with 'center' (column means, shape (n_cols,)) and 'comps' + (unit-norm components as rows, shape (n_comps_eff, n_cols)). + """ + data = np.asarray(matrix, dtype=np.float64) + center = data.mean(axis=0) + centered = data - center + n_rows, n_cols = centered.shape + + data_dim = min(n_rows, n_cols) + n_comps_eff = max(1, min(int(n_comps), data_dim)) + + provided: List[Optional[np.ndarray]] = [] + if start_vectors is not None: + provided = [None if sv is None else np.asarray(sv, dtype=np.float64).ravel() + for sv in start_vectors] + + # Deterministic cold-start draws (see START-VECTOR POLICY above). A fresh + # fixed-seed generator per call keeps repeated calls bit-identical. + rng = np.random.default_rng(_POWERIT_START_SEED) + + comps = [] + deflated = centered + for comp_idx in range(n_comps_eff): + start = provided[comp_idx] if comp_idx < len(provided) else None + if start is not None and not np.any(start): + # wrapped-pca parity (pca.clj:122-123): all-zero (or empty) start + # vectors are treated as missing. + start = None + if start is None: + # Clojure: rand-starting-vec draws uniform[0,1) per column + # (pca.clj:79-82); ours is the deterministic equivalent. + start = rng.random(n_cols) + pc = _power_iteration(deflated, iters=iters, start_vector=start) + comps.append(pc) + if comp_idx < n_comps_eff - 1: + deflated = _factor_matrix(deflated, pc) + + return {'center': center, 'comps': np.array(comps)} + + def pca_project_dataframe(df: pd.DataFrame, n_comps: int = 2) -> Tuple[Dict[str, np.ndarray], Dict[str, np.ndarray]]: """ @@ -81,20 +304,52 @@ def pca_project_dataframe(df: pd.DataFrame, # TODO(julien): try removing random_state to see if results are deterministic without it # (sklearn's full SVD solver is deterministic; randomized solver needs a seed). - + # + # Seeding history: the Clojure implementation never fixes a seed anywhere. + # Its k-means is deterministic by construction (first-k-distinct init) and + # its PCA power iteration draws an UNSEEDED random start vector on cold + # start only (warm-started from the previous tick's eigenvectors after + # that). The original Clojure author's note on this exact problem, verbatim + # (math/src/polismath/math/pca.clj:80-81): + # + # ;; Should really throw a parallelizable random number generator in the equation here... + # ;; With seeds fed in and persisted... XXX + # + # Verified 2026-07-05: the Python batch pipeline is bit-for-bit + # deterministic across 5 consecutive runs on vw + biodiversity (only + # math_tick, a wall-clock version counter, varies) — see the + # "Determinism verification" entry (2026-07-04/05) in + # docs/CLJ-PARITY-FIXES-JOURNAL.md. + + # Solver switch (read at call time — see _resolve_impl_flag): + # POLISMATH_PCA_IMPL=powerit (default) legacy/Clojure-parity power iteration + # POLISMATH_PCA_IMPL=sklearn improved exact-SVD path + # The imputation above and sparsity scaling below are IDENTICAL for both; + # only the eigen-solver differs. + impl = _resolve_impl_flag(PCA_IMPL_ENV_VAR, PCA_IMPL_DEFAULT, PCA_IMPL_CHOICES) + # Perform PCA with error handling # TODO(julien): use function that compute projections and PCAs in one pass. try: - from sklearn.decomposition import PCA + if impl == PCA_IMPL_SKLEARN: + from sklearn.decomposition import PCA - pca = PCA(n_components=n_comps, random_state=42) - projections = pca.fit_transform(matrix_data_no_nan) - projections = np.ascontiguousarray(projections) + pca = PCA(n_components=n_comps, random_state=42) + projections = pca.fit_transform(matrix_data_no_nan) - pca_results = { - 'center': pca.mean_, - 'comps': pca.components_ - } + pca_results = { + 'center': pca.mean_, + 'comps': pca.components_ + } + else: + # Legacy/Clojure-parity solver (default). Comps are unit vectors; + # projections are (X - center) @ compsᵀ, exactly like sklearn's + # fit_transform convention. + pca_results = powerit_pca(matrix_data_no_nan, n_comps=n_comps) + projections = ((matrix_data_no_nan - pca_results['center']) + @ pca_results['comps'].T) + + projections = np.ascontiguousarray(projections) except Exception as e: print(f"Error in PCA computation: {e}") @@ -123,5 +378,66 @@ def pca_project_dataframe(df: pd.DataFrame, # Create fallback projections (all zeros) n_proj = min(n_cols, 2) proj_dict = {pid: np.zeros(n_proj) for pid in df.index} - - return pca_results, proj_dict \ No newline at end of file + + return pca_results, proj_dict + + +# ============================================================================= +# D12: Comment projection / extremity (Clojure parity) +# ============================================================================= +# +# Port of Clojure `pca-project-cmnts` (math/src/polismath/math/pca.clj:167-178) +# and the extremity step from `with-proj-and-extremtiy` +# (math/src/polismath/math/conversation.clj:341-352). + +def pca_project_cmnts(center: np.ndarray, comps: np.ndarray) -> np.ndarray: + """ + Project each comment into the 2D PCA space. + + Clojure (`pca-project-cmnts`, pca.clj:167-178) calls + `sparsity-aware-project-ptpts` on a synthetic vote matrix where row `i` + has value `-1` at column `i` and `nil` everywhere else. + + For comment `i`, the sparsity-aware reduce (pca.clj:134-157) collapses to: + n_votes = 1 (only column i is non-nil) + p1 = (-1 - center[i]) * pc1[i] + p2 = (-1 - center[i]) * pc2[i] + scale = sqrt(n_cmnts / max(1, 1)) = sqrt(n_cmnts) + Final row: + proj[i] = sqrt(n_cmnts) * (-1 - center[i]) * [pc1[i], pc2[i]] + = -sqrt(n_cmnts) * (1 + center[i]) * [pc1[i], pc2[i]] + + Args: + center: PCA center (column means), shape (n_cmnts,). + comps: PCA components, shape (n_components, n_cmnts). Typically + n_components == 2. + + Returns: + Array of shape (n_cmnts, n_components) — projection per comment, in + the same column order as `center` / `comps`. + """ + n_cmnts = len(center) + if n_cmnts == 0: + return np.zeros((0, comps.shape[0] if comps.ndim == 2 else 0)) + scale = np.sqrt(n_cmnts) + coefs = -scale * (1.0 + center) # shape (n_cmnts,) + return coefs[:, None] * comps.T # shape (n_cmnts, n_components) + + +def compute_comment_extremity(cmnt_proj: np.ndarray) -> np.ndarray: + """ + Per-comment extremity = L2 norm of each projection row. + + Clojure parity: `with-proj-and-extremtiy` (conversation.clj:347-349) maps + `matrix/length` over each row of `pca-project-cmnts`. `matrix/length` is + Euclidean norm. + + Args: + cmnt_proj: shape (n_cmnts, n_components). + + Returns: + Shape (n_cmnts,) — extremity per comment. + """ + if cmnt_proj.size == 0: + return np.zeros(0) + return np.linalg.norm(cmnt_proj, axis=1) \ No newline at end of file diff --git a/delphi/polismath/pca_kmeans_rep/repness.py b/delphi/polismath/pca_kmeans_rep/repness.py index bbc5fb1f4..56808eb7f 100644 --- a/delphi/polismath/pca_kmeans_rep/repness.py +++ b/delphi/polismath/pca_kmeans_rep/repness.py @@ -7,10 +7,7 @@ import numpy as np import pandas as pd -from typing import Dict, List, Optional, Tuple, Union, Any -from copy import deepcopy -import math -from scipy import stats +from typing import Any, Dict, Iterable, List, Optional, Tuple from polismath.utils.general import AGREE, DISAGREE @@ -66,474 +63,46 @@ def z_score_sig_95(z: float) -> bool: return z > Z_95 -def prop_test(succ: int, n: int) -> float: - """ - One-proportion z-test, matching Clojure's stats/prop-test (stats.clj:10-15). - - Clojure formula: - (let [[succ n] (map inc [succ n])] - (* 2 (sqrt n) (+ (/ succ n) -0.5))) - - Which simplifies to: 2 * sqrt(n+1) * ((succ+1)/(n+1) - 0.5) - - This is a Wilson-score-like test with built-in +1 pseudocount (Laplace - smoothing). Unlike the standard z-test ((p - p0) / sqrt(p0*(1-p0)/n)), - the +1 terms regularize extreme values for small samples, preventing - spurious significance in small Polis groups. - - Note: the pseudocount here (+1 to succ and n, i.e. Beta(1,1)) is - independent of the PSEUDO_COUNT used for pa/pd computation (Beta(2,2)). - Clojure's prop-test takes raw success counts, not pre-smoothed - probabilities. - - Args: - succ: Number of successes (e.g. agrees `na` or disagrees `nd`) - n: Number of *trials counted as votes for this test*. In all current - callers, this is `ns = na + nd` (AGREE + DISAGREE) — PASS votes - are NOT included, matching what Clojure passes as `n-trials`. This - is a Polis-pipeline convention, not a generic z-test signature; - if you call this from elsewhere, supply `na + nd` rather than - a "total votes seen including pass" count. - - Returns: - Z-score. Positive when the smoothed proportion (succ+1)/(n+1) > 0.5 - (equivalent to succ >= n/2). This differs slightly from the raw-ratio - condition succ/n > 0.5 because of the +1 pseudocount applied to both - numerator and denominator. - - No n=0 short-circuit (Clojure parity — stats.clj:10-15 has no guard): - prop_test(0, 0) → (1, 1) after +1 → 2*sqrt(1)*(1/1 - 0.5) = 1.0. - """ - # Apply +1 pseudocount to both numerator and denominator - succ_pc = succ + 1 - n_pc = n + 1 - return 2 * math.sqrt(n_pc) * (succ_pc / n_pc - 0.5) - - -def two_prop_test(succ_in: int, succ_out: int, pop_in: int, pop_out: int) -> float: - """ - Two-proportion z-test with +1 pseudocount on all inputs. - - Matches Clojure's stats/two-prop-test (stats.clj:18-33): - (let [[succ-in succ-out pop-in pop-out] (map inc [succ-in succ-out pop-in pop-out]) - pi1 (/ succ-in pop-in) - pi2 (/ succ-out pop-out) - pi-hat (/ (+ succ-in succ-out) (+ pop-in pop-out))] - ...) - - The +1 pseudocount (Laplace smoothing) regularizes the z-score for small - samples, preventing extreme values when group sizes are tiny. - - Args: - succ_in: Number of successes in the group (e.g., agrees) - succ_out: Number of successes outside the group - pop_in: Total votes in the group - pop_out: Total votes outside the group - - Returns: - Z-score (positive means group proportion > other proportion) - """ - # No pop_in/pop_out short-circuit: Clojure's (map inc ...) increments all - # four inputs unconditionally, so pop=0 becomes pop=1 and the test proceeds. - # The only early-return is pi_hat == 1 below, matching Clojure. - - # Add +1 pseudocount to all four inputs (Clojure: map inc) - s1 = succ_in + 1 - s2 = succ_out + 1 - p1 = pop_in + 1 - p2 = pop_out + 1 - - pi1 = s1 / p1 - pi2 = s2 / p2 - pi_hat = (s1 + s2) / (p1 + p2) - - if pi_hat == 1.0: - # Clojure note (stats.clj:26-27): "this isn't quite right... could - # actually solve this using limits" — returning 0 for now, matching Clojure. - return 0.0 - - se = math.sqrt(pi_hat * (1 - pi_hat) * (1/p1 + 1/p2)) - if se == 0: - return 0.0 - return (pi1 - pi2) / se - - -def comment_stats(votes: np.ndarray, group_members: List[int]) -> Dict[str, Any]: - """ - Calculate basic stats for a comment within a group. - - Args: - votes: Array of votes (-1, 0, 1, or None) for the comment - group_members: Indices of group members - - Returns: - Dictionary of statistics - """ - # Filter votes to only include group members - group_votes = votes[group_members] - - # Count agrees, disagrees, and total votes - n_agree = np.sum(group_votes == AGREE) - n_disagree = np.sum(group_votes == DISAGREE) - n_votes = n_agree + n_disagree - - # Calculate probabilities with pseudocounts (Bayesian smoothing) - p_agree = (n_agree + PSEUDO_COUNT/2) / (n_votes + PSEUDO_COUNT) if n_votes > 0 else 0.5 - p_disagree = (n_disagree + PSEUDO_COUNT/2) / (n_votes + PSEUDO_COUNT) if n_votes > 0 else 0.5 - - # Calculate significance tests — pass raw counts, matching Clojure's - # (stats/prop-test na ns) and (stats/prop-test nd ns) (repness.clj:74-75) - # No n_votes>0 guard — Clojure parity (stats.clj:10-15 has no n=0 short-circuit; - # prop_test handles n=0 via the +1 pseudocount → returns 1.0) - p_agree_test = prop_test(n_agree, n_votes) - p_disagree_test = prop_test(n_disagree, n_votes) - - # Return stats - return { - 'na': n_agree, - 'nd': n_disagree, - 'ns': n_votes, - 'pa': p_agree, - 'pd': p_disagree, - 'pat': p_agree_test, - 'pdt': p_disagree_test - } - - -def add_comparative_stats(comment_stats: Dict[str, Any], - other_stats: Dict[str, Any]) -> Dict[str, Any]: - """ - Add comparative statistics between a group and others. - - Args: - comment_stats: Statistics for the group - other_stats: Statistics for other groups combined - - Returns: - Enhanced statistics with comparative measures - """ - result = deepcopy(comment_stats) - - # Calculate representativeness ratios - result['ra'] = result['pa'] / other_stats['pa'] if other_stats['pa'] > 0 else 1.0 - result['rd'] = result['pd'] / other_stats['pd'] if other_stats['pd'] > 0 else 1.0 - - # Calculate representativeness tests — pass raw counts, matching Clojure's - # (stats/two-prop-test (:na in-stats) (sum :na rest-stats) - # (:ns in-stats) (sum :ns rest-stats)) (repness.clj:97-100) - result['rat'] = two_prop_test( - result['na'], other_stats['na'], - result['ns'], other_stats['ns'] - ) - - result['rdt'] = two_prop_test( - result['nd'], other_stats['nd'], - result['ns'], other_stats['ns'] - ) - - return result - - -def repness_metric(stats: Dict[str, Any], key_prefix: str) -> float: - """ - Composite representativeness score, matching Clojure's repness-metric. - - Clojure (math/src/polismath/math/repness.clj:191-193): - (defn repness-metric - [{:keys [repness repness-test p-success p-test]}] - (* repness repness-test p-success p-test)) - - For Python the keys are looked up via key_prefix: - 'a' (agree) → ra * rat * pa * pat - 'd' (disagree) → rd * rdt * pd * pdt - - This is a *signed* product of 4 values — there is no abs(). A negative - z-score (pat / rat / pdt / rdt) flips the sign of the metric, exactly as - in Clojure. Downstream `select_rep_comments` sorts candidates by this - metric in descending order and keeps the top N, so negative metrics rank - at the bottom of the candidate pool. They are not actively *filtered* - here, though — fallback paths (e.g. fewer than the requested N candidates - pass significance) can still surface a negative-metric comment. Callers - that need strict positive-metric semantics should gate at the call site. - - Args: - stats: Statistics for a comment/group - key_prefix: 'a' for agreement, 'd' for disagreement - - Returns: - Composite representativeness score (signed product of 4 values). - """ - p = stats[f'p{key_prefix}'] - p_test = stats[f'p{key_prefix}t'] - r = stats[f'r{key_prefix}'] - r_test = stats[f'r{key_prefix}t'] - return r * r_test * p * p_test - - -def finalize_cmt_stats(stats: Dict[str, Any]) -> Dict[str, Any]: - """ - Finalize comment stats and classify as agree/disagree, matching Clojure. - - Clojure (math/src/polismath/math/repness.clj:173-180): - (defn finalize-cmt-stats - [tid {:keys [... rat rdt ...]}] - (let [[...] (if (> rat rdt) - [na ns pa pat ra rat :agree] - [nd ns pd pdt rd rdt :disagree])] - ...)) - - Pure comparison of the two two-prop z-scores. No probability/ratio - threshold logic — strict `rat > rdt` (rat == rdt falls through to disagree). - Always populates `agree_metric` / `disagree_metric` (used downstream by - selection routines that rank candidates). - - Args: - stats: Statistics for a comment/group - - Returns: - Finalized statistics with `repful`, `agree_metric`, `disagree_metric`. - """ - result = deepcopy(stats) - result['agree_metric'] = repness_metric(stats, 'a') - result['disagree_metric'] = repness_metric(stats, 'd') - result['repful'] = 'agree' if stats['rat'] > stats['rdt'] else 'disagree' - return result - - -def passes_by_test(stats: Dict[str, Any], repful: str, p_thresh: float = 0.5) -> bool: - """ - Check if comment passes significance tests. - - Args: - stats: Statistics for a comment/group - repful: 'agree' or 'disagree' - p_thresh: Probability threshold - - Returns: - True if passes significance tests - """ - key_prefix = 'a' if repful == 'agree' else 'd' - p = stats[f'p{key_prefix}'] - p_test = stats[f'p{key_prefix}t'] - r_test = stats[f'r{key_prefix}t'] - - # Check if proportion is high enough - if p < p_thresh: - return False - - # Check significance tests - return z_score_sig_90(p_test) and z_score_sig_90(r_test) - - -def best_agree(all_stats: List[Dict[str, Any]]) -> List[Dict[str, Any]]: - """ - Filter for best agreement comments. - - Args: - all_stats: List of comment statistics - - Returns: - Filtered list of comments that are best representatives by agreement - """ - # Filter to comments more agreed with than disagreed with - agree_stats = [s for s in all_stats if s['pa'] > s['pd']] - - # Filter to comments that pass significance tests - passing = [s for s in agree_stats if passes_by_test(s, 'agree')] - - if passing: - return passing - else: - return agree_stats - - -def best_disagree(all_stats: List[Dict[str, Any]]) -> List[Dict[str, Any]]: - """ - Filter for best disagreement comments. - - Args: - all_stats: List of comment statistics - - Returns: - Filtered list of comments that are best representatives by disagreement - """ - # Filter to comments more disagreed with than agreed with - disagree_stats = [s for s in all_stats if s['pd'] > s['pa']] - - # Filter to comments that pass significance tests - passing = [s for s in disagree_stats if passes_by_test(s, 'disagree')] - - if passing: - return passing - else: - return disagree_stats - - -def select_rep_comments(all_stats: List[Dict[str, Any]], - agree_count: int = 3, - disagree_count: int = 2) -> List[Dict[str, Any]]: - """ - Select representative comments for a group. - - Args: - all_stats: List of comment statistics - agree_count: Number of agreement comments to select - disagree_count: Number of disagreement comments to select - - Returns: - List of selected representative comments - """ - if not all_stats: - return [] - - # Start with best agreement comments - agree_comments = best_agree(all_stats) - - # Sort by agreement metric - agree_comments = sorted( - agree_comments, - key=lambda s: s['agree_metric'], - reverse=True - ) - - # Start with best disagreement comments - disagree_comments = best_disagree(all_stats) - - # Sort by disagreement metric - disagree_comments = sorted( - disagree_comments, - key=lambda s: s['disagree_metric'], - reverse=True - ) - - # Select top comments - selected = [] - - # Add agreement comments - for i, cmt in enumerate(agree_comments): - if i < agree_count: - cmt_copy = deepcopy(cmt) - cmt_copy['repful'] = 'agree' - selected.append(cmt_copy) - - # Add disagreement comments - for i, cmt in enumerate(disagree_comments): - if i < disagree_count: - cmt_copy = deepcopy(cmt) - cmt_copy['repful'] = 'disagree' - selected.append(cmt_copy) - - # If we couldn't find enough, try to add more from the other category - if len(selected) < agree_count + disagree_count: - # Add more agreement comments if needed - if len(selected) < agree_count + disagree_count and len(agree_comments) > agree_count: - for i in range(agree_count, min(len(agree_comments), agree_count + disagree_count)): - cmt_copy = deepcopy(agree_comments[i]) - cmt_copy['repful'] = 'agree' - selected.append(cmt_copy) - - # Add more disagreement comments if needed - if len(selected) < agree_count + disagree_count and len(disagree_comments) > disagree_count: - for i in range(disagree_count, min(len(disagree_comments), agree_count + disagree_count)): - cmt_copy = deepcopy(disagree_comments[i]) - cmt_copy['repful'] = 'disagree' - selected.append(cmt_copy) - - # If still not enough, at least ensure one comment - if not selected and all_stats: - # Just take the first one - cmt_copy = deepcopy(all_stats[0]) - cmt_copy['repful'] = cmt_copy.get('repful', 'agree') - selected.append(cmt_copy) - - return selected - - -def calculate_kl_divergence(p: np.ndarray, q: np.ndarray) -> float: - """ - Calculate Kullback-Leibler divergence between two probability distributions. - - Args: - p: First probability distribution - q: Second probability distribution - - Returns: - KL divergence - """ - # Replace zeros to avoid division by zero - p = np.where(p == 0, 1e-10, p) - q = np.where(q == 0, 1e-10, q) - - # numpy stubs: np.where widens p to ndarray|bool_, so np.sum is typed bool_. - # See pyright #2811. - return np.sum(p * np.log(p / q)) # pyright: ignore[reportReturnType] - - -def select_consensus_comments(all_stats: List[Dict[str, Any]]) -> List[Dict[str, Any]]: - """ - Select comments with broad consensus. - - Args: - all_stats: List of comment statistics for all groups - - Returns: - List of consensus comments - """ - # Group by comment - by_comment = {} - for stat in all_stats: - cid = stat['comment_id'] - if cid not in by_comment: - by_comment[cid] = [] - by_comment[cid].append(stat) - - # Comments that have stats for all groups - consensus_candidates = [] - - for cid, stats in by_comment.items(): - # Check if all groups mostly agree - all_agree = all(s['pa'] > 0.6 for s in stats) - - if all_agree: - # Calculate average agreement - avg_agree = sum(s['pa'] for s in stats) / len(stats) - - # Add as consensus candidate - consensus_candidates.append({ - 'comment_id': cid, - 'avg_agree': avg_agree, - 'repful': 'consensus', - 'stats': stats - }) - - # Sort by average agreement - consensus_candidates.sort(key=lambda x: x['avg_agree'], reverse=True) - - # Take top 2 - return consensus_candidates[:2] - - # ============================================================================= # Vectorized DataFrame-native functions for multi-group operations # ============================================================================= def prop_test_vectorized(succ: pd.Series, n: pd.Series) -> pd.Series: """ - Vectorized one-proportion z-test, matching Clojure's stats/prop-test. + Vectorized one-proportion z-test, matching Clojure's stats/prop-test + (math/src/polismath/math/stats.clj:10-15). + + Scalar equivalent (the formula this implements element-wise): - Formula: 2 * sqrt(n+1) * ((succ+1)/(n+1) - 0.5) + def prop_test(succ, n): + return 2 * sqrt(n + 1) * ((succ + 1) / (n + 1) - 0.5) - See prop_test() docstring for derivation and rationale. + Wilson-score-like test with built-in +1 pseudocount (Laplace / Beta(1,1) + smoothing). The +1 terms regularize extreme values for small samples, + preventing spurious significance in small Polis groups. Unlike the standard + z-test ((p - p0) / sqrt(p0*(1-p0)/n)), this formulation never divides by + zero — n=0 collapses to `2*sqrt(1)*(1/1 - 0.5) = 1.0` after smoothing. + + No n=0 short-circuit (Clojure parity — stats.clj:10-15 has no guard). + + Note: the pseudocount here (Beta(1,1)) is independent of the PSEUDO_COUNT + used for pa/pd computation (Beta(2,2)). prop_test takes RAW success and + trial counts, not pre-smoothed probabilities. Args: - succ: Series of success counts (e.g. `na` or `nd` per row) - n: Series of trial counts. In all current callers this is `ns = na + nd` - (AGREE + DISAGREE per row) — PASS votes are NOT included, matching - what Clojure passes as `n-trials`. See scalar `prop_test()` for the - same convention. + succ: Series of success counts (e.g. `na` or `nd` per row). + n: Series of trial counts. In all current callers this is `ns` = the + count of ALL non-nil votes INCLUDING PASS (`notna().sum()`), + matching Clojure's `count-votes` with no vote arg + (`(count (filter identity votes))` — 0/PASS is truthy in Clojure, + repness.clj:56-61; ns-PASS fix 2026-06-11). If you call this from + elsewhere, supply the PASS-inclusive non-nil count, NOT `na + nd`. Returns: - Series of z-scores + Series of z-scores. Positive when the smoothed proportion (succ+1)/(n+1) + > 0.5 (equivalent to succ >= n/2). Differs slightly from raw-ratio + succ/n > 0.5 because of the +1 pseudocount on both numerator and + denominator. """ succ_pc = succ + 1 n_pc = n + 1 @@ -549,19 +118,40 @@ def prop_test_vectorized(succ: pd.Series, n: pd.Series) -> pd.Series: def two_prop_test_vectorized(succ_in: pd.Series, succ_out: pd.Series, pop_in: pd.Series, pop_out: pd.Series) -> pd.Series: """ - Vectorized two-proportion z-test with +1 pseudocount on all inputs. + Vectorized two-proportion z-test with +1 pseudocount on all inputs, + matching Clojure's stats/two-prop-test + (math/src/polismath/math/stats.clj:18-33). + + Scalar equivalent (the formula this implements element-wise): + + def two_prop_test(succ_in, succ_out, pop_in, pop_out): + s1, s2 = succ_in + 1, succ_out + 1 + p1, p2 = pop_in + 1, pop_out + 1 + pi1, pi2 = s1 / p1, s2 / p2 + pi_hat = (s1 + s2) / (p1 + p2) + if pi_hat == 1.0: + return 0.0 # Clojure: "could solve via limits" (stats.clj:26-27) + se = sqrt(pi_hat * (1 - pi_hat) * (1/p1 + 1/p2)) + return (pi1 - pi2) / se - Matches Clojure's stats/two-prop-test (stats.clj:18-33). - See two_prop_test() scalar version for formula details. + +1 pseudocount (Laplace / Beta(1,1)) regularizes z-scores for small samples; + Clojure increments all four inputs unconditionally via (map inc ...) so + pop=0 becomes pop=1 and the test proceeds. The only early-return is + pi_hat == 1. + + No pop_in/pop_out short-circuit (Clojure parity). Vectorized handling: + - pi_hat == 1 → SE = 0 → z = NaN → fillna(0.0). + - pi_hat > 1 (na > pop, unreachable in real data) → sqrt of negative → NaN → 0.0. + - Division by zero → ±inf → replaced with 0.0. Args: - succ_in: Series of success counts in the group - succ_out: Series of success counts outside the group - pop_in: Series of total vote counts in the group - pop_out: Series of total vote counts outside the group + succ_in: Series of success counts in the group (e.g. agrees). + succ_out: Series of success counts outside the group. + pop_in: Series of total vote counts in the group. + pop_out: Series of total vote counts outside the group. Returns: - Series of z-scores + Series of z-scores (positive means group proportion > other proportion). """ # Add +1 pseudocount to all four inputs (Clojure: map inc) s1 = succ_in + 1 @@ -589,8 +179,9 @@ def compute_group_comment_stats_df(votes_long: pd.DataFrame, """ Compute vote counts and probabilities for all (group, comment) pairs. - This is the vectorized version of comment_stats() that operates on all - groups and comments simultaneously. + Vectorized port of Clojure's per-(group, comment) `comment-stats` recipe + (math/src/polismath/math/repness.clj:64-100). Operates on all groups and + comments simultaneously. Args: votes_long: Long-format DataFrame with columns: @@ -603,7 +194,8 @@ def compute_group_comment_stats_df(votes_long: pd.DataFrame, DataFrame indexed by (group_id, comment) with columns: - na: number of agrees - nd: number of disagrees - - ns: number of votes (agrees + disagrees) + - ns: number of votes (agrees + disagrees + PASS, Clojure parity; + see repness.clj:56-61, :70) - pa: probability of agree (with pseudocount smoothing) - pd: probability of disagree (with pseudocount smoothing) - pat: proportion test z-score for agree @@ -632,11 +224,17 @@ def compute_group_comment_stats_df(votes_long: pd.DataFrame, # Compute total counts per comment BEFORE filtering to group members # This matches the old behavior where "other" included ALL participants # not in the current group (even those not in any cluster) + # + # total_votes counts agree + disagree + PASS, matching Clojure's + # `count-votes` (math/src/polismath/math/repness.clj:56-61, :70). + # `count-votes` called with no `vote` arg uses `identity` as the filter + # predicate; in Clojure 0 is truthy, so PASS (0) votes are kept. NaN + # entries are already dropped above. Use size() to count non-NaN rows. total_counts = votes_only.groupby('comment').agg( total_agree=('vote', lambda x: (x == AGREE).sum()), total_disagree=('vote', lambda x: (x == DISAGREE).sum()), + total_votes=('vote', 'size'), ) - total_counts['total_votes'] = total_counts['total_agree'] + total_counts['total_disagree'] # Now add group column and filter to only group members votes_with_group = votes_only.copy() @@ -655,12 +253,18 @@ def compute_group_comment_stats_df(votes_long: pd.DataFrame, # Get all group IDs all_group_ids = [group['id'] for group in group_clusters] - # Compute vote counts per (group, comment) for votes from group members + # Compute vote counts per (group, comment) for votes from group members. + # + # ns counts agree + disagree + PASS, matching Clojure's `count-votes` + # (math/src/polismath/math/repness.clj:56-61, :70). `count-votes` with + # no `vote` arg uses `identity` as filter; in Clojure 0 is truthy, so + # PASS (0) votes count. NaN entries were already dropped above. Use + # size() to count non-NaN rows. group_counts = votes_in_groups.groupby(['group_id', 'comment']).agg( na=('vote', lambda x: (x == AGREE).sum()), nd=('vote', lambda x: (x == DISAGREE).sum()), + ns=('vote', 'size'), ) - group_counts['ns'] = group_counts['na'] + group_counts['nd'] # Create full index with all (group, comment) combinations to match old behavior # Old implementation: for each group, iterate over ALL comments (that have any votes) @@ -726,7 +330,10 @@ def compute_group_comment_stats_df(votes_long: pd.DataFrame, # Compute metrics # Clojure (repness.clj:191-193): (* repness repness-test p-success p-test) # agree_metric = ra * rat * pa * pat - # disagree_metric = rd * rdt * pd * pdt (signed product — see scalar repness_metric) + # disagree_metric = rd * rdt * pd * pdt + # Signed product — no abs(). Negative z-scores flip the sign of the metric; + # downstream selection sorts descending, so negative-metric comments rank + # at the bottom of the candidate pool but are not filtered here. stats_df['agree_metric'] = (stats_df['ra'] * stats_df['rat'] * stats_df['pa'] * stats_df['pat']) stats_df['disagree_metric'] = (stats_df['rd'] * stats_df['rdt'] @@ -739,159 +346,110 @@ def compute_group_comment_stats_df(votes_long: pd.DataFrame, return stats_df -def select_rep_comments_df(stats_df: pd.DataFrame, - agree_count: int = 3, - disagree_count: int = 2) -> pd.DataFrame: - """ - Select representative comments for a single group from a DataFrame. +# ============================================================================= +# D10: Selection helpers (Clojure parity for select-rep-comments) +# ============================================================================= +# +# Ports of Clojure's `select-rep-comments` and its three predicates from +# math/src/polismath/math/repness.clj:133-281. Operate on per-(group, comment) +# dict rows produced by `compute_group_comment_stats_df` (via to_dict('records')). +# Per-row dict ops + small per-group iteration (rather than vectorized) because +# `beats_best_agr` has a 4-branch decision against a moving "current best" +# that updates during iteration; vectorizing would require multiple passes +# without saving lines (per-group N typically <500 comments). - DataFrame-native version of select_rep_comments(). - Args: - stats_df: DataFrame with comment statistics for ONE group - agree_count: Number of agreement comments to select - disagree_count: Number of disagreement comments to select +def passes_by_test(s: Dict[str, Any]) -> bool: + """ + Clojure passes-by-test? (repness.clj:165-170). - Returns: - DataFrame of selected representative comments + True iff the agree side OR the disagree side passes z-sig-90 on BOTH + the proportion test (pat/pdt) and the representativeness test (rat/rdt). + No probability gate — pre-D10 Python's `pa >= 0.5` gate was a Python-only + over-restriction without a Clojure analog. + + (or (and (z-sig-90? rat) (z-sig-90? pat)) + (and (z-sig-90? rdt) (z-sig-90? pdt))) """ - if stats_df.empty: - return stats_df - - total_wanted = agree_count + disagree_count - - # Best agree: pa > pd and passes significance tests - agree_candidates = stats_df[stats_df['pa'] > stats_df['pd']].copy() - if not agree_candidates.empty: - # Check significance: pat > Z_90 and rat > Z_90 - passing_agree = agree_candidates[ - (agree_candidates['pat'] > Z_90) & - (agree_candidates['rat'] > Z_90) & - (agree_candidates['pa'] >= 0.5) - ] - if not passing_agree.empty: - agree_candidates = passing_agree - - # Best disagree: pd > pa and passes significance tests - disagree_candidates = stats_df[stats_df['pd'] > stats_df['pa']].copy() - if not disagree_candidates.empty: - passing_disagree = disagree_candidates[ - (disagree_candidates['pdt'] > Z_90) & - (disagree_candidates['rdt'] > Z_90) & - (disagree_candidates['pd'] >= 0.5) - ] - if not passing_disagree.empty: - disagree_candidates = passing_disagree - - # Sort candidates by metric - if not agree_candidates.empty: - agree_candidates = agree_candidates.sort_values('agree_metric', ascending=False) - if not disagree_candidates.empty: - disagree_candidates = disagree_candidates.sort_values('disagree_metric', ascending=False) - - # Select top N from each category - selected_parts = [] - - if not agree_candidates.empty: - top_agree = agree_candidates.head(agree_count).copy() - top_agree['repful'] = 'agree' - selected_parts.append(top_agree) - - if not disagree_candidates.empty: - top_disagree = disagree_candidates.head(disagree_count).copy() - top_disagree['repful'] = 'disagree' - selected_parts.append(top_disagree) - - if selected_parts: - selected = pd.concat(selected_parts, ignore_index=False) - else: - selected = pd.DataFrame() - - # If we couldn't find enough, try to fill from available candidates - # This matches the exact behavior of the old select_rep_comments() function: - # - First fallback adds agree_comments[agree_count:min(len, total_wanted)] regardless of - # whether we exceed total_wanted (up to disagree_count more agrees) - # - Second fallback only runs if STILL < total_wanted - if len(selected) < total_wanted: - # Try to add more agree comments - # Old code: range(agree_count, min(len(agree_comments), agree_count + disagree_count)) - if not agree_candidates.empty and len(agree_candidates) > agree_count: - extra_limit = min(len(agree_candidates), total_wanted) - extra_agrees = agree_candidates.iloc[agree_count:extra_limit].copy() - extra_agrees['repful'] = 'agree' - selected = pd.concat([selected, extra_agrees], ignore_index=False) - - # Try to add more disagree comments (only if still not enough) - # Old code: range(disagree_count, min(len(disagree_comments), agree_count + disagree_count)) - if len(selected) < total_wanted and not disagree_candidates.empty and len(disagree_candidates) > disagree_count: - extra_limit = min(len(disagree_candidates), total_wanted) - extra_disagrees = disagree_candidates.iloc[disagree_count:extra_limit].copy() - extra_disagrees['repful'] = 'disagree' - selected = pd.concat([selected, extra_disagrees], ignore_index=False) - - # Fallback: if still empty, take first row - if selected.empty and not stats_df.empty: - selected = stats_df.head(1).copy() - selected['repful'] = selected['repful'].iloc[0] if 'repful' in selected.columns else 'agree' - - return selected - - -def select_consensus_comments_df(stats_df: pd.DataFrame, - n_groups: int) -> List[Dict[str, Any]]: + return ( + (z_score_sig_90(s['rat']) and z_score_sig_90(s['pat'])) + or (z_score_sig_90(s['rdt']) and z_score_sig_90(s['pdt'])) + ) + + +def beats_best_by_test(s: Dict[str, Any], current_best_z: Optional[float]) -> bool: """ - Select consensus comments from DataFrame. + Clojure beats-best-by-test? (repness.clj:133-139). - Args: - stats_df: DataFrame with all (group, comment) statistics - n_groups: Number of groups + True if `s` has a more-representative max(rat, rdt) than `current_best_z`, + OR if there is no current best yet. Strict `>` (Clojure: `>`). - Returns: - List of consensus comment dicts + (or (nil? current-best-z) + (> (max rat rdt) current-best-z)) """ - if stats_df.empty: - return [] - - # Group by comment and check if all groups have high agreement - stats_reset = stats_df.reset_index() - comment_stats = stats_reset.groupby('comment').agg( - min_pa=('pa', 'min'), - avg_pa=('pa', 'mean'), - group_count=('group_id', 'count') - ) + if current_best_z is None: + return True + return max(s['rat'], s['rdt']) > current_best_z - # Filter to comments where all groups agree (pa > 0.6 for all) - # and present in all groups - consensus = comment_stats[ - (comment_stats['min_pa'] > 0.6) & - (comment_stats['group_count'] == n_groups) - ].copy() - - if consensus.empty: - return [] - - # Sort by average agreement and take top 2 - consensus = consensus.nlargest(2, 'avg_pa') - - # Convert to list of dicts using _stats_row_to_dict for legacy format - result = [] - for comment_id in consensus.index: - comment_rows = stats_reset[stats_reset['comment'] == comment_id] - # Convert each row to legacy dict format - stats_list = [_stats_row_to_dict(row) for _, row in comment_rows.iterrows()] - result.append({ - 'comment_id': comment_id, - 'avg_agree': consensus.loc[comment_id, 'avg_pa'], - 'repful': 'consensus', - 'stats': stats_list - }) - return result +def beats_best_agr(s: Dict[str, Any], + current_best: Optional[Dict[str, Any]]) -> bool: + """ + Clojure beats-best-agr? (repness.clj:142-162). + Four mutually exclusive branches: -def _stats_row_to_dict(row: pd.Series) -> Dict[str, Any]: - """Convert a stats DataFrame row to the legacy dict format.""" - return { + 1. `na == 0 and nd == 0`: reject. Comments with no votes never enter the + best-agree slot (Clojure: `(= 0 na nd)` → false). + 2. `current_best` exists AND `current_best['ra'] > 1.0`: compare the + 4-way signed product `ra * rat * pa * pat`. New row must beat the + current best on this product. + 3. `current_best` exists (else, i.e. `current_best['ra'] <= 1.0`): + compare `pa * pat` only — "shoot for something generally agreed upon" + when the current best isn't representative enough. + 4. No `current_best`: accept if `z90(pat)` OR `(ra > 1.0 AND pa > 0.5)`. + + `current_best` here is the RAW stats row (Clojure stores raw at + repness.clj:250 so this comparator keeps the `ra/rat/pa/pat` surface). + """ + if s['na'] == 0 and s['nd'] == 0: # Branch 1. + return False + if current_best is not None and current_best['ra'] > 1.0: # Branch 2. + return (s['ra'] * s['rat'] * s['pa'] * s['pat']) > ( + current_best['ra'] * current_best['rat'] + * current_best['pa'] * current_best['pat'] + ) + if current_best is not None: # Branch 3. + return (s['pa'] * s['pat']) > (current_best['pa'] * current_best['pat']) + # Branch 4. + return z_score_sig_90(s['pat']) or (s['ra'] > 1.0 and s['pa'] > 0.5) + + +def _finalize_row_for_output(row: Dict[str, Any], *, + is_best_agree: bool = False) -> Dict[str, Any]: + """ + Format a per-(group, comment) stats row for the final repness output + (math blob `repness` / `group_repness`). + + Mirrors Clojure `finalize-cmt-stats` (repness.clj:173-188) plus the + best-agree flagging at repness.clj:262-264. + + When `is_best_agree=True`, two extra keys are added: + - `best_agree`: True + - `n_agree`: the raw `na` (preserves the agree count even when the + row is classified as 'disagree' by `rat > rdt`). + + Key naming uses Python convention (underscored). Clojure-style hyphens + (`repful-for`, `n-agree`, etc.) are deferred to a future math-blob + alignment PR (see PLAN.md "Pending — needs team discussion"). + + `agree_metric` / `disagree_metric` are read directly from the row + (produced by `compute_group_comment_stats_df`) rather than recomputed. + Recomputing here would duplicate the formula at repness.clj:191-193 in + two places and risk drift if it ever changes (decision D10.8.3). + """ + repful = 'agree' if row['rat'] > row['rdt'] else 'disagree' + finalized: Dict[str, Any] = { 'comment_id': row['comment'], 'group_id': row['group_id'], 'na': int(row['na']), @@ -907,11 +465,314 @@ def _stats_row_to_dict(row: pd.Series) -> Dict[str, Any]: 'rdt': row['rdt'], 'agree_metric': row['agree_metric'], 'disagree_metric': row['disagree_metric'], - 'repful': row['repful'], + 'repful': repful, + } + if is_best_agree: + finalized['best_agree'] = True + finalized['n_agree'] = int(row['na']) + return finalized + + +def select_rep_comments_df(stats_df: pd.DataFrame, + mod_out: Optional[Iterable[int]] = None + ) -> Tuple[pd.DataFrame, Optional[Dict[str, Any]]]: + """ + Select representative comments for a single group (Clojure parity). + + Single-pass reduce over the group's (gid, tid) rows, mirroring + `select-rep-comments` in math/src/polismath/math/repness.clj:212-281. + + Per-row state {sufficient, best, best_agree}: + - `passes_by_test(row)` → append finalized row to `sufficient`. + - `:sufficient` still empty AND `beats_best_by_test` → update `best`. + - `beats_best_agr(row, best_agree)` → store RAW row as new `best_agree`. + + Final assembly (decision S2 / D10.4): + - `sufficient` non-empty: dedup best_agree from sufficient → sort by + agree/disagree metric (descending, signed product per repness.clj:191) + → take up to 5 (post-prepend → 4 sufficient max) → agrees-before- + disagrees on the sufficient slice. The best-agree dict is returned + SEPARATELY so the DataFrame stays clean (no NaN best_agree/n_agree + columns when the slot is empty). + - Else: `(empty_df, best_agree_dict)` if best_agree exists, else + `(single_row_df_for_best, None)` if best exists, else + `(empty_df, None)`. + + Args: + stats_df: DataFrame with comment statistics for ONE group, schema + as produced by `compute_group_comment_stats_df`. + mod_out: Optional iterable of tids to exclude (moderated-out comments). + Filter applied before the reduce (Clojure repness.clj:222). + + Returns: + `(rep_df, best_agree_dict)` tuple: + - `rep_df`: DataFrame of finalized rep-comment rows in math-blob + shape (see `_finalize_row_for_output`) — does NOT include the + best-agree slot, and carries NO `best_agree`/`n_agree` columns. + Already ordered agrees-before-disagrees and capped so that + `len(rep_df) + (1 if best_agree_dict else 0) <= 5`. + - `best_agree_dict`: Standalone finalized dict for the best-agree + slot (with `best_agree=True` and `n_agree=`), or `None` if + no candidate qualified. The caller is responsible for prepending + it to the flat output list. + """ + empty_df: pd.DataFrame = pd.DataFrame() + + if stats_df.empty: + return empty_df, None + + # `is not None`, not truthiness: mod_out may be a numpy array / pandas + # Index, whose bare truth value raises for len>1 (Copilot 2026-07-04). + mod_out_set = set(mod_out) if mod_out is not None else set() + sufficient: List[Dict[str, Any]] = [] + best: Optional[Dict[str, Any]] = None + # Track best's max(rat, rdt) as a sidecar scalar so we never have to mutate + # `best` itself with synthetic comparison keys. Avoids the leak/pop dance + # of stashing a `_max_rt` inside the finalized dict (decision D10.8.4). + best_max_rt: Optional[float] = None + best_agree: Optional[Dict[str, Any]] = None + + # Sort by `comment` (tid) ascending BEFORE iterating, so ties in + # `beats_best_by_test` (max(rat, rdt) tied) and in `beats_best_agr` + # (Branch 2/3 product tied) resolve deterministically. The chosen order + # matches Clojure's named-matrix column iteration: after normalization + # the columns are insertion-ordered, and for cold-start that's tid + # ascending (see Clojure named_matrix.clj:130-131 — insertion order). + # All Clojure beats-*? predicates use strict `>` so the FIRST row at a + # tied score wins; sorting ascending here mirrors that (decision D10.8.1). + iter_df = stats_df.sort_values('comment', kind='mergesort') + + for row in iter_df.to_dict('records'): + if row['comment'] in mod_out_set: + continue + if passes_by_test(row): + sufficient.append(_finalize_row_for_output(row)) + # Update `best` only while sufficient is still empty (Clojure parity). + if not sufficient: + if beats_best_by_test(row, best_max_rt): + best = _finalize_row_for_output(row) + best_max_rt = max(row['rat'], row['rdt']) + # `best_agree` stores RAW row (Clojure repness.clj:250) so subsequent + # `beats_best_agr` calls keep the ra/rat/pa/pat surface. + if beats_best_agr(row, best_agree): + best_agree = row + + # Build the standalone best-agree dict (or None) once — used in every + # assembly branch below. + best_agree_dict: Optional[Dict[str, Any]] = ( + _finalize_row_for_output(best_agree, is_best_agree=True) + if best_agree is not None else None + ) + + # Assembly. + if not sufficient: + if best_agree_dict is not None: + # Best-agree slot returned separately; rep_df stays empty. + return empty_df, best_agree_dict + if best is not None: + return pd.DataFrame([best]), None + return empty_df, None + + # Sufficient non-empty path. + best_agree_tid = best_agree['comment'] if best_agree is not None else None + # Dedup best_agree from sufficient (caller will re-prepend it). + deduped = [s for s in sufficient if s['comment_id'] != best_agree_tid] + + # Sort each row by its winning-side metric (signed product, per Clojure + # repness.clj:191-193). Clojure (repness.clj:191-200) sorts by a single + # `:repness-metric` field that `finalize-cmt-stats` populates per the + # winning side. We achieve equivalent ranking by reading `agree_metric` + # for repful=='agree' rows and `disagree_metric` otherwise — same + # comparator value, just a different key per row (decision D10.8.2). + def _sort_key(s: Dict[str, Any]) -> float: + return s['agree_metric'] if s['repful'] == 'agree' else s['disagree_metric'] + deduped.sort(key=_sort_key, reverse=True) + + # TODO(parity-eviction): the cap of 5 INCLUDING the best-agree slot can + # evict the 5th-highest-metric `sufficient` entry — a strong dissenting + # view may be silently dropped by a weak agree-priority one. Mirrors + # Clojure exactly for parity; flagged in PLAN.md + # "Pending — needs team discussion". + cap = 5 - (1 if best_agree_dict is not None else 0) + capped = deduped[:cap] + + # agrees-before-disagrees (Clojure repness.clj:203-209). Stable partition. + agrees = [c for c in capped if c['repful'] == 'agree'] + disagrees = [c for c in capped if c['repful'] == 'disagree'] + rep_df = pd.DataFrame(agrees + disagrees) + + return rep_df, best_agree_dict + + +def _assemble_rep_comments(stats_df: pd.DataFrame, + mod_out: Optional[Iterable[int]] = None + ) -> List[Dict[str, Any]]: + """Thin wrapper around `select_rep_comments_df` that returns the flat + output list (best-agree slot prepended, then the DataFrame's rows, + then re-partitioned agrees-before-disagrees so a `repful='disagree'` + best-agree slot lands in the disagrees section as in pre-S2). + + Decision S2: `select_rep_comments_df` returns a `(rep_df, best_agree_dict)` + tuple so the DataFrame stays clean (no NaN extra-key columns). Most + callers — including `conv_repness` and the D10 synthetic tests — want + the flat List[Dict] form, so we keep one place that does the prepend + and the final agrees-before-disagrees stable partition. + """ + rep_df, best_agree_dict = select_rep_comments_df(stats_df, mod_out=mod_out) + head: List[Dict[str, Any]] = [best_agree_dict] if best_agree_dict is not None else [] + tail: List[Dict[str, Any]] = ( + rep_df.to_dict('records') if not rep_df.empty else [] + ) + combined = head + tail + # Re-run agrees-before-disagrees stable partition so the best-agree slot + # ends up in the correct section per its own `repful`. This mirrors the + # pre-S2 behaviour where best_agree was prepended into a single list and + # then partitioned (Clojure repness.clj:203-209). + agrees = [c for c in combined if c['repful'] == 'agree'] + disagrees = [c for c in combined if c['repful'] == 'disagree'] + return agrees + disagrees + + +# ============================================================================= +# D11: Consensus comment selection (Clojure parity) +# ============================================================================= +# +# Ports of Clojure's `consensus-stats` and `select-consensus-comments` +# (math/src/polismath/math/repness.clj:284-323). +# +# Conceptually different from rep-comment selection: consensus stats are +# computed over the FULL conversation (no group split — `add-comparitive-stats` +# is NOT called). Two independent top-5 lists are then built — one for "agree +# consensus" (pa > 0.5 AND z-sig-90 on pat) ordered by `pa * pat`, one for +# "disagree consensus" (pd > 0.5 AND z-sig-90 on pdt) ordered by `pd * pdt`. + +def consensus_stats_df(vote_matrix_df: pd.DataFrame, + mod_out: Optional[Iterable[int]] = None + ) -> pd.DataFrame: + """ + Compute per-comment consensus stats across the whole conversation. + + Vectorized port of Clojure `consensus-stats` (repness.clj:284-290). Unlike + `compute_group_comment_stats_df`, no group split and no `ra/rd/rat/rdt` + (Clojure's `add-comparitive-stats` is not called here). + + Args: + vote_matrix_df: Wide-format vote matrix (participants × comments). + Values in {AGREE, DISAGREE, PASS, NaN}. + mod_out: Optional iterable of tids to exclude. Belt-and-braces with + D15 column-zeroing: moderated-out columns auto-fail the `pa > 0.5` + filter downstream (na=nd=0 → pa=pd=0.5), but the explicit filter + matches Clojure's behaviour (repness.clj:296). + + Returns: + DataFrame indexed by tid with columns [na, nd, ns, pa, pd, pat, pdt]. + """ + # Per-column counts. `vote_matrix_df` may have NaN for unvoted cells; + # those count as neither agree nor disagree. + na = (vote_matrix_df == AGREE).sum(axis=0).astype(int) + nd = (vote_matrix_df == DISAGREE).sum(axis=0).astype(int) + # ns counts all non-nil votes (incl. PASS) — Clojure parity, repness.clj:56-61. + ns = vote_matrix_df.notna().sum(axis=0).astype(int) + + df = pd.DataFrame({'na': na, 'nd': nd, 'ns': ns}) + df.index.name = 'tid' + + # pa, pd with PSEUDO_COUNT smoothing. + # Scalar equivalent: pa = (na + 1) / (ns + 2), pd = (nd + 1) / (ns + 2) + df['pa'] = (df['na'] + PSEUDO_COUNT / 2) / (df['ns'] + PSEUDO_COUNT) + df['pd'] = (df['nd'] + PSEUDO_COUNT / 2) / (df['ns'] + PSEUDO_COUNT) + zero_mask = df['ns'] == 0 + df.loc[zero_mask, 'pa'] = 0.5 + df.loc[zero_mask, 'pd'] = 0.5 + + # Proportion-test z-scores. + df['pat'] = prop_test_vectorized(df['na'], df['ns']) + df['pdt'] = prop_test_vectorized(df['nd'], df['ns']) + + # `is not None`, not truthiness: mod_out may be a numpy array / pandas + # Index, whose bare truth value raises for len>1 (Copilot 2026-07-04). + if mod_out is not None: + mod_out_set = set(mod_out) + df = df[~df.index.isin(mod_out_set)] + + return df + + +def select_consensus_comments_df( + cons_stats: pd.DataFrame, +) -> Dict[str, List[Dict[str, Any]]]: + """ + Select consensus comments (Clojure parity). + + Port of Clojure `select-consensus-comments` (repness.clj:293-323). Returns + two independent top-5 lists — one for agree consensus, one for disagree + consensus. + + Filters and ordering: + - Agree: `pa > 0.5 AND z-sig-90(pat)`, sorted desc by `am = pa * pat`. + - Disagree: `pd > 0.5 AND z-sig-90(pdt)`, sorted desc by `dm = pd * pdt`. + + Since `ns` counts all non-nil votes including PASS (ns ≥ na+nd), + `pa + pd = (na+nd+PSEUDO_COUNT)/(ns+PSEUDO_COUNT) ≤ 1`, so pa and pd + cannot both exceed 0.5 — the same tid cannot appear in both lists. (The + equality pa+pd=1 holds only for PASS-free comments.) + + Args: + cons_stats: DataFrame indexed by tid with cols [na, nd, ns, pa, pd, + pat, pdt], as produced by `consensus_stats_df`. + + Returns: + Dict shape `{'agree': [entries], 'disagree': [entries]}`. Each entry + is `{tid, n-success, n-trials, p-success, p-test}` — EXACTLY the + Clojure blob shape (repness.clj:181 + the ::consensus s/keys spec). + This narrows the S1 deferral (2026-07-04): consensus entries flow + raw into `result['consensus']` in to_dict / to_dynamo_dict, where + server-helpers.ts:298-313 and client-report's + majorityStrict.jsx:23-27 pluck `tid` — Python-convention keys broke + both. Rep-comment entries keep `comment_id` until the deferred + math-blob alignment PR. + """ + if cons_stats.empty: + return {'agree': [], 'disagree': []} + + df = cons_stats.copy() + df['am'] = df['pa'] * df['pat'] + df['dm'] = df['pd'] * df['pdt'] + + agree_filter = (df['pa'] > 0.5) & (df['pat'] > Z_90) + disagree_filter = (df['pd'] > 0.5) & (df['pdt'] > Z_90) + + agree_top = df[agree_filter].nlargest(5, 'am') + disagree_top = df[disagree_filter].nlargest(5, 'dm') + + def _agree_entry(tid: Any, row: pd.Series) -> Dict[str, Any]: + return { + 'tid': int(tid), + 'n-success': int(row['na']), + 'n-trials': int(row['ns']), + 'p-success': float(row['pa']), + 'p-test': float(row['pat']), + } + + def _disagree_entry(tid: Any, row: pd.Series) -> Dict[str, Any]: + return { + 'tid': int(tid), + 'n-success': int(row['nd']), + 'n-trials': int(row['ns']), + 'p-success': float(row['pd']), + 'p-test': float(row['pdt']), + } + + return { + 'agree': [_agree_entry(tid, row) for tid, row in agree_top.iterrows()], + 'disagree': [_disagree_entry(tid, row) for tid, row in disagree_top.iterrows()], } -def conv_repness(vote_matrix_df: pd.DataFrame, group_clusters: List[Dict[str, Any]]) -> Dict[str, Any]: +def conv_repness(vote_matrix_df: pd.DataFrame, + group_clusters: List[Dict[str, Any]], + mod_out: Optional[Iterable[int]] = None, + ) -> Dict[str, Any]: """ Calculate representativeness for all comments and groups. @@ -921,19 +782,23 @@ def conv_repness(vote_matrix_df: pd.DataFrame, group_clusters: List[Dict[str, An vote_matrix_df: pd.DataFrame of matrix of votes (participants × comments) Values should be AGREE (1), DISAGREE (-1), PASS (0), or NaN (unvoted) group_clusters: List of group clusters, each with 'id' and 'members' + mod_out: Optional iterable of tids to exclude (moderated-out comments). + Forwarded to `select_rep_comments_df` and `consensus_stats_df`. + See `Conversation.mod_out_tids`. Returns: Dictionary with representativeness data for each group: - comment_ids: list of comment IDs - group_repness: dict mapping group_id -> list of representative comments - - consensus_comments: list of consensus comments + - consensus_comments: dict `{'agree': [...], 'disagree': [...]}` after + D11 (was a flat list pre-D11; Clojure parity per repness.clj:322-323) - comment_repness: list of all comment repness data """ # Create empty-result structure in case we need to return early empty_result = { 'comment_ids': vote_matrix_df.columns.tolist(), 'group_repness': {group['id']: [] for group in group_clusters}, - 'consensus_comments': [], + 'consensus_comments': {'agree': [], 'disagree': []}, 'comment_repness': [] } @@ -998,25 +863,27 @@ def conv_repness(vote_matrix_df: pd.DataFrame, group_clusters: List[Dict[str, An continue try: - rep_df = select_rep_comments_df(group_stats) - # Convert to list of dicts only at the end - rep_comments = [_stats_row_to_dict(row) for _, row in rep_df.iterrows()] - result['group_repness'][group_id] = rep_comments + # `select_rep_comments_df` now returns `(rep_df, best_agree_dict)` + # (decision S2) so the DataFrame stays clean. Use the + # `_assemble_rep_comments` wrapper to get the flat List[Dict] the + # math blob expects (best-agree prepended, agrees-before-disagrees + # partition applied). Forward `mod_out` from conv_repness (D11 + # added this kwarg). + result['group_repness'][group_id] = _assemble_rep_comments( + group_stats, mod_out=mod_out) except Exception as e: print(f"Error selecting representative comments for group {group_id}: {e}") result['group_repness'][group_id] = [] - # Add consensus comments if there are multiple groups + # Consensus comments (D11 / PR 9). Whole-conversation stats, not per-group. + # Clojure runs this unconditionally (conversation.clj:706-709) — no + # `len(group_clusters) > 1` guard. try: - if len(group_clusters) > 1: - result['consensus_comments'] = select_consensus_comments_df( - stats_df, len(group_clusters) - ) - else: - result['consensus_comments'] = [] + cons_stats = consensus_stats_df(vote_matrix_df, mod_out=mod_out) + result['consensus_comments'] = select_consensus_comments_df(cons_stats) except Exception as e: print(f"Error selecting consensus comments: {e}") - result['consensus_comments'] = [] + result['consensus_comments'] = {'agree': [], 'disagree': []} return result diff --git a/delphi/tests/conftest.py b/delphi/tests/conftest.py index fc5e9dc73..110a048ed 100644 --- a/delphi/tests/conftest.py +++ b/delphi/tests/conftest.py @@ -56,7 +56,27 @@ def require_dynamodb( try: client.list_tables(Limit=1) except Exception as exc: - pytest.fail(f"DynamoDB is not available at {endpoint}: {exc}") + # In CI, DynamoDB is a provisioned service — its absence is an + # infrastructure failure that must fail LOUDLY (a silent skip would + # disable the only end-to-end gate; the 2026-07-05 consensus-float + # crash was caught precisely because CI runs this). + # Locally, DynamoDB is opt-in (e.g. + # `docker run --rm -d -p 8002:8000 amazon/dynamodb-local` + + # `DYNAMODB_ENDPOINT=http://localhost:8002`) — skip gracefully so + # the e2e test no longer needs a blanket --ignore in local runs. + # GITHUB_ACTIONS, not CI: local supply-chain wrappers (pmg) inject + # CI=true into wrapped package-manager runs, which would force the + # loud-fail path on developer machines (observed 2026-07-05). + msg = f"DynamoDB is not available at {endpoint}: {exc}" + if os.environ.get("GITHUB_ACTIONS"): + pytest.fail(msg) + pytest.skip( + f"{msg} — to run this test locally, start DynamoDB and point the " + "test at it:\n" + " docker run --rm -d --name delphi-test-dynamo -p 8002:8000 " + "amazon/dynamodb-local\n" + " DYNAMODB_ENDPOINT=http://localhost:8002 uv run pytest " + ) def require_s3( diff --git a/delphi/tests/test_benchmarks_importable.py b/delphi/tests/test_benchmarks_importable.py new file mode 100644 index 000000000..b3ba34af2 --- /dev/null +++ b/delphi/tests/test_benchmarks_importable.py @@ -0,0 +1,21 @@ +"""Benchmarks must stay importable as the production API evolves. + +PR 14a deleted the scalar repness functions; `bench_repness.py` still +imported `comment_stats`, so running any benchmark in that module crashed +with ImportError. Plain import tests catch this class of drift at CI time +(Copilot review 2026-07-04, g2). +""" + +import importlib + +import pytest + + +@pytest.mark.parametrize('module_name', [ + 'polismath.benchmarks.bench_repness', + 'polismath.benchmarks.bench_pca', + 'polismath.benchmarks.bench_update_votes', + 'polismath.benchmarks.benchmark_utils', +]) +def test_benchmark_module_imports(module_name): + importlib.import_module(module_name) diff --git a/delphi/tests/test_clusters.py b/delphi/tests/test_clusters.py index 8918106b6..96cceaa42 100644 --- a/delphi/tests/test_clusters.py +++ b/delphi/tests/test_clusters.py @@ -17,7 +17,7 @@ assign_points_to_clusters, update_cluster_centers, filter_empty_clusters, cluster_step, most_distal, split_cluster, clean_start_clusters, kmeans, distance_matrix, silhouette, clusters_to_dict, clusters_from_dict, - cluster_dataframe + cluster_dataframe, calculate_silhouette_sklearn ) @@ -496,6 +496,37 @@ def test_silhouette_edge_cases(self): assert silhouette(data, singleton_clusters) == 0.0 +class TestCalculateSilhouetteSklearn: + """Tests for the sklearn-backed calculate_silhouette_sklearn helper. + + sklearn's silhouette_score requires 2 <= n_labels <= n_samples - 1. The + group-clustering k-selection loop can feed it as many labels as samples + (e.g. only two base clusters -> k=2 group clustering => 2 points / 2 + labels), which sklearn rejects with ValueError. The helper must treat any + n_labels >= n_samples clustering as undefined and return the neutral 0.0 + sentinel instead of raising. (Regression: powerit PCA collapsing a small + conversation to two base clusters crashed recompute; see #2591.) + """ + + def test_two_samples_two_labels_returns_zero_not_raise(self): + data = np.array([[0.0, 0.0], [1.0, 1.0]]) + labels = np.array([0, 1]) # n_labels (2) >= n_samples (2) + assert calculate_silhouette_sklearn(data, labels) == 0.0 + + def test_single_label_still_returns_zero(self): + data = np.array([[0.0, 0.0], [1.0, 1.0], [2.0, 2.0]]) + assert calculate_silhouette_sklearn(data, np.array([0, 0, 0])) == 0.0 + + def test_valid_clustering_still_scores(self): + # 3 samples, 2 labels is the smallest sklearn-valid case; the guard + # must NOT swallow it — a real coefficient in [-1, 1] is returned. + data = np.array([[0.0, 0.0], [0.1, 0.1], [5.0, 5.0]]) + labels = np.array([0, 0, 1]) + score = calculate_silhouette_sklearn(data, labels) + assert -1.0 <= score <= 1.0 + assert score != 0.0 # a genuinely computed, non-sentinel score + + class TestClusterSerialization: """Tests for cluster serialization functions.""" diff --git a/delphi/tests/test_discrepancy_fixes.py b/delphi/tests/test_discrepancy_fixes.py index 9e7390d9e..62bf63da4 100644 --- a/delphi/tests/test_discrepancy_fixes.py +++ b/delphi/tests/test_discrepancy_fixes.py @@ -29,6 +29,7 @@ import math import numpy as np +import pandas as pd import pytest import pytest_check as check @@ -39,11 +40,28 @@ Z_95, z_score_sig_90, z_score_sig_95, - prop_test, - two_prop_test, - repness_metric, - finalize_cmt_stats, + prop_test_vectorized, + two_prop_test_vectorized, + # D10 selection helpers (PR 8) + passes_by_test, + beats_best_by_test, + beats_best_agr, + select_rep_comments_df, + _assemble_rep_comments, + # D11 consensus helpers (PR 9) + consensus_stats_df, + select_consensus_comments_df, ) +from polismath.pca_kmeans_rep.pca import ( + pca_project_cmnts, + compute_comment_extremity, +) +from polismath.conversation.conversation import ( + importance_metric, + priority_metric, + META_PRIORITY, +) +from polismath.utils.general import AGREE, DISAGREE from polismath.regression import get_dataset_files, get_blob_variants from polismath.regression.datasets import discover_datasets from conftest import _get_requested_datasets, make_dataset_params, parse_dataset_blob_id @@ -584,14 +602,21 @@ def test_repness_not_empty(self, conv, dataset_name): check.greater(len(repness['comment_repness']), 0, "comment_repness should not be empty") - @pytest.mark.xfail(reason="D5/D6: z-values differ → different significance decisions → different sets") - def test_significance_sets_match_clojure(self, conv, clojure_blob, dataset_name): + def test_significance_sets_match_clojure(self, request, conv, clojure_blob, dataset_name): """Post-significance-filtering comment sets should match Clojure per group. Both sides apply z-sig-90? to their z-values and select top comments. - With D9 the gate semantics match (>, no abs), but the z-values - themselves differ until D5 (prop test) and D6 (two-prop test) are fixed. + biodiversity-cold_start matches exactly since the gid label-swap fix + (2026-07-05) and gates; other variants remain xfailed on residual + per-(gid, tid) group-membership/stat divergence. """ + if request.node.callspec.id != 'biodiversity-cold_start': + request.applymarker(pytest.mark.xfail( + raises=AssertionError, + strict=False, + reason="residual per-(gid, tid) group-membership/stat " + "divergence (gid label swap fixed 2026-07-05; " + "biodiversity-cold_start gates)")) clojure_repness = clojure_blob.get('repness', {}) if not clojure_repness: pytest.skip("No repness in Clojure blob") @@ -618,7 +643,11 @@ def test_significance_sets_match_clojure(self, conv, clojure_blob, dataset_name) check.equal(len(mismatches), 0, f"{len(mismatches)} groups differ in selected rep comments") - @pytest.mark.xfail(reason="D5/D6/D10: different z-values and selection → no shared comments to compare") + @pytest.mark.xfail(reason="residual per-(gid, tid) group-membership divergence: the gid " + "0↔1 label swap was FIXED 2026-07-05 (group size re-sort removed) " + "and did not resolve this test on any variant — groups contain " + "slightly different participants, so exact z-values differ. " + "Deferred to clustering-membership / sequential-parity work.") def test_z_values_match_clojure(self, conv, clojure_blob, dataset_name): """Z-score values for shared rep comments should match Clojure. @@ -698,29 +727,31 @@ class TestD5ProportionTest: """ def test_prop_test_matches_clojure_formula(self): - """prop_test(succ, n) should match Clojure's formula for known inputs.""" - test_cases = [ - (12, 13), # High success rate - (5, 8), # Moderate - (0, 10), # All failures - (10, 10), # All successes - (1, 2), # Tiny sample - (50, 100), # Larger sample - (0, 1), # Single trial, no success - (1, 1), # Single trial, success - ] - for succ, n in test_cases: - # Clojure formula: 2 * sqrt(n+1) * ((succ+1)/(n+1) - 0.5) - expected = 2 * math.sqrt(n + 1) * ((succ + 1) / (n + 1) - 0.5) - result = prop_test(succ, n) - check.almost_equal(result, expected, abs=1e-10, - msg=f"prop_test({succ}, {n}): got {result:.6f}, expected {expected:.6f}") - - def test_prop_test_edge_cases(self): - """prop_test n=0: no short-circuit, +1 pseudocount yields 1.0 (Clojure parity).""" - # Clojure stats.clj:10-15 has no n=0 guard. After (map inc ...), (0, 0) - # becomes (1, 1), giving 2*sqrt(1)*(1/1 - 0.5) = 1.0. - assert prop_test(0, 0) == 1.0 + """prop_test_vectorized(succ, n) should match Clojure's formula for known + inputs, including the n=0 boundary (no short-circuit; +1 pseudocount → 1.0).""" + # (succ, n, label_for_diagnostic) + cases = pd.DataFrame([ + (12, 13, "high success rate"), + (5, 8, "moderate"), + (0, 10, "all failures"), + (10, 10, "all successes"), + (1, 2, "tiny sample"), + (50, 100, "larger sample"), + (0, 1, "single trial, no success"), + (1, 1, "single trial, success"), + (0, 0, "n=0 boundary (no short-circuit; +1 pseudocount → 1.0)"), + ], columns=['succ', 'n', 'label']) + + # Clojure formula: 2 * sqrt(n+1) * ((succ+1)/(n+1) - 0.5) + cases['expected'] = (2 * np.sqrt(cases['n'] + 1) + * ((cases['succ'] + 1) / (cases['n'] + 1) - 0.5)) + cases['actual'] = prop_test_vectorized(cases['succ'], cases['n']) + cases['diff'] = (cases['actual'] - cases['expected']).abs() + + mismatches = cases[cases['diff'] > 1e-10] + assert mismatches.empty, ( + f"{len(mismatches)}/{len(cases)} prop_test_vectorized mismatches:\n" + + mismatches.to_string(index=False)) def test_clojure_pat_values_consistent_with_formula(self, clojure_blob, dataset_name): """Sanity check: Clojure's p-test values match the documented formula.""" @@ -752,7 +783,13 @@ def test_clojure_pat_values_consistent_with_formula(self, clojure_blob, dataset_ print(f"[{dataset_name}] pat consistency: {total - mismatches}/{total} match formula (max_diff={max_diff:.4f})") check.equal(mismatches, 0, f"Clojure p-test values don't match formula for {mismatches}/{total}") - @pytest.mark.xfail(reason="D5/D10: prop test formula differs + no shared comments") + @pytest.mark.xfail(reason="gid 0↔1 label swap + group-membership divergence on cold_start " + "(per workflow Investigation C 2026-06-11 — PR #2524 D14 verified " + "only k count, not per-(gid, tid) memberships). D10 unlocks shared " + "comments but per-(gid, tid) pat values still differ because the " + "swapped/divergent groups contain different participants. Fix " + "requires canonical-group-id sorting or set-based comparison " + "infrastructure.") def test_pat_values_match_clojure_blob(self, conv, clojure_blob, dataset_name): """p-test (Clojure) vs pat (Python) for shared rep comments.""" clojure_repness = clojure_blob.get('repness', {}) @@ -813,56 +850,65 @@ def _clojure_two_prop_test(succ_in, succ_out, pop_in, pop_out): return (pi1 - pi2) / math.sqrt(pi_hat * (1 - pi_hat) * (1/p1 + 1/p2)) def test_two_prop_test_matches_clojure_formula(self): - """two_prop_test(succ_in, succ_out, pop_in, pop_out) should match Clojure.""" - # Test cases: (succ_in, succ_out, pop_in, pop_out) - test_cases = [ - (10, 15, 20, 30), # typical case - (0, 0, 10, 10), # no successes in either group - (5, 5, 10, 10), # identical groups - (10, 0, 10, 10), # all success in group, none outside - (1, 1, 1, 1), # minimal counts - (50, 20, 100, 200), # asymmetric sizes - (0, 10, 20, 30), # no success in group, some outside - ] - - for succ_in, succ_out, pop_in, pop_out in test_cases: - expected = self._clojure_two_prop_test(succ_in, succ_out, pop_in, pop_out) - result = two_prop_test(succ_in, succ_out, pop_in, pop_out) - check.almost_equal( - result, expected, abs=0.001, - msg=f"two_prop_test({succ_in},{succ_out},{pop_in},{pop_out}): " - f"got={result:.4f}, expected={expected:.4f}") - - def test_two_prop_test_edge_cases(self): - """Edge cases: pi_hat=1 returns 0; pop=0 with pop=0 short-circuit removed. - - Clojure (stats.clj:18-33) increments ALL four inputs by 1 (no special- - casing of pop=0). Each case below happens to return 0 because of the - pi_hat==1 guard, NOT because pop=0 — verify by tracing the math. - """ - # (5,5,0,10) → s1=6,s2=6,p1=1,p2=11 → pi_hat = 12/12 = 1.0 → 0 via guard - check.equal(two_prop_test(5, 5, 0, 10), 0.0) - # (5,5,10,0) → s1=6,s2=6,p1=11,p2=1 → pi_hat = 12/12 = 1.0 → 0 via guard - check.equal(two_prop_test(5, 5, 10, 0), 0.0) - # (0,0,0,0) → s1=1,s2=1,p1=1,p2=1 → pi_hat = 2/2 = 1.0 → 0 via guard - check.equal(two_prop_test(0, 0, 0, 0), 0.0) - # Real pop=0 (no pi_hat=1 collapse): (5,5,0,100) gives a large positive z, - # confirming the +1-pseudocount path runs instead of short-circuiting. - check.greater(two_prop_test(5, 5, 0, 100), 10.0, - "pop_in=0 should NOT short-circuit to 0; +1 pseudocount produces large positive z") + """two_prop_test_vectorized should match Clojure's per-row formula, including + edge cases that exercise the pi_hat==1 guard and the no-pop=0 short-circuit.""" + cases = pd.DataFrame([ + # (succ_in, succ_out, pop_in, pop_out, label) + (10, 15, 20, 30, "typical case"), + (0, 0, 10, 10, "no successes in either group"), + (5, 5, 10, 10, "identical groups"), + (10, 0, 10, 10, "all success in group, none outside"), + (1, 1, 1, 1, "minimal counts"), + (50, 20, 100, 200, "asymmetric sizes"), + (0, 10, 20, 30, "no success in group, some outside"), + # pi_hat==1 boundary cases (Clojure: returns 0; vectorized: NaN → 0.0) + (5, 5, 0, 10, "pop_in=0, succ saturates → pi_hat=1 guard"), + (5, 5, 10, 0, "pop_out=0, succ saturates → pi_hat=1 guard"), + (0, 0, 0, 0, "all zero → pi_hat=1 guard"), + ], columns=['succ_in', 'succ_out', 'pop_in', 'pop_out', 'label']) + + cases['expected'] = cases.apply( + lambda r: self._clojure_two_prop_test( + r['succ_in'], r['succ_out'], r['pop_in'], r['pop_out']), + axis=1) + cases['actual'] = two_prop_test_vectorized( + cases['succ_in'], cases['succ_out'], cases['pop_in'], cases['pop_out']) + cases['diff'] = (cases['actual'] - cases['expected']).abs() + + mismatches = cases[cases['diff'] > 1e-3] + assert mismatches.empty, ( + f"{len(mismatches)}/{len(cases)} two_prop_test mismatches:\n" + + mismatches.to_string(index=False)) + + # Pin the no-pop=0-short-circuit behavior: real pop=0 (no pi_hat=1 collapse) + # → (5,5,0,100) produces a large positive z, confirming the +1-pseudocount + # path runs instead of short-circuiting. + no_pi_hat_collapse = two_prop_test_vectorized( + pd.Series([5]), pd.Series([5]), pd.Series([0]), pd.Series([100])).iloc[0] + check.greater(no_pi_hat_collapse, 10.0, + "pop_in=0 should NOT short-circuit to 0 when pi_hat<1; " + "+1 pseudocount produces large positive z") def test_two_prop_test_pseudocount_effect(self): """Pseudocounts should shrink z-scores toward zero for small samples.""" - # With small n, the +1 pseudocount has a large effect - # succ=1, pop=1 → without pseudocount: p=1.0 (extreme) - # With pseudocount: (1+1)/(1+1) = 1.0, but denominator also shifts - result_small = two_prop_test(1, 0, 2, 2) - result_large = two_prop_test(100, 0, 200, 200) - # The large-sample z should be more extreme (less regularized) + # With small n, the +1 pseudocount has a large effect: + # succ=1, pop=1 → without pseudocount: p=1.0 (extreme); with pseudocount, + # both numerator and denominator shift. + results = two_prop_test_vectorized( + pd.Series([1, 100]), # succ_in: small, large + pd.Series([0, 0]), # succ_out: zero in both + pd.Series([2, 200]), # pop_in: small, large + pd.Series([2, 200]), # pop_out: small, large + ) + result_small, result_large = results.iloc[0], results.iloc[1] check.greater(abs(result_large), abs(result_small), "Large samples should produce more extreme z-scores than small ones") - @pytest.mark.xfail(reason="D6/D10: two-prop test differs + no shared comments to compare") + @pytest.mark.xfail(reason="residual per-(gid, tid) group-membership divergence: the gid " + "0↔1 label swap was FIXED 2026-07-05 (group size re-sort removed) " + "and did not resolve this test on any variant — groups contain " + "slightly different participants, so exact rat values differ. " + "Deferred to clustering-membership / sequential-parity work.") def test_rat_values_match_clojure_blob(self, conv, clojure_blob, dataset_name): """repness-test (Clojure) vs rat (Python) for shared rep comments. @@ -916,24 +962,39 @@ class TestD7RepnessMetric: """ def test_metric_formula_is_product(self): - """repness_metric should use product formula (ra * rat * pa * pat).""" - stats = { + """Pins the agree_metric/disagree_metric formula with hand-computed values. + + Clojure repness-metric (repness.clj:191-193): + (* repness repness-test p-success p-test) + Production code mirrors this in compute_group_comment_stats_df: + stats_df['agree_metric'] = stats_df['ra'] * stats_df['rat'] + * stats_df['pa'] * stats_df['pat'] + stats_df['disagree_metric'] = stats_df['rd'] * stats_df['rdt'] + * stats_df['pd'] * stats_df['pdt'] + Signed product — no abs(). Negative z-scores flip the sign. + """ + df = pd.DataFrame([{ 'pa': 0.8, 'pat': 2.5, 'ra': 1.3, 'rat': 1.8, 'pd': 0.2, 'pdt': -1.5, 'rd': 0.7, 'rdt': -0.9, - } - - # Clojure formula for agree: ra * rat * pa * pat - expected_agree = stats['ra'] * stats['rat'] * stats['pa'] * stats['pat'] - # Current Python formula: pa * (|pat| + |rat|) - current_python = stats['pa'] * (abs(stats['pat']) + abs(stats['rat'])) - - result = repness_metric(stats, 'a') - print(f"agree_metric: current={result:.4f}, expected(Clojure)={expected_agree:.4f}, current_formula={current_python:.4f}") - - check.almost_equal(result, expected_agree, abs=0.01, - msg=f"agree_metric should be ra*rat*pa*pat={expected_agree:.4f}, got {result:.4f}") - - @pytest.mark.xfail(reason="D7/D10: metric formula differs + no shared comments") + }]) + + agree_metric = (df['ra'] * df['rat'] * df['pa'] * df['pat']).iloc[0] + disagree_metric = (df['rd'] * df['rdt'] * df['pd'] * df['pdt']).iloc[0] + + # Hand-computed reference values. + check.almost_equal(agree_metric, 4.68, abs=1e-10, + msg=f"agree_metric (1.3 * 1.8 * 0.8 * 2.5) = 4.68, got {agree_metric}") + # Two negatives cancel — signed product. + check.almost_equal(disagree_metric, 0.189, abs=1e-10, + msg=f"disagree_metric (0.7 * -0.9 * 0.2 * -1.5) = 0.189, got {disagree_metric}") + + @pytest.mark.xfail(reason="gid 0↔1 label swap + group-membership divergence on cold_start " + "(per workflow Investigation C 2026-06-11 — PR #2524 D14 verified " + "only k count, not per-(gid, tid) memberships). D10 unlocks shared " + "comments but per-(gid, tid) repness metrics still differ because " + "the swapped/divergent groups contain different participants. Fix " + "requires canonical-group-id sorting or set-based comparison " + "infrastructure.") def test_repness_metric_matches_clojure_blob(self, conv, clojure_blob, dataset_name): """repness (Clojure) vs agree/disagree_metric (Python) for shared comments.""" clojure_repness = clojure_blob.get('repness', {}) @@ -979,98 +1040,54 @@ class TestD8FinalizeStats: Clojure uses simple rat > rdt → 'agree'; else → 'disagree' """ - def test_repful_uses_rat_vs_rdt(self): - """repful classification should use rat > rdt (Clojure logic). - - Case where the OLD Python 3-branch logic disagrees with Clojure: - pa > 0.5 AND ra > 1.0 → old Python says 'agree', - but rat < rdt → Clojure says 'disagree'. + def test_repful_classification_boundary(self): + """Pin the repful classification logic: agree iff rat > rdt (strict), else disagree. + + Production code (compute_group_comment_stats_df): + stats_df['repful'] = np.where(stats_df['rat'] > stats_df['rdt'], + 'agree', 'disagree') + Clojure (repness.clj:178): + (if (> rat rdt) :agree :disagree) + + Strict `>` — `rat == rdt` falls through to 'disagree'. Covers: + - rat < rdt → 'disagree' (case where old Python 3-branch wrongly said 'agree') + - rat > rdt → 'agree' (case where old Python wrongly said 'disagree') + - rat == rdt → 'disagree' (strict >, non-zero boundary) + - rat == rdt == 0 → 'disagree' (all-zero boundary, distinct from above) + - negative z-scores: comparison works on signed values (-0.5 > -2.0) """ - stats = { - 'pa': 0.6, 'pat': 1.0, 'ra': 1.2, 'rat': 0.5, - 'pd': 0.4, 'pdt': -0.5, 'rd': 0.8, 'rdt': 1.5, - 'agree_metric': 0.0, - 'disagree_metric': 0.0, - } - result = finalize_cmt_stats(stats) - # Clojure: rat (0.5) < rdt (1.5) → 'disagree' - check.equal(result['repful'], 'disagree', - f"repful should be 'disagree' when rat < rdt, got '{result['repful']}'") - - def test_repful_uses_rat_vs_rdt_inverse(self): - """Inverse case: Clojure says 'agree' where old Python 3-branch said 'disagree'. - - pd > 0.5 AND rd > 1.0 → old Python says 'disagree', but rat > rdt → Clojure 'agree'. + cases = pd.DataFrame([ + (0.5, 1.5, 'disagree', "rat < rdt: old Python 3-branch would say agree"), + (1.5, 0.5, 'agree', "rat > rdt: old Python 3-branch would say disagree"), + (1.5, 1.5, 'disagree', "rat == rdt non-zero (strict >)"), + (0.0, 0.0, 'disagree', "rat == rdt == 0 boundary"), + (-0.5, -2.0, 'agree', "negative z-scores: -0.5 > -2.0"), + ], columns=['rat', 'rdt', 'expected', 'label']) + + cases['actual'] = np.where(cases['rat'] > cases['rdt'], 'agree', 'disagree') + + mismatches = cases[cases['actual'] != cases['expected']] + assert mismatches.empty, ( + f"{len(mismatches)}/{len(cases)} repful mismatches:\n" + + mismatches.to_string(index=False)) + + def test_repful_matches_clojure_blob(self, request, conv, clojure_blob, dataset_name): + """repful-for (Clojure) vs repful (Python) for shared rep comments. + + Gates on 9/11 variants since the gid label-swap fix (2026-07-05 + removal of the group size re-sort). Residual known-bad: two + incremental variants with deeper trajectory divergence + (pakistan-incremental: Clojure blob PCA computed on a comment + subset; vw-incremental: in-conv trajectory divergence) — deferred + to the sequential-parity work. """ - stats = { - 'pa': 0.4, 'pat': -0.5, 'ra': 0.8, 'rat': 1.5, - 'pd': 0.6, 'pdt': 1.0, 'rd': 1.2, 'rdt': 0.5, - 'agree_metric': 0.0, - 'disagree_metric': 0.0, - } - result = finalize_cmt_stats(stats) - check.equal(result['repful'], 'agree', - f"repful should be 'agree' when rat > rdt, got '{result['repful']}'") - - def test_repful_strict_greater_than(self): - """Clojure uses strict (> rat rdt) — when rat == rdt, falls through to disagree.""" - stats = { - 'pa': 0.5, 'pat': 0.0, 'ra': 1.0, 'rat': 1.5, - 'pd': 0.5, 'pdt': 0.0, 'rd': 1.0, 'rdt': 1.5, - 'agree_metric': 0.0, - 'disagree_metric': 0.0, - } - result = finalize_cmt_stats(stats) - # Clojure: (> 1.5 1.5) is false → :disagree branch - check.equal(result['repful'], 'disagree', - f"rat == rdt should yield 'disagree' (strict >), got '{result['repful']}'") - - def test_repful_negative_z_scores(self): - """Comparison works with negative z-scores: e.g. rat=-0.5 > rdt=-2.0 → 'agree'.""" - stats = { - 'pa': 0.3, 'pat': -1.0, 'ra': 0.5, 'rat': -0.5, - 'pd': 0.7, 'pdt': -2.0, 'rd': 1.5, 'rdt': -2.0, - 'agree_metric': 0.0, - 'disagree_metric': 0.0, - } - result = finalize_cmt_stats(stats) - # -0.5 > -2.0 → 'agree' - check.equal(result['repful'], 'agree', - f"rat=-0.5 > rdt=-2.0 should yield 'agree', got '{result['repful']}'") - - def test_finalize_cmt_stats_keeps_metrics(self): - """Regression: finalize_cmt_stats must still populate agree_metric / disagree_metric.""" - stats = { - 'pa': 0.8, 'pat': 3.0, 'ra': 1.5, 'rat': 2.0, - 'pd': 0.2, 'pdt': -1.0, 'rd': 0.5, 'rdt': -0.5, - } - result = finalize_cmt_stats(stats) - check.is_in('agree_metric', result) - check.is_in('disagree_metric', result) - check.is_in('repful', result) - # Sanity: with rat=2.0 > rdt=-0.5, repful is 'agree' - check.equal(result['repful'], 'agree') - - def test_repful_both_zero(self): - """Boundary: rat == rdt == 0 should fall through to 'disagree' (strict >). - - Distinct from `test_repful_strict_greater_than` (rat==rdt==1.5): - this case pins the all-zero boundary specifically. - """ - stats = { - 'pa': 0.5, 'pat': 0.0, 'ra': 1.0, 'rat': 0.0, - 'pd': 0.5, 'pdt': 0.0, 'rd': 1.0, 'rdt': 0.0, - 'agree_metric': 0.0, - 'disagree_metric': 0.0, - } - result = finalize_cmt_stats(stats) - # Clojure: (> 0 0) is false → :disagree branch - check.equal(result['repful'], 'disagree', - f"rat == rdt == 0 should yield 'disagree' (strict >), got '{result['repful']}'") - - @pytest.mark.xfail(reason="D8/D10: repful logic differs + no shared comments") - def test_repful_matches_clojure_blob(self, conv, clojure_blob, dataset_name): - """repful-for (Clojure) vs repful (Python) for shared rep comments.""" + if request.node.callspec.id in ('vw-incremental', 'pakistan-incremental'): + request.applymarker(pytest.mark.xfail( + raises=AssertionError, + strict=False, + reason="residual incremental trajectory divergence (gid " + "label swap fixed 2026-07-05; sequential-parity " + "work)")) clojure_repness = clojure_blob.get('repness', {}) if not clojure_repness: pytest.skip("No repness in Clojure blob") @@ -1114,9 +1131,23 @@ class TestD10RepCommentSelection: Clojure selects up to 5 total, agrees first, with beats-best-by-test logic """ - @pytest.mark.xfail(reason="D10: Different selection logic than Clojure") - def test_rep_comments_match_clojure(self, conv, clojure_blob, dataset_name): - """Selected representative comments per group should match Clojure.""" + def test_rep_comments_match_clojure(self, request, conv, clojure_blob, dataset_name): + """Selected representative comments per group should match Clojure. + + biodiversity-cold_start matches exactly since the gid label-swap + fix (2026-07-05) and gates. Other variants remain xfailed: the + selection is highly sensitive to residual per-(gid, tid) + group-membership/stat divergence. D10 selection LOGIC is verified + by TestD10PassesByTest, TestD10BeatsBestByTest, TestD10BeatsBestAgr, + TestD10SelectRepCommentsBoundary. + """ + if request.node.callspec.id != 'biodiversity-cold_start': + request.applymarker(pytest.mark.xfail( + raises=AssertionError, + strict=False, + reason="residual per-(gid, tid) group-membership/stat " + "divergence (gid label swap fixed 2026-07-05; " + "biodiversity-cold_start gates)")) clojure_repness = clojure_blob.get('repness', {}) if not clojure_repness: pytest.skip("No repness in Clojure blob") @@ -1147,6 +1178,504 @@ def test_rep_comments_match_clojure(self, conv, clojure_blob, dataset_name): f"Only {matching_groups}/{total_groups} groups have matching rep comments") +# ---------------------------------------------------------------------------- +# D10 — Synthetic unit tests for the new selection helpers +# ---------------------------------------------------------------------------- +# +# Pin the Clojure-parity semantics of `passes_by_test`, `beats_best_by_test`, +# `beats_best_agr`, and `select_rep_comments_df`. Synthetic 1-group fixtures +# only — no real datasets, no Clojure blob dependency. +# +# References: +# - Clojure `select-rep-comments`: math/src/polismath/math/repness.clj:212-281 +# - Helpers `passes-by-test?` :165, `beats-best-by-test?` :133, +# `beats-best-agr?` :142, `finalize-cmt-stats` :173, `repness-metric` :191. +# ---------------------------------------------------------------------------- + + +def _stats_row(tid, na, nd, pa, pd_, pat, pdt, ra, rd, rat, rdt, *, ns=None, + agree_metric=None, disagree_metric=None, repful=None, group_id=0): + """Build a single stats DataFrame row matching the schema produced by + `compute_group_comment_stats_df`. Defaults derived per Clojure recipe.""" + if ns is None: + ns = na + nd + if agree_metric is None: + agree_metric = ra * rat * pa * pat + if disagree_metric is None: + disagree_metric = rd * rdt * pd_ * pdt + if repful is None: + repful = 'agree' if rat > rdt else 'disagree' + return { + 'group_id': group_id, 'comment': tid, + 'na': na, 'nd': nd, 'ns': ns, + 'pa': pa, 'pd': pd_, + 'pat': pat, 'pdt': pdt, + 'ra': ra, 'rd': rd, + 'rat': rat, 'rdt': rdt, + 'agree_metric': agree_metric, 'disagree_metric': disagree_metric, + 'repful': repful, + } + + +class TestD10PassesByTest: + """`passes-by-test?` (repness.clj:165) — OR'd on (rat, pat) and (rdt, pdt). + + NO probability threshold (`pa >= 0.5` was a Python-only over-restriction + in the pre-D10 botched port — Clojure has no such gate).""" + + def test_agree_side_significant_passes(self): + row = _stats_row(1, na=8, nd=2, pa=0.75, pd_=0.25, pat=2.0, pdt=-2.0, + ra=2.0, rd=0.5, rat=2.0, rdt=-2.0) # rat,pat > Z_90 + assert passes_by_test(row) + + def test_disagree_side_significant_passes(self): + row = _stats_row(2, na=2, nd=8, pa=0.25, pd_=0.75, pat=-2.0, pdt=2.0, + ra=0.5, rd=2.0, rat=-2.0, rdt=2.0) # rdt,pdt > Z_90 + assert passes_by_test(row) + + def test_neither_side_significant_fails(self): + row = _stats_row(3, na=5, nd=5, pa=0.5, pd_=0.5, pat=0.5, pdt=0.5, + ra=1.0, rd=1.0, rat=0.5, rdt=0.5) + assert not passes_by_test(row) + + def test_no_pa_threshold_gate(self): + """Pre-D10 Python added `pa >= 0.5` — Clojure has no such gate. A row + with pa=0.4 that's otherwise significant on the agree side must pass.""" + row = _stats_row(4, na=4, nd=6, pa=0.42, pd_=0.58, pat=2.0, pdt=-2.0, + ra=2.0, rd=0.5, rat=2.0, rdt=-2.0) + assert passes_by_test(row), "no pa>=0.5 gate (Clojure parity)" + + +class TestD10BeatsBestByTest: + """`beats-best-by-test?` (repness.clj:133) — max(rat, rdt) > current_best_z.""" + + def test_none_best_always_beats(self): + row = _stats_row(1, na=5, nd=2, pa=0.6, pd_=0.4, pat=1.0, pdt=-1.0, + ra=1.2, rd=0.8, rat=2.0, rdt=0.5) + assert beats_best_by_test(row, None) + + def test_max_rat_rdt_used(self): + row = _stats_row(1, na=5, nd=2, pa=0.6, pd_=0.4, pat=1.0, pdt=-1.0, + ra=1.2, rd=0.8, rat=2.0, rdt=0.5) + # max = 2.0 + assert beats_best_by_test(row, 1.5) + assert not beats_best_by_test(row, 2.5) + + def test_strict_greater_than(self): + row = _stats_row(1, na=5, nd=2, pa=0.6, pd_=0.4, pat=1.0, pdt=-1.0, + ra=1.2, rd=0.8, rat=2.0, rdt=0.5) + assert not beats_best_by_test(row, 2.0), "strict > (Clojure parity)" + + +class TestD10BeatsBestAgr: + """`beats-best-agr?` (repness.clj:142) — 4-branch agree priority logic.""" + + def test_na_nd_zero_always_rejected(self): + """Branch 1: (= 0 na nd) → false. Unvoted comments excluded from best-agree + regardless of stats.""" + unvoted = _stats_row(1, na=0, nd=0, pa=0.5, pd_=0.5, pat=1.0, pdt=1.0, + ra=1.0, rd=1.0, rat=1.0, rdt=1.0) + assert not beats_best_agr(unvoted, None) + other = _stats_row(2, na=5, nd=2, pa=0.6, pd_=0.4, pat=1.0, pdt=-1.0, + ra=1.2, rd=0.8, rat=2.0, rdt=0.5) + assert not beats_best_agr(unvoted, other) + + def test_branch_2_ra_gt_1_uses_4way_product(self): + """Branch 2: current_best AND current_best.ra > 1.0 → compare ra*rat*pa*pat.""" + big_ra_best = _stats_row(1, na=10, nd=0, pa=0.9, pd_=0.1, pat=2.0, pdt=-2.0, + ra=2.0, rd=0.5, rat=2.0, rdt=-2.0) + # ra*rat*pa*pat = 2.0*2.0*0.9*2.0 = 7.2 + bigger = _stats_row(2, na=15, nd=0, pa=0.94, pd_=0.06, pat=3.0, pdt=-3.0, + ra=2.5, rd=0.4, rat=2.5, rdt=-2.5) + # 2.5*2.5*0.94*3.0 = 17.625 > 7.2 + smaller = _stats_row(3, na=5, nd=0, pa=0.86, pd_=0.14, pat=1.5, pdt=-1.5, + ra=1.5, rd=0.6, rat=1.5, rdt=-1.5) + # 1.5*1.5*0.86*1.5 ≈ 2.9 < 7.2 + assert beats_best_agr(bigger, big_ra_best) + assert not beats_best_agr(smaller, big_ra_best) + + def test_branch_3_ra_le_1_uses_pa_pat_product(self): + """Branch 3: current_best AND current_best.ra <= 1.0 → compare pa*pat only.""" + weak_best = _stats_row(1, na=5, nd=4, pa=0.55, pd_=0.45, pat=1.0, pdt=-1.0, + ra=0.9, rd=1.1, rat=1.0, rdt=-1.0) + # pa*pat = 0.55 + bigger = _stats_row(2, na=6, nd=2, pa=0.7, pd_=0.3, pat=1.2, pdt=-1.2, + ra=1.0, rd=1.0, rat=0.5, rdt=-0.5) + # pa*pat = 0.84 > 0.55 + assert beats_best_agr(bigger, weak_best) + + def test_branch_4_no_best_accepts_via_z_sig_pat(self): + """Branch 4 / no current_best: accept if z90(pat) is true.""" + row = _stats_row(1, na=6, nd=4, pa=0.58, pd_=0.42, pat=1.5, pdt=-1.5, + ra=0.9, rd=1.1, rat=1.0, rdt=-1.0) # pat=1.5 > Z_90=1.2816 + assert beats_best_agr(row, None) + + def test_branch_4_no_best_accepts_via_ra_gt_1_and_pa_gt_half(self): + """Branch 4 / no current_best: accept if ra > 1.0 AND pa > 0.5 + (even when pat not significant).""" + row = _stats_row(1, na=5, nd=4, pa=0.55, pd_=0.45, pat=0.5, pdt=-0.5, + ra=1.2, rd=0.8, rat=0.5, rdt=-0.5) + assert beats_best_agr(row, None) + + def test_branch_4_no_best_rejects_when_neither(self): + """Branch 4 / no current_best: reject if neither z90(pat) nor + (ra > 1.0 AND pa > 0.5).""" + row = _stats_row(1, na=4, nd=5, pa=0.45, pd_=0.55, pat=0.5, pdt=0.5, + ra=0.8, rd=1.2, rat=0.5, rdt=0.5) + assert not beats_best_agr(row, None) + + +class TestD10SelectRepCommentsBoundary: + """`select_rep_comments_df` Clojure-parity boundaries.""" + + def test_empty_input_returns_empty(self): + result = _assemble_rep_comments(pd.DataFrame()) + assert len(result) == 0 + + def test_single_unvoted_row_falls_through_to_best(self): + """`beats_best_by_test` does NOT filter na=nd=0; only `beats_best_agr` + Branch 1 does. So a sole na=nd=0 row still ends up in the `:best` + fallback (Clojure parity — repness.clj:244-247). The `:best_agree` + slot stays empty (Branch 1 rejects). Output is [best], not []. + """ + rows = [ + _stats_row(1, na=0, nd=0, pa=0.5, pd_=0.5, pat=0.0, pdt=0.0, + ra=1.0, rd=1.0, rat=0.0, rdt=0.0), + ] + result = _assemble_rep_comments(pd.DataFrame(rows)) + assert len(result) == 1 + assert result[0]['comment_id'] == 1 + # NOT the best-agree slot (Branch 1 rejected na=nd=0). + assert 'best_agree' not in result[0] + + def test_sufficient_empty_best_agree_only(self): + """Sufficient empty + best_agree exists → returns [best_agree_finalized].""" + # passes_by_test fails (pat=pdt below z90, rat=rdt below z90). + # beats_best_agr triggers via Branch 4: z90(pat) is true (pat=1.5). + rows = [ + _stats_row(1, na=6, nd=4, pa=0.58, pd_=0.42, pat=1.5, pdt=-1.5, + ra=0.9, rd=1.1, rat=1.0, rdt=-1.0), + # Filler row to make this not trivially the only one — also fails + # passes_by_test and beats_best_agr. + _stats_row(2, na=3, nd=5, pa=0.4, pd_=0.6, pat=-0.5, pdt=0.5, + ra=0.8, rd=1.2, rat=-0.3, rdt=0.3), + ] + result = _assemble_rep_comments(pd.DataFrame(rows)) + assert len(result) == 1 + # New select_rep_comments_df returns (rep_df, best_agree_dict); the + # `_assemble_rep_comments` wrapper returns the flat List[Dict] + # (decision S2). + row = result[0] + assert row['comment_id'] == 1 + # best_agree flag emitted in Python-convention key naming (decision S1 / Q2). + assert row.get('best_agree') is True, "best_agree slot should be flagged" + assert row.get('n_agree') == 6, "n_agree should be na from the raw best-agree row" + + def test_take_5_cap_agrees_before_disagrees(self): + """7 sufficient candidates (4 agree-passing, 3 disagree-passing). + Sort by metric desc → take 5 → agrees-before-disagrees.""" + rows = [ + # 4 agree-passing, large to small agree_metric + _stats_row(1, na=9, nd=1, pa=0.83, pd_=0.17, pat=2.5, pdt=-2.5, + ra=2.0, rd=0.5, rat=2.5, rdt=-2.5), # agree_metric ~10.4 + _stats_row(2, na=8, nd=2, pa=0.75, pd_=0.25, pat=2.0, pdt=-2.0, + ra=1.8, rd=0.55, rat=2.0, rdt=-2.0), # ~5.4 + _stats_row(3, na=7, nd=3, pa=0.67, pd_=0.33, pat=1.5, pdt=-1.5, + ra=1.5, rd=0.6, rat=1.5, rdt=-1.5), # ~2.27 + _stats_row(4, na=6, nd=4, pa=0.58, pd_=0.42, pat=1.3, pdt=-1.3, + ra=1.3, rd=0.7, rat=1.3, rdt=-1.3), # ~1.27 + # 3 disagree-passing, large to small disagree_metric + _stats_row(5, na=1, nd=9, pa=0.17, pd_=0.83, pat=-2.5, pdt=2.5, + ra=0.5, rd=2.0, rat=-2.5, rdt=2.5), # ~10.4 + _stats_row(6, na=2, nd=8, pa=0.25, pd_=0.75, pat=-2.0, pdt=2.0, + ra=0.55, rd=1.8, rat=-2.0, rdt=2.0), # ~5.4 + _stats_row(7, na=3, nd=7, pa=0.33, pd_=0.67, pat=-1.5, pdt=1.5, + ra=0.6, rd=1.5, rat=-1.5, rdt=1.5), # ~2.27 + ] + result = _assemble_rep_comments(pd.DataFrame(rows)) + assert len(result) == 5 + # Agrees-before-disagrees: all agrees precede all disagrees in the output. + repful_values = [r['repful'] for r in result] # List[Dict] per S2 + last_agree_idx = -1 + first_disagree_idx = len(repful_values) + for i, v in enumerate(repful_values): + if v == 'agree': + last_agree_idx = i + elif v == 'disagree' and first_disagree_idx == len(repful_values): + first_disagree_idx = i + assert last_agree_idx < first_disagree_idx, \ + f"agrees must come before disagrees, got order: {repful_values}" + + def test_take_5_eviction_when_best_agree_outside_sufficient(self): + """The eviction edge case (flagged in PLAN for future review). + + Sufficient has 5 entries, best_agree is OUTSIDE sufficient (failed + passes_by_test). Prepending best_agree pushes total to 6, take(5) drops + the lowest-metric sufficient entry. + + Fixture design (subtle): + - tid 1: best_agree slot. Fails passes_by_test (rat=1.0, pat=1.0 + both below Z_90=1.2816). Branch 4 accepts via ra>1.0 AND pa>0.5. + Its agree_metric (ra*rat*pa*pat = 0.9) is LARGER than every + sufficient row's metric, so subsequent rows can't beat it via + Branch 2. + - tid 2-6: pass passes_by_test (rat,pat at 1.3 > Z_90), with + DECREASING agree_metrics all SMALLER than 0.9, so Branch 2 keeps + tid 1 as best_agree throughout. + + Expected: tid 1 prepended, sort gives [tid 5, 4, 3, 2, 6] (desc by + agree_metric), take(5) drops tid 6 (smallest metric). + + See PLAN.md "Pending — needs team discussion": take-5 eviction. + """ + rows = [ + # best_agree slot: fails passes_by_test, qualifies via Branch 4 + # (ra=1.5>1 AND pa=0.6>0.5). agree_metric = 1.5*1.0*0.6*1.0 = 0.9. + _stats_row(1, na=6, nd=4, pa=0.6, pd_=0.4, pat=1.0, pdt=-1.0, + ra=1.5, rd=0.7, rat=1.0, rdt=-1.0), + # 5 sufficient rows, each with agree_metric < 0.9. + # ra=1.0, rat=1.3, pa=0.5, pat=1.3 → agree_metric = 0.845 + _stats_row(2, na=5, nd=5, pa=0.5, pd_=0.5, pat=1.3, pdt=-1.3, + ra=1.0, rd=1.0, rat=1.3, rdt=-1.3), + # ra=0.9, rat=1.3, pa=0.4, pat=1.3 → agree_metric = 0.609 + _stats_row(3, na=4, nd=6, pa=0.4, pd_=0.6, pat=1.3, pdt=-1.3, + ra=0.9, rd=1.1, rat=1.3, rdt=-1.3), + # ra=0.8, rat=1.3, pa=0.3, pat=1.3 → agree_metric = 0.406 + _stats_row(4, na=3, nd=7, pa=0.3, pd_=0.7, pat=1.3, pdt=-1.3, + ra=0.8, rd=1.2, rat=1.3, rdt=-1.3), + # ra=0.7, rat=1.3, pa=0.2, pat=1.3 → agree_metric = 0.237 + _stats_row(5, na=2, nd=8, pa=0.2, pd_=0.8, pat=1.3, pdt=-1.3, + ra=0.7, rd=1.3, rat=1.3, rdt=-1.3), + # ra=0.6, rat=1.3, pa=0.15, pat=1.3 → agree_metric = 0.152 (smallest, evicted) + _stats_row(6, na=1, nd=9, pa=0.15, pd_=0.85, pat=1.3, pdt=-1.3, + ra=0.6, rd=1.4, rat=1.3, rdt=-1.3), + ] + result = _assemble_rep_comments(pd.DataFrame(rows)) + assert len(result) == 5 + tids = [r['comment_id'] for r in result] + # best_agree (tid 1) prepended at position 0. + assert tids[0] == 1, f"best-agree slot at position 0, got {tids[0]}" + # Tid 6 (smallest sufficient metric) evicted. + assert 6 not in tids, f"lowest-metric sufficient should be evicted, got {tids}" + # Rest are tids 2-5 in some agree-first ordering. + assert set(tids[1:]) == {2, 3, 4, 5}, f"expected tids 2-5 to remain, got {tids[1:]}" + # best_agree flag on position 0. + assert result[0].get('best_agree') is True + assert result[0].get('n_agree') == 6 # raw na from tid 1 + + +class TestD10TestGaps: + """Additional D10 coverage filling gaps identified in decisions D10.8. + + These pin behaviours not previously asserted: + - mod_out filtering on the best-agree path. + - Deterministic tiebreak (lowest tid wins) on `beats_best_by_test` + max(rat,rdt) ties and on the `_sort_key` agree_metric ties. + - Disagree-only path through assembly + agrees-before-disagrees no-op. + - All-uninformative `ns=0` rows: passes_by_test fails, Branch 1 rejects + best_agree; best may still get set via Branch 4 / beats_best_by_test. + - Negative-ra rows handled correctly by Branch 2 (signed 4-way product). + """ + + # --- Deliverable 3: deterministic max(rat, rdt) tiebreak ------------------ + + def test_tied_max_rt_uses_deterministic_tiebreak(self): + """Two rows with identical max(rat, rdt) — strict `>` means the FIRST + iterated row wins. Sorting by `comment` (tid) ascending makes that + the LOWER tid (Clojure named-matrix insertion order parity, decision + D10.8.1).""" + rows = [ + # Pass through `best` slot (neither passes passes_by_test — + # rat/rdt below Z_90), tied max(rat, rdt) = 1.0. + # Insert in REVERSE tid order to prove we sort, not just take input order. + _stats_row(7, na=4, nd=2, pa=0.55, pd_=0.45, pat=0.5, pdt=-0.5, + ra=1.0, rd=1.0, rat=1.0, rdt=-1.0), + _stats_row(3, na=4, nd=2, pa=0.55, pd_=0.45, pat=0.5, pdt=-0.5, + ra=1.0, rd=1.0, rat=1.0, rdt=-1.0), + ] + result = _assemble_rep_comments(pd.DataFrame(rows)) + assert len(result) == 1 + # Tid 3 (lower) wins the `best` slot under the tid-ascending tiebreak. + assert result[0]['comment_id'] == 3, \ + f"lowest-tid wins tied max(rat,rdt); got {result[0]['comment_id']}" + + # --- Deliverable 5: 5 gap tests ------------------------------------------- + + def test_mod_out_excludes_best_agree_candidate(self): + """A `mod_out` tid that would otherwise own the best-agree slot is + filtered before the reduce (Clojure repness.clj:222). The next-best + candidate becomes best_agree.""" + rows = [ + # tid 1: would-be best_agree (ra=2.0>1, pa=0.8>0.5 → Branch 4 accepts; + # strong ra*rat*pa*pat = 2.0*2.0*0.8*2.0 = 6.4 so it dominates Branch 2). + _stats_row(1, na=8, nd=2, pa=0.8, pd_=0.2, pat=2.0, pdt=-2.0, + ra=2.0, rd=0.5, rat=2.0, rdt=-2.0), + # tid 2: next-best (ra*rat*pa*pat = 1.5*1.5*0.7*1.5 ≈ 2.36). + _stats_row(2, na=6, nd=3, pa=0.7, pd_=0.3, pat=1.5, pdt=-1.5, + ra=1.5, rd=0.6, rat=1.5, rdt=-1.5), + # tid 3: weaker. + _stats_row(3, na=5, nd=4, pa=0.55, pd_=0.45, pat=1.0, pdt=-1.0, + ra=1.1, rd=0.9, rat=1.0, rdt=-1.0), + ] + df = pd.DataFrame(rows) + # Without mod_out: tid 1 wins best_agree. + baseline = _assemble_rep_comments(df) + baseline_best_agree = next(r for r in baseline if r.get('best_agree')) + assert baseline_best_agree['comment_id'] == 1 + # With tid 1 moderated out: tid 2 must win best_agree, tid 1 absent. + result = _assemble_rep_comments(df, mod_out=[1]) + tids = [r['comment_id'] for r in result] + assert 1 not in tids, f"mod_out tid 1 must be excluded, got {tids}" + flagged = [r for r in result if r.get('best_agree')] + assert len(flagged) == 1, "exactly one best_agree slot" + assert flagged[0]['comment_id'] == 2, \ + f"next-best (tid 2) should become best_agree, got {flagged[0]['comment_id']}" + + def test_mod_out_accepts_ndarray(self): + """`mod_out` typed Optional[Iterable[int]] — callers may pass a numpy + array or pandas Index (e.g. sourced from a DataFrame column). Bare + `if mod_out:` truthiness raises 'truth value of an array is + ambiguous' for len>1 arrays; the check must be `is not None` + (Copilot review 2026-07-04, verified).""" + rows = [ + _stats_row(1, na=8, nd=2, pa=0.8, pd_=0.2, pat=2.0, pdt=-2.0, + ra=2.0, rd=0.5, rat=2.0, rdt=-2.0), + _stats_row(2, na=6, nd=3, pa=0.7, pd_=0.3, pat=1.5, pdt=-1.5, + ra=1.5, rd=0.6, rat=1.5, rdt=-1.5), + _stats_row(3, na=5, nd=4, pa=0.55, pd_=0.45, pat=1.0, pdt=-1.0, + ra=1.1, rd=0.9, rat=1.0, rdt=-1.0), + ] + df = pd.DataFrame(rows) + # len-2 ndarray: bare truthiness would raise ValueError. + result = _assemble_rep_comments(df, mod_out=np.array([1, 3])) + tids = [r['comment_id'] for r in result] + assert 1 not in tids and 3 not in tids, \ + f"ndarray mod_out tids must be excluded, got {tids}" + assert 2 in tids + + def test_tied_agree_metric_in_sort_uses_deterministic_tiebreak(self): + """Two `sufficient` rows with identical `agree_metric` resolve + deterministically. `list.sort` is stable in CPython, so the lower-tid + row (which entered `sufficient` first thanks to the tid-ascending + iter sort) appears first after descending sort by metric. + + Decision D10.8.1: lowest tid wins ties.""" + # Both rows pass passes_by_test (rat,pat at 2.0 > Z_90). + # Identical agree_metric: ra*rat*pa*pat is the SAME for both. + # ra=1.5, rat=2.0, pa=0.7, pat=2.0 → agree_metric = 4.2 (both). + # Insert in REVERSE tid order to prove the deterministic outcome + # comes from the sort, not the input order. + rows = [ + _stats_row(9, na=7, nd=3, pa=0.7, pd_=0.3, pat=2.0, pdt=-2.0, + ra=1.5, rd=0.6, rat=2.0, rdt=-2.0), + _stats_row(2, na=7, nd=3, pa=0.7, pd_=0.3, pat=2.0, pdt=-2.0, + ra=1.5, rd=0.6, rat=2.0, rdt=-2.0), + ] + result = _assemble_rep_comments(pd.DataFrame(rows)) + # 2 sufficient rows; one of them is also best_agree. + # Order: best_agree (tid 2, lowest tid wins beats_best_agr ties via + # strict-> first-row-wins) prepended, then deduped sufficient (tid 9). + tids = [r['comment_id'] for r in result] + assert tids[0] == 2, \ + f"lowest-tid wins tied beats_best_agr Branch 2 product; got {tids}" + assert result[0].get('best_agree') is True + + def test_disagree_only_group(self): + """All sufficient rows are `repful='disagree'`. Sort works on + `disagree_metric`; agrees-before-disagrees partition is a no-op.""" + rows = [ + # 3 disagree-passing rows (rdt,pdt > Z_90), descending disagree_metric. + _stats_row(1, na=1, nd=9, pa=0.17, pd_=0.83, pat=-2.5, pdt=2.5, + ra=0.5, rd=2.0, rat=-2.5, rdt=2.5), # disagree_metric ≈ 8.6 + _stats_row(2, na=2, nd=8, pa=0.25, pd_=0.75, pat=-2.0, pdt=2.0, + ra=0.55, rd=1.8, rat=-2.0, rdt=2.0), # ≈ 5.4 + _stats_row(3, na=3, nd=7, pa=0.33, pd_=0.67, pat=-1.5, pdt=1.5, + ra=0.6, rd=1.5, rat=-1.5, rdt=1.5), # ≈ 2.27 + ] + result = _assemble_rep_comments(pd.DataFrame(rows)) + # All rows have repful='disagree' (rdt > rat for each). + assert len(result) >= 1 + assert all(r['repful'] == 'disagree' for r in result), \ + f"all rows should be disagree, got {[r['repful'] for r in result]}" + assert len(result) <= 5, "take-5 cap holds" + # best_agree may also be present (Branch 4 doesn't require agree side + # to dominate — z90(pat) is false here, ra<1 for all, so Branch 4 + # rejects all candidates and best_agree stays None for all entries). + # No row should be flagged as best_agree given the fixture. + assert not any(r.get('best_agree') for r in result), \ + "no row qualifies for best_agree under Branch 4 with ra<1 and pat None=True on first row, best gets set, + so output is exactly [best].""" + # Build via _stats_row but override pat/pdt to match the n=0 collapse: + # prop_test_vectorized(0, 0) = 2*sqrt(1)*(1/1 - 0.5) = 1.0 < Z_90. + rows = [ + _stats_row(1, na=0, nd=0, pa=0.5, pd_=0.5, pat=1.0, pdt=1.0, + ra=1.0, rd=1.0, rat=0.5, rdt=0.5, ns=0), + _stats_row(2, na=0, nd=0, pa=0.5, pd_=0.5, pat=1.0, pdt=1.0, + ra=1.0, rd=1.0, rat=0.3, rdt=0.4, ns=0), + ] + result = _assemble_rep_comments(pd.DataFrame(rows)) + # passes_by_test fails (pat=1.0 1.0) compares the SIGNED 4-way product + `ra * rat * pa * pat`. A candidate with negative `ra` and negative + `rat` produces a positive product that can beat the current best, + while a candidate with single negative factor produces a negative + product that cannot.""" + # Set up so iteration order: tid 1 (current_best), tid 2 (negative + # single factor, should NOT beat), tid 3 (two negatives → positive, + # should beat ONLY if its product is larger). + current_best_row = _stats_row( + 1, na=8, nd=2, pa=0.8, pd_=0.2, pat=2.0, pdt=-2.0, + ra=2.0, rd=0.5, rat=2.0, rdt=-2.0) + # current_best product = 2.0*2.0*0.8*2.0 = 6.4. + + # Single negative factor → negative product → loses on strict >. + single_neg = _stats_row( + 2, na=1, nd=1, pa=0.5, pd_=0.5, pat=1.0, pdt=1.0, + ra=-0.5, rd=1.0, rat=1.0, rdt=1.0) + # product = -0.5*1.0*0.5*1.0 = -0.25 < 6.4 → does NOT beat. + assert not beats_best_agr(single_neg, current_best_row), \ + "single negative factor → negative product loses Branch 2" + + # Two negatives → positive product. Make it LARGER than 6.4. + # ra=-5.0, rat=-2.0, pa=0.9, pat=2.0 → -5 * -2 * 0.9 * 2 = 18.0 > 6.4. + two_neg = _stats_row( + 3, na=5, nd=5, pa=0.9, pd_=0.1, pat=2.0, pdt=-2.0, + ra=-5.0, rd=0.2, rat=-2.0, rdt=-2.0) + assert beats_best_agr(two_neg, current_best_row), \ + "two negative factors → positive 18.0 > 6.4 wins Branch 2" + + # Two negatives but product NOT larger → loses. + two_neg_small = _stats_row( + 4, na=1, nd=1, pa=0.5, pd_=0.5, pat=1.0, pdt=1.0, + ra=-1.0, rd=1.0, rat=-1.0, rdt=1.0) + # product = -1 * -1 * 0.5 * 1 = 0.5 < 6.4 → does NOT beat. + assert not beats_best_agr(two_neg_small, current_best_row), \ + "two negatives but small positive product (0.5) still loses to 6.4" + + # ============================================================================ # D11 — Consensus Comment Selection # ============================================================================ @@ -1158,29 +1687,254 @@ class TestD11ConsensusSelection: Clojure uses per-comment pa > 0.5, top 5 agree + 5 disagree with z-test scores """ - @pytest.mark.xfail(reason="D11: Different consensus selection logic than Clojure") - def test_consensus_matches_clojure(self, conv, clojure_blob, dataset_name): - """Consensus comments should match Clojure's selection.""" + def test_consensus_matches_clojure(self, request, conv, clojure_blob, dataset_name): + """Consensus selection should match Clojure on cold_start. + + After D11 (PR 9), Python's `consensus_comments` is a dict + `{'agree': [...], 'disagree': [...]}` mirroring Clojure's shape. + Consensus stats are whole-conversation (no group split), so unlike + rep-comments this is NOT affected by upstream PCA/KMeans + group-membership divergence. The ns-PASS divergence + (DISCOVERY 2026-06-11) was fixed by switching `ns` from `na + nd` + to `notna().sum()` — matches Clojure `(count (filter identity ...))` + in repness.clj:56-61. + """ + # Per-variant xfail (g5, 2026-07-04): known-bad INCREMENTAL variants + # only. biodiversity-incremental was documented 2026-06-11 (residual + # upstream PCA/KMeans group-membership divergence affecting which + # participants are in-conv at the incremental step). Scoping the + # previously-blanket xfail(strict=False) then UNMASKED + # bg2018-incremental and pakistan-incremental (private datasets) — + # failures the blanket had silently absorbed, undocumented until + # 2026-07-04. Same incremental-divergence family; resolution belongs + # to the sequential-parity work (replay infra / warm-start port). + # ALL cold_start variants and vw-incremental match Clojure exactly + # and MUST keep gating. + _known_bad_incremental = ('biodiversity', 'bg2018', 'pakistan') + _callspec = request.node.callspec.id + if 'incremental' in _callspec and any( + ds in _callspec for ds in _known_bad_incremental): + request.applymarker(pytest.mark.xfail( + raises=AssertionError, + strict=False, + reason="known-bad incremental variant (biodiversity: journal " + "2026-06-11; bg2018/pakistan: unmasked 2026-07-04 when " + "the blanket xfail was scoped per-variant): residual " + "upstream incremental divergence, deferred to the " + "sequential-parity work")) + clj_consensus = clojure_blob.get('consensus', {}) if not clj_consensus: pytest.skip("No consensus in Clojure blob") - # Clojure consensus has 'agree' and 'disagree' keys clj_agree_tids = set(e['tid'] for e in clj_consensus.get('agree', [])) clj_disagree_tids = set(e['tid'] for e in clj_consensus.get('disagree', [])) clj_all = clj_agree_tids | clj_disagree_tids - py_consensus = conv.repness.get('consensus_comments', []) if conv.repness else [] - py_tids = set(int(c['comment_id']) for c in py_consensus) + py_consensus = (conv.repness.get('consensus_comments', {}) + if conv.repness else {}) + py_agree_tids = set(int(c['tid']) + for c in py_consensus.get('agree', [])) + py_disagree_tids = set(int(c['tid']) + for c in py_consensus.get('disagree', [])) + py_all = py_agree_tids | py_disagree_tids + + print(f"[{dataset_name}] Consensus Clojure: " + f"agree={sorted(clj_agree_tids)}, " + f"disagree={sorted(clj_disagree_tids)}") + print(f"[{dataset_name}] Consensus Python: " + f"agree={sorted(py_agree_tids)}, " + f"disagree={sorted(py_disagree_tids)}") + overlap = len(clj_all & py_all) + print(f"[{dataset_name}] Consensus overlap: {overlap}/{len(clj_all)}") - print(f"[{dataset_name}] Consensus: Clojure agree={sorted(clj_agree_tids)}, disagree={sorted(clj_disagree_tids)}") - print(f"[{dataset_name}] Consensus: Python={sorted(py_tids)}") + check.equal(py_agree_tids, clj_agree_tids, + f"Agree consensus mismatch") + check.equal(py_disagree_tids, clj_disagree_tids, + f"Disagree consensus mismatch") - overlap = len(clj_all & py_tids) - print(f"[{dataset_name}] Consensus overlap: {overlap}/{len(clj_all)}") - check.equal(py_tids, clj_all, - f"Consensus mismatch: Python={sorted(py_tids)}, Clojure={sorted(clj_all)}") +class TestD11ConsensusStatsDf: + """`consensus_stats_df` — whole-conversation per-comment stats (no group split).""" + + @staticmethod + def _vote_matrix(per_comment_votes): + """Helper: build a vote matrix from {tid: [vote_per_participant]}.""" + return pd.DataFrame(per_comment_votes) + + def test_basic_counts(self): + """na/nd/ns counted correctly across all participants.""" + # 5 participants, 3 comments + # tid 1: 4 agrees, 1 disagree → na=4, nd=1, ns=5 + # tid 2: 2 agrees, 3 disagrees → na=2, nd=3, ns=5 + # tid 3: 1 agree, 2 disagrees, 2 NaN (pass/unvoted) → na=1, nd=2, ns=3 + votes = pd.DataFrame({ + 1: [AGREE, AGREE, AGREE, AGREE, DISAGREE], + 2: [AGREE, AGREE, DISAGREE, DISAGREE, DISAGREE], + 3: [AGREE, DISAGREE, DISAGREE, np.nan, np.nan], + }) + df = consensus_stats_df(votes) + assert df.loc[1, 'na'] == 4 and df.loc[1, 'nd'] == 1 and df.loc[1, 'ns'] == 5 + assert df.loc[2, 'na'] == 2 and df.loc[2, 'nd'] == 3 and df.loc[2, 'ns'] == 5 + assert df.loc[3, 'na'] == 1 and df.loc[3, 'nd'] == 2 and df.loc[3, 'ns'] == 3 + + def test_pseudocount_pa_pd(self): + """pa/pd use Beta(2,2) smoothing: (na+1)/(ns+2).""" + votes = pd.DataFrame({1: [AGREE, AGREE, AGREE, AGREE, DISAGREE]}) + df = consensus_stats_df(votes) + # na=4, ns=5 → pa = 5/7 ≈ 0.714 + assert abs(df.loc[1, 'pa'] - 5/7) < 1e-10 + # nd=1, ns=5 → pd = 2/7 ≈ 0.286 + assert abs(df.loc[1, 'pd'] - 2/7) < 1e-10 + + def test_ns_zero_uses_uninformative_prior(self): + """When ns=0 (no agree/disagree at all), pa=pd=0.5.""" + votes = pd.DataFrame({1: [np.nan, np.nan, np.nan]}) + df = consensus_stats_df(votes) + assert df.loc[1, 'pa'] == 0.5 + assert df.loc[1, 'pd'] == 0.5 + + def test_mod_out_filters_tids(self): + """`mod_out` removes tids from the output.""" + votes = pd.DataFrame({ + 1: [AGREE, AGREE, AGREE], + 2: [AGREE, AGREE, AGREE], + 3: [AGREE, AGREE, AGREE], + }) + df = consensus_stats_df(votes, mod_out={2}) + assert 1 in df.index + assert 2 not in df.index + assert 3 in df.index + + def test_mod_out_accepts_ndarray(self): + """Same `is not None` requirement as select_rep_comments_df: a len>1 + numpy array as mod_out must filter, not raise 'truth value of an + array is ambiguous' (Copilot review 2026-07-04, verified).""" + votes = pd.DataFrame({ + 1: [AGREE, AGREE, AGREE], + 2: [AGREE, AGREE, AGREE], + 3: [AGREE, AGREE, AGREE], + }) + df = consensus_stats_df(votes, mod_out=np.array([2, 3])) + assert 1 in df.index + assert 2 not in df.index + assert 3 not in df.index + + def test_ns_includes_pass_votes(self): + """Clojure parity: ns counts all non-nil votes incl. PASS (repness.clj:56-61).""" + votes = pd.DataFrame({ + 1: [AGREE, AGREE, DISAGREE, 0, 0], # 2A, 1D, 2P → ns=5 + }) + df = consensus_stats_df(votes) + assert df.loc[1, 'na'] == 2 + assert df.loc[1, 'nd'] == 1 + assert df.loc[1, 'ns'] == 5, f"ns should include PASS (Clojure parity); got {df.loc[1, 'ns']}" + + +class TestD11SelectConsensusBoundary: + """`select_consensus_comments_df` Clojure-parity boundaries.""" + + @staticmethod + def _stats(rows): + """Helper: build a stats DataFrame from list of (tid, na, nd, ns, pa, pd, pat, pdt).""" + df = pd.DataFrame(rows, columns=['tid', 'na', 'nd', 'ns', 'pa', 'pd', 'pat', 'pdt']) + return df.set_index('tid') + + def test_empty_input_returns_empty_lists(self): + result = select_consensus_comments_df(pd.DataFrame(columns=['na', 'nd', 'ns', 'pa', 'pd', 'pat', 'pdt'])) + assert result == {'agree': [], 'disagree': []} + + def test_clear_agree_consensus(self): + """Comments with pa > 0.5 AND z-sig-90(pat) land in 'agree'.""" + stats = self._stats([ + (1, 9, 1, 10, 0.83, 0.17, 2.5, -2.5), # pa>0.5, pat z90 → agree + (2, 8, 2, 10, 0.75, 0.25, 2.0, -2.0), # agree + ]) + result = select_consensus_comments_df(stats) + agree_tids = [e['tid'] for e in result['agree']] + assert 1 in agree_tids and 2 in agree_tids + assert result['disagree'] == [] + + def test_clear_disagree_consensus(self): + """Comments with pd > 0.5 AND z-sig-90(pdt) land in 'disagree'.""" + stats = self._stats([ + (1, 1, 9, 10, 0.17, 0.83, -2.5, 2.5), # pd>0.5, pdt z90 → disagree + (2, 2, 8, 10, 0.25, 0.75, -2.0, 2.0), + ]) + result = select_consensus_comments_df(stats) + disagree_tids = [e['tid'] for e in result['disagree']] + assert 1 in disagree_tids and 2 in disagree_tids + assert result['agree'] == [] + + def test_divisive_no_consensus(self): + """Comments split ~50/50 with low z-scores → neither list populated.""" + stats = self._stats([ + (1, 5, 5, 10, 0.5, 0.5, 0.0, 0.0), + (2, 4, 6, 10, 0.42, 0.58, -0.4, 0.4), + ]) + result = select_consensus_comments_df(stats) + assert result['agree'] == [] + assert result['disagree'] == [] + + def test_top_5_cap_per_side(self): + """Each list capped at 5 entries.""" + # 7 high-agree comments + rows = [] + for i, am in enumerate([2.5, 2.3, 2.1, 1.9, 1.7, 1.5, 1.4]): + rows.append((i + 1, 9, 1, 10, 0.83, 0.17, am, -am)) + stats = self._stats(rows) + result = select_consensus_comments_df(stats) + assert len(result['agree']) == 5 + # Highest am at front: pa*pat = 0.83 * 2.5 = 2.075 + assert result['agree'][0]['tid'] == 1 + + def test_entry_keys_match_clojure_blob(self): + """Per-entry keys: tid, n-success, n-trials, p-success, p-test — + EXACTLY the Clojure blob shape (repness.clj:181 + ::consensus spec). + + Narrows the S1 deferral (2026-07-04): consensus entries are new in + D11 and flow raw into `result['consensus']` in to_dict / + to_dynamo_dict, where server-helpers.ts:298-313 and client-report's + majorityStrict.jsx:23-27 pluck `tid`. Python-convention keys would + break both consumers. Rep-comment entries keep `comment_id` until + the deferred math-blob alignment PR.""" + stats = self._stats([(1, 9, 1, 10, 0.83, 0.17, 2.5, -2.5)]) + result = select_consensus_comments_df(stats) + entry = result['agree'][0] + assert set(entry.keys()) == {'tid', 'n-success', 'n-trials', 'p-success', 'p-test'} + assert entry['tid'] == 1 + # For agree side, n-success = na, p-success = pa, p-test = pat + assert entry['n-success'] == 9 + assert entry['n-trials'] == 10 + assert abs(entry['p-success'] - 0.83) < 1e-10 + assert abs(entry['p-test'] - 2.5) < 1e-10 + + def test_disagree_entry_uses_d_keys(self): + """For disagree side, n-success = nd, p-success = pd, p-test = pdt.""" + stats = self._stats([(1, 1, 9, 10, 0.17, 0.83, -2.5, 2.5)]) + result = select_consensus_comments_df(stats) + entry = result['disagree'][0] + assert entry['n-success'] == 9 # = nd + assert abs(entry['p-success'] - 0.83) < 1e-10 # = pd + assert abs(entry['p-test'] - 2.5) < 1e-10 # = pdt + + def test_mutually_exclusive_lists(self): + """With ns ≥ na+nd (ns includes PASS post-ns-PASS fix), + pa + pd = (na+nd+PSEUDO_COUNT)/(ns+PSEUDO_COUNT) ≤ 1, so pa and pd + cannot both exceed 0.5 — the same tid cannot appear in both lists. + (The equality pa+pd=1 only holds for PASS-free comments, as in this + fixture.)""" + # PASS-free rows: na+nd = ns here (but the invariant above holds + # generally, PASS or not). + stats = self._stats([ + (1, 7, 3, 10, 0.67, 0.33, 1.5, -1.5), # agree side + (2, 3, 7, 10, 0.33, 0.67, -1.5, 1.5), # disagree side + ]) + result = select_consensus_comments_df(stats) + agree_tids = {e['tid'] for e in result['agree']} + disagree_tids = {e['tid'] for e in result['disagree']} + assert agree_tids & disagree_tids == set(), \ + f"agree and disagree lists must be disjoint, got overlap {agree_tids & disagree_tids}" # ============================================================================ @@ -1194,9 +1948,36 @@ class TestD12CommentPriorities: Clojure computes priorities based on PCA extremity and importance. """ - @pytest.mark.xfail(reason="D12: Comment priorities not implemented in Python") - def test_comment_priorities_exist(self, conv, clojure_blob, dataset_name): - """Python should produce comment-priorities matching Clojure.""" + def test_comment_priorities_exist(self, request, conv, clojure_blob, dataset_name): + """Python should produce comment-priorities matching Clojure. + + Per D12.6: Clojure's `(if 0 ...)` truthiness quirk means every tid + takes the meta branch, so Clojure cold_start priorities are all + META_PRIORITY^2 = 49.0 for vw/biodiversity. Python now mirrors this + bug + (priority_metric returns META_PRIORITY**2 unconditionally), so both + sides should yield identical all-constant 49.0. Spearman is not + meaningful when both sides have zero variance — we instead verify + the constant-value parity directly. + """ + # Per-variant xfail (g5, refined 2026-07-05): known-bad only where + # the Clojure incremental blob has VARIED priorities (no truthy-0 + # bug there), so Python's all-49 mirror can't match. FLI and bg2050 + # incremental blobs carry the all-49 signature and DO match — they + # gate. All cold_start variants gate. Once the Clojure bug (#2571) + # is fixed upstream, drop the Python mirror and this xfail. + _varied_priority_incrementals = ( + 'vw-incremental', 'biodiversity-incremental', + 'bg2018-incremental', 'engage-incremental', + 'pakistan-incremental') + if request.node.callspec.id in _varied_priority_incrementals: + request.applymarker(pytest.mark.xfail( + raises=AssertionError, + strict=False, + reason="D12.6: this Clojure incremental blob has varied " + "priorities (no truthy-0 bug there); Python's " + "all-49 mirror cannot match. See issue #2571.")) + clj_priorities = clojure_blob.get('comment-priorities', {}) check.greater(len(clj_priorities), 0, f"Clojure has {len(clj_priorities)} comment priorities") @@ -1209,11 +1990,183 @@ def test_comment_priorities_exist(self, conv, clojure_blob, dataset_name): return py_priorities = conv.comment_priorities - # Compare rankings (Spearman correlation would be ideal, but check overlap first) - common_tids = set(str(k) for k in clj_priorities.keys()) & set(str(k) for k in py_priorities.keys()) - print(f"[{dataset_name}] Common priority tids: {len(common_tids)}/{len(clj_priorities)}") + # Normalize keys to int for comparison. + clj_p = {int(k): v for k, v in clj_priorities.items()} + py_p = {int(k): v for k, v in py_priorities.items()} + common_tids = set(clj_p.keys()) & set(py_p.keys()) + print(f"[{dataset_name}] Common priority tids: {len(common_tids)}/{len(clj_p)}") check.greater(len(common_tids), 0, "Should have common priority tids") + tids_sorted = sorted(common_tids) + clj_vals = [clj_p[t] for t in tids_sorted] + py_vals = [py_p[t] for t in tids_sorted] + clj_unique = set(clj_vals) + py_unique = set(py_vals) + print(f"[{dataset_name}] clj_vals sample: {clj_vals[:5]}, " + f"min={min(clj_vals)}, max={max(clj_vals)}, " + f"unique={len(clj_unique)}") + print(f"[{dataset_name}] py_vals sample: {py_vals[:5]}, " + f"min={min(py_vals)}, max={max(py_vals)}, " + f"unique={len(py_unique)}") + + # D12.6 Clojure-parity-bug mirror: both sides should return + # META_PRIORITY**2 = 49.0 for every tid. + META_PRIORITY_SQ = META_PRIORITY ** 2 + check.equal(len(clj_unique), 1, + f"Clojure priorities should be all-constant (bug); got {len(clj_unique)} unique") + check.equal(len(py_unique), 1, + f"Python priorities should be all-constant (bug mirror); got {len(py_unique)} unique") + if len(clj_unique) == 1: + (clj_const,) = clj_unique + check.almost_equal(clj_const, META_PRIORITY_SQ, abs=1e-9, + msg=f"Clojure constant priority should be META_PRIORITY**2={META_PRIORITY_SQ}") + if len(py_unique) == 1: + (py_const,) = py_unique + check.almost_equal(py_const, META_PRIORITY_SQ, abs=1e-9, + msg=f"Python constant priority should be META_PRIORITY**2={META_PRIORITY_SQ}") + + +class TestD12PriorityExtremityAlignment: + """`_compute_comment_priorities` must fail closed on a PCA/columns desync. + + `dict(zip(rating_mat.columns, extremity_arr))` silently truncates when + the PCA output was computed on a different column set than the current + rating_mat (e.g. moderation changed between recomputes). Silent + truncation assigns E=0 to the overflow tids — wrong priorities with no + signal. The guard logs an error and returns {} (server falls back to + uniform routing — degraded but honest). (Copilot review 2026-07-04, g4.) + """ + + def _conv_with_desync(self): + conv = Conversation(conversation_id='ztest-desync') + # 3 comments in the rating matrix... + conv.rating_mat = pd.DataFrame( + [[1.0, -1.0, 0.0], [1.0, 1.0, -1.0]], + index=[0, 1], columns=[10, 11, 12], + ) + conv.raw_rating_mat = conv.rating_mat.copy() + # ...but PCA computed on only 2 (stale center/comps). + conv.pca = { + 'center': np.array([0.5, -0.5]), + 'comps': np.array([[0.7, 0.7], [0.7, -0.7]]), + } + conv.group_clusters = [] + conv.meta_tids = set() + return conv + + def test_desync_returns_empty_and_logs(self, caplog): + conv = self._conv_with_desync() + import logging + with caplog.at_level(logging.ERROR): + result = conv._compute_comment_priorities() + assert result == {}, ( + f"desynced PCA/columns must fail closed (empty priorities), " + f"got {result!r} — silent zip truncation assigns E=0 to " + f"overflow tids" + ) + assert any('extremity' in r.message.lower() or + 'priorit' in r.message.lower() + for r in caplog.records), \ + "expected an ERROR log naming the priorities/extremity desync" + + +class TestD12PCAProjectComments: + """`pca_project_cmnts` and `compute_comment_extremity` — Clojure parity.""" + + def test_pca_project_cmnts_shape(self): + """Output shape (n_cmnts, n_components).""" + center = np.array([0.1, 0.2, 0.3, 0.4]) + comps = np.array([[1.0, 0.0, 0.5, 0.5], + [0.0, 1.0, 0.5, -0.5]]) + proj = pca_project_cmnts(center, comps) + assert proj.shape == (4, 2) + + def test_pca_project_cmnts_formula(self): + """For comment i: proj[i] = -sqrt(n_cmnts) * (1 + center[i]) * [pc1[i], pc2[i]].""" + center = np.array([0.1, 0.2, 0.3, 0.4]) + comps = np.array([[1.0, 0.5, -0.5, 0.0], + [0.0, 0.5, 0.5, 1.0]]) + proj = pca_project_cmnts(center, comps) + n_cmnts = 4 + scale = np.sqrt(n_cmnts) + for i in range(n_cmnts): + expected = -scale * (1 + center[i]) * comps[:, i] + assert np.allclose(proj[i], expected), \ + f"proj[{i}] = {proj[i]} vs expected {expected}" + + def test_pca_project_cmnts_empty(self): + """Empty inputs return shape (0, n_comps).""" + center = np.zeros(0) + comps = np.zeros((2, 0)) + proj = pca_project_cmnts(center, comps) + assert proj.shape == (0, 2) + + def test_compute_comment_extremity_l2_norm(self): + """Extremity = L2 norm of each projection row.""" + cmnt_proj = np.array([[3.0, 4.0], + [0.0, 0.0], + [-1.0, 1.0]]) + ext = compute_comment_extremity(cmnt_proj) + assert np.allclose(ext, [5.0, 0.0, np.sqrt(2)]) + + def test_compute_comment_extremity_empty(self): + """Empty input → empty output.""" + ext = compute_comment_extremity(np.zeros((0, 2))) + assert ext.shape == (0,) + + +class TestD12PriorityMetrics: + """`importance_metric` and `priority_metric` — Clojure parity.""" + + def test_importance_metric_formula(self): + """`(1 - p) * (E + 1) * a` where p = (P+1)/(S+2), a = (A+1)/(S+2).""" + # Clojure ref values from conversation.clj:335: + # `(float (importance-metric 1 0 1 0))` — A=1, P=0, S=1, E=0 + # p = 1/3, a = 2/3, return = (2/3)*(1)*(2/3) = 4/9 ≈ 0.4444 + assert abs(importance_metric(1, 0, 1, 0) - 4 / 9) < 1e-10 + + def test_importance_metric_high_extremity_boosts(self): + """Higher extremity → higher importance.""" + baseline = importance_metric(5, 1, 8, 0.0) + boosted = importance_metric(5, 1, 8, 2.0) + assert boosted > baseline + + def test_priority_metric_meta_constant(self): + """Meta comments return META_PRIORITY^2 = 49 (Clojure parity).""" + # is_meta=True → inner = 7, return = 49 + assert priority_metric(True, 5, 2, 10, 1.5) == META_PRIORITY ** 2 + assert priority_metric(True, 0, 0, 0, 0) == META_PRIORITY ** 2 + + @pytest.mark.xfail(reason="Clojure parity bug mirror (D12.6): priority_metric always " + "returns META_PRIORITY**2 until upstream Clojure bug resolves. " + "Tests pin the semantically-correct formula and will pass again " + "when we revert the mirror.") + def test_priority_metric_non_meta_squared(self): + """Non-meta: return = (importance * (1 + 8*2^(-S/5)))^2.""" + # A=20, P=3, S=20, E=0 — ref from conversation.clj:337 + A, P, S, E = 20, 3, 20, 0 + imp = importance_metric(A, P, S, E) + decay = 1 + 8 * (2 ** (-S / 5)) + expected = (imp * decay) ** 2 + assert abs(priority_metric(False, A, P, S, E) - expected) < 1e-10 + + def test_priority_metric_decay_factor_lets_new_bubble_up(self): + """For low-S (new) comments, the decay factor is larger → priority boost.""" + # Two comments with identical importance metrics but different S. + # importance depends on A, P, S, E; to isolate the decay factor, + # pick A,P,E values that give same `(1 - (P+1)/(S+2)) * (E+1) * (A+1)/(S+2)`? + # Hard to isolate, so just test that the decay factor itself increases for low S. + new_decay = 1 + 8 * (2 ** (-1 / 5)) # S=1 + old_decay = 1 + 8 * (2 ** (-100 / 5)) # S=100 + assert new_decay > old_decay + assert new_decay > 1.0 + # Old comments fade toward 1 (no boost). + assert old_decay < 1.01 + + def test_meta_priority_constant_value(self): + """META_PRIORITY = 7 (Clojure conversation.clj:319).""" + assert META_PRIORITY == 7 + # ============================================================================ # D15 — Moderation Handling @@ -1620,36 +2573,11 @@ def test_z_thresholds_are_one_tailed(self): check.almost_equal(Z_95, 1.6449, abs=0.001, msg=f"Z_95={Z_95}, expected 1.6449 (one-tailed)") - def test_prop_test_matches_clojure_formula_synthetic(self): - """prop_test(succ, n) should produce 2*sqrt(n+1)*((succ+1)/(n+1) - 0.5).""" - # Small n: 5 successes out of 8 trials - succ, n = 5, 8 - expected = 2 * 3.0 * (6.0 / 9.0 - 0.5) # = 1.0 - result = prop_test(succ, n) - assert abs(result - expected) < 1e-10, f"prop_test({succ}, {n})={result}, expected {expected}" - - def test_clojure_repness_metric_product(self): - """Python's repness_metric matches Clojure (* repness repness-test p-success p-test). - - Verifies the actual production function, not a re-implementation of the formula. - """ - stats = { - 'pa': 0.8, 'pat': 3.0, 'ra': 1.5, 'rat': 2.0, - 'pd': 0.2, 'pdt': -1.0, 'rd': 0.5, 'rdt': -0.5, - } - # Agree: (* ra rat pa pat) = 1.5 * 2.0 * 0.8 * 3.0 = 7.2 - assert repness_metric(stats, 'a') == pytest.approx(7.2) - # Disagree (same product, no (1-pd) trick): (* rd rdt pd pdt) - # = 0.5 * -0.5 * 0.2 * -1.0 = 0.05 (two negatives cancel — signed product) - assert repness_metric(stats, 'd') == pytest.approx(0.05) - - def test_clojure_repful_uses_rat_vs_rdt(self): - """Clojure determines repful by comparing rat vs rdt.""" - # rat > rdt → agree - assert (2.0 > 1.0) # rat=2.0, rdt=1.0 → agree - - # rat < rdt → disagree - assert (0.5 < 1.5) # rat=0.5, rdt=1.5 → disagree + # prop_test / repness_metric / repful formula tests are covered by + # TestD5ProportionTest::test_prop_test_matches_clojure_formula, + # TestD7RepnessMetric::test_metric_formula_is_product, and + # TestD8FinalizeStats::test_repful_classification_boundary respectively + # (migrated to vectorized in PR 14a). # ============================================================================ @@ -1666,58 +2594,60 @@ def test_clojure_repful_uses_rat_vs_rdt(self): # isolating each computation stage from upstream divergence. # ============================================================================ +def _blob_repness_rows(clojure_blob): + """Flatten the Clojure blob's `repness` dict into a list of per-(gid, tid) rows + for vectorized comparison. Each row is `{gid, tid, **entry_keys}`.""" + return [{'gid': gid, **entry} + for gid, entries in clojure_blob.get('repness', {}).items() + for entry in entries] + + @pytest.mark.clojure_comparison class TestD5BlobInjection: - """D5: Verify prop_test against real Clojure blob p-test values. + """D5: Verify prop_test_vectorized against real Clojure blob p-test values. - For each repness entry in the blob, extract n-success and n-trials, - feed to Python's prop_test(), compare to blob's p-test. + Collect (n-success, n-trials, p-test) from every repness entry in the blob, + run a single vectorized call, compare element-wise. Tests the actual + production code path (same call shape as `compute_group_comment_stats_df`). """ def test_prop_test_matches_blob_p_test(self, clojure_blob, dataset_name): - """prop_test(n_success, n_trials) should match blob's p-test for every repness entry.""" - repness = clojure_blob.get('repness', {}) - if not repness: + """prop_test_vectorized(n_success, n_trials) should match blob's p-test + for every repness entry.""" + rows = _blob_repness_rows(clojure_blob) + if not rows: pytest.skip(f"No repness in Clojure blob for {dataset_name}") - mismatches = [] - total = 0 - for gid, entries in repness.items(): - for entry in entries: - n_success = entry['n-success'] - n_trials = entry['n-trials'] - expected_p_test = entry['p-test'] - actual = prop_test(n_success, n_trials) - total += 1 - if abs(actual - expected_p_test) > 1e-4: - mismatches.append( - f"group={gid} tid={entry['tid']}: " - f"prop_test({n_success}, {n_trials})={actual:.6f}, " - f"blob p-test={expected_p_test:.6f}") + df = pd.DataFrame(rows)[['gid', 'tid', 'n-success', 'n-trials', 'p-test']] + df['actual'] = prop_test_vectorized(df['n-success'], df['n-trials']) + df['diff'] = (df['actual'] - df['p-test']).abs() - assert not mismatches, ( - f"[{dataset_name}] {len(mismatches)}/{total} p-test mismatches:\n" - + "\n".join(mismatches[:10])) + mismatches = df[df['diff'] > 1e-4] + assert mismatches.empty, ( + f"[{dataset_name}] {len(mismatches)}/{len(df)} p-test mismatches:\n" + + mismatches.head(10).to_string(index=False)) @pytest.mark.clojure_comparison class TestD6BlobInjection: - """D6: Verify two_prop_test against real Clojure blob repness-test values. + """D6: Verify two_prop_test_vectorized against real Clojure blob + repness-test values. For each repness entry, reconstruct the two_prop_test inputs from - group-votes (group counts vs total-minus-group), compare to blob's - repness-test. + group-votes (group counts vs total-minus-group), collect into a DataFrame, + and run a single vectorized call. Tests the actual production code path. """ def test_two_prop_test_matches_blob_repness_test(self, clojure_blob, dataset_name): - """two_prop_test should match blob's repness-test for every repness entry.""" + """two_prop_test_vectorized should match blob's repness-test for every + repness entry.""" repness = clojure_blob.get('repness', {}) group_votes = clojure_blob.get('group-votes', {}) if not repness or not group_votes: pytest.skip(f"No repness or group-votes in blob for {dataset_name}") - # Precompute total votes across ALL groups for each comment - all_group_votes = {} + # Precompute total votes across ALL groups for each comment. + all_group_votes: dict = {} for other_gid, other_gv_data in group_votes.items(): for tid_str, counts in other_gv_data.get('votes', {}).items(): if tid_str not in all_group_votes: @@ -1726,15 +2656,12 @@ def test_two_prop_test_matches_blob_repness_test(self, clojure_blob, dataset_nam all_group_votes[tid_str]['D'] += counts['D'] all_group_votes[tid_str]['S'] += counts['S'] - mismatches = [] - total = 0 + rows = [] for gid, entries in repness.items(): gv = group_votes.get(gid, {}).get('votes', {}) for entry in entries: tid_str = str(entry['tid']) repful = entry['repful-for'] - expected_rt = entry['repness-test'] - group_cv = gv.get(tid_str, {'A': 0, 'D': 0, 'S': 0}) total_cv = all_group_votes.get(tid_str, {'A': 0, 'D': 0, 'S': 0}) @@ -1745,20 +2672,23 @@ def test_two_prop_test_matches_blob_repness_test(self, clojure_blob, dataset_nam succ_in = group_cv['D'] succ_out = total_cv['D'] - group_cv['D'] - pop_in = group_cv['S'] - pop_out = total_cv['S'] - group_cv['S'] + rows.append({ + 'gid': gid, 'tid': entry['tid'], 'repful': repful, + 'succ_in': succ_in, 'succ_out': succ_out, + 'pop_in': group_cv['S'], + 'pop_out': total_cv['S'] - group_cv['S'], + 'expected': entry['repness-test'], + }) - actual = two_prop_test(succ_in, succ_out, pop_in, pop_out) - total += 1 - if abs(actual - expected_rt) > 1e-4: - mismatches.append( - f"group={gid} tid={entry['tid']} ({repful}): " - f"two_prop_test({succ_in},{succ_out},{pop_in},{pop_out})={actual:.6f}, " - f"blob repness-test={expected_rt:.6f}") + df = pd.DataFrame(rows) + df['actual'] = two_prop_test_vectorized( + df['succ_in'], df['succ_out'], df['pop_in'], df['pop_out']) + df['diff'] = (df['actual'] - df['expected']).abs() - assert not mismatches, ( - f"[{dataset_name}] {len(mismatches)}/{total} repness-test mismatches:\n" - + "\n".join(mismatches[:10])) + mismatches = df[df['diff'] > 1e-4] + assert mismatches.empty, ( + f"[{dataset_name}] {len(mismatches)}/{len(df)} repness-test mismatches:\n" + + mismatches.head(10).to_string(index=False)) @pytest.mark.clojure_comparison @@ -1767,25 +2697,184 @@ class TestD4BlobInjection: def test_p_success_matches_blob(self, clojure_blob, dataset_name): """(n_success + 1) / (n_trials + 2) should match blob's p-success.""" - repness = clojure_blob.get('repness', {}) - if not repness: + rows = _blob_repness_rows(clojure_blob) + if not rows: pytest.skip(f"No repness in blob for {dataset_name}") - mismatches = [] - total = 0 - for gid, entries in repness.items(): - for entry in entries: - ns = entry['n-success'] - nt = entry['n-trials'] - expected = entry['p-success'] - actual = (ns + PSEUDO_COUNT / 2) / (nt + PSEUDO_COUNT) - total += 1 - if abs(actual - expected) > 1e-4: - mismatches.append( - f"group={gid} tid={entry['tid']}: " - f"pa=({ns}+1)/({nt}+2)={actual:.6f}, " - f"blob p-success={expected:.6f}") + df = pd.DataFrame(rows)[['gid', 'tid', 'n-success', 'n-trials', 'p-success']] + df['actual'] = ((df['n-success'] + PSEUDO_COUNT / 2) + / (df['n-trials'] + PSEUDO_COUNT)) + df['diff'] = (df['actual'] - df['p-success']).abs() + + mismatches = df[df['diff'] > 1e-4] + assert mismatches.empty, ( + f"[{dataset_name}] {len(mismatches)}/{len(df)} p-success mismatches:\n" + + mismatches.head(10).to_string(index=False)) + + +class TestD11D12Serialization: + """Round-trip tests for the D11/D12 plumb-through in to_dict / to_dynamo_dict. + + Investigation B (2026-06-11) discovered that both serializers were hardcoding + ``result['consensus']`` to an empty dict regardless of + ``self.repness['consensus_comments']``, so the D11 consensus dict never + reached client-report's Majority view and never landed in the DynamoDB + math blob. ``comment_priorities`` (D12) was already conditionally plumbed + via ``hasattr/if`` guards; we lock that in with a regression test so a + future cleanup doesn't silently revert to the empty-default shape. + """ + + @staticmethod + def _make_conversation_with_repness(consensus_comments, priorities): + """Build a Conversation with just enough state to exercise the + serializers. Empty rating matrices and empty group_clusters mean the + rest of to_dict/to_dynamo_dict iterates over zero rows/cols (cheap) + while the consensus + priorities fields still flow through end-to-end. + """ + conv = Conversation(conversation_id='ztest-serialization') + conv.repness = { + 'comment_ids': [], + 'group_repness': {}, + 'comment_repness': [], + 'consensus_comments': consensus_comments, + } + conv.comment_priorities = priorities + return conv + + def test_to_dict_surfaces_consensus_comments(self): + """``to_dict()`` must surface ``self.repness['consensus_comments']`` into + ``result['consensus']``. Pre-fix this slot was hardcoded + ``{'agree': [], 'disagree': [], 'comment-stats': {}}`` and the D11 + selection was silently dropped on the floor.""" + consensus = { + 'agree': [ + {'tid': 1, 'n-success': 3, 'n-trials': 4, + 'p-success': 0.7, 'p-test': 1.5} + ], + 'disagree': [ + {'tid': 2, 'n-success': 2, 'n-trials': 5, + 'p-success': 0.42, 'p-test': 1.1} + ], + } + conv = self._make_conversation_with_repness(consensus, {}) + + result = conv.to_dict() + + assert result['consensus'] == consensus, ( + "to_dict() must plumb self.repness['consensus_comments'] into " + "result['consensus']; got " + repr(result['consensus'])) + + def test_to_dict_surfaces_comment_priorities(self): + """``to_dict()`` must surface ``self.comment_priorities`` (D12). This is + a regression lock: the field is currently conditionally plumbed via + ``hasattr/if``; a future cleanup must not revert to the hardcoded + empty default.""" + priorities = {1: 0.42, 2: 1.7, 3: 0.0} + conv = self._make_conversation_with_repness( + {'agree': [], 'disagree': []}, priorities) + + result = conv.to_dict() + + # The to_dict key uses underscore form (see line ~1706); no rename + # happens on the way out, unlike most Clojure-format fields. + assert 'comment_priorities' in result, ( + "to_dict() must emit 'comment_priorities' when " + "self.comment_priorities is populated; keys = " + + repr(sorted(result.keys()))) + assert result['comment_priorities'] == priorities + + def test_to_dynamo_dict_surfaces_both(self): + """``to_dynamo_dict()`` must surface BOTH consensus comments (D11) and + comment priorities (D12). The DynamoDB shape uses underscore keys + (``consensus``, ``comment_priorities``); the consensus inner shape + matches whatever ``self.repness['consensus_comments']`` holds + (Clojure-style ``agree``/``disagree`` lists).""" + consensus = { + 'agree': [ + {'tid': 11, 'n-success': 8, 'n-trials': 10, + 'p-success': 0.83, 'p-test': 2.1} + ], + 'disagree': [], + } + # Priorities use comment-id keys; the serializer coerces KEYS to int + # when possible and preserves VALUES as Decimal (2026-07-04 fix — + # the old int(value) coercion floored sub-1 priorities to 0, which + # the TS server's weighted routing reads as "no priority data"). + priorities = {7: 1.5, 9: 0.25} + conv = self._make_conversation_with_repness(consensus, priorities) + + result = conv.to_dynamo_dict() + + # Values land Decimal-converted (boto3 boundary — the raw-float write + # crashed CI's e2e run 2026-07-05); compare structure and numeric + # values, not float identity. + got = result['consensus'] + assert set(got.keys()) == {'agree', 'disagree'} + assert got['disagree'] == [] + assert len(got['agree']) == 1 + for k, v in consensus['agree'][0].items(): + assert float(got['agree'][0][k]) == pytest.approx(float(v)), ( + f"consensus entry key {k}: {got['agree'][0][k]!r} != {v!r}") + assert 'comment_priorities' in result, ( + "to_dynamo_dict() must emit 'comment_priorities' when " + "self.comment_priorities is populated; keys = " + + repr(sorted(result.keys()))) + # Values land as Decimal (boto3-safe) with full precision — assert + # the post-serialization shape to lock in what actually lands in + # DynamoDB. + from decimal import Decimal + assert result['comment_priorities'] == { + 7: Decimal('1.5'), 9: Decimal('0.25')} + + +class TestGroupIdOrderMatchesClojure: + """Group-cluster ids must preserve first-k-distinct encounter order over + base-cluster centers — Clojure parity (`init-clusters`, clusters.clj:55-64; + output `sort-by :id`, conversation.clj:437; merge lineage keeps the larger + cluster's id but NEVER re-sorts by size). + + Python's former size-descending re-sort + id reassignment caused the + gid 0↔1 label swap confirmed by the S3-4 trace (2026-06-11): Python g0 ∩ + Clojure g1 = 50/50 on vw-cold_start, sizes [50, 17] vs Clojure [17, 50]. + The base level already preserves k-means id order for exactly this + reason (K-inv); the group level must too. + """ + + def _conv_with_ordered_proj(self): + conv = Conversation(conversation_id='ztest-gid-order') + # proj key order defines base-center row order (K-inv invariant). + # Row 0 (left side, SMALL group) is encountered FIRST, row 1 (right + # side, LARGE group) second → group-level first-2-distinct init = + # (L, R) → group id 0 must be the L group even though it is smaller + # (2 vs 3 members). + conv.proj = { + 0: [-1.0, 0.05], # L (small group) + 1: [1.0, 0.05], # R (large group) + 2: [1.0, 0.0], # R + 3: [1.0, -0.05], # R + 4: [-1.0, -0.05], # L + } + # Focus the test on id assignment: bypass the in-conv vote-count + # machinery (instance attribute shadows the bound method). + conv._get_in_conv_participants = lambda: {0, 1, 2, 3, 4} + return conv - assert not mismatches, ( - f"[{dataset_name}] {len(mismatches)}/{total} p-success mismatches:\n" - + "\n".join(mismatches[:10])) + def test_group_id_zero_is_first_encountered_not_biggest(self): + conv = self._conv_with_ordered_proj() + conv._compute_clusters() + groups = conv.group_clusters + assert len(groups) == 2, f"expected k=2, got {len(groups)}" + + # Resolve group members down to participant ids via base clusters. + base_by_id = {b['id']: b for b in conv.base_clusters} + members0 = sorted(p for bid in groups[0]['members'] + for p in base_by_id[bid]['members']) + members1 = sorted(p for bid in groups[1]['members'] + for p in base_by_id[bid]['members']) + + assert [g['id'] for g in groups] == [0, 1] + assert members0 == [0, 4], ( + f"group id 0 must be the FIRST-ENCOUNTERED (smaller, L) group " + f"per Clojure first-k-distinct order; got members {members0} — " + f"a size re-sort promotes the larger group instead") + assert members1 == [1, 2, 3] diff --git a/delphi/tests/test_dynamodb_consensus_roundtrip.py b/delphi/tests/test_dynamodb_consensus_roundtrip.py new file mode 100644 index 000000000..1d89039cf --- /dev/null +++ b/delphi/tests/test_dynamodb_consensus_roundtrip.py @@ -0,0 +1,360 @@ +""" +Tests for D11 cascade fix in `delphi/polismath/database/dynamodb.py`. + +Investigation B (2026-06-11) found three sites in the DynamoDB writer/reader +that either dropped the new D11 `consensus_comments` dict shape +(`{'agree': [...], 'disagree': [...]}`) or defaulted to the obsolete empty +list. These tests assert that, given the new shape, the writer preserves +BOTH agree and disagree lists into the `Delphi_PCAResults` table, and that +the reader defaults to the new dict shape when no item is present. + +The boto3 `Table` resource is replaced with a `unittest.mock.MagicMock`, +so no DynamoDB process is required. +""" + +from unittest.mock import MagicMock + +import pytest + +from polismath.database.dynamodb import DynamoDBClient + + +# --------------------------------------------------------------------------- +# Helpers +# --------------------------------------------------------------------------- + +# Entries use the Clojure blob shape (tid + hyphenated stats keys) — the +# shape `select_consensus_comments_df` emits since 2026-07-04 (narrowed S1 +# deferral; server-helpers.ts and majorityStrict.jsx pluck `tid`). +_AGREE_ENTRY = { + 'tid': 1, + 'n-success': 3, + 'n-trials': 5, + 'p-success': 0.6, + 'p-test': 1.5, +} + +_DISAGREE_ENTRY = { + 'tid': 2, + 'n-success': 4, + 'n-trials': 5, + 'p-success': 0.8, + 'p-test': 2.0, +} + + +def _make_consensus_dict(): + """Return a fresh D11-shape consensus dict (copy per test).""" + return { + 'agree': [dict(_AGREE_ENTRY)], + 'disagree': [dict(_DISAGREE_ENTRY)], + } + + +class _StubConversation: + """Minimal Conversation-like stub for the writer's legacy branch. + + The writer only touches a handful of attributes for the PCAResults + write path, so we keep this stub deliberately tiny. The DynamoDB + `Delphi_PCAResults` write is independent of group_clusters / + comment_priorities / etc. — those affect other tables we are not + exercising here. + """ + + def __init__(self, repness): + self.conversation_id = 42 + self.participant_count = 10 + self.comment_count = 3 + self.group_clusters = [] + self.pca = {} + self.repness = repness + # `consensus` attribute is intentionally NOT set — Site 2 must + # source from `self.repness['consensus_comments']`, not from + # the deprecated `self.consensus` attribute. + + +def _client_with_pca_results_only(): + """Build a DynamoDBClient with ONLY the PCAResults table mocked. + + All other tables are None so the writer short-circuits early at each + later step. This keeps the test focused on the consensus write site. + """ + client = DynamoDBClient() + pca_results_table = MagicMock(name='Delphi_PCAResults') + client.tables = { + 'Delphi_PCAConversationConfig': None, + 'Delphi_PCAResults': pca_results_table, + 'Delphi_KMeansClusters': None, + 'Delphi_CommentRouting': None, + 'Delphi_RepresentativeComments': None, + 'Delphi_ParticipantProjections': None, + } + return client, pca_results_table + + +# --------------------------------------------------------------------------- +# Site 1 — `dynamo_data` branch (preferred path) +# --------------------------------------------------------------------------- + +class TestSite1DynamoDataBranch: + """When `conv.to_dynamo_dict()` returns the new shape, both lists land.""" + + def test_writes_both_agree_and_disagree(self): + client, pca_results_table = _client_with_pca_results_only() + + # Stub the conversation so the writer takes the `dynamo_data` branch. + conv = MagicMock(name='Conversation') + conv.conversation_id = 42 + # REAL `to_dynamo_dict()` shape (verified 2026-07-04): consensus is + # TOP-LEVEL `result['consensus']`; `repness` carries only + # `comment_repness`. The previous stub nested `consensus_comments` + # inside `repness` — matching the writer's (buggy) read path instead + # of the producer, so the test passed while production silently + # wrote the empty default. + conv.to_dynamo_dict.return_value = { + 'participant_count': 10, + 'comment_count': 3, + 'group_count': 0, + 'pca': {}, + 'math_tick': 30000, + 'consensus': _make_consensus_dict(), + 'repness': { + 'comment_repness': [], + }, + } + + ok = client.write_conversation(conv) + assert ok is True + + assert pca_results_table.put_item.called, \ + "Writer did not call put_item on Delphi_PCAResults" + item = pca_results_table.put_item.call_args.kwargs['Item'] + + written = item['consensus_comments'] + assert isinstance(written, dict), \ + f"Expected dict shape, got {type(written).__name__}: {written!r}" + assert 'agree' in written, f"Missing 'agree' key: {written!r}" + assert 'disagree' in written, f"Missing 'disagree' key: {written!r}" + # The writer Decimal-converts at the boto3 boundary (belt-and-braces + # with to_dynamo_dict's own conversion — the raw-float write crashed + # CI's e2e run 2026-07-05). Compare keys and numeric values, not types. + for side, expected_entries in (('agree', [_AGREE_ENTRY]), + ('disagree', [_DISAGREE_ENTRY])): + got_entries = written[side] + assert len(got_entries) == len(expected_entries) + for got, expected in zip(got_entries, expected_entries): + assert set(got.keys()) == set(expected.keys()) + for k, v in expected.items(): + assert float(got[k]) == pytest.approx(float(v)), \ + f"{side} entry key {k}: {got[k]!r} != {v!r}" + + def test_missing_consensus_uses_dict_default(self): + """No top-level consensus in dynamo_data → empty dict, not list, + not crash.""" + client, pca_results_table = _client_with_pca_results_only() + + conv = MagicMock(name='Conversation') + conv.conversation_id = 42 + conv.to_dynamo_dict.return_value = { + 'participant_count': 10, + 'comment_count': 3, + 'group_count': 0, + 'pca': {}, + 'math_tick': 30000, + # 'consensus' intentionally absent + } + + ok = client.write_conversation(conv) + assert ok is True + + item = pca_results_table.put_item.call_args.kwargs['Item'] + assert item['consensus_comments'] == {'agree': [], 'disagree': []} + + +# --------------------------------------------------------------------------- +# Site 2 — legacy branch (no `to_dynamo_dict`) +# --------------------------------------------------------------------------- + +class TestSite2LegacyBranch: + """When `to_dynamo_dict` is absent, the writer sources from conv.repness.""" + + def test_writes_both_agree_and_disagree_from_repness(self): + client, pca_results_table = _client_with_pca_results_only() + + # Plain object without `to_dynamo_dict` → legacy branch. + conv = _StubConversation( + repness={'consensus_comments': _make_consensus_dict()} + ) + + ok = client.write_conversation(conv) + assert ok is True + + item = pca_results_table.put_item.call_args.kwargs['Item'] + written = item['consensus_comments'] + assert isinstance(written, dict), \ + f"Legacy branch produced non-dict: {type(written).__name__}: {written!r}" + assert set(written.keys()) >= {'agree', 'disagree'} + # _replace_floats_with_decimals converts floats to Decimal but + # preserves the list structure and integer fields. Confirm the + # tids round-trip cleanly. + assert [c['tid'] for c in written['agree']] == [1] + assert [c['tid'] for c in written['disagree']] == [2] + + def test_missing_repness_uses_dict_default(self): + client, pca_results_table = _client_with_pca_results_only() + + # `repness` is None — writer must default to the dict shape. + conv = _StubConversation(repness=None) + + ok = client.write_conversation(conv) + assert ok is True + + item = pca_results_table.put_item.call_args.kwargs['Item'] + assert item['consensus_comments'] == {'agree': [], 'disagree': []} + + +# --------------------------------------------------------------------------- +# Site 3 — reader default +# --------------------------------------------------------------------------- + +class TestSite3ReaderDefault: + """`read_math_by_tick` must default consensus to the new dict shape.""" + + def test_missing_consensus_comments_returns_dict(self): + client = DynamoDBClient() + analysis_table = MagicMock(name='Delphi_PCAResults') + # Returned Item lacks `consensus_comments` entirely. + analysis_table.get_item.return_value = { + 'Item': { + 'participant_count': 10, + 'comment_count': 3, + 'pca': {'center': [], 'components': []}, + } + } + client.tables = { + 'Delphi_PCAResults': analysis_table, + 'Delphi_KMeansClusters': None, + 'Delphi_CommentRouting': None, + 'Delphi_RepresentativeComments': None, + 'Delphi_ParticipantProjections': None, + } + + result = client.read_math_by_tick('42', 30000) + assert result['consensus'] == {'agree': [], 'disagree': []} + + def test_present_consensus_comments_round_trip(self): + """When the stored Item already has the dict shape, it's returned verbatim.""" + client = DynamoDBClient() + analysis_table = MagicMock(name='Delphi_PCAResults') + stored = _make_consensus_dict() + analysis_table.get_item.return_value = { + 'Item': { + 'participant_count': 10, + 'comment_count': 3, + 'pca': {'center': [], 'components': []}, + 'consensus_comments': stored, + } + } + client.tables = { + 'Delphi_PCAResults': analysis_table, + 'Delphi_KMeansClusters': None, + 'Delphi_CommentRouting': None, + 'Delphi_RepresentativeComments': None, + 'Delphi_ParticipantProjections': None, + } + + result = client.read_math_by_tick('42', 30000) + assert result['consensus'] == stored + + def test_legacy_list_consensus_normalized_to_dict(self): + """Pre-D11 blobs stored consensus as a (hardcoded-empty) LIST. + The reader must normalize it to the dict shape so downstream + consumers never see a list (Copilot review 2026-07-04, g3).""" + client = DynamoDBClient() + analysis_table = MagicMock(name='Delphi_PCAResults') + analysis_table.get_item.return_value = { + 'Item': { + 'participant_count': 10, + 'comment_count': 3, + 'pca': {'center': [], 'components': []}, + 'consensus_comments': [], # legacy list shape + } + } + client.tables = { + 'Delphi_PCAResults': analysis_table, + 'Delphi_KMeansClusters': None, + 'Delphi_CommentRouting': None, + 'Delphi_RepresentativeComments': None, + 'Delphi_ParticipantProjections': None, + } + + result = client.read_math_by_tick('42', 30000) + assert result['consensus'] == {'agree': [], 'disagree': []}, \ + f"legacy list must normalize to dict, got {result['consensus']!r}" + + def test_present_but_none_consensus_normalized_to_dict(self): + """`consensus_comments` present-but-`None` (or any non-dict) must + normalize to the dict shape. A present key with value None makes + `.get(..., default)` return None (not the default), so the reader + must guard on "not a dict", not just "is a list" (Copilot review + on #2591). Otherwise `result['consensus']` is None and breaks the + post-D11 contract that both keys are always present.""" + client = DynamoDBClient() + analysis_table = MagicMock(name='Delphi_PCAResults') + analysis_table.get_item.return_value = { + 'Item': { + 'participant_count': 10, + 'comment_count': 3, + 'pca': {'center': [], 'components': []}, + 'consensus_comments': None, # present-but-None + } + } + client.tables = { + 'Delphi_PCAResults': analysis_table, + 'Delphi_KMeansClusters': None, + 'Delphi_CommentRouting': None, + 'Delphi_RepresentativeComments': None, + 'Delphi_ParticipantProjections': None, + } + + result = client.read_math_by_tick('42', 30000) + assert result['consensus'] == {'agree': [], 'disagree': []}, \ + f"present-but-None must normalize to dict, got {result['consensus']!r}" + + +# --------------------------------------------------------------------------- +# Round-trip — write then read on the same in-memory store +# --------------------------------------------------------------------------- + +class TestRoundTrip: + """Smoke-test: writer output, fed back through the reader, preserves shape.""" + + def test_write_then_read_preserves_both_lists(self): + client, pca_results_table = _client_with_pca_results_only() + + # Record what the writer puts (REAL to_dynamo_dict shape: top-level + # consensus, repness with only comment_repness — verified 2026-07-04). + conv = MagicMock(name='Conversation') + conv.conversation_id = 42 + conv.to_dynamo_dict.return_value = { + 'participant_count': 10, + 'comment_count': 3, + 'group_count': 0, + 'pca': {}, + 'math_tick': 30000, + 'consensus': _make_consensus_dict(), + 'repness': { + 'comment_repness': [], + }, + } + client.write_conversation(conv) + written_item = pca_results_table.put_item.call_args.kwargs['Item'] + + # Replay the written Item back through the reader. + pca_results_table.get_item.return_value = {'Item': written_item} + + result = client.read_math_by_tick('42', 30000) + consensus = result['consensus'] + assert isinstance(consensus, dict) + assert [c['tid'] for c in consensus['agree']] == [1] + assert [c['tid'] for c in consensus['disagree']] == [2] diff --git a/delphi/tests/test_dynamodb_float_serialization.py b/delphi/tests/test_dynamodb_float_serialization.py index c8b33756c..1edf325d6 100644 --- a/delphi/tests/test_dynamodb_float_serialization.py +++ b/delphi/tests/test_dynamodb_float_serialization.py @@ -149,3 +149,119 @@ def test_repness_records_serialize(self): _assert_dynamodb_serializable( put_item, context="Delphi_RepresentativeComments Item" ) + + +# --------------------------------------------------------------------------- +# Layer 2b — comment priorities: value-preserving AND serializable +# --------------------------------------------------------------------------- + +class TestToDynamoDictPrioritiesSerialization: + """`to_dynamo_dict` must preserve priority VALUES, not truncate them. + + The old code did `int(priority)`: harmless today (the D12.6 bug-mirror + makes every priority exactly 49.0) but a landmine for the day issue + #2571 resolves and the real formula returns — real-data priorities span + ~0.18–31.46 (decisions doc D12.6), so `int()` floors sub-1 priorities + to 0. The TS server's weighted routing treats 0 as "no priority data": + those comments would silently never be routed. Values must round-trip + as Decimal (raw floats crash boto3's TypeSerializer). + """ + + # Real-data-shaped values: sub-1 (floors to 0 under int()), fractional + # mid-range (loses 46% of its weight under int()), and the current + # bug-mirror constant. + _PRIORITIES = {10: 0.18, 11: 31.46, 12: 49.0} + + def _conversation_with_priorities(self): + conv = Conversation(conversation_id='ztest-priorities') + conv.repness = conv_repness(_vote_matrix(), _groups()) + conv.comment_priorities = dict(self._PRIORITIES) + return conv + + def test_priorities_preserve_values(self): + conv = self._conversation_with_priorities() + dynamo_data = conv.to_dynamo_dict() + + priorities = dynamo_data['comment_priorities'] + assert priorities, "expected non-empty comment_priorities" + + for tid, expected in self._PRIORITIES.items(): + got = priorities[tid] + assert float(got) == pytest.approx(expected, abs=1e-9), ( + f"priority for tid {tid} not preserved: expected {expected}, " + f"got {got!r} (int() truncation floors sub-1 priorities to 0)" + ) + + def test_priorities_serialize_for_dynamodb(self): + conv = self._conversation_with_priorities() + dynamo_data = conv.to_dynamo_dict() + + for tid, value in dynamo_data['comment_priorities'].items(): + _assert_dynamodb_serializable( + value, context=f"comment_priorities[{tid}]" + ) + # The CommentRouting write path (dynamodb.py step 4) writes this + # value raw into `'priority': priorities.get(comment_id, 0)` — + # it must already be a DynamoDB scalar at this point. + +# --------------------------------------------------------------------------- +# Layer 2c — consensus entries: serializable through writer Site 1 +# --------------------------------------------------------------------------- + +class TestToDynamoDictConsensusSerialization: + """Consensus entries flowing through writer Site 1 must be boto3-safe. + + Caught by CI's test_math_pipeline_runs_e2e (2026-07-05): once the writer + read top-level `consensus` (the key to_dynamo_dict actually emits), REAL + D11 data flowed for the first time — carrying float p-success/p-test — + and boto3 rejected the Delphi_PCAResults put_item ("Float types are not + supported"). Only the LEGACY writer branch Decimal-converted; the + pre-formatted branch wrote `dynamo_data['consensus']` raw. Locally + invisible: the e2e test needs DynamoDB (skip list) and the round-trip + tests use MagicMock (no TypeSerializer) — hence this real-serializer pin. + """ + + _CONSENSUS = { + 'agree': [ + {'tid': 1, 'n-success': 3, 'n-trials': 5, + 'p-success': 0.6, 'p-test': 1.5}, + ], + 'disagree': [ + {'tid': 2, 'n-success': 4, 'n-trials': 5, + 'p-success': 0.8, 'p-test': 2.0}, + ], + } + + def _conversation_with_consensus(self): + conv = Conversation(conversation_id='ztest-consensus-decimal') + conv.repness = conv_repness(_vote_matrix(), _groups()) + # Inject non-empty consensus (the tiny fixture matrix does not clear + # the pa>0.5 & z-sig-90 filters on its own — avoid a vacuous test). + conv.repness['consensus_comments'] = { + side: [dict(e) for e in entries] + for side, entries in self._CONSENSUS.items() + } + conv.comment_priorities = {} + return conv + + def test_consensus_serializes_for_dynamodb(self): + conv = self._conversation_with_consensus() + dynamo_data = conv.to_dynamo_dict() + + consensus = dynamo_data['consensus'] + assert consensus['agree'] and consensus['disagree'], \ + "expected non-empty consensus (vacuous test otherwise)" + + # Exactly what writer Site 1 puts into the Delphi_PCAResults Item. + _assert_dynamodb_serializable( + consensus, context="Delphi_PCAResults consensus_comments") + + def test_consensus_values_preserved(self): + conv = self._conversation_with_consensus() + consensus = conv.to_dynamo_dict()['consensus'] + + entry = consensus['agree'][0] + assert entry['tid'] == 1 + assert float(entry['p-success']) == pytest.approx(0.6, abs=1e-9) + assert float(entry['p-test']) == pytest.approx(1.5, abs=1e-9) + diff --git a/delphi/tests/test_legacy_clojure_regression.py b/delphi/tests/test_legacy_clojure_regression.py index 6ad2c6aed..3eab53a5d 100644 --- a/delphi/tests/test_legacy_clojure_regression.py +++ b/delphi/tests/test_legacy_clojure_regression.py @@ -122,7 +122,7 @@ def test_basic_outputs(self, conversation_data): if conv.repness and 'comment_repness' in conv.repness: check.greater(len(conv.repness['comment_repness']), 0, "Should have representative comments") - def test_pca_components_match_clojure(self, conversation_data): + def test_pca_components_match_clojure(self, request, conversation_data): """ Test that PCA components match the Clojure implementation. @@ -133,6 +133,23 @@ def test_pca_components_match_clojure(self, conversation_data): Note: The centers will be negated due to vote sign convention difference (Python: agree=+1, Clojure: agree=-1), but the eigenvectors should match. """ + # Pre-existing CCR failures, verified identical on edge 722640eb0 + # (2026-07-04). Marked per-variant so every other variant keeps + # gating and an XPASS is visible the day the upstream fix lands. + _known_bad = { + 'bg2050-incremental': + "pre-existing: PC2 angle 10.71° vs ≤10° tolerance — " + "incremental PCA drift (D1 sign-flip/replay territory, " + "needs replay infra; journal 'incremental PCA dimensions')", + 'pakistan-incremental': + "pre-existing: PCA shape (2, 9030) vs Clojure (2, 194) — " + "incremental blob computed on a comment subset (large-conv " + "sampling/moderation divergence; journal 'incremental PCA " + "dimensions')", + } + if request.node.callspec.id in _known_bad: + request.applymarker(pytest.mark.xfail( + strict=False, reason=_known_bad[request.node.callspec.id])) import numpy as np conv = conversation_data['conv'] @@ -187,7 +204,7 @@ def test_pca_components_match_clojure(self, conversation_data): check.less_equal(norm_angle_deg, 10.0, f"PC{i+1} angle difference should be ≤10° (got {norm_angle_deg:.2f}°)") - def test_group_clustering(self, conversation_data): + def test_group_clustering(self, request, conversation_data): """ Test that group clustering matches the Clojure implementation. @@ -197,6 +214,14 @@ def test_group_clustering(self, conversation_data): Both sides are unfolded to participant-level membership for comparison. """ + # Pre-existing CCR failure, verified identical on edge 722640eb0 + # (2026-07-04). Per-variant so the other datasets keep gating. + if request.node.callspec.id == 'bg2018-cold_start': + request.applymarker(pytest.mark.xfail( + strict=False, + reason="pre-existing: bg2018 cold_start group membership " + "divergence (same family as the gid 0↔1 label-swap / " + "clustering-stability queue, S3-4 2026-06-11)")) conv = conversation_data['conv'] clojure_output = conversation_data['clojure_output'] dataset_name = conversation_data['dataset_name'] @@ -275,8 +300,7 @@ def test_group_clustering(self, conversation_data): check.is_true(result['overall_match'], f"Clustering should match Clojure output (distribution + membership)") - @pytest.mark.xfail(raises=AssertionError, strict=True, reason="D12: Comment priorities not yet implemented in Python") - def test_comment_priorities(self, conversation_data): + def test_comment_priorities(self, request, conversation_data): """ Test that comment priorities match the Clojure implementation exactly. @@ -288,6 +312,23 @@ def test_comment_priorities(self, conversation_data): clojure_output = conversation_data['clojure_output'] dataset_name = conversation_data['dataset_name'] + # Per-variant xfail (g5, refined 2026-07-05): known-bad only where + # the Clojure incremental blob has VARIED priorities (no truthy-0 + # bug there). FLI and bg2050 incremental blobs carry the all-49 + # signature and match Python's mirror — they gate, as do all + # cold_start variants. Drop this once the Clojure bug (#2571) is + # fixed and the Python mirror is removed. + _varied_priority_incrementals = ( + 'vw-incremental', 'biodiversity-incremental', + 'bg2018-incremental', 'engage-incremental', + 'pakistan-incremental') + if request.node.callspec.id in _varied_priority_incrementals: + request.applymarker(pytest.mark.xfail( + raises=AssertionError, strict=False, + reason="D12.6: this Clojure incremental blob has varied " + "priorities (no truthy-0 bug there); Python's " + "all-49 mirror cannot match. See issue #2571.")) + print(f"\n[{dataset_name}] Testing comment priorities...") has_python_priorities = hasattr(conv, 'comment_priorities') diff --git a/delphi/tests/test_legacy_repness_comparison.py b/delphi/tests/test_legacy_repness_comparison.py index e920aafde..a31668015 100644 --- a/delphi/tests/test_legacy_repness_comparison.py +++ b/delphi/tests/test_legacy_repness_comparison.py @@ -194,11 +194,21 @@ def _compare_results(self, py_results: Dict[str, Any], clj_results: Dict[str, An # Look for consensus comments if they exist if 'consensus-comments' in clj_repness: clj_consensus = clj_repness.get('consensus-comments', []) - py_consensus = py_results.get('consensus_comments', []) + # B2 fix (D11 sub-agent review): post-D11 (PR 9) Python's + # consensus_comments is a dict `{agree: [...], disagree: [...]}`, + # not a flat list. Flatten for the ID extraction below. + py_consensus_dict = py_results.get('consensus_comments', {}) + if isinstance(py_consensus_dict, dict): + py_consensus = (py_consensus_dict.get('agree', []) + + py_consensus_dict.get('disagree', [])) + else: + py_consensus = py_consensus_dict # legacy fallback # Extract comment IDs clj_consensus_ids = [str(c.get('comment-id', c.get('tid', c.get('comment_id', '')))) for c in clj_consensus] - py_consensus_ids = [str(c.get('comment_id', '')) for c in py_consensus] + # Python consensus entries use `tid` (Clojure blob shape, + # 2026-07-04); `comment_id` fallback covers pre-fix blobs. + py_consensus_ids = [str(c.get('tid', c.get('comment_id', ''))) for c in py_consensus] consensus_matches = set(clj_consensus_ids) & set(py_consensus_ids) consensus_total = len(set(clj_consensus_ids) | set(py_consensus_ids)) diff --git a/delphi/tests/test_old_format_repness.py b/delphi/tests/test_old_format_repness.py deleted file mode 100644 index 94459e8bd..000000000 --- a/delphi/tests/test_old_format_repness.py +++ /dev/null @@ -1,557 +0,0 @@ -""" -Tests for the representativeness module's backwards-compatible interface. - -These tests verify the single-group, single-comment "old format" API -that wraps the new DataFrame-native implementation. -""" - -import math -import numpy as np -import pandas as pd -import sys -import os - -# Add the parent directory to the path to import the module -sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), '..'))) - -from polismath.pca_kmeans_rep.repness import ( - PSEUDO_COUNT, - z_score_sig_90, z_score_sig_95, prop_test, two_prop_test, - comment_stats, add_comparative_stats, repness_metric, finalize_cmt_stats, - passes_by_test, best_agree, best_disagree, select_rep_comments, - select_consensus_comments, conv_repness, -) -from polismath.conversation.conversation import Conversation - - -class TestStatisticalFunctions: - """Tests for the statistical utility functions.""" - - def test_z_score_significance(self): - """Test z-score significance checks.""" - # 90% confidence — one-tailed, strict >, matching Clojure - assert z_score_sig_90(2.0) - assert not z_score_sig_90(1.2816) # boundary: not significant (strict >) - assert not z_score_sig_90(-1.2816) # negative: not significant (one-tailed) - assert not z_score_sig_90(1.0) - assert not z_score_sig_90(1.28) - - # 95% confidence — one-tailed, strict >, matching Clojure - assert z_score_sig_95(2.5) - assert not z_score_sig_95(1.6449) # boundary: not significant (strict >) - assert not z_score_sig_95(-1.6449) # negative: not significant (one-tailed) - assert not z_score_sig_95(1.5) - assert not z_score_sig_95(1.64) - - def test_prop_test(self): - """Test one-proportion z-test (Clojure formula: 2*sqrt(n+1)*((succ+1)/(n+1) - 0.5)).""" - # 70 successes out of 100 - assert np.isclose(prop_test(70, 100), - 2 * math.sqrt(101) * (71/101 - 0.5), atol=0.01) - # 10 successes out of 50 - assert np.isclose(prop_test(10, 50), - 2 * math.sqrt(51) * (11/51 - 0.5), atol=0.01) - - # Edge case: n=0 → 1.0 (Clojure parity — see scalar prop_test docstring) - assert prop_test(0, 0) == 1.0 - - def test_two_prop_test(self): - """Test two-proportion z-test with +1 pseudocounts (Clojure parity).""" - # two_prop_test(succ_in, succ_out, pop_in, pop_out) — raw counts - # After +1: pi1=71/101≈0.703, pi2=51/101≈0.505, z≈2.88 - assert np.isclose(two_prop_test(70, 50, 100, 100), 2.88, atol=0.1) - - # Equal proportions → z ≈ 0 - assert np.isclose(two_prop_test(25, 25, 50, 50), 0.0, atol=0.1) - - # pop_in=0 / pop_out=0: Clojure (stats.clj:18-33) increments all four - # inputs by 1 (no short-circuit), so pop=0 → pop=1 and the test proceeds. - # With succ_in=succ_out=5, pop_in=0, pop_out=100 → z ≈ 18.35 (positive). - # Symmetric case → z ≈ -18.35. - assert np.isclose(two_prop_test(5, 5, 0, 100), 18.3476, atol=0.01) - assert np.isclose(two_prop_test(5, 5, 100, 0), -18.3476, atol=0.01) - - -class TestCommentStats: - """Tests for comment statistics functions (old single-array interface).""" - - def test_comment_stats(self): - """Test basic comment statistics calculation.""" - # Create test votes: 3 agrees, 1 disagree, 1 pass - votes = np.array([1, 1, 1, -1, None]) - group_members = [0, 1, 2, 3, 4] - - stats = comment_stats(votes, group_members) - - assert stats['na'] == 3 - assert stats['nd'] == 1 - assert stats['ns'] == 4 - - # Check probabilities (with pseudocounts) - n_agree = 3 - n_disagree = 1 - n_votes = 4 - p_agree = (n_agree + PSEUDO_COUNT/2) / (n_votes + PSEUDO_COUNT) - p_disagree = (n_disagree + PSEUDO_COUNT/2) / (n_votes + PSEUDO_COUNT) - - assert np.isclose(stats['pa'], p_agree) - assert np.isclose(stats['pd'], p_disagree) - - # Test with no votes - empty_votes = np.array([None, None]) - empty_stats = comment_stats(empty_votes, [0, 1]) - - assert empty_stats['na'] == 0 - assert empty_stats['nd'] == 0 - assert empty_stats['ns'] == 0 - assert np.isclose(empty_stats['pa'], 0.5) - assert np.isclose(empty_stats['pd'], 0.5) - - def test_add_comparative_stats(self): - """Test adding comparative statistics.""" - # Group stats: 80% agree - group_stats = { - 'na': 8, - 'nd': 2, - 'ns': 10, - 'pa': 0.8, - 'pd': 0.2, - 'pat': 3.0, - 'pdt': -3.0 - } - - # Other group stats: 40% agree - other_stats = { - 'na': 4, - 'nd': 6, - 'ns': 10, - 'pa': 0.4, - 'pd': 0.6, - 'pat': -1.0, - 'pdt': 1.0 - } - - result = add_comparative_stats(group_stats, other_stats) - - # Check representativeness ratios - assert np.isclose(result['ra'], 0.8 / 0.4) - assert np.isclose(result['rd'], 0.2 / 0.6) - - # Test edge case with zero probability - other_stats_zero = { - 'na': 0, - 'nd': 10, - 'ns': 10, - 'pa': 0.0, - 'pd': 1.0, - 'pat': -5.0, - 'pdt': 5.0 - } - - result_zero = add_comparative_stats(group_stats, other_stats_zero) - assert np.isclose(result_zero['ra'], 1.0) # Should default to 1.0 - - def test_repness_metric(self): - """Test representativeness metric calculation.""" - stats = { - 'pa': 0.8, - 'pd': 0.2, - 'pat': 3.0, - 'pdt': -3.0, - 'ra': 2.0, - 'rd': 0.33, - 'rat': 2.5, - 'rdt': -2.5 - } - - # Calculate agree metric - agree_metric = repness_metric(stats, 'a') - # Clojure (repness.clj:191-193): (* ra rat pa pat) - expected_agree = 2.0 * 2.5 * 0.8 * 3.0 # = 12.0 - assert np.isclose(agree_metric, expected_agree) - - # Calculate disagree metric - disagree_metric = repness_metric(stats, 'd') - # Clojure: (* rd rdt pd pdt) — signed product, two negatives cancel - expected_disagree = 0.33 * (-2.5) * 0.2 * (-3.0) # = 0.495 - assert np.isclose(disagree_metric, expected_disagree) - - def test_finalize_cmt_stats(self): - """Test finalizing comment statistics.""" - # Stats where agree is more representative - agree_stats = { - 'pa': 0.8, - 'pd': 0.2, - 'pat': 3.0, - 'pdt': -3.0, - 'ra': 2.0, - 'rd': 0.33, - 'rat': 2.5, - 'rdt': -2.5 - } - - finalized_agree = finalize_cmt_stats(agree_stats) - - assert 'agree_metric' in finalized_agree - assert 'disagree_metric' in finalized_agree - assert finalized_agree['repful'] == 'agree' - - # Stats where disagree is more representative - disagree_stats = { - 'pa': 0.2, - 'pd': 0.8, - 'pat': -3.0, - 'pdt': 3.0, - 'ra': 0.33, - 'rd': 2.0, - 'rat': -2.5, - 'rdt': 2.5 - } - - finalized_disagree = finalize_cmt_stats(disagree_stats) - assert finalized_disagree['repful'] == 'disagree' - - -class TestSelectionFunctions: - """Tests for representative comment selection functions.""" - - def test_passes_by_test(self): - """Test checking if comments pass significance tests.""" - # Create stats that pass significance tests - passing_stats = { - 'pa': 0.8, - 'pd': 0.2, - 'pat': 3.0, - 'pdt': -3.0, - 'ra': 2.0, - 'rd': 0.33, - 'rat': 3.0, - 'rdt': -3.0 - } - - assert passes_by_test(passing_stats, 'agree') - assert not passes_by_test(passing_stats, 'disagree') - - # Create stats that don't pass (not significant) - failing_stats = { - 'pa': 0.8, - 'pd': 0.2, - 'pat': 1.0, # Below 90% threshold - 'pdt': -1.0, - 'ra': 2.0, - 'rd': 0.33, - 'rat': 1.0, # Below 90% threshold - 'rdt': -1.0 - } - - assert not passes_by_test(failing_stats, 'agree') - - def test_best_agree(self): - """Test filtering for best agreement comments.""" - # Create a mix of stats - stats = [ - { # Passes tests, high agreement - 'comment_id': 'c1', - 'pa': 0.8, 'pd': 0.2, - 'pat': 3.0, 'pdt': -3.0, - 'rat': 3.0, 'rdt': -3.0 - }, - { # Doesn't pass tests - 'comment_id': 'c2', - 'pa': 0.6, 'pd': 0.4, - 'pat': 1.0, 'pdt': -1.0, - 'rat': 1.0, 'rdt': -1.0 - }, - { # Not agreement (more disagree) - 'comment_id': 'c3', - 'pa': 0.3, 'pd': 0.7, - 'pat': -2.0, 'pdt': 2.0, - 'rat': -2.0, 'rdt': 2.0 - }, - { # Passes tests, moderate agreement - 'comment_id': 'c4', - 'pa': 0.7, 'pd': 0.3, - 'pat': 2.5, 'pdt': -2.5, - 'rat': 2.5, 'rdt': -2.5 - } - ] - - best = best_agree(stats) - - # Should return 2 comments that pass tests - assert len(best) == 2 - comment_ids = [s['comment_id'] for s in best] - assert 'c1' in comment_ids - assert 'c4' in comment_ids - assert 'c3' not in comment_ids - - def test_best_disagree(self): - """Test filtering for best disagreement comments.""" - # Create a mix of stats - stats = [ - { # Not disagreement (more agree) - 'comment_id': 'c1', - 'pa': 0.8, 'pd': 0.2, - 'pat': 3.0, 'pdt': -3.0, - 'rat': 3.0, 'rdt': -3.0 - }, - { # Disagreement but doesn't pass tests - 'comment_id': 'c2', - 'pa': 0.4, 'pd': 0.6, - 'pat': -1.0, 'pdt': 1.0, - 'rat': -1.0, 'rdt': 1.0 - }, - { # Passes tests, high disagreement - 'comment_id': 'c3', - 'pa': 0.2, 'pd': 0.8, - 'pat': -3.0, 'pdt': 3.0, - 'rat': -3.0, 'rdt': 3.0 - } - ] - - best = best_disagree(stats) - - # Should return 1 comment that passes tests - assert len(best) == 1 - assert best[0]['comment_id'] == 'c3' - - def test_select_rep_comments(self): - """Test selecting representative comments.""" - # Create a mix of stats - stats = [ - { # Strong agree - 'comment_id': 'c1', - 'pa': 0.9, 'pd': 0.1, - 'pat': 4.0, 'pdt': -4.0, - 'rat': 4.0, 'rdt': -4.0, - 'agree_metric': 7.2, - 'disagree_metric': 0.9 - }, - { # Moderate agree - 'comment_id': 'c2', - 'pa': 0.7, 'pd': 0.3, - 'pat': 2.0, 'pdt': -2.0, - 'rat': 2.0, 'rdt': -2.0, - 'agree_metric': 2.8, - 'disagree_metric': 1.2 - }, - { # Weak agree - 'comment_id': 'c3', - 'pa': 0.6, 'pd': 0.4, - 'pat': 1.0, 'pdt': -1.0, - 'rat': 1.0, 'rdt': -1.0, - 'agree_metric': 1.2, - 'disagree_metric': 0.8 - }, - { # Strong disagree - 'comment_id': 'c4', - 'pa': 0.1, 'pd': 0.9, - 'pat': -4.0, 'pdt': 4.0, - 'rat': -4.0, 'rdt': 4.0, - 'agree_metric': 0.8, - 'disagree_metric': 7.2 - }, - { # Moderate disagree - 'comment_id': 'c5', - 'pa': 0.3, 'pd': 0.7, - 'pat': -2.0, 'pdt': 2.0, - 'rat': -2.0, 'rdt': 2.0, - 'agree_metric': 1.2, - 'disagree_metric': 2.8 - } - ] - - # Set 'repful' for all stats to match the implementation - for stat in stats: - if stat.get('agree_metric', 0) >= stat.get('disagree_metric', 0): - stat['repful'] = 'agree' - else: - stat['repful'] = 'disagree' - - # Select with default counts - selected = select_rep_comments(stats) - - # Check that we get some representative comments - assert len(selected) > 0 - - # Verify that comments are properly marked - agree_comments = [s for s in selected if s['repful'] == 'agree'] - disagree_comments = [s for s in selected if s['repful'] == 'disagree'] - - # Make sure we have both types of comments if available - assert len(agree_comments) > 0 - assert len(disagree_comments) > 0 - - # Check that the order is by metrics - if len(agree_comments) >= 2: - assert agree_comments[0]['agree_metric'] >= agree_comments[1]['agree_metric'] - - if len(disagree_comments) >= 2: - assert disagree_comments[0]['disagree_metric'] >= disagree_comments[1]['disagree_metric'] - - # Test with different counts - selected_custom = select_rep_comments(stats, agree_count=2, disagree_count=1) - - assert len(selected_custom) == 3 - agree_count = sum(1 for s in selected_custom if s['repful'] == 'agree') - disagree_count = sum(1 for s in selected_custom if s['repful'] == 'disagree') - - assert agree_count == 2 - assert disagree_count == 1 - - # Test with empty stats - assert select_rep_comments([]) == [] - - -class TestConsensusAndGroupRepness: - """Tests for consensus and group representativeness functions.""" - - def test_select_consensus_comments(self): - """Test selecting consensus comments.""" - # Create stats for groups - group1_stats = [ - { - 'comment_id': 'c1', - 'group_id': 1, - 'pa': 0.8, 'pd': 0.2 - }, - { - 'comment_id': 'c2', - 'group_id': 1, - 'pa': 0.7, 'pd': 0.3 - } - ] - - group2_stats = [ - { - 'comment_id': 'c1', - 'group_id': 2, - 'pa': 0.85, 'pd': 0.15 - }, - { - 'comment_id': 'c2', - 'group_id': 2, - 'pa': 0.6, 'pd': 0.4 - }, - { - 'comment_id': 'c3', - 'group_id': 2, - 'pa': 0.9, 'pd': 0.1 - } - ] - - # Combine stats - all_stats = group1_stats + group2_stats - - consensus = select_consensus_comments(all_stats) - - # Comments with high agreement across all groups should be consensus - assert len(consensus) > 0 - - # Verify comment IDs in consensus list - both c1 and c2 have high agreement - consensus_ids = [c['comment_id'] for c in consensus] - - # At least one of these should be in the consensus - assert 'c1' in consensus_ids or 'c2' in consensus_ids - - # NOTE: The implementation actually sorts by average agreement - # c3 has the highest average agreement (0.9) but is only in one group - # So it's actually expected that c3 could be in the consensus - # Just verify that the implementation is consistent in its behavior - - # Check all consensus comments have the correct label - for comment in consensus: - assert comment['repful'] == 'consensus' - - -class TestIntegration: - """Integration tests for the representativeness module.""" - - def test_conv_repness(self): - """Test the main representativeness calculation function.""" - # Create a test vote matrix - vote_data = np.array([ - [1, 1, -1, None], # Participant 1 - [1, 1, -1, 1], # Participant 2 - [-1, -1, 1, -1], # Participant 3 - [-1, -1, 1, 1] # Participant 4 - ]) - - row_names = ['p1', 'p2', 'p3', 'p4'] - col_names = ['c1', 'c2', 'c3', 'c4'] - - vote_matrix = pd.DataFrame(vote_data, index=row_names, columns=col_names) - - # Create group clusters - group_clusters = [ - {'id': 1, 'members': ['p1', 'p2']}, # Group 1: mostly agrees with c1, c2 - {'id': 2, 'members': ['p3', 'p4']} # Group 2: mostly agrees with c3 - ] - - # Calculate representativeness - repness_result = conv_repness(vote_matrix, group_clusters) - - # Check result structure - assert 'comment_ids' in repness_result - assert 'group_repness' in repness_result - assert 'consensus_comments' in repness_result - - # Check group repness - assert 1 in repness_result['group_repness'] - assert 2 in repness_result['group_repness'] - - # Group 1 should identify c1/c2 as representative - group1_rep_ids = [s['comment_id'] for s in repness_result['group_repness'][1]] - assert 'c1' in group1_rep_ids or 'c2' in group1_rep_ids - - # Group 2 should identify c3 as representative - group2_rep_ids = [s['comment_id'] for s in repness_result['group_repness'][2]] - assert 'c3' in group2_rep_ids - - def test_participant_stats(self): - """Test participant statistics calculation via vectorized method.""" - # Create a test vote matrix - vote_data = np.array([ - [1, 1, -1, None], # Participant 1 - [1, 1, -1, 1], # Participant 2 - [-1, -1, 1, -1], # Participant 3 - [-1, -1, 1, 1] # Participant 4 - ]) - - row_names = ['p1', 'p2', 'p3', 'p4'] - col_names = ['c1', 'c2', 'c3', 'c4'] - - vote_matrix = pd.DataFrame(vote_data, index=row_names, columns=col_names) - - # Create group clusters. _compute_participant_info_optimized only - # reads 'id' and 'members'; 'center' is unused but kept to mirror - # the production cluster schema. - group_clusters = [ - {'id': 1, 'members': ['p1', 'p2'], 'center': [0.0]}, - {'id': 2, 'members': ['p3', 'p4'], 'center': [0.0]} - ] - - # Calculate participant stats using vectorized method - conv = Conversation("test") - ptpt_stats = conv._compute_participant_info_optimized(vote_matrix, group_clusters) - - # Check result structure - assert 'participant_ids' in ptpt_stats - assert 'stats' in ptpt_stats - - # Check participant stats - for ptpt_id in row_names: - assert ptpt_id in ptpt_stats['stats'] - stats = ptpt_stats['stats'][ptpt_id] - - assert 'n_agree' in stats - assert 'n_disagree' in stats - assert 'n_votes' in stats - assert 'group' in stats - assert 'group_correlations' in stats - - # Check specific stats - p1_stats = ptpt_stats['stats']['p1'] - assert p1_stats['n_agree'] == 2 - assert p1_stats['n_disagree'] == 1 - assert p1_stats['group'] == 1 diff --git a/delphi/tests/test_participant_info.py b/delphi/tests/test_participant_info.py index 84ede4f45..c8caaa8a6 100644 --- a/delphi/tests/test_participant_info.py +++ b/delphi/tests/test_participant_info.py @@ -659,6 +659,23 @@ def test_vectorized_matches_per_participant_corrcoef(dataset_name): ) +# PGR (Python golden) deferral — same treatment as test_regression.py +# (2026-06-11 decision): goldens shift on every Clojure-parity fix. The +# stored private-dataset goldens predate the gid label-swap fix +# (2026-07-05), which re-orders group ids to Clojure encounter order — +# per-(pid, group) correlations are keyed by gid, so all comparisons +# against pre-fix goldens fail by design, not by regression. +# REACTIVATION: re-record goldens + remove this mark at the +# Python-vs-Python phase. +_goldens_deferred = pytest.mark.skip( + reason="PGR goldens deferred during Clojure-parity phase (2026-06-11 " + "decision); stored goldens predate the gid label-swap fix " + "(2026-07-05). Re-record + reactivate at the Python-vs-Python " + "phase.", +) + + +@_goldens_deferred @_skip_golden @pytest.mark.use_discovered_datasets def test_participant_info_matches_golden(dataset_name): diff --git a/delphi/tests/test_pipeline_integrity.py b/delphi/tests/test_pipeline_integrity.py index 60d007e45..19a2e10de 100644 --- a/delphi/tests/test_pipeline_integrity.py +++ b/delphi/tests/test_pipeline_integrity.py @@ -195,10 +195,15 @@ def test_full_pipeline(dataset_name: str) -> None: print(f" Agree: {comment.get('pa', 0):.2f}, Disagree: {comment.get('pd', 0):.2f}") print(f" Metrics: A={comment.get('agree_metric', 0):.2f}, D={comment.get('disagree_metric', 0):.2f}") - # Check consensus comments + # Check consensus comments. Post-D11 (PR 9), shape is + # `{'agree': [...], 'disagree': [...]}` matching Clojure. print("\n Consensus Comments:") - for i, comment in enumerate(updated_conv.repness.get('consensus_comments', [])): - print(f" - Comment {i+1}: ID {comment.get('comment_id')}, Avg Agree: {comment.get('avg_agree', 0):.2f}") + consensus = updated_conv.repness.get('consensus_comments', {}) + for side in ('agree', 'disagree'): + for i, comment in enumerate(consensus.get(side, [])): + print(f" - {side} #{i+1}: ID {comment.get('tid')}, " + f"p-success={comment.get('p-success', 0):.2f}, " + f"p-test={comment.get('p-test', 0):.2f}") else: print(" No representativeness results available") diff --git a/delphi/tests/test_powerit_pca.py b/delphi/tests/test_powerit_pca.py new file mode 100644 index 000000000..81cb682a3 --- /dev/null +++ b/delphi/tests/test_powerit_pca.py @@ -0,0 +1,303 @@ +""" +Tests for the Clojure-parity power-iteration PCA port (`powerit_pca`). + +Clojure reference: math/src/polismath/math/pca.clj + - power-iteration (l.38-56): fixed iteration count (default 100), exact + eigenvalue-equality early exit, ones-padding of short start vectors. + - factor-matrix (l.66-76): per-component Gram-Schmidt deflation. + - powerit-pca (l.86-105): column-mean centering, n_comps clamped to + min(rows, cols), per-component start vector (provided or random). + - wrapped-pca (l.108-124): all-zero start vectors are treated as missing. + +Also covers the POLISMATH_PCA_IMPL env-var switch in pca_project_dataframe: +'powerit' (default, legacy/Clojure-parity) vs 'sklearn' (improved path). +""" + +import inspect + +import numpy as np +import pandas as pd +import pytest + +from polismath.pca_kmeans_rep.pca import ( + pca_project_dataframe, + powerit_pca, +) + + +# --------------------------------------------------------------------------- +# Helpers +# --------------------------------------------------------------------------- + +def _structured_data(n_rows: int = 60, n_cols: int = 12, seed: int = 7) -> np.ndarray: + """Dense data with a well-separated spectrum (fast power-iteration convergence).""" + rng = np.random.default_rng(seed) + v1 = rng.normal(size=n_cols) + v1 /= np.linalg.norm(v1) + v2 = rng.normal(size=n_cols) + v2 -= v1 * np.dot(v1, v2) + v2 /= np.linalg.norm(v2) + a = rng.normal(size=n_rows) + b = rng.normal(size=n_rows) + noise = rng.normal(scale=0.05, size=(n_rows, n_cols)) + return 5.0 * np.outer(a, v1) + 2.0 * np.outer(b, v2) + noise + + +def _votes_like_data(n_rows: int = 80, n_cols: int = 30, seed: int = 11) -> np.ndarray: + """Ternary vote-like matrix with two opinion groups (dense, no NaN).""" + rng = np.random.default_rng(seed) + group = rng.integers(0, 2, size=n_rows) + lean = np.where(group[:, None] == 0, 0.6, -0.6) + raw = lean + rng.normal(scale=0.8, size=(n_rows, n_cols)) + return np.sign(np.round(raw)).clip(-1, 1) + + +def _eigh_top_components(data: np.ndarray, n: int = 2) -> np.ndarray: + """Reference principal components via numpy.linalg.eigh of the scatter matrix.""" + centered = data - data.mean(axis=0) + scatter = centered.T @ centered + _, vecs = np.linalg.eigh(scatter) # ascending eigenvalues + return vecs[:, ::-1][:, :n].T # top-n as rows + + +def _angle_deg(u: np.ndarray, v: np.ndarray) -> float: + """Angle between two vectors in degrees, ignoring sign (eigenvector convention).""" + cos = abs(np.dot(u, v)) / (np.linalg.norm(u) * np.linalg.norm(v)) + return float(np.degrees(np.arccos(np.clip(cos, -1.0, 1.0)))) + + +def _vote_dataframe_with_nans(seed: int = 3) -> pd.DataFrame: + """Small vote DataFrame with NaNs, for pca_project_dataframe flag tests.""" + rng = np.random.default_rng(seed) + data = _votes_like_data(n_rows=25, n_cols=10, seed=seed).astype(float) + mask = rng.random(data.shape) < 0.3 + data[mask] = np.nan + # Belt: guarantee at least one vote per column so imputation is exercised, + # not the all-NaN-column fallback. + data[0, :] = 1.0 + return pd.DataFrame(data, + index=[f"p{i}" for i in range(data.shape[0])], + columns=[f"c{j}" for j in range(data.shape[1])]) + + +# --------------------------------------------------------------------------- +# Clojure-parity semantics +# --------------------------------------------------------------------------- + +class TestPoweritPcaCorrectness: + + def test_default_iters_matches_clojure(self): + """Clojure default: power-iteration iters=100 (pca.clj:43), and the + production pipeline also passes :pca-iters 100 (conversation.clj:146).""" + sig = inspect.signature(powerit_pca) + assert sig.parameters['iters'].default == 100 + + def test_center_is_column_mean(self): + data = _structured_data() + result = powerit_pca(data) + np.testing.assert_array_equal(result['center'], data.mean(axis=0)) + + def test_components_match_numpy_eigh(self): + """PC1/PC2 must match eigh of the scatter matrix (up to sign).""" + data = _structured_data() + result = powerit_pca(data, n_comps=2) + reference = _eigh_top_components(data, 2) + assert result['comps'].shape == (2, data.shape[1]) + pc1_angle = _angle_deg(result['comps'][0], reference[0]) + pc2_angle = _angle_deg(result['comps'][1], reference[1]) + # Well-separated spectrum + 100 fixed iterations => machine precision. + assert pc1_angle < 1e-3 + assert pc2_angle < 1e-3 + + def test_components_unit_norm_and_orthogonal(self): + data = _structured_data(seed=13) + comps = powerit_pca(data, n_comps=2)['comps'] + np.testing.assert_allclose(np.linalg.norm(comps, axis=1), 1.0, atol=1e-12) + # Gram-Schmidt deflation (factor-matrix) removes the PC1 direction. + assert abs(np.dot(comps[0], comps[1])) < 1e-8 + + def test_n_comps_clamped_to_data_dim(self): + """Clojure clamps to (min n-comps (min rows cols)) — pca.clj:93,96.""" + data = _structured_data(n_rows=3, n_cols=5, seed=5) + result = powerit_pca(data, n_comps=4) + assert result['comps'].shape == (3, 5) + + def test_votes_like_data_matches_eigh(self): + """Same check on ternary vote-like data (the production regime).""" + data = _votes_like_data() + comps = powerit_pca(data, n_comps=2)['comps'] + reference = _eigh_top_components(data, 2) + assert _angle_deg(comps[0], reference[0]) < 0.01 + assert _angle_deg(comps[1], reference[1]) < 0.01 + + +class TestPoweritPcaDeterminismAndStartVectors: + + def test_two_calls_bit_identical(self): + """Cold start must be deterministic (project invariant, 2026-07-05 + determinism verification) — unlike Clojure's unseeded (rand).""" + data = _votes_like_data(seed=17) + r1 = powerit_pca(data, n_comps=2) + r2 = powerit_pca(data, n_comps=2) + np.testing.assert_array_equal(r1['center'], r2['center']) + np.testing.assert_array_equal(r1['comps'], r2['comps']) + + def test_start_vectors_honored(self): + """With iters=0 power iteration does exactly one multiplication, so the + output is a direct function of the start vector: normalise(XᵀX·start).""" + data = _structured_data(seed=19) + centered = data - data.mean(axis=0) + rng = np.random.default_rng(23) + start = rng.random(data.shape[1]) + + result = powerit_pca(data, n_comps=1, iters=0, start_vectors=[start]) + expected = centered.T @ (centered @ start) + expected /= np.linalg.norm(expected) + np.testing.assert_allclose(result['comps'][0], expected, rtol=1e-12, atol=1e-12) + + # A different start vector must give a measurably different output. + other = rng.random(data.shape[1]) + result_other = powerit_pca(data, n_comps=1, iters=0, start_vectors=[other]) + assert _angle_deg(result['comps'][0], result_other['comps'][0]) > 0.001 + + def test_short_start_vector_padded_with_ones(self): + """Clojure pads short start vectors with 1s when new comments have + added columns (pca.clj:46-49).""" + data = _structured_data(seed=29) + n_cols = data.shape[1] + short = np.array([0.5, -0.25, 0.75]) + padded = np.concatenate([short, np.ones(n_cols - short.size)]) + + r_short = powerit_pca(data, n_comps=1, iters=0, start_vectors=[short]) + r_padded = powerit_pca(data, n_comps=1, iters=0, start_vectors=[padded]) + np.testing.assert_allclose(r_short['comps'][0], r_padded['comps'][0], + rtol=1e-12, atol=1e-12) + + def test_zero_start_vector_treated_as_missing(self): + """wrapped-pca maps all-zero start vectors to nil (pca.clj:122-123), + which powerit-pca replaces with a fresh start; ours is deterministic.""" + data = _votes_like_data(seed=31) + r_zero = powerit_pca(data, n_comps=2, start_vectors=[np.zeros(data.shape[1])]) + r_cold = powerit_pca(data, n_comps=2) + np.testing.assert_array_equal(r_zero['comps'], r_cold['comps']) + + def test_warm_start_converges_to_same_components(self): + """Warm-starting from the previous tick's comps (conversation.clj:385) + must land on the same components as a cold start (up to sign). + + NOT bit-identical: 100 FIXED iterations (no convergence criterion, + Clojure parity) leave truncation error that depends on the start. + Measured on this data: 0° (PC1) / 4.2e-5° (PC2, flatter residual + spectrum after deflation). Bound 1e-3° = ~24x headroom while still + far below any behaviorally relevant angle.""" + data = _votes_like_data(seed=37) + cold = powerit_pca(data, n_comps=2) + warm = powerit_pca(data, n_comps=2, start_vectors=list(cold['comps'])) + assert _angle_deg(cold['comps'][0], warm['comps'][0]) < 1e-3 + assert _angle_deg(cold['comps'][1], warm['comps'][1]) < 1e-3 + + +# --------------------------------------------------------------------------- +# POLISMATH_PCA_IMPL flag in pca_project_dataframe +# --------------------------------------------------------------------------- + +class TestPcaImplFlag: + + def test_default_is_powerit(self, monkeypatch): + """With the env var unset, pca_project_dataframe must use powerit_pca + (bit-identical comps) — legacy/parity mode is the default.""" + monkeypatch.delenv('POLISMATH_PCA_IMPL', raising=False) + df = _vote_dataframe_with_nans() + + pca_results, proj = pca_project_dataframe(df, n_comps=2) + + # Replicate the documented imputation: NaN -> column nanmean. + matrix = df.to_numpy(copy=True) + col_means = np.nanmean(matrix, axis=0) + nan_idx = np.where(np.isnan(matrix)) + matrix[nan_idx] = col_means[nan_idx[1]] + expected = powerit_pca(matrix, n_comps=2) + + np.testing.assert_array_equal(pca_results['comps'], expected['comps']) + np.testing.assert_array_equal(pca_results['center'], expected['center']) + assert len(proj) == df.shape[0] + + def test_sklearn_flag_selects_sklearn(self, monkeypatch): + """POLISMATH_PCA_IMPL=sklearn keeps the improved sklearn path: valid + shapes, and NOT bit-identical to the powerit solver output.""" + df = _vote_dataframe_with_nans() + + monkeypatch.setenv('POLISMATH_PCA_IMPL', 'powerit') + powerit_results, powerit_proj = pca_project_dataframe(df, n_comps=2) + + monkeypatch.setenv('POLISMATH_PCA_IMPL', 'sklearn') + sk_results, sk_proj = pca_project_dataframe(df, n_comps=2) + + for results, proj in ((powerit_results, powerit_proj), (sk_results, sk_proj)): + assert results['comps'].shape == (2, df.shape[1]) + assert results['center'].shape == (df.shape[1],) + assert np.all(np.isfinite(results['comps'])) + assert np.all(np.isfinite(results['center'])) + assert len(proj) == df.shape[0] + assert all(p.shape == (2,) for p in proj.values()) + + # Different solvers: same subspace but not the same bits. + assert not np.array_equal(powerit_results['comps'], sk_results['comps']) + # ... yet they must agree on the actual components (loose angle check; + # tight agreement is asserted in test_agreement_with_sklearn below). + for i in range(2): + assert _angle_deg(powerit_results['comps'][i], sk_results['comps'][i]) < 1.0 + + def test_invalid_flag_value_falls_back_to_default(self, monkeypatch): + df = _vote_dataframe_with_nans() + monkeypatch.setenv('POLISMATH_PCA_IMPL', 'powerit') + expected, _ = pca_project_dataframe(df, n_comps=2) + monkeypatch.setenv('POLISMATH_PCA_IMPL', 'not-a-solver') + got, _ = pca_project_dataframe(df, n_comps=2) + np.testing.assert_array_equal(got['comps'], expected['comps']) + + def test_flag_read_at_call_time(self, monkeypatch): + """The env var must be read per call (not cached at import time).""" + df = _vote_dataframe_with_nans() + monkeypatch.setenv('POLISMATH_PCA_IMPL', 'powerit') + r_powerit, _ = pca_project_dataframe(df, n_comps=2) + monkeypatch.setenv('POLISMATH_PCA_IMPL', 'sklearn') + r_sklearn, _ = pca_project_dataframe(df, n_comps=2) + assert not np.array_equal(r_powerit['comps'], r_sklearn['comps']) + + def test_pipeline_determinism_under_default(self, monkeypatch): + """Two identical pipeline calls under the default impl are bit-identical + (the 2026-07-05 determinism verification must keep holding).""" + monkeypatch.delenv('POLISMATH_PCA_IMPL', raising=False) + df = _vote_dataframe_with_nans(seed=41) + r1, p1 = pca_project_dataframe(df, n_comps=2) + r2, p2 = pca_project_dataframe(df, n_comps=2) + np.testing.assert_array_equal(r1['comps'], r2['comps']) + for pid in p1: + np.testing.assert_array_equal(p1[pid], p2[pid]) + + +# --------------------------------------------------------------------------- +# Agreement between the two solvers +# --------------------------------------------------------------------------- + +class TestSolverAgreement: + + def test_agreement_with_sklearn(self): + """powerit and sklearn solve the same eigenproblem; on vote-like data + with 100 fixed iterations they agree far below the 10° CCR tolerance. + Measured on this data: 0° (PC1) / 2.1e-3° (PC2, fixed-iters + truncation on the flatter post-deflation spectrum). Bound 0.1° = + ~47x headroom for BLAS/platform variation, still 100x below the CCR + tolerance and well under 1°.""" + from sklearn.decomposition import PCA + + data = _votes_like_data(n_rows=120, n_cols=40, seed=43) + powerit_comps = powerit_pca(data, n_comps=2)['comps'] + + sk = PCA(n_components=2, random_state=42) + sk.fit(data) + + for i in range(2): + angle = _angle_deg(powerit_comps[i], sk.components_[i]) + assert angle < 0.1, f"PC{i+1} angle powerit vs sklearn: {angle:.2e}°" diff --git a/delphi/tests/test_regression.py b/delphi/tests/test_regression.py index 4c8c18fde..e4558847c 100644 --- a/delphi/tests/test_regression.py +++ b/delphi/tests/test_regression.py @@ -26,6 +26,23 @@ reason="Golden snapshot tests disabled (SKIP_GOLDEN=1)", ) +# PGR (Python golden record) deferral — per Julien's 2026-06-11 decision +# (D10_D11_D12_GOLDENS_DECISIONS.md "Goldens re-record" + deferred-PRs +# handoff): goldens shift on every Clojure-parity fix and only add noise +# during the parity phase. The existing private-dataset goldens predate the +# D10/D11/D12 stack, so these comparisons fail by design, not by regression. +# REACTIVATION CONDITION: remove this mark and re-record all goldens +# (`uv run python scripts/regression_recorder.py `) when the +# Python-vs-Python refactor comparison phase begins — i.e. after the gid +# 0↔1 label-swap fix lands and batch outputs stabilize. +# (S3-5 2026-06-11 claimed this mark was applied; it never was — added +# 2026-07-04.) +_goldens_deferred = pytest.mark.skip( + reason="PGR goldens deferred during Clojure-parity phase (2026-06-11 " + "decision); stored goldens predate the D10/D11/D12 stack. " + "Re-record + reactivate at the Python-vs-Python phase.", +) + def _check_golden_exists(dataset_name: str): """ @@ -56,6 +73,7 @@ def _check_golden_exists(dataset_name: str): ) +@_goldens_deferred @_skip_golden @pytest.mark.use_discovered_datasets def test_conversation_regression(dataset_name): @@ -106,6 +124,7 @@ def test_conversation_regression(dataset_name): ) +@_goldens_deferred @_skip_golden @pytest.mark.use_discovered_datasets def test_conversation_stages_individually(dataset_name): diff --git a/delphi/tests/test_repness_smoke.py b/delphi/tests/test_repness_smoke.py index 99834025a..5361a0ecc 100644 --- a/delphi/tests/test_repness_smoke.py +++ b/delphi/tests/test_repness_smoke.py @@ -100,14 +100,20 @@ def test_repness_structure(self, dataset_name: str, conversation): assert 'repful' in comment # 'agree', 'disagree', or other type logger.debug(f"Group {group_id}: {len(comments)} representative comments") - # Check consensus comments if present + # Check consensus comments if present. Post-D11 (PR 9), shape is + # `{'agree': [...], 'disagree': [...]}` matching Clojure (repness.clj:322-323). if 'consensus_comments' in repness_results: consensus = repness_results['consensus_comments'] - logger.debug(f"Consensus comments: {len(consensus)}") - - if len(consensus) > 0: - comment = consensus[0] - assert 'comment_id' in comment + agree = consensus.get('agree', []) + disagree = consensus.get('disagree', []) + logger.debug(f"Consensus: {len(agree)} agree, {len(disagree)} disagree") + + # Consensus entries use the Clojure blob shape (2026-07-04, + # narrowed S1 deferral): tid + hyphenated stats keys. Rep-comment + # entries above keep `comment_id` until the math-blob alignment PR. + for entry in agree + disagree: + assert set(entry.keys()) == { + 'tid', 'n-success', 'n-trials', 'p-success', 'p-test'} logger.debug("✓ Representativeness structure validated") diff --git a/delphi/tests/test_repness_unit.py b/delphi/tests/test_repness_unit.py index 094a839b0..6733ea19d 100644 --- a/delphi/tests/test_repness_unit.py +++ b/delphi/tests/test_repness_unit.py @@ -7,20 +7,17 @@ import pandas as pd import sys import os -import math # Add the parent directory to the path to import the module sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), '..'))) from polismath.pca_kmeans_rep.repness import ( PSEUDO_COUNT, - z_score_sig_90, z_score_sig_95, prop_test, two_prop_test, - comment_stats, add_comparative_stats, repness_metric, finalize_cmt_stats, - passes_by_test, best_agree, best_disagree, select_rep_comments, - calculate_kl_divergence, select_consensus_comments, conv_repness, + z_score_sig_90, z_score_sig_95, conv_repness, # DataFrame-native vectorized functions - prop_test_vectorized, two_prop_test_vectorized, compute_group_comment_stats_df + prop_test_vectorized, two_prop_test_vectorized, compute_group_comment_stats_df, ) +from polismath.utils.general import AGREE, DISAGREE, PASS from polismath.conversation.conversation import Conversation @@ -43,437 +40,6 @@ def test_z_score_significance(self): assert not z_score_sig_95(1.5) assert not z_score_sig_95(1.64) - def test_prop_test(self): - """Test one-proportion z-test (Clojure formula: 2*sqrt(n+1)*((succ+1)/(n+1) - 0.5)).""" - # 70 successes out of 100: 2*sqrt(101)*((71/101)-0.5) = ~4.08 - assert np.isclose(prop_test(70, 100), - 2 * math.sqrt(101) * (71/101 - 0.5), atol=0.01) - # 10 successes out of 50: 2*sqrt(51)*((11/51)-0.5) = ~-4.06 - assert np.isclose(prop_test(10, 50), - 2 * math.sqrt(51) * (11/51 - 0.5), atol=0.01) - - # Edge case: n=0 → Clojure (stats.clj:10-15) has no guard; the +1 - # pseudocount turns (0, 0) into (1, 1), giving 2*sqrt(1)*(1/1 - 0.5) = 1.0. - assert prop_test(0, 0) == 1.0 - # Single trial: 2*sqrt(2)*((2/2)-0.5) = 2*1.414*0.5 = 1.414 - assert np.isclose(prop_test(1, 1), - 2 * math.sqrt(2) * 0.5, atol=0.01) - - def test_two_prop_test(self): - """Test two-proportion z-test with +1 pseudocounts (Clojure parity).""" - # two_prop_test(succ_in, succ_out, pop_in, pop_out) — raw counts - # Clojure adds +1 to all 4 inputs (stats.clj:20) - - # succ_in=70, succ_out=50, pop_in=100, pop_out=100 - # After +1: pi1=71/101≈0.703, pi2=51/101≈0.505, z≈2.88 - assert np.isclose(two_prop_test(70, 50, 100, 100), 2.88, atol=0.1) - - # Equal proportions → z ≈ 0 - assert np.isclose(two_prop_test(25, 25, 50, 50), 0.0, atol=0.1) - - # pop_in=0 / pop_out=0: Clojure (stats.clj:18-33) applies (map inc ...) - # to ALL FOUR inputs including the populations, so pop=0 becomes pop=1 - # and the test proceeds. With succ_in=succ_out=5, pop_in=0, pop_out=100: - # after +1, pi1=6/1=6, pi2=6/101≈0.0594, pi_hat=12/102≈0.1176, giving - # a very large positive z-score. The symmetric case is negative. - assert np.isclose(two_prop_test(5, 5, 0, 100), 18.3476, atol=0.01) - assert np.isclose(two_prop_test(5, 5, 100, 0), -18.3476, atol=0.01) - - -class TestCommentStats: - """Tests for comment statistics functions.""" - - def test_comment_stats(self): - """Test basic comment statistics calculation.""" - # Create test votes: 3 agrees, 1 disagree, 1 pass - votes = np.array([1, 1, 1, -1, None]) - group_members = [0, 1, 2, 3, 4] - - stats = comment_stats(votes, group_members) - - assert stats['na'] == 3 - assert stats['nd'] == 1 - assert stats['ns'] == 4 - - # Check probabilities (with pseudocounts) - n_agree = 3 - n_disagree = 1 - n_votes = 4 - p_agree = (n_agree + PSEUDO_COUNT/2) / (n_votes + PSEUDO_COUNT) - p_disagree = (n_disagree + PSEUDO_COUNT/2) / (n_votes + PSEUDO_COUNT) - - assert np.isclose(stats['pa'], p_agree) - assert np.isclose(stats['pd'], p_disagree) - - # Test with no votes - empty_votes = np.array([None, None]) - empty_stats = comment_stats(empty_votes, [0, 1]) - - assert empty_stats['na'] == 0 - assert empty_stats['nd'] == 0 - assert empty_stats['ns'] == 0 - assert np.isclose(empty_stats['pa'], 0.5) - assert np.isclose(empty_stats['pd'], 0.5) - # Clojure parity: with no votes, prop_test(0, 0) = 1.0 (no short-circuit). - # comment_stats should propagate that — no upstream gate on n_votes==0. - assert np.isclose(empty_stats['pat'], 1.0) - assert np.isclose(empty_stats['pdt'], 1.0) - - def test_add_comparative_stats(self): - """Test adding comparative statistics.""" - # Group stats: 80% agree - group_stats = { - 'na': 8, - 'nd': 2, - 'ns': 10, - 'pa': 0.8, - 'pd': 0.2, - 'pat': 3.0, - 'pdt': -3.0 - } - - # Other group stats: 40% agree - other_stats = { - 'na': 4, - 'nd': 6, - 'ns': 10, - 'pa': 0.4, - 'pd': 0.6, - 'pat': -1.0, - 'pdt': 1.0 - } - - result = add_comparative_stats(group_stats, other_stats) - - # Check representativeness ratios - assert np.isclose(result['ra'], 0.8 / 0.4) - assert np.isclose(result['rd'], 0.2 / 0.6) - - # Test edge case with zero probability - other_stats_zero = { - 'na': 0, - 'nd': 10, - 'ns': 10, - 'pa': 0.0, - 'pd': 1.0, - 'pat': -5.0, - 'pdt': 5.0 - } - - result_zero = add_comparative_stats(group_stats, other_stats_zero) - assert np.isclose(result_zero['ra'], 1.0) # Should default to 1.0 - - def test_repness_metric(self): - """Test representativeness metric calculation.""" - stats = { - 'pa': 0.8, - 'pd': 0.2, - 'pat': 3.0, - 'pdt': -3.0, - 'ra': 2.0, - 'rd': 0.33, - 'rat': 2.5, - 'rdt': -2.5 - } - - # Clojure formula (repness.clj:191-193): (* repness repness-test p-success p-test) - # → agree: ra * rat * pa * pat - # → disagree: rd * rdt * pd * pdt (signed product — two negatives cancel) - agree_metric = repness_metric(stats, 'a') - expected_agree = 2.0 * 2.5 * 0.8 * 3.0 # = 12.0 - assert np.isclose(agree_metric, expected_agree) - - disagree_metric = repness_metric(stats, 'd') - expected_disagree = 0.33 * (-2.5) * 0.2 * (-3.0) # = 0.495 - assert np.isclose(disagree_metric, expected_disagree) - - def test_finalize_cmt_stats(self): - """Test finalizing comment statistics.""" - # Stats where agree is more representative - agree_stats = { - 'pa': 0.8, - 'pd': 0.2, - 'pat': 3.0, - 'pdt': -3.0, - 'ra': 2.0, - 'rd': 0.33, - 'rat': 2.5, - 'rdt': -2.5 - } - - finalized_agree = finalize_cmt_stats(agree_stats) - - assert 'agree_metric' in finalized_agree - assert 'disagree_metric' in finalized_agree - assert finalized_agree['repful'] == 'agree' - - # Stats where disagree is more representative - disagree_stats = { - 'pa': 0.2, - 'pd': 0.8, - 'pat': -3.0, - 'pdt': 3.0, - 'ra': 0.33, - 'rd': 2.0, - 'rat': -2.5, - 'rdt': 2.5 - } - - finalized_disagree = finalize_cmt_stats(disagree_stats) - assert finalized_disagree['repful'] == 'disagree' - - -class TestSelectionFunctions: - """Tests for representative comment selection functions.""" - - def test_passes_by_test(self): - """Test checking if comments pass significance tests.""" - # Create stats that pass significance tests - passing_stats = { - 'pa': 0.8, - 'pd': 0.2, - 'pat': 3.0, - 'pdt': -3.0, - 'ra': 2.0, - 'rd': 0.33, - 'rat': 3.0, - 'rdt': -3.0 - } - - assert passes_by_test(passing_stats, 'agree') - assert not passes_by_test(passing_stats, 'disagree') - - # Create stats that don't pass (not significant) - failing_stats = { - 'pa': 0.8, - 'pd': 0.2, - 'pat': 1.0, # Below 90% threshold - 'pdt': -1.0, - 'ra': 2.0, - 'rd': 0.33, - 'rat': 1.0, # Below 90% threshold - 'rdt': -1.0 - } - - assert not passes_by_test(failing_stats, 'agree') - - def test_best_agree(self): - """Test filtering for best agreement comments.""" - # Create a mix of stats - stats = [ - { # Passes tests, high agreement - 'comment_id': 'c1', - 'pa': 0.8, 'pd': 0.2, - 'pat': 3.0, 'pdt': -3.0, - 'rat': 3.0, 'rdt': -3.0 - }, - { # Doesn't pass tests - 'comment_id': 'c2', - 'pa': 0.6, 'pd': 0.4, - 'pat': 1.0, 'pdt': -1.0, - 'rat': 1.0, 'rdt': -1.0 - }, - { # Not agreement (more disagree) - 'comment_id': 'c3', - 'pa': 0.3, 'pd': 0.7, - 'pat': -2.0, 'pdt': 2.0, - 'rat': -2.0, 'rdt': 2.0 - }, - { # Passes tests, moderate agreement - 'comment_id': 'c4', - 'pa': 0.7, 'pd': 0.3, - 'pat': 2.5, 'pdt': -2.5, - 'rat': 2.5, 'rdt': -2.5 - } - ] - - best = best_agree(stats) - - # Should return 2 comments that pass tests - assert len(best) == 2 - comment_ids = [s['comment_id'] for s in best] - assert 'c1' in comment_ids - assert 'c4' in comment_ids - assert 'c3' not in comment_ids - - def test_best_disagree(self): - """Test filtering for best disagreement comments.""" - # Create a mix of stats - stats = [ - { # Not disagreement (more agree) - 'comment_id': 'c1', - 'pa': 0.8, 'pd': 0.2, - 'pat': 3.0, 'pdt': -3.0, - 'rat': 3.0, 'rdt': -3.0 - }, - { # Disagreement but doesn't pass tests - 'comment_id': 'c2', - 'pa': 0.4, 'pd': 0.6, - 'pat': -1.0, 'pdt': 1.0, - 'rat': -1.0, 'rdt': 1.0 - }, - { # Passes tests, high disagreement - 'comment_id': 'c3', - 'pa': 0.2, 'pd': 0.8, - 'pat': -3.0, 'pdt': 3.0, - 'rat': -3.0, 'rdt': 3.0 - } - ] - - best = best_disagree(stats) - - # Should return 1 comment that passes tests - assert len(best) == 1 - assert best[0]['comment_id'] == 'c3' - - def test_select_rep_comments(self): - """Test selecting representative comments.""" - # Create a mix of stats - stats = [ - { # Strong agree - 'comment_id': 'c1', - 'pa': 0.9, 'pd': 0.1, - 'pat': 4.0, 'pdt': -4.0, - 'rat': 4.0, 'rdt': -4.0, - 'agree_metric': 7.2, - 'disagree_metric': 0.9 - }, - { # Moderate agree - 'comment_id': 'c2', - 'pa': 0.7, 'pd': 0.3, - 'pat': 2.0, 'pdt': -2.0, - 'rat': 2.0, 'rdt': -2.0, - 'agree_metric': 2.8, - 'disagree_metric': 1.2 - }, - { # Weak agree - 'comment_id': 'c3', - 'pa': 0.6, 'pd': 0.4, - 'pat': 1.0, 'pdt': -1.0, - 'rat': 1.0, 'rdt': -1.0, - 'agree_metric': 1.2, - 'disagree_metric': 0.8 - }, - { # Strong disagree - 'comment_id': 'c4', - 'pa': 0.1, 'pd': 0.9, - 'pat': -4.0, 'pdt': 4.0, - 'rat': -4.0, 'rdt': 4.0, - 'agree_metric': 0.8, - 'disagree_metric': 7.2 - }, - { # Moderate disagree - 'comment_id': 'c5', - 'pa': 0.3, 'pd': 0.7, - 'pat': -2.0, 'pdt': 2.0, - 'rat': -2.0, 'rdt': 2.0, - 'agree_metric': 1.2, - 'disagree_metric': 2.8 - } - ] - - # Set 'repful' for all stats to match the implementation - for stat in stats: - if stat.get('agree_metric', 0) >= stat.get('disagree_metric', 0): - stat['repful'] = 'agree' - else: - stat['repful'] = 'disagree' - - # Select with default counts - selected = select_rep_comments(stats) - - # Check that we get some representative comments - assert len(selected) > 0 - - # Verify that comments are properly marked - agree_comments = [s for s in selected if s['repful'] == 'agree'] - disagree_comments = [s for s in selected if s['repful'] == 'disagree'] - - # Make sure we have both types of comments if available - assert len(agree_comments) > 0 - assert len(disagree_comments) > 0 - - # Check that the order is by metrics - if len(agree_comments) >= 2: - assert agree_comments[0]['agree_metric'] >= agree_comments[1]['agree_metric'] - - if len(disagree_comments) >= 2: - assert disagree_comments[0]['disagree_metric'] >= disagree_comments[1]['disagree_metric'] - - # Test with different counts - selected_custom = select_rep_comments(stats, agree_count=2, disagree_count=1) - - assert len(selected_custom) == 3 - agree_count = sum(1 for s in selected_custom if s['repful'] == 'agree') - disagree_count = sum(1 for s in selected_custom if s['repful'] == 'disagree') - - assert agree_count == 2 - assert disagree_count == 1 - - # Test with empty stats - assert select_rep_comments([]) == [] - - -class TestConsensusAndGroupRepness: - """Tests for consensus and group representativeness functions.""" - - def test_select_consensus_comments(self): - """Test selecting consensus comments.""" - # Create stats for groups - group1_stats = [ - { - 'comment_id': 'c1', - 'group_id': 1, - 'pa': 0.8, 'pd': 0.2 - }, - { - 'comment_id': 'c2', - 'group_id': 1, - 'pa': 0.7, 'pd': 0.3 - } - ] - - group2_stats = [ - { - 'comment_id': 'c1', - 'group_id': 2, - 'pa': 0.85, 'pd': 0.15 - }, - { - 'comment_id': 'c2', - 'group_id': 2, - 'pa': 0.6, 'pd': 0.4 - }, - { - 'comment_id': 'c3', - 'group_id': 2, - 'pa': 0.9, 'pd': 0.1 - } - ] - - # Combine stats - all_stats = group1_stats + group2_stats - - consensus = select_consensus_comments(all_stats) - - # Comments with high agreement across all groups should be consensus - assert len(consensus) > 0 - - # Verify comment IDs in consensus list - both c1 and c2 have high agreement - consensus_ids = [c['comment_id'] for c in consensus] - - # At least one of these should be in the consensus - assert 'c1' in consensus_ids or 'c2' in consensus_ids - - # NOTE: The implementation actually sorts by average agreement - # c3 has the highest average agreement (0.9) but is only in one group - # So it's actually expected that c3 could be in the consensus - # Just verify that the implementation is consistent in its behavior - - # Check all consensus comments have the correct label - for comment in consensus: - assert comment['repful'] == 'consensus' - class TestIntegration: """Integration tests for the representativeness module.""" @@ -571,6 +137,23 @@ def test_participant_stats(self): class TestVectorizedFunctions: """Tests for DataFrame-native vectorized functions.""" + @staticmethod + def _prop_test_reference(succ, n): + """Closed-form Clojure prop-test (stats.clj:10-15). +1 pseudocount, no n=0 guard.""" + return 2 * np.sqrt(n + 1) * ((succ + 1) / (n + 1) - 0.5) + + @staticmethod + def _two_prop_test_reference(succ_in, succ_out, pop_in, pop_out): + """Closed-form Clojure two-prop-test (stats.clj:18-33). +1 pseudocount on all 4.""" + s1, s2 = succ_in + 1, succ_out + 1 + p1, p2 = pop_in + 1, pop_out + 1 + pi1, pi2 = s1 / p1, s2 / p2 + pi_hat = (s1 + s2) / (p1 + p2) + if pi_hat == 1.0: + return 0.0 + se = np.sqrt(pi_hat * (1 - pi_hat) * (1/p1 + 1/p2)) + return (pi1 - pi2) / se + def test_prop_test_vectorized(self): """Test vectorized one-proportion z-test (Clojure formula).""" succ = pd.Series([70, 10, 50]) @@ -578,23 +161,20 @@ def test_prop_test_vectorized(self): result = prop_test_vectorized(succ, n) - # Compare with scalar version - assert np.isclose(result.iloc[0], prop_test(70, 100), atol=0.01) - assert np.isclose(result.iloc[1], prop_test(10, 50), atol=0.01) - assert np.isclose(result.iloc[2], prop_test(50, 100), atol=0.01) + # Compare with closed-form reference + for i, (s, m) in enumerate(zip(succ, n)): + assert np.isclose(result.iloc[i], self._prop_test_reference(s, m), atol=0.01) def test_prop_test_vectorized_edge_cases(self): """Vectorized prop test n=0 → 1.0 (Clojure parity, no short-circuit). - Cross-checks scalar/vectorized agreement on the n=0 boundary. + (0, 0) → (1, 1) after +1 → 2*sqrt(1)*(1/1 - 0.5) = 1.0. """ succ = pd.Series([0, 70]) n = pd.Series([0, 100]) result = prop_test_vectorized(succ, n) - # Clojure parity: (0, 0) → (1, 1) after +1 → 2*sqrt(1)*(1/1 - 0.5) = 1.0 - assert np.isclose(result.iloc[0], prop_test(0, 0), atol=1e-10) assert np.isclose(result.iloc[0], 1.0, atol=1e-10) assert not np.isnan(result.iloc[1]) # normal case @@ -608,16 +188,18 @@ def test_two_prop_test_vectorized(self): result = two_prop_test_vectorized(succ_in, succ_out, pop_in, pop_out) - # Compare with scalar version - assert np.isclose(result.iloc[0], two_prop_test(70, 50, 100, 100), atol=0.01) - assert np.isclose(result.iloc[1], two_prop_test(10, 15, 50, 50), atol=0.01) + # Compare with closed-form reference + for i, (sin, sout, pin, pout) in enumerate(zip(succ_in, succ_out, pop_in, pop_out)): + assert np.isclose(result.iloc[i], + self._two_prop_test_reference(sin, sout, pin, pout), + atol=0.01) def test_two_prop_test_vectorized_edge_cases(self): """Vectorized two-prop test: pop=0 must match Clojure parity (no short-circuit). Clojure (stats.clj:18-33) applies (map inc ...) to all four inputs, so - pop=0 → pop=1 and the test proceeds. We pair each pop=0 case with the - scalar version to confirm scalar/vectorized agreement. + pop=0 → pop=1 and the test proceeds. Hand-verified reference values pin + the behavior on the two relevant boundaries. """ # Row 0: (5, 5, 0, 100) — pop_in=0 → expect large positive z (≈18.35) # Row 1: (5, 5, 0, 10) — pop_in=0 AND pi_hat=1 by coincidence → 0 @@ -628,10 +210,12 @@ def test_two_prop_test_vectorized_edge_cases(self): result = two_prop_test_vectorized(succ_in, succ_out, pop_in, pop_out) - assert np.isclose(result.iloc[0], two_prop_test(5, 5, 0, 100), atol=0.01) - assert np.isclose(result.iloc[1], two_prop_test(5, 5, 0, 10), atol=0.01) + # Row 0: closed-form via the reference helper. + assert np.isclose(result.iloc[0], + self._two_prop_test_reference(5, 5, 0, 100), atol=0.01) assert np.isclose(result.iloc[0], 18.3476, atol=0.01) - assert result.iloc[1] == 0.0 # pi_hat=1 coincidence after +1 pseudocount + # Row 1: pi_hat=1 coincidence after +1 → 0.0 (closed-form returns 0). + assert result.iloc[1] == 0.0 def test_compute_group_comment_stats_df(self): """Test vectorized computation of group/comment statistics.""" @@ -710,8 +294,9 @@ def test_compute_group_comment_stats_df_empty(self): assert stats_df.empty - def test_compute_group_comment_stats_matches_scalar(self): - """Test that vectorized results match scalar function results.""" + def test_compute_group_comment_stats_consistency_with_conv_repness(self): + """Sanity: per-(gid, tid) pa/pd from compute_group_comment_stats_df match + the values surfaced in conv_repness's comment_repness output.""" # Create test data vote_data = np.array([ [1, 1, -1], # p1 @@ -750,4 +335,125 @@ def test_compute_group_comment_stats_matches_scalar(self): df_row = stats_df.loc[(gid, tid)] assert np.isclose(entry['pa'], df_row['pa'], atol=1e-10) - assert np.isclose(entry['pd'], df_row['pd'], atol=1e-10) \ No newline at end of file + assert np.isclose(entry['pd'], df_row['pd'], atol=1e-10) + + +class TestNsIncludesPassVotes: + """ns / total_votes must count agree + disagree + PASS (Clojure parity). + + Clojure (math/src/polismath/math/repness.clj:56-61, :70): + (defn- count-votes [votes & [vote]] + (let [filt-fn (if vote #(= vote %) identity)] + (count (filter filt-fn votes)))) + ... + :ns (fnk [votes] (count-votes votes)) + + `count-votes` is called with no `vote` arg → `filt-fn = identity`. In + Clojure, 0 is truthy, so `(filter identity ...)` keeps every non-nil + entry — including PASS (0). Therefore ns = na + nd + np (PASS count). + + Python had ns = na + nd, silently dropping PASS. Every downstream metric + (pa, pd, pat, pdt, ra, rd, rat, rdt, agree_metric, disagree_metric, + consensus stats) was off whenever PASS votes existed. D5 BlobInjection + tests bypassed `compute_group_comment_stats_df` entirely (they feed a + pre-baked stats blob), so the bug was invisible there — pure-formula + tests are the only way to RED it. + """ + + def test_ns_includes_pass_votes(self): + """ns counts AGREE + DISAGREE + PASS, not just AGREE + DISAGREE.""" + # 5 ptpts, 1 comment, mixed votes: 2 agree, 1 disagree, 2 pass. + # Clojure ns = count of all non-nil = 5. + # Buggy Python ns = na + nd = 3. + votes_long = pd.DataFrame({ + 'participant': ['p1', 'p2', 'p3', 'p4', 'p5'], + 'comment': ['c1'] * 5, + 'vote': [AGREE, AGREE, DISAGREE, PASS, PASS], + }) + group_clusters = [{'id': 0, 'members': ['p1', 'p2', 'p3', 'p4', 'p5']}] + + stats_df = compute_group_comment_stats_df(votes_long, group_clusters) + row = stats_df.loc[(0, 'c1')] + + assert row['na'] == 2 + assert row['nd'] == 1 + assert row['ns'] == 5, ( + f"ns should include PASS (Clojure parity); got {row['ns']}" + ) + + def test_ns_all_pass_column(self): + """All-PASS column: na=0, nd=0, ns=3 (not 0).""" + votes_long = pd.DataFrame({ + 'participant': ['p1', 'p2', 'p3'], + 'comment': ['c1'] * 3, + 'vote': [PASS, PASS, PASS], + }) + group_clusters = [{'id': 0, 'members': ['p1', 'p2', 'p3']}] + + stats_df = compute_group_comment_stats_df(votes_long, group_clusters) + row = stats_df.loc[(0, 'c1')] + + assert row['na'] == 0 + assert row['nd'] == 0 + assert row['ns'] == 3, ( + f"All-PASS column should still have ns=3 (Clojure parity); " + f"got {row['ns']}" + ) + + def test_ns_mixed_with_nan_only_explicit_votes_count(self): + """NaN (unvoted) must NOT count; only explicit AGREE/DISAGREE/PASS do.""" + # 6 ptpts on c1: 1 agree, 1 disagree, 2 pass, 2 unvoted (NaN). + # Clojure parity: ns = 4 (the 4 explicit votes). NaN never counts. + votes_long = pd.DataFrame({ + 'participant': ['p1', 'p2', 'p3', 'p4', 'p5', 'p6'], + 'comment': ['c1'] * 6, + 'vote': [AGREE, DISAGREE, PASS, PASS, np.nan, np.nan], + }) + group_clusters = [{'id': 0, 'members': ['p1', 'p2', 'p3', 'p4', 'p5', 'p6']}] + + stats_df = compute_group_comment_stats_df(votes_long, group_clusters) + row = stats_df.loc[(0, 'c1')] + + assert row['na'] == 1 + assert row['nd'] == 1 + assert row['ns'] == 4, ( + f"ns must include PASS but exclude NaN; got {row['ns']}" + ) + + def test_other_votes_includes_other_group_pass(self): + """`other_votes` = total_votes - ns must include PASS in BOTH halves. + + Two groups, one comment. Group 0 votes [AGREE, PASS], group 1 votes + [DISAGREE, PASS]. Total na=1, nd=1, total_votes (Clojure) = 4. + Group 0: na=1, nd=0, ns=2 → other_votes=2 (the group-1 disagree + pass). + Group 1: na=0, nd=1, ns=2 → other_votes=2 (the group-0 agree + pass). + """ + votes_long = pd.DataFrame({ + 'participant': ['p1', 'p2', 'p3', 'p4'], + 'comment': ['c1'] * 4, + 'vote': [AGREE, PASS, DISAGREE, PASS], + }) + group_clusters = [ + {'id': 0, 'members': ['p1', 'p2']}, + {'id': 1, 'members': ['p3', 'p4']}, + ] + + stats_df = compute_group_comment_stats_df(votes_long, group_clusters) + + g0 = stats_df.loc[(0, 'c1')] + assert g0['na'] == 1 + assert g0['nd'] == 0 + assert g0['ns'] == 2, f"group 0 ns should include its PASS; got {g0['ns']}" + assert g0['other_votes'] == 2, ( + f"group 0 other_votes should include group-1 PASS; " + f"got {g0['other_votes']}" + ) + + g1 = stats_df.loc[(1, 'c1')] + assert g1['na'] == 0 + assert g1['nd'] == 1 + assert g1['ns'] == 2, f"group 1 ns should include its PASS; got {g1['ns']}" + assert g1['other_votes'] == 2, ( + f"group 1 other_votes should include group-0 PASS; " + f"got {g1['other_votes']}" + ) \ No newline at end of file