Skip to content

feat(math): group-k-smoother in clojure-legacy engine mode (PR-D smoother)#2620

Draft
jucor wants to merge 1 commit into
spr/edge/be96f1dafrom
spr/edge/d4dbbfa9
Draft

feat(math): group-k-smoother in clojure-legacy engine mode (PR-D smoother)#2620
jucor wants to merge 1 commit into
spr/edge/be96f1dafrom
spr/edge/d4dbbfa9

Conversation

@jucor

@jucor jucor commented Jul 18, 2026

Copy link
Copy Markdown
Collaborator

Ports Clojure's :group-k-smoother (conversation.clj:454-478), which damps K
(the number of opinion groups) so it only switches to a new best value after
:group-k-buffer (= 4, conversation.clj:154) consecutive ticks agree. Default
'improved' mode keeps the existing best_k logic bit-for-bit.

  • New polismath/pca_kmeans_rep/group_k_smoother.py: pure
    group_k_smoother_update(prev_state, silhouettes_by_k, buffer=4) ->
    (new_state, smoothed_k). Implements the exact Clojure rule, INCLUDING:
    • HIGHER-k-wins tie-break (Clojure apply max-key ... (keys ...) returns
      the LAST maximal arg; array-map keys iterate ascending — so ties go to
      the higher k). conversation.py's improved best_k uses strict '>' (LOWER
      k wins) and is left untouched.
    • the Fix stale group-k-smoother crash on large conversations #2536 clamp (conversation.clj:469-478): a carried smoothed_k absent
      from the current clusterings falls back to this_k — smoothed_k is always
      a valid key, so downstream never KeyErrors.
    • first-tick acceptance of this_k (smoothed_k None), the cold-start
      invariant that keeps the two modes identical on tick 1.
      Only the top-level smoother is ported; Python has no subgroups
      (subgroup_clusters is hardcoded {}), so the parallel subgroup smoother
      (conversation.clj:520-560) is intentionally omitted.
  • Conversation._compute_clusters: in legacy mode computes silhouettes_by_k from
    group_clusterings, runs the smoother against prev_group_k_smoother (threaded
    via recompute(), PR-A), picks group_clusters = group_clusterings[smoothed_k],
    and stores the new smoother state + per-k clusterings on the conv (NOT
    persisted — conv_man.clj:52-74). Improved mode: selected_k = best_k, and the
    new fields stay {} (inert).

Semantic judgment call (integrator, please verify): the degenerate early-return
paths in _compute_clusters (<2 in-conv ptpts / <2 base clusters) do NOT touch
the smoother state, so a degenerate tick preserves the prior in-memory smoother
memory rather than resetting it; the clamp protects the next real tick.

TDD RED evidence (before implementing group_k_smoother.py):
tests/test_group_k_smoother.py collection error:
"ModuleNotFoundError: No module named
'polismath.pca_kmeans_rep.group_k_smoother'"

Tests: 12 new — 10 pure-function (buffer counting, reset-on-change, brief
alternation, clamp present/absent, first-tick, higher-k tie-break) + 2 pipeline
integration (legacy no-flicker-then-switch-after-4 via silhouette stub over 9
chained update_votes ticks: smoothed_k == [2,2,2,2,2,2,2,3,3]; improved mode
leaves the smoother inert). Full suite 434 passed, 17 skipped, 47 xfailed.

Co-Authored-By: Claude Fable 5 noreply@anthropic.com

commit-id:d4dbbfa9


Stack:


⚠️ Part of a stack created by spr. Do not merge manually using the UI - doing so may have unexpected results.

@jucor
jucor marked this pull request as draft July 18, 2026 01:48
@jucor
jucor force-pushed the spr/edge/d4dbbfa9 branch from e289f05 to 9128aa8 Compare July 18, 2026 01:59
@jucor jucor changed the title feat(math): group-k-smoother in clojure-legacy engine mode (PR-D smoother) feat(math): group-k-smoother in clojure-legacy engine mode (PR-D smoo… Jul 18, 2026
@jucor
jucor changed the base branch from spr/edge/b39add7e to edge July 18, 2026 13:27
@jucor
jucor force-pushed the spr/edge/d4dbbfa9 branch from 9128aa8 to d22ebc2 Compare July 18, 2026 13:28
@jucor jucor changed the title feat(math): group-k-smoother in clojure-legacy engine mode (PR-D smoo… feat(math): group-k-smoother in clojure-legacy engine mode (PR-D smoother) Jul 18, 2026
@jucor
jucor changed the base branch from edge to spr/edge/81a957ec July 18, 2026 13:28
@jucor
jucor force-pushed the spr/edge/d4dbbfa9 branch from d22ebc2 to c49fbb9 Compare July 18, 2026 13:32
…ther)

Ports Clojure's :group-k-smoother (conversation.clj:454-478), which damps K
(the number of opinion groups) so it only switches to a new best value after
:group-k-buffer (= 4, conversation.clj:154) consecutive ticks agree. Default
'improved' mode keeps the existing best_k logic bit-for-bit.

- New polismath/pca_kmeans_rep/group_k_smoother.py: pure
  group_k_smoother_update(prev_state, silhouettes_by_k, buffer=4) ->
  (new_state, smoothed_k). Implements the exact Clojure rule, INCLUDING:
    * HIGHER-k-wins tie-break (Clojure `apply max-key ... (keys ...)` returns
      the LAST maximal arg; array-map keys iterate ascending — so ties go to
      the higher k). conversation.py's improved best_k uses strict '>' (LOWER
      k wins) and is left untouched.
    * the #2536 clamp (conversation.clj:469-478): a carried smoothed_k absent
      from the current clusterings falls back to this_k — smoothed_k is always
      a valid key, so downstream never KeyErrors.
    * first-tick acceptance of this_k (smoothed_k None), the cold-start
      invariant that keeps the two modes identical on tick 1.
  Only the top-level smoother is ported; Python has no subgroups
  (subgroup_clusters is hardcoded {}), so the parallel subgroup smoother
  (conversation.clj:520-560) is intentionally omitted.
- Conversation._compute_clusters: in legacy mode computes silhouettes_by_k from
  group_clusterings, runs the smoother against prev_group_k_smoother (threaded
  via recompute(), PR-A), picks group_clusters = group_clusterings[smoothed_k],
  and stores the new smoother state + per-k clusterings on the conv (NOT
  persisted — conv_man.clj:52-74). Improved mode: selected_k = best_k, and the
  new fields stay {} (inert).

Semantic judgment call (integrator, please verify): the degenerate early-return
paths in _compute_clusters (<2 in-conv ptpts / <2 base clusters) do NOT touch
the smoother state, so a degenerate tick preserves the prior in-memory smoother
memory rather than resetting it; the clamp protects the next real tick.

TDD RED evidence (before implementing group_k_smoother.py):
  tests/test_group_k_smoother.py collection error:
  "ModuleNotFoundError: No module named
   'polismath.pca_kmeans_rep.group_k_smoother'"

Tests: 12 new — 10 pure-function (buffer counting, reset-on-change, brief
alternation, clamp present/absent, first-tick, higher-k tie-break) + 2 pipeline
integration (legacy no-flicker-then-switch-after-4 via silhouette stub over 9
chained update_votes ticks: smoothed_k == [2,2,2,2,2,2,2,3,3]; improved mode
leaves the smoother inert). Full suite 434 passed, 17 skipped, 47 xfailed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

commit-id:d4dbbfa9
@github-actions

Copy link
Copy Markdown

Delphi Coverage Report

File Stmts Miss Cover
init.py 2 0 100%
benchmarks/bench_pca.py 128 107 16%
benchmarks/bench_repness.py 81 65 20%
benchmarks/bench_update_votes.py 38 28 26%
benchmarks/benchmark_utils.py 34 24 29%
components/init.py 1 0 100%
components/config.py 165 133 19%
conversation/init.py 2 0 100%
conversation/conversation.py 1130 258 77%
conversation/manager.py 131 42 68%
database/init.py 1 0 100%
database/dynamodb.py 395 189 52%
database/postgres.py 306 206 33%
pca_kmeans_rep/init.py 5 0 100%
pca_kmeans_rep/clusters.py 257 21 92%
pca_kmeans_rep/corr.py 98 17 83%
pca_kmeans_rep/group_k_smoother.py 26 0 100%
pca_kmeans_rep/pca.py 139 18 87%
pca_kmeans_rep/repness.py 208 6 97%
regression/init.py 4 0 100%
regression/clojure_comparer.py 188 20 89%
regression/comparer.py 887 661 25%
regression/datasets.py 135 27 80%
regression/recorder.py 36 27 25%
regression/utils.py 138 94 32%
replay/init.py 2 0 100%
replay/driver.py 51 2 96%
replay/real_data.py 19 2 89%
replay/schedule.py 125 6 95%
replay/stepcompare.py 76 1 99%
replay/store.py 119 11 91%
replay/types.py 65 8 88%
run_math_pipeline.py 261 114 56%
umap_narrative/500_generate_embedding_umap_cluster.py 210 109 48%
umap_narrative/501_calculate_comment_extremity.py 112 53 53%
umap_narrative/502_calculate_priorities.py 135 135 0%
umap_narrative/700_datamapplot_for_layer.py 502 502 0%
umap_narrative/701_static_datamapplot_for_layer.py 310 310 0%
umap_narrative/702_consensus_divisive_datamapplot.py 432 432 0%
umap_narrative/801_narrative_report_batch.py 785 785 0%
umap_narrative/802_process_batch_results.py 268 268 0%
umap_narrative/803_check_batch_status.py 183 183 0%
umap_narrative/llm_factory_constructor/init.py 2 2 0%
umap_narrative/llm_factory_constructor/model_provider.py 192 192 0%
umap_narrative/polismath_commentgraph/init.py 1 0 100%
umap_narrative/polismath_commentgraph/cli.py 270 270 0%
umap_narrative/polismath_commentgraph/core/init.py 3 3 0%
umap_narrative/polismath_commentgraph/core/clustering.py 108 108 0%
umap_narrative/polismath_commentgraph/core/embedding.py 104 104 0%
umap_narrative/polismath_commentgraph/lambda_handler.py 219 219 0%
umap_narrative/polismath_commentgraph/schemas/init.py 2 0 100%
umap_narrative/polismath_commentgraph/schemas/dynamo_models.py 160 9 94%
umap_narrative/polismath_commentgraph/tests/conftest.py 17 17 0%
umap_narrative/polismath_commentgraph/tests/test_clustering.py 74 74 0%
umap_narrative/polismath_commentgraph/tests/test_embedding.py 55 55 0%
umap_narrative/polismath_commentgraph/tests/test_storage.py 87 87 0%
umap_narrative/polismath_commentgraph/utils/init.py 3 0 100%
umap_narrative/polismath_commentgraph/utils/converter.py 283 237 16%
umap_narrative/polismath_commentgraph/utils/group_data.py 354 336 5%
umap_narrative/polismath_commentgraph/utils/storage.py 584 518 11%
umap_narrative/reset_conversation.py 159 50 69%
umap_narrative/run_pipeline.py 453 312 31%
utils/engine_mode.py 9 0 100%
utils/general.py 62 41 34%
Total 11391 7498 34%

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant