Skip to content

feat(math): clojure-legacy blob emission + selection parity — Clojure-exact math_main at step 0#2649

Draft
jucor wants to merge 1 commit into
spr/edge/d1ae732bfrom
spr/edge/0add28f3
Draft

feat(math): clojure-legacy blob emission + selection parity — Clojure-exact math_main at step 0#2649
jucor wants to merge 1 commit into
spr/edge/d1ae732bfrom
spr/edge/0add28f3

Conversation

@jucor

@jucor jucor commented Jul 22, 2026

Copy link
Copy Markdown
Collaborator

Aligns clojure-legacy emission and selection semantics with Clojure's
prep-main blob (conv_man.clj:45-74), per the step-0 battery diagnosis
(journal 2026-07-22 session 3; fingerprints in docs/divergences.json).
Battery effect: vw:uniform8 (8 steps), vw:single-cut and
biodiversity:uniform8 (8 steps) went from DIVERGENCE to full MATCH;
vw:front-loaded6 remains open (base-cluster count edge at small N).

Emission (Conversation.to_dict, legacy-gated, improved unchanged):

  • group-clusters: members emitted as BASE-CLUSTER ids (Clojure folded
    form); unfolded pids stay under the snake alias group_clusters.
  • votes-base: per-base-cluster A/D/S bucket lists over sort-by-id
    clustered members (agg-bucket-votes-for-tid, conversation.clj:601-608)
    instead of whole-matrix int totals.
  • pca: comment-projection + comment-extremity emitted (existing D12 fns).
  • Sign parity: pca.center, base-clusters.x/y, group-clusters centers,
    comment-projection negated at emission — Delphi's matrix is the
    NEGATION of Clojure's (AGREE=+1 vs -1); comps are covariance-derived
    and already equal (verified max|clj+py| = 1e-16 on center).
  • repness: {gid: [finalize-cmt-stats-shaped entries]} (repness.clj:173),
    internal dict preserved under repness_full for from_dict round-trip.
  • mod-in/mod-out/lastModTimestamp: null until moderation applied.
  • tids emitted in Clojure column (first-vote arrival) order with pca
    arrays permuted alongside; from_dict inverts the center sign and
    restores the arrival tracker in legacy mode.

Selection/stat parity:

  • group-aware-consensus multiplies (A+1)/(S+2) over EVERY group — a
    zero-S group contributes 1/2, not skipped (conversation.clj:639-641).
  • repness rest-stats sum over the OTHER GROUPS only (repness.clj:125),
    excluding unclustered voters from the comparison domain.
  • Exact-score ties in repness/consensus selection resolve by Clojure's
    stable sort over named-matrix column order: update_votes tracks
    first-vote arrival order, conv_repness accepts tid_order and the
    selectors iterate as-given in legacy mode.

Also: skip the three math-tree-hashing harness tests in delphi-only CI
images (no /math in the container; run 29881478306's 3 failures).

TDD: tests/test_legacy_blob_shape.py (25 tests, RED observed per fix).

commit-id:0add28f3


Stack:


⚠️ Part of a stack created by spr. Do not merge manually using the UI - doing so may have unexpected results.

This was referenced Jul 22, 2026
@jucor jucor changed the title feat(math): clojure-legacy blob emission + selection parity — Clojure-exact math_main at step 0 feat(math): clojure-legacy blob emission + selection parity — Clojure… Jul 22, 2026
@jucor
jucor changed the base branch from spr/edge/d1ae732b to edge July 22, 2026 02:10
@jucor jucor changed the title feat(math): clojure-legacy blob emission + selection parity — Clojure… feat(math): clojure-legacy blob emission + selection parity — Clojure-exact math_main at step 0 Jul 22, 2026
@jucor
jucor changed the base branch from edge to spr/edge/d1ae732b July 22, 2026 02:10
@jucor
jucor force-pushed the spr/edge/0add28f3 branch from f5a2c36 to aa5a214 Compare July 22, 2026 02:31
@jucor jucor changed the title feat(math): clojure-legacy blob emission + selection parity — Clojure-exact math_main at step 0 feat(math): clojure-legacy blob emission + selection parity — Clojure… Jul 22, 2026
@jucor
jucor changed the base branch from spr/edge/d1ae732b to edge July 22, 2026 03:54
@jucor
jucor changed the base branch from edge to spr/edge/d1ae732b July 22, 2026 03:54
…-exact math_main at step 0

Aligns clojure-legacy emission and selection semantics with Clojure's
prep-main blob (conv_man.clj:45-74), per the step-0 battery diagnosis
(journal 2026-07-22 session 3; fingerprints in docs/divergences.json).
Battery effect: vw:uniform8 (8 steps), vw:single-cut and
biodiversity:uniform8 (8 steps) went from DIVERGENCE to full MATCH;
vw:front-loaded6 remains open (base-cluster count edge at small N).

Emission (Conversation.to_dict, legacy-gated, improved unchanged):
- group-clusters: members emitted as BASE-CLUSTER ids (Clojure folded
  form); unfolded pids stay under the snake alias group_clusters.
- votes-base: per-base-cluster A/D/S bucket lists over sort-by-id
  clustered members (agg-bucket-votes-for-tid, conversation.clj:601-608)
  instead of whole-matrix int totals.
- pca: comment-projection + comment-extremity emitted (existing D12 fns).
- Sign parity: pca.center, base-clusters.x/y, group-clusters centers,
  comment-projection negated at emission — Delphi's matrix is the
  NEGATION of Clojure's (AGREE=+1 vs -1); comps are covariance-derived
  and already equal (verified max|clj+py| = 1e-16 on center).
- repness: {gid: [finalize-cmt-stats-shaped entries]} (repness.clj:173),
  internal dict preserved under repness_full for from_dict round-trip.
- mod-in/mod-out/lastModTimestamp: null until moderation applied.
- tids emitted in Clojure column (first-vote arrival) order with pca
  arrays permuted alongside; from_dict inverts the center sign and
  restores the arrival tracker in legacy mode.

Selection/stat parity:
- group-aware-consensus multiplies (A+1)/(S+2) over EVERY group — a
  zero-S group contributes 1/2, not skipped (conversation.clj:639-641).
- repness rest-stats sum over the OTHER GROUPS only (repness.clj:125),
  excluding unclustered voters from the comparison domain.
- Exact-score ties in repness/consensus selection resolve by Clojure's
  stable sort over named-matrix column order: update_votes tracks
  first-vote arrival order, conv_repness accepts tid_order and the
  selectors iterate as-given in legacy mode.

Also: skip the three math-tree-hashing harness tests in delphi-only CI
images (no /math in the container; run 29881478306's 3 failures).

TDD: tests/test_legacy_blob_shape.py (25 tests, RED observed per fix).

commit-id:0add28f3
@jucor jucor changed the title feat(math): clojure-legacy blob emission + selection parity — Clojure-exact math_main at step 0 feat(math): clojure-legacy blob emission + selection parity — Clojure… Jul 22, 2026
@jucor
jucor changed the base branch from spr/edge/d1ae732b to edge July 22, 2026 06:16
@jucor
jucor force-pushed the spr/edge/0add28f3 branch from aa5a214 to 5e139c1 Compare July 22, 2026 06:16
@jucor jucor changed the title feat(math): clojure-legacy blob emission + selection parity — Clojure… feat(math): clojure-legacy blob emission + selection parity — Clojure-exact math_main at step 0 Jul 22, 2026
@jucor
jucor changed the base branch from edge to spr/edge/d1ae732b July 22, 2026 06:16
@github-actions

Copy link
Copy Markdown

Delphi Coverage Report

File Stmts Miss Cover
init.py 2 0 100%
benchmarks/bench_pca.py 128 107 16%
benchmarks/bench_repness.py 81 65 20%
benchmarks/bench_update_votes.py 38 28 26%
benchmarks/benchmark_utils.py 34 24 29%
components/init.py 1 0 100%
components/config.py 165 133 19%
conversation/init.py 2 0 100%
conversation/conversation.py 1299 261 80%
conversation/manager.py 131 42 68%
database/init.py 1 0 100%
database/dynamodb.py 395 189 52%
database/postgres.py 310 166 46%
pca_kmeans_rep/init.py 5 0 100%
pca_kmeans_rep/clusters.py 257 21 92%
pca_kmeans_rep/corr.py 98 17 83%
pca_kmeans_rep/group_k_smoother.py 28 0 100%
pca_kmeans_rep/legacy_kmeans.py 151 6 96%
pca_kmeans_rep/pca.py 130 18 86%
pca_kmeans_rep/repness.py 223 6 97%
poller/init.py 4 0 100%
poller/math_writer.py 41 1 98%
poller/service.py 235 62 74%
poller/worker_pool.py 75 0 100%
regression/init.py 4 0 100%
regression/clojure_comparer.py 188 20 89%
regression/comparer.py 887 649 27%
regression/datasets.py 135 27 80%
regression/recorder.py 36 27 25%
regression/utils.py 130 85 35%
replay/init.py 2 0 100%
replay/certify.py 452 121 73%
replay/crosslang.py 128 3 98%
replay/driver.py 51 2 96%
replay/prodclone.py 184 16 91%
replay/real_data.py 19 2 89%
replay/schedule.py 125 6 95%
replay/stepcompare.py 77 1 99%
replay/store.py 120 11 91%
replay/types.py 65 8 88%
run_math_pipeline.py 261 114 56%
umap_narrative/500_generate_embedding_umap_cluster.py 210 109 48%
umap_narrative/501_calculate_comment_extremity.py 112 53 53%
umap_narrative/502_calculate_priorities.py 135 135 0%
umap_narrative/700_datamapplot_for_layer.py 502 502 0%
umap_narrative/701_static_datamapplot_for_layer.py 310 310 0%
umap_narrative/702_consensus_divisive_datamapplot.py 432 432 0%
umap_narrative/801_narrative_report_batch.py 785 785 0%
umap_narrative/802_process_batch_results.py 268 268 0%
umap_narrative/803_check_batch_status.py 183 183 0%
umap_narrative/llm_factory_constructor/init.py 2 2 0%
umap_narrative/llm_factory_constructor/model_provider.py 192 192 0%
umap_narrative/polismath_commentgraph/init.py 1 0 100%
umap_narrative/polismath_commentgraph/cli.py 270 270 0%
umap_narrative/polismath_commentgraph/core/init.py 3 3 0%
umap_narrative/polismath_commentgraph/core/clustering.py 108 108 0%
umap_narrative/polismath_commentgraph/core/embedding.py 104 104 0%
umap_narrative/polismath_commentgraph/lambda_handler.py 219 219 0%
umap_narrative/polismath_commentgraph/schemas/init.py 2 0 100%
umap_narrative/polismath_commentgraph/schemas/dynamo_models.py 160 9 94%
umap_narrative/polismath_commentgraph/tests/conftest.py 17 17 0%
umap_narrative/polismath_commentgraph/tests/test_clustering.py 74 74 0%
umap_narrative/polismath_commentgraph/tests/test_embedding.py 55 55 0%
umap_narrative/polismath_commentgraph/tests/test_storage.py 87 87 0%
umap_narrative/polismath_commentgraph/utils/init.py 3 0 100%
umap_narrative/polismath_commentgraph/utils/converter.py 283 237 16%
umap_narrative/polismath_commentgraph/utils/group_data.py 354 336 5%
umap_narrative/polismath_commentgraph/utils/storage.py 584 518 11%
umap_narrative/reset_conversation.py 159 50 69%
umap_narrative/run_pipeline.py 453 312 31%
utils/engine_mode.py 9 0 100%
utils/env_flags.py 13 0 100%
utils/general.py 62 41 34%
utils/serialization.py 9 5 44%
Total 12858 7654 40%

@jucor jucor changed the title feat(math): clojure-legacy blob emission + selection parity — Clojure-exact math_main at step 0 feat(math): clojure-legacy blob emission + selection parity — Clojure… Jul 22, 2026
@jucor
jucor changed the base branch from spr/edge/d1ae732b to edge July 22, 2026 06:50
@jucor jucor changed the title feat(math): clojure-legacy blob emission + selection parity — Clojure… feat(math): clojure-legacy blob emission + selection parity — Clojure-exact math_main at step 0 Jul 22, 2026
@jucor
jucor changed the base branch from edge to spr/edge/d1ae732b July 22, 2026 06:50
@jucor jucor changed the title feat(math): clojure-legacy blob emission + selection parity — Clojure-exact math_main at step 0 feat(math): clojure-legacy blob emission + selection parity — Clojure… Jul 22, 2026
@jucor
jucor changed the base branch from spr/edge/d1ae732b to edge July 22, 2026 08:11
@jucor jucor changed the title feat(math): clojure-legacy blob emission + selection parity — Clojure… feat(math): clojure-legacy blob emission + selection parity — Clojure-exact math_main at step 0 Jul 22, 2026
@jucor
jucor changed the base branch from edge to spr/edge/d1ae732b July 22, 2026 08:11
@jucor jucor changed the title feat(math): clojure-legacy blob emission + selection parity — Clojure-exact math_main at step 0 feat(math): clojure-legacy blob emission + selection parity — Clojure… Jul 22, 2026
@jucor
jucor changed the base branch from spr/edge/d1ae732b to edge July 22, 2026 09:58
@jucor jucor changed the title feat(math): clojure-legacy blob emission + selection parity — Clojure… feat(math): clojure-legacy blob emission + selection parity — Clojure-exact math_main at step 0 Jul 22, 2026
@jucor
jucor changed the base branch from edge to spr/edge/d1ae732b July 22, 2026 09:58
@jucor jucor changed the title feat(math): clojure-legacy blob emission + selection parity — Clojure-exact math_main at step 0 feat(math): clojure-legacy blob emission + selection parity — Clojure… Jul 22, 2026
@jucor
jucor changed the base branch from spr/edge/d1ae732b to edge July 22, 2026 11:21
@jucor jucor changed the title feat(math): clojure-legacy blob emission + selection parity — Clojure… feat(math): clojure-legacy blob emission + selection parity — Clojure-exact math_main at step 0 Jul 22, 2026
@jucor
jucor changed the base branch from edge to spr/edge/d1ae732b July 22, 2026 11:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant