Skip to content

feat(math): Q11 — replicate vectorz distance cancellation in legacy kmeans#2654

Draft
jucor wants to merge 1 commit into
spr/edge/00d8bc52from
spr/edge/7ae9c688
Draft

feat(math): Q11 — replicate vectorz distance cancellation in legacy kmeans#2654
jucor wants to merge 1 commit into
spr/edge/00d8bc52from
spr/edge/7ae9c688

Conversation

@jucor

@jucor jucor commented Jul 22, 2026

Copy link
Copy Markdown
Collaborator

Clojure's kmeans distances go through vectorz's matrix/distance on
ArraySubVector row views, which computes d^2 = |a|^2+|b|^2-2ab; the
cancellation floors true distances below ~1e-8 to EXACTLY 0.0, so
near-coincident points TIE against multiple clusters and min-key's
last-wins tie-break merges them into the LATER cluster (the emptied one
drops). Verified in-process on the vw every-vote step-57 pair (true
distance 4.66e-15 -> both cluster distances 0.0 -> merge), via the new
cache-safe diagnostic math/dev/proj_probe.clj, which also reproduces the
recording chain bit-exactly. CLOJURE_QUIRKS.md Q11.

_euclidean in legacy_kmeans.py now uses the same formula (module runs
only in clojure-legacy mode — gated by construction). numpy reproduces
the 0.0 collapse bit-for-bit on the real pair. The pre-Q11
all-singletons expectation in test_base_cluster_lineage.py pinned
python's old diff-based distance, not Clojure — updated to the verified
contract (merges only ever at Q11-distance 0.0).

TDD: RED on the real step-57 pair (unit + warm-start merge integration).

commit-id:7ae9c688


Stack:


⚠️ Part of a stack created by spr. Do not merge manually using the UI - doing so may have unexpected results.

This was referenced Jul 22, 2026
…means

Clojure's kmeans distances go through vectorz's matrix/distance on
ArraySubVector row views, which computes d^2 = |a|^2+|b|^2-2ab; the
cancellation floors true distances below ~1e-8 to EXACTLY 0.0, so
near-coincident points TIE against multiple clusters and min-key's
last-wins tie-break merges them into the LATER cluster (the emptied one
drops). Verified in-process on the vw every-vote step-57 pair (true
distance 4.66e-15 -> both cluster distances 0.0 -> merge), via the new
cache-safe diagnostic math/dev/proj_probe.clj, which also reproduces the
recording chain bit-exactly. CLOJURE_QUIRKS.md Q11.

_euclidean in legacy_kmeans.py now uses the same formula (module runs
only in clojure-legacy mode — gated by construction). numpy reproduces
the 0.0 collapse bit-for-bit on the real pair. The pre-Q11
all-singletons expectation in test_base_cluster_lineage.py pinned
python's old diff-based distance, not Clojure — updated to the verified
contract (merges only ever at Q11-distance 0.0).

TDD: RED on the real step-57 pair (unit + warm-start merge integration).

commit-id:7ae9c688
@jucor jucor changed the title feat(math): Q11 — replicate vectorz distance cancellation in legacy kmeans feat(math): Q11 — replicate vectorz distance cancellation in legacy k… Jul 22, 2026
@jucor
jucor changed the base branch from spr/edge/00d8bc52 to edge July 22, 2026 09:58
@jucor
jucor force-pushed the spr/edge/7ae9c688 branch from 20e58ae to 2ac4c32 Compare July 22, 2026 09:58
@jucor jucor changed the title feat(math): Q11 — replicate vectorz distance cancellation in legacy k… feat(math): Q11 — replicate vectorz distance cancellation in legacy kmeans Jul 22, 2026
@jucor
jucor changed the base branch from edge to spr/edge/00d8bc52 July 22, 2026 09:58
@github-actions

Copy link
Copy Markdown

Delphi Coverage Report

File Stmts Miss Cover
init.py 2 0 100%
benchmarks/bench_pca.py 128 107 16%
benchmarks/bench_repness.py 81 65 20%
benchmarks/bench_update_votes.py 38 28 26%
benchmarks/benchmark_utils.py 34 24 29%
components/init.py 1 0 100%
components/config.py 165 133 19%
conversation/init.py 2 0 100%
conversation/conversation.py 1304 261 80%
conversation/manager.py 131 42 68%
database/init.py 1 0 100%
database/dynamodb.py 395 189 52%
database/postgres.py 310 166 46%
pca_kmeans_rep/init.py 5 0 100%
pca_kmeans_rep/clusters.py 257 21 92%
pca_kmeans_rep/corr.py 98 17 83%
pca_kmeans_rep/group_k_smoother.py 28 0 100%
pca_kmeans_rep/legacy_kmeans.py 154 5 97%
pca_kmeans_rep/pca.py 133 18 86%
pca_kmeans_rep/repness.py 224 6 97%
poller/init.py 4 0 100%
poller/math_writer.py 41 1 98%
poller/service.py 235 62 74%
poller/worker_pool.py 75 0 100%
regression/init.py 4 0 100%
regression/clojure_comparer.py 188 20 89%
regression/comparer.py 887 649 27%
regression/datasets.py 135 27 80%
regression/recorder.py 36 27 25%
regression/utils.py 130 85 35%
replay/init.py 2 0 100%
replay/certify.py 452 121 73%
replay/crosslang.py 128 3 98%
replay/driver.py 51 2 96%
replay/prodclone.py 184 16 91%
replay/real_data.py 21 2 90%
replay/schedule.py 125 6 95%
replay/stepcompare.py 77 1 99%
replay/store.py 120 11 91%
replay/types.py 65 8 88%
run_math_pipeline.py 261 114 56%
umap_narrative/500_generate_embedding_umap_cluster.py 210 109 48%
umap_narrative/501_calculate_comment_extremity.py 112 53 53%
umap_narrative/502_calculate_priorities.py 135 135 0%
umap_narrative/700_datamapplot_for_layer.py 502 502 0%
umap_narrative/701_static_datamapplot_for_layer.py 310 310 0%
umap_narrative/702_consensus_divisive_datamapplot.py 432 432 0%
umap_narrative/801_narrative_report_batch.py 785 785 0%
umap_narrative/802_process_batch_results.py 268 268 0%
umap_narrative/803_check_batch_status.py 183 183 0%
umap_narrative/llm_factory_constructor/init.py 2 2 0%
umap_narrative/llm_factory_constructor/model_provider.py 192 192 0%
umap_narrative/polismath_commentgraph/init.py 1 0 100%
umap_narrative/polismath_commentgraph/cli.py 270 270 0%
umap_narrative/polismath_commentgraph/core/init.py 3 3 0%
umap_narrative/polismath_commentgraph/core/clustering.py 108 108 0%
umap_narrative/polismath_commentgraph/core/embedding.py 104 104 0%
umap_narrative/polismath_commentgraph/lambda_handler.py 219 219 0%
umap_narrative/polismath_commentgraph/schemas/init.py 2 0 100%
umap_narrative/polismath_commentgraph/schemas/dynamo_models.py 160 9 94%
umap_narrative/polismath_commentgraph/tests/conftest.py 17 17 0%
umap_narrative/polismath_commentgraph/tests/test_clustering.py 74 74 0%
umap_narrative/polismath_commentgraph/tests/test_embedding.py 55 55 0%
umap_narrative/polismath_commentgraph/tests/test_storage.py 87 87 0%
umap_narrative/polismath_commentgraph/utils/init.py 3 0 100%
umap_narrative/polismath_commentgraph/utils/converter.py 283 237 16%
umap_narrative/polismath_commentgraph/utils/group_data.py 354 336 5%
umap_narrative/polismath_commentgraph/utils/storage.py 584 518 11%
umap_narrative/reset_conversation.py 159 50 69%
umap_narrative/run_pipeline.py 453 312 31%
utils/clj_hash.py 44 0 100%
utils/engine_mode.py 9 0 100%
utils/env_flags.py 13 0 100%
utils/general.py 62 41 34%
utils/serialization.py 9 5 44%
Total 12916 7653 41%

@jucor jucor changed the title feat(math): Q11 — replicate vectorz distance cancellation in legacy kmeans feat(math): Q11 — replicate vectorz distance cancellation in legacy k… Jul 22, 2026
@jucor
jucor changed the base branch from spr/edge/00d8bc52 to edge July 22, 2026 11:20
@jucor jucor changed the title feat(math): Q11 — replicate vectorz distance cancellation in legacy k… feat(math): Q11 — replicate vectorz distance cancellation in legacy kmeans Jul 22, 2026
@jucor
jucor changed the base branch from edge to spr/edge/00d8bc52 July 22, 2026 11:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant