Skip to content

Commit f5f7e76

Browse files
claudeclaude
committed
test(clam): CHAODA outlier-discrimination spike — single-method LFD is below the PROBE-CHAODA-1000G bar
Runs the "1-day spike substitute" named in the genetics-probes-v1 spec (AdaWorldAPI/lance-graph): a kernel smoke test for the claim "CHAODA detects novel variants without a trained classifier." Synthesises a 5-lane Gaussian mixture (matching the probe's 5-lane variant feature vector) — three tight "common" clusters plus eight deliberately extreme "novel" outliers — thermometer-encodes each lane into 48 bits so Hamming distance is monotone in per-lane L1 magnitude (the honest bridge from ordinal features to the Hamming-metric CLAM default), builds the shipped ClamTree, and scores via anomaly_scores. MEASURED (deterministic, seed-fixed): mean cluster score = 0.6749, mean outlier score = 0.7500 frac cluster >= 0.5 = 0.733, frac outlier >= 0.5 = 0.750 ROC-AUC (Mann-Whitney U) = 0.6240 FINDING: the shipped single-method leaf-LFD anomaly_scores reaches only AUC ~ 0.62 on the EASIEST possible case (clean synthetic clusters with far outliers) — well below the probe's >= 0.85 bar. The cause is mechanical: leaf LFD = log2(|B(c,r)|/|B(c,r/2)|) measures intra-leaf geometry complexity, not inter-leaf isolation, so an isolated singleton lands in a leaf whose LFD is comparable to a dense cluster's, and global min-max normalisation compresses both into the same band. The CHAODA ensemble of Ishaq et al. 2021 combines several graph-based signals (relative/component cardinality, graph neighbourhood, random-walk stationary distribution, vertex degree); only the LFD signal is shipped here. PROBE-CHAODA-1000G therefore needs the multi-method ensemble or an augmented signal before it can pass — not merely genomic fixtures. The test locks robust, wide-tolerance invariants (valid range, bit-exact determinism, correct polarity, better-than-chance lower bound) plus one tripwire (auc < 0.85) that fails by design if a future multi-method port lifts the signal to the probe bar, forcing a cross-repo FINDING update rather than letting the claim silently rot. This is the evidence-before-build payoff: the gap is caught before any adapter-genetics-experimental (D-GEN-1..4) spend. https://claude.ai/code/session_01VysoWJ6vsyg3wEGc5v7T5v
1 parent cb77a31 commit f5f7e76

1 file changed

Lines changed: 184 additions & 0 deletions

File tree

‎src/hpc/clam.rs‎

Lines changed: 184 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2499,6 +2499,190 @@ mod tests {
24992499
}
25002500
}
25012501

2502+
// ── CHAODA outlier-discrimination spike (PROBE-CHAODA-1000G kernel smoke test) ──
2503+
//
2504+
// Thermometer-encode each of 5 continuous lanes into 48 bits so Hamming
2505+
// distance is monotone in per-lane L1 magnitude (the honest bridge from
2506+
// ordinal features to the Hamming-metric CLAM default). 5 lanes x 6 bytes
2507+
// = 30 bytes/vector.
2508+
const SPIKE_LANES: usize = 5;
2509+
const SPIKE_LEVELS: usize = 48; // 48 bits = 6 bytes per lane
2510+
const SPIKE_VEC_LEN: usize = SPIKE_LANES * (SPIKE_LEVELS / 8);
2511+
2512+
fn thermometer_encode(lanes: &[f64; SPIKE_LANES]) -> Vec<u8> {
2513+
let mut out = vec![0u8; SPIKE_VEC_LEN];
2514+
for (l, &v) in lanes.iter().enumerate() {
2515+
let q = (v.clamp(0.0, 1.0) * SPIKE_LEVELS as f64).round() as usize;
2516+
let base_bit = l * SPIKE_LEVELS;
2517+
for b in 0..q {
2518+
let bit = base_bit + b;
2519+
out[bit / 8] |= 1 << (bit % 8);
2520+
}
2521+
}
2522+
out
2523+
}
2524+
2525+
// Box-Muller standard normal from a SplitMix64 uniform stream.
2526+
fn next_gaussian(rng: &mut SplitMix64) -> f64 {
2527+
let u1 = ((rng.next_u64() >> 11) as f64 / (1u64 << 53) as f64).max(1e-12);
2528+
let u2 = (rng.next_u64() >> 11) as f64 / (1u64 << 53) as f64;
2529+
(-2.0 * u1.ln()).sqrt() * (std::f64::consts::TAU * u2).cos()
2530+
}
2531+
2532+
/// Synthesise a 5-lane Gaussian mixture: tight "common" clusters plus
2533+
/// far "novel" outliers. Returns (bytes, outlier_index_set).
2534+
fn make_genetics_like_mixture() -> (Vec<u8>, Vec<usize>) {
2535+
let mut rng = SplitMix64::new(0x6E_65_74_69_63_73); // "netics"
2536+
let centers: [[f64; SPIKE_LANES]; 3] = [
2537+
[0.20, 0.25, 0.15, 0.30, 0.22],
2538+
[0.50, 0.55, 0.48, 0.52, 0.50],
2539+
[0.78, 0.72, 0.80, 0.75, 0.82],
2540+
];
2541+
let sigma = 0.025;
2542+
let per_cluster = 40;
2543+
let mut data = Vec::new();
2544+
let mut idx = 0usize;
2545+
for center in &centers {
2546+
for _ in 0..per_cluster {
2547+
let mut lanes = *center;
2548+
for lane in lanes.iter_mut() {
2549+
*lane = (*lane + sigma * next_gaussian(&mut rng)).clamp(0.0, 1.0);
2550+
}
2551+
data.extend_from_slice(&thermometer_encode(&lanes));
2552+
idx += 1;
2553+
}
2554+
}
2555+
// 8 novel outliers placed in regions far from every cluster center.
2556+
let outlier_lanes: [[f64; SPIKE_LANES]; 8] = [
2557+
[0.02, 0.97, 0.03, 0.95, 0.05],
2558+
[0.98, 0.02, 0.96, 0.04, 0.99],
2559+
[0.05, 0.05, 0.98, 0.02, 0.50],
2560+
[0.95, 0.95, 0.02, 0.98, 0.03],
2561+
[0.50, 0.02, 0.05, 0.97, 0.95],
2562+
[0.03, 0.50, 0.97, 0.05, 0.02],
2563+
[0.99, 0.50, 0.99, 0.50, 0.01],
2564+
[0.01, 0.99, 0.50, 0.99, 0.99],
2565+
];
2566+
let mut outlier_indices = Vec::new();
2567+
for lanes in &outlier_lanes {
2568+
data.extend_from_slice(&thermometer_encode(lanes));
2569+
outlier_indices.push(idx);
2570+
idx += 1;
2571+
}
2572+
(data, outlier_indices)
2573+
}
2574+
2575+
/// Kernel smoke test for the `PROBE-CHAODA-1000G` claim (genetics-probes-v1
2576+
/// in AdaWorldAPI/lance-graph): *"CHAODA detects novel variants without a
2577+
/// trained classifier."*
2578+
///
2579+
/// FINDING (RUN 2026-06-16): the shipped single-method leaf-LFD
2580+
/// `anomaly_scores` achieves only **ROC-AUC ≈ 0.62** separating deliberately
2581+
/// extreme outliers from tight Gaussian clusters — the *easiest* possible
2582+
/// case. That is well below the probe's ≥ 0.85 bar. The cause is mechanical:
2583+
/// leaf LFD = log₂(|B(c,r)|/|B(c,r/2)|) measures *intra-leaf* geometry
2584+
/// complexity, not *inter-leaf* isolation, so an isolated singleton lands in
2585+
/// a leaf whose LFD is comparable to a dense cluster's, and the global
2586+
/// min-max normalisation compresses both into the same score band. The
2587+
/// CHAODA ensemble of Ishaq et al. 2021 combines several graph-based signals
2588+
/// (relative/component cardinality, graph neighbourhood, random-walk
2589+
/// stationary distribution, vertex degree); only the LFD signal is shipped
2590+
/// here. PROBE-CHAODA-1000G therefore needs the multi-method ensemble (or an
2591+
/// augmented signal) before it can pass — not just genomic fixtures.
2592+
///
2593+
/// This test locks the *robust* invariants — valid range, bit-exact
2594+
/// determinism, correct polarity (outliers ≥ cluster mean), better-than-
2595+
/// chance lower bound — with wide tolerance: the AUC may drift anywhere in
2596+
/// [0.5, 0.85) without breaking. The one tripwire is `auc < 0.85`: it does
2597+
/// not assert the measured 0.62, but it does fail by design if a future
2598+
/// change (e.g. a multi-method CHAODA ensemble) lifts the single-method
2599+
/// signal to the probe bar — forcing whoever does that to update the
2600+
/// PROBE-CHAODA-1000G FINDING in lance-graph rather than letting the
2601+
/// cross-repo claim silently rot.
2602+
#[test]
2603+
fn test_chaoda_flags_novel_outliers_in_genetics_like_mixture() {
2604+
let (data, outliers) = make_genetics_like_mixture();
2605+
let count = data.len() / SPIKE_VEC_LEN;
2606+
let tree = ClamTree::build(&data, SPIKE_VEC_LEN, 3);
2607+
let scores = tree.anomaly_scores(&data, SPIKE_VEC_LEN);
2608+
assert_eq!(scores.len(), count);
2609+
2610+
let is_outlier = |i: usize| outliers.contains(&i);
2611+
let (mut sum_out, mut n_out) = (0.0f64, 0usize);
2612+
let (mut sum_clu, mut n_clu) = (0.0f64, 0usize);
2613+
let (mut out_high, mut clu_high) = (0usize, 0usize); // score >= 0.5
2614+
for s in &scores {
2615+
assert!(s.score >= 0.0 && s.score <= 1.0);
2616+
if is_outlier(s.index) {
2617+
sum_out += s.score;
2618+
n_out += 1;
2619+
if s.score >= 0.5 {
2620+
out_high += 1;
2621+
}
2622+
} else {
2623+
sum_clu += s.score;
2624+
n_clu += 1;
2625+
if s.score >= 0.5 {
2626+
clu_high += 1;
2627+
}
2628+
}
2629+
}
2630+
let mean_out = sum_out / n_out as f64;
2631+
let mean_clu = sum_clu / n_clu as f64;
2632+
let frac_out_high = out_high as f64 / n_out as f64;
2633+
let frac_clu_high = clu_high as f64 / n_clu as f64;
2634+
2635+
// ROC-AUC via the Mann-Whitney U statistic (ties count 0.5). This is the
2636+
// exact number PROBE-CHAODA-1000G gates on (>= 0.85 to pass).
2637+
let mut u = 0.0f64;
2638+
for a in &scores {
2639+
if !is_outlier(a.index) {
2640+
continue;
2641+
}
2642+
for b in &scores {
2643+
if is_outlier(b.index) {
2644+
continue;
2645+
}
2646+
if a.score > b.score {
2647+
u += 1.0;
2648+
} else if (a.score - b.score).abs() < 1e-12 {
2649+
u += 0.5;
2650+
}
2651+
}
2652+
}
2653+
let auc = u / (n_out as f64 * n_clu as f64);
2654+
eprintln!(
2655+
"[CHAODA-spike] n_clu={n_clu} n_out={n_out} mean_clu={mean_clu:.4} mean_out={mean_out:.4} \
2656+
frac_clu>=0.5={frac_clu_high:.3} frac_out>=0.5={frac_out_high:.3} ROC_AUC={auc:.4}"
2657+
);
2658+
2659+
// Determinism: rebuild + rescore must be bit-identical (no-randomness invariant).
2660+
let tree2 = ClamTree::build(&data, SPIKE_VEC_LEN, 3);
2661+
let scores2 = tree2.anomaly_scores(&data, SPIKE_VEC_LEN);
2662+
for (a, b) in scores.iter().zip(scores2.iter()) {
2663+
assert_eq!(a.score.to_bits(), b.score.to_bits(), "non-deterministic score");
2664+
}
2665+
2666+
// Robust, forward-compatible invariants (see the doc comment for the
2667+
// measured AUC ≈ 0.62 finding; we deliberately do NOT assert the ceiling).
2668+
assert!(
2669+
mean_out >= mean_clu,
2670+
"polarity wrong: outliers ({mean_out:.4}) below cluster mean ({mean_clu:.4})"
2671+
);
2672+
assert!(
2673+
auc > 0.5,
2674+
"leaf-LFD anomaly signal is not better than chance (AUC={auc:.4})"
2675+
);
2676+
// Documents the gap to the PROBE-CHAODA-1000G bar without making the
2677+
// test brittle to a future multi-method CHAODA port that raises the AUC.
2678+
assert!(
2679+
auc < 0.85,
2680+
"single-method leaf-LFD unexpectedly met the >= 0.85 probe bar \
2681+
(AUC={auc:.4}); if a multi-method CHAODA ensemble was added, update \
2682+
this assertion AND the PROBE-CHAODA-1000G FINDING in lance-graph"
2683+
);
2684+
}
2685+
25022686
// ── rho_nn_candidates tests ──────────────────────────────────
25032687

25042688
#[test]

0 commit comments

Comments
 (0)