Follow-up spun out of #43. Now that sparse kmer8 (#43) decouples the resident k-mer screen from 4^k, the KMER_SIZE_MAX = 8 cap is no longer a memory wall — it's just an index-width artifact. Worth raising experimentally to probe whether higher k yields legitimate fine-resolution ASVs or screening artifacts. Low priority; gated on sourcing a truth set.
Why the cap exists (and why it's now soft)
KMER_SIZE_MAX = 8 is set because 4^8 = 65536 is the largest k-mer index that fits a u16 (kmers.rs:24, :106). Pre-#43 the dense 4^k array also made k≥8 a memory non-starter; sparse removed that wall. So raising the cap is now mechanical, not a redesign.
Engineering scope (~half day, feature-gated)
- Widen sparse index
u16 → u32 in KmerScreen::Sparse and kord (kmers.rs:146, :129). (u32,u8) pads to 8 B → sparse footprint ~doubles to ~12 KB/raw, still trivial vs dense 4^9 = 256 KB. Only resident change.
- Fix the sparse build path.
assign_kmer8_sparse builds the dense 4^k u16 vector then filters (kmers.rs:113-120); fine at k8 (128 KB transient/raw) but at k12 it's 4^12 × 2 = 32 MB/raw just to zero-and-scan — wall would explode even though the resident result stays tiny. Build must go direct (hash/sort the ~seqlen windows), skipping the dense intermediate.
- Bump
KMER_SIZE_MAX + docs/asserts. Keep default at 8; put higher k behind an --experimental-kmer guard so the production path can't select an unvalidated screen.
The actual question (needs a truth set)
The k-mer distance is a screen: larger k makes it stricter, so more pairs are screened out and never aligned/collapsed → more ASVs almost by construction. Per #15, k had ~no effect on the final table across k5–7, suggesting the screen wasn't binding there; k=10/12 probes whether we've crossed into the regime where the screen starts dropping legitimate alignments. The extra ASVs split two ways:
- Legitimate: real variants a looser screen was collapsing (true positives previously hidden).
- Artifact: pairs that should collapse but the stricter screen now skips → PCR/sequencing-error daughters survive as spurious ASVs (false positives, typically 1–2 Hamming off a real member at low abundance).
Can't distinguish from the table alone. More ASVs is the hypothesis to disprove, not the win.
Validation plan (the gating dependency)
- Truth set: public PacBio 16S mock community (Zymo / ATCC / PacBio-provided) with known member count + reference seqs.
- Metric: sensitivity vs false-positive rate as a function of k — do extra high-k ASVs map to genuine reference members/known strain variants, or sit 1–2 Hamming off a real member at low abundance (artifact signature)?
scripts/compare_asvs.py already reports nearest-baseline Hamming + abundance for only_post, so most of the harness exists.
- Sweep k=8/9/10/12 against the mock once the truth set is in hand.
Acceptance
- Cap raise behind experimental flag, default unchanged at 8, no regression to k≤8 paths.
- Sweep produces a sens/FP-vs-k curve against the mock; conclusion documented (which k, if any, adds real resolution).
Relates to #43 (sparse made this feasible), #15 (k ~no effect on table at k5–7).
Follow-up spun out of #43. Now that sparse
kmer8(#43) decouples the resident k-mer screen from4^k, theKMER_SIZE_MAX = 8cap is no longer a memory wall — it's just an index-width artifact. Worth raising experimentally to probe whether higher k yields legitimate fine-resolution ASVs or screening artifacts. Low priority; gated on sourcing a truth set.Why the cap exists (and why it's now soft)
KMER_SIZE_MAX = 8is set because4^8 = 65536is the largest k-mer index that fits au16(kmers.rs:24, :106). Pre-#43 the dense4^karray also made k≥8 a memory non-starter; sparse removed that wall. So raising the cap is now mechanical, not a redesign.Engineering scope (~half day, feature-gated)
u16 → u32inKmerScreen::Sparseandkord(kmers.rs:146, :129).(u32,u8)pads to 8 B → sparse footprint ~doubles to ~12 KB/raw, still trivial vs dense4^9 = 256 KB. Only resident change.assign_kmer8_sparsebuilds the dense4^ku16 vector then filters (kmers.rs:113-120); fine at k8 (128 KB transient/raw) but at k12 it's4^12 × 2 = 32 MB/rawjust to zero-and-scan — wall would explode even though the resident result stays tiny. Build must go direct (hash/sort the ~seqlen windows), skipping the dense intermediate.KMER_SIZE_MAX+ docs/asserts. Keep default at 8; put higher k behind an--experimental-kmerguard so the production path can't select an unvalidated screen.The actual question (needs a truth set)
The k-mer distance is a screen: larger k makes it stricter, so more pairs are screened out and never aligned/collapsed → more ASVs almost by construction. Per #15, k had ~no effect on the final table across k5–7, suggesting the screen wasn't binding there; k=10/12 probes whether we've crossed into the regime where the screen starts dropping legitimate alignments. The extra ASVs split two ways:
Can't distinguish from the table alone. More ASVs is the hypothesis to disprove, not the win.
Validation plan (the gating dependency)
scripts/compare_asvs.pyalready reports nearest-baseline Hamming + abundance foronly_post, so most of the harness exists.Acceptance
Relates to #43 (sparse made this feasible), #15 (k ~no effect on table at k5–7).