test(wfa): pin ends-free divergence (#102) + warn WFA concordance gates are fixture-scale#56
Merged
Merged
Conversation
Add wfa_endsfree_known_divergence: a deterministic, asserting unit test for the documented WFA ends-free suboptimality (upstream WFA2-lib #102, suboptimal endsfree alignment when match score != 0). s2 = s1 with the leading base dropped, so the true optimum credits a free leading end-gap (score 255); WFA mis-places it as a penalized internal gap (247). Asserts the strict inequality so a future fork/upstream fix fails loudly and prompts updating to equality. Also cite #102 in the raw_align doc comment and note the edit cap does NOT mask this (low-edit pairs finish under budget) — it's the mechanism behind WFA's low-abundance ASV over-calls at scale (#51). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The Illumina/PacBio WFA concordance gates pass on the small committed fixtures, which is reassuring but deceptive: WFA's ends-free alignment is genuinely suboptimal (WFA2-lib #102) and over-calls low-abundance ASVs at scale (issue #51 — jaccard ~0.92-0.95 unbounded, parity-to-slower capped). A green WFA gate means "did not regress on the fixture", NOT "WFA == NW". Add an explicit CAUTION comment block to each WFA gate and a visible ::warning:: annotation in each WFA compare step so the caveat surfaces in the CI run summary rather than only in YAML comments. NW stays the default + error-model backend; WFA stays experimental. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two related changes hardening the experimental WFA backend's test/CI story, following the issue #51 scale validation (full MiSeq SOP + PacBio 95-sample, both k, all three pool modes).
1. Deterministic regression guard for the known ends-free divergence
nwalign.rs::wfa_endsfree_known_divergencepins the upstream WFA2-lib #102 suboptimality (suboptimal ends-free alignment when match score ≠ 0) with a concrete minimal counterexample:s2 = s1with the leading base dropped, so the true optimum credits a free leading end-gap (align_endsfree= 255) while WFA mis-places it as a penalized internal gap (247). Asserts the strict inequality so a future fork/upstream fix fails loudly and prompts updating the guard to assert equality. #102 is now cited in theraw_aligndoc comment, noting the edit cap does not mask this (low-edit pairs finish under budget — it's the mechanism behind the low-abundance ASV over-calls at scale).2. Warn that the WFA concordance gates are fixture-scale only
The Illumina/PacBio WFA gates pass on the small committed fixtures, which is reassuring but deceptive: the divergence is below the count thresholds at fixture scale but at production scale WFA over-calls low-abundance ASVs (jaccard ~0.92–0.95 unbounded, parity-to-slower capped). Added a CAUTION comment block to each WFA gate plus a visible
::warning::annotation in each compare step so the caveat surfaces in the CI run summary, not just in YAML comments.A green WFA gate means "did not regress on the fixture", NOT "WFA == NW". NW remains the default + error-model backend; WFA stays experimental.
Test plan
cargo test --release wfa_endsfree_known_divergencepasses (asserts 255 vs 247).sweep_wfa_parity/wfa_diagnose(ignored) unchanged.Context
Closes the deferred test item under issue #51. Related: #49 (WFA dependency), #53 (edit-budget cap). BiWFA and the #102 pattern/text-swap trick were both evaluated and ruled out (swap fixes 8 / breaks 53 over 20k random pairs — doesn't generalize); the only real fix is a match-aware wavefront cost (fork), tracked separately.
🤖 Generated with Claude Code