dev: isolated C++ WFA2-lib FFI oracle for #102 (no-ship)

dev/wfa-ffi-oracle/.gitignore

@@ -0,0 +1,5 @@
+/target
+Cargo.lock
+wfa2lib_src/
+oracle
+*.o

dev/wfa-ffi-oracle/Cargo.toml

@@ -0,0 +1,15 @@
+# Standalone oracle crate — NOT part of the dada2-rs build.
+# The empty [workspace] table makes this its own workspace root so no ancestor
+# Cargo.toml can ever pull it in. It uses C++ WFA2-lib via FFI ON PURPOSE, only
+# as a test oracle for upstream issue #102 — nothing here ships. See README.md.
+[package]
+name = "wfa-ffi-oracle"
+version = "0.0.0"
+edition = "2021"
+publish = false
+
+[workspace]
+
+[dependencies]
+# TODO: add a Rust FFI binding to the *C++* WFA2-lib here (see README "Wiring").
+# Left empty so the stub in src/main.rs compiles standalone today.

dev/wfa-ffi-oracle/README.md

@@ -0,0 +1,109 @@
+# wfa-ffi-oracle
+
+A **throwaway oracle** for the WFA ends-free + nonzero-match-score divergence
+(upstream WFA2-lib [#102](https://github.com/smarco/WFA2-lib/issues/102)). It
+exists to answer one question:
+
+> Does the **latest C++ WFA2-lib** return the *optimal* ends-free alignment when
+> the match score is nonzero (DADA2 uses match = +5), or does it reproduce the
+> same suboptimality we see in the pure-Rust port?
+
+## Results (WFA2-lib `bcf473a`, 2026-06) — #102 is **partially** fixed
+
+Run via `./run.sh` (C harness `oracle.c` linked against the C++ static lib).
+With `match=-5` the WFA penalty score equals the DADA2 score, so any result
+below the known optimum is a genuine optimality miss, not a penalty-space tie:
+
+| case | mode | optimum | C++ | verdict |
+|------|------|---------|-----|---------|
+| leading-gap 52nt | Mode 1 (free end-gap crediting) | 255 | 255 | ✅ fixed |
+| `GCGG`/`CG` (the #102 thread reproducer) | Mode 1 | 10 | 10 | ✅ fixed |
+| leading-gap 55nt | Mode 1 | 275 | 275 | ✅ fixed |
+| trailing interior gap | Mode 2 (mismatch + free-end indel over interior gap) | 272 | **271** | ❌ still off by 1 |
+| near-end interior gap | Mode 2 | 347 | **346** | ❌ still off by 1 |
+
+**3/5.** Current C++ landed the leading/trailing free-end-gap fix (**Mode 1**,
+the common case — including the issue's own `GCGG`/`CG`) but still exhibits
+**Mode 2**: its CIGARs end `…X​M​D` / `…X​M​I`, i.e. it prefers `[mismatch +
+free-end indel]` over the higher-scoring `[interior indel + trailing matches]`.
+
+**Implication.** Porting the C++ ends-free fix into `wfa2lib-rs::termination.rs`
+would remove the *bulk* (Mode 1) of our divergence, but **not** make WFA a clean
+drop-in: Mode 2 survives even upstream, so WFA stays non-identical to NW and can
+still flip the occasional low-abundance ASV. Option (c) stands — NW remains the
+error-model backend; this porting is worthwhile only if WFA's *speed* ever
+justifies it (currently it does not; see `project_wfa_edit_budget_cap_issue51`).
+
+## Two harnesses in this dir
+
+- **`oracle.c` + `run.sh`** — the **working** oracle: links the C++ WFA2-lib
+  static lib directly (simplest path for an FFI oracle). `run.sh` clones+builds
+  WFA2-lib at a pinned commit, compiles, and runs. This produced the results
+  above.
+- **`Cargo.toml` + `src/main.rs`** — an optional pure-Rust-binding stub (its own
+  `[workspace]`, never built by the root). Kept as the skeleton if we later want
+  the oracle as a Rust FFI crate instead of a C harness.
+
+## Why this is isolated under `dev/` and never ships
+
+- The production aligner is the **pure-Rust** `wfa2lib-rs` (a re-implementation,
+  not an FFI binding). We deliberately avoided a C/C++ build dependency
+  (cc/bindgen, cross-compile friction). See memory
+  `project_wfa_dependency_wfa2lib_rs`.
+- This crate uses **FFI into the C++ library on purpose**, but *only as a test
+  oracle*. Nothing here is a dependency of dada2-rs. It is a standalone Cargo
+  project (its own `[workspace]`), so the root `cargo build`/`cargo test` never
+  touches it and it cannot leak a C++ toolchain requirement into CI or releases.
+
+## The decision it gates
+
+The #102 bug is **not** the match-score setting (the Rust port already exposes
+`match_` with Eizenga reward-conversion, and our global score already matches
+NW's optimum). The bug is that the ends-free **termination** ignores the match
+reward — `wfa2lib-rs/src/termination.rs::termination_endsfree` is purely
+length-based. So:
+
+1. **If the latest C++ lib gets these reproducers right** → the fix is real and
+   we port the match-aware termination logic into the pure-Rust
+   `termination.rs` (shipping crate stays 100% Rust at runtime).
+2. **If the C++ lib is *also* wrong** → no point porting; #102 is unfixed
+   upstream and WFA stays experimental with NW as the error-model backend.
+
+**Do not port any termination code until this oracle confirms (1).**
+
+## The reproducers
+
+Both come from `dada2-rs` `src/nwalign.rs` (DADA2 scoring: match +5, mismatch
+-4, gap -8, ends-free on both sequences):
+
+| case | pattern / text | optimal (NW `align_endsfree`) | pure-Rust WFA |
+|------|----------------|-------------------------------|---------------|
+| leading-gap (Mode 1) | `s2 = s1` minus leading base, 52nt | **255** (free leading end-gap, all 51 cols match) | 247 (penalized internal gap) |
+| #102 minimal | pattern `GCGG`, text `CG` | optimal `GCGG / -CG-` | suboptimal `-GCGG / CG--` |
+
+`src/main.rs` hard-codes these and prints, per case: the C++ WFA score + CIGAR
+vs the known optimum, and PASS/FAIL (PASS = C++ matches the optimum = the fix is
+real).
+
+## Wiring the FFI binding (TODO)
+
+`src/main.rs` currently runs as a **stub** (no external dep) so it compiles and
+documents the expected values today. To make it a live oracle, add a binding to
+the C++ WFA2-lib and replace the `wfa_endsfree_score()` stub. Options, in order
+of preference:
+
+1. An existing Rust binding crate to **WFA2-lib (C++)** that exposes ends-free +
+   settable match score (verify it tracks a recent enough WFA2-lib that includes
+   any #102 fix; the older crates.io `libwfa` binds the *original* C WFA and
+   predates Eizenga match support — not usable here).
+2. A minimal hand-written `bindgen`/`cc` shim against a local WFA2-lib checkout
+   (pin the exact commit/tag tested in this README when results are recorded).
+
+Record the WFA2-lib version/commit actually tested next to the results.
+
+## Run
+
+```
+cd dev/wfa-ffi-oracle
+cargo run            # once a binding is wired; prints PASS/FAIL per reproducer
+```

dev/wfa-ffi-oracle/oracle.c

@@ -0,0 +1,78 @@
+/* Oracle for WFA2-lib ends-free + nonzero-match-score divergence (issue #102).
+ *
+ * Links the *C++* WFA2-lib static lib (built by run.sh) and asks, on the
+ * committed dada2-rs reproducers, whether the C++ library returns the OPTIMAL
+ * ends-free alignment under DADA2 scoring (match +5) — i.e. whether #102 is
+ * fixed upstream. Nothing here ships; this is a throwaway oracle (see README).
+ *
+ * Penalty config mirrors the pure-Rust adapter exactly:
+ *   match=-5 (reward), mismatch=4, gap_opening=0, gap_extension=8, ends-free
+ *   on both ends of both sequences. With match=-5 the WFA penalty score equals
+ *   the DADA2 score, so a WFA result below the known optimum is a genuine
+ *   optimality miss, not a penalty-space tie.
+ */
+#include <stdio.h>
+#include <string.h>
+#include "wavefront/wavefront_align.h"
+
+/* DADA2 score of a WFA CIGAR op string; leading/trailing indels are free. */
+static int dada2_score(const char *ops, int n) {
+    int lead = 0, trail = 0;
+    while (lead < n && (ops[lead] == 'I' || ops[lead] == 'D')) lead++;
+    while (trail < n && (ops[n - 1 - trail] == 'I' || ops[n - 1 - trail] == 'D')) trail++;
+    int s = 0;
+    for (int i = 0; i < n; i++) {
+        char o = ops[i];
+        if (o == 'M') s += 5;
+        else if (o == 'X') s += -4;
+        else if (o == 'I' || o == 'D') s += (i < lead || i >= n - trail) ? 0 : -8;
+    }
+    return s;
+}
+
+static int run(const char *name, const char *p, const char *t, int optimal) {
+    int pl = strlen(p), tl = strlen(t);
+    wavefront_aligner_attr_t a = wavefront_aligner_attr_default;
+    a.distance_metric = gap_affine;
+    a.affine_penalties.match = -5;       /* reward — the #102 trigger */
+    a.affine_penalties.mismatch = 4;
+    a.affine_penalties.gap_opening = 0;  /* linear gap of 8 == DADA2 gap_p */
+    a.affine_penalties.gap_extension = 8;
+    a.alignment_form.span = alignment_endsfree;
+    a.alignment_form.pattern_begin_free = pl;
+    a.alignment_form.pattern_end_free = pl;
+    a.alignment_form.text_begin_free = tl;
+    a.alignment_form.text_end_free = tl;
+    wavefront_aligner_t *wf = wavefront_aligner_new(&a);
+    wavefront_align(wf, p, pl, t, tl);
+    cigar_t *c = wf->cigar;
+    int n = c->end_offset - c->begin_offset;
+    const char *ops = c->operations + c->begin_offset;
+    int sc = dada2_score(ops, n);
+    int pass = (sc == optimal);
+    printf("[%-24s] opt=%d  C++=%d  %s\n    cigar=%.*s\n",
+           name, optimal, sc, pass ? "PASS (optimal)" : "FAIL (suboptimal)", n, ops);
+    wavefront_aligner_delete(wf);
+    return pass;
+}
+
+int main(void) {
+    int ok = 0, tot = 0;
+    /* Mode 1 — free leading/trailing end-gap crediting (the headline #102 symptom). */
+    tot++; ok += run("Mode1 leading (255)",
+        "AACAGCGCAAACCAACTCGCTAGCTAGCAAAATCTTGTGTTTCTGCCTAGCG",
+        "ACAGCGCAAACCAACTCGCTAGCTAGCAAAATCTTGTGTTTCTGCCTAGCG", 255);
+    tot++; ok += run("#102 thread GCGG/CG (10)", "GCGG", "CG", 10);
+    tot++; ok += run("Mode1 leading (275)",
+        "AACTAGACATCAAAGCCACAGAATTTGGCAGGAGATACAAGAAGTACTCCAGCTC",
+        "AAACTAGACATCAAAGCCACAGAATTTGGCAGGAGATACAAGAAGTACTCCAGCTC", 275);
+    /* Mode 2 — mismatch + free-end indel preferred over higher-scoring interior gap. */
+    tot++; ok += run("Mode2 trailing (272)",
+        "TACACGCTACCTTGGAGACCTCCTGTGCTTGCAGCCCATCAATCTTTTTACACAAGG",
+        "TACACGCTACCTTGGAGACCTCCTGTGCTTGCAGCCCATCAATCTTTTTACACAGG", 272);
+    tot++; ok += run("Mode2 near-end (347)",
+        "TAACGACGAACGAGTTTCTAAACCAAAGAACGTAACAAGGAGTTGCCTCTTAGCCTTGGCCTACGCAGGAA",
+        "TAACGACGAACGAGTTTCTAAACCAAAGAACGTAACAAGGAGTTGCCTCTTAGCCTTGGCCTACGCAGGGAA", 347);
+    printf("\n%d/%d optimal.  PASS = #102 fixed for that case; FAIL = still suboptimal upstream.\n", ok, tot);
+    return ok == tot ? 0 : 1;
+}

dev/wfa-ffi-oracle/run.sh

@@ -0,0 +1,23 @@
+#!/usr/bin/env bash
+# Build the C++ WFA2-lib at a pinned commit and run the #102 oracle against it.
+# Nothing here ships; this is a throwaway test oracle (see README.md).
+set -euo pipefail
+cd "$(dirname "$0")"
+
+# Pinned to the commit tested when results were recorded in the README. Bump
+# deliberately (and re-record results) to re-check a newer WFA2-lib.
+WFA_COMMIT="${WFA_COMMIT:-bcf473a}"
+SRC=wfa2lib_src
+
+if [ ! -d "$SRC" ]; then
+  git clone https://github.com/smarco/WFA2-lib "$SRC"
+fi
+git -C "$SRC" fetch -q origin && git -C "$SRC" checkout -q "$WFA_COMMIT"
+
+# The Makefile expects these output dirs to exist.
+mkdir -p "$SRC/build/cpp" "$SRC/lib"
+make -C "$SRC" lib_wfa >/dev/null
+
+cc -Wall -I"$SRC" oracle.c "$SRC/lib/libwfa.a" -o oracle -lm -lpthread
+echo "WFA2-lib commit: $(git -C "$SRC" rev-parse --short HEAD)"
+./oracle

dev/wfa-ffi-oracle/src/main.rs

@@ -0,0 +1,105 @@
+//! Oracle for WFA2-lib ends-free + nonzero-match-score divergence (issue #102).
+//!
+//! Asks whether the *C++* WFA2-lib returns the OPTIMAL ends-free alignment with
+//! match score = +5 (DADA2 scoring), or reproduces the suboptimality we observe
+//! in the pure-Rust `wfa2lib-rs`. See README.md. Nothing here ships — it is a
+//! standalone FFI oracle used only to decide whether the upstream fix is real
+//! and therefore worth porting into the pure-Rust `termination.rs`.
+//!
+//! Status: STUB. `wfa_endsfree_score` is not yet wired to the C++ library, so
+//! the harness prints the known values and the expected verdict shape. Wire the
+//! FFI binding (README "Wiring the FFI binding") and replace the stub.
+
+/// DADA2 ends-free scoring used throughout dada2-rs (`src/nwalign.rs`).
+const MATCH: i32 = 5;
+const MISMATCH: i32 = -4;
+const GAP: i32 = -8;
+
+/// A reproducer: pattern/text, plus the optimal ends-free score that
+/// `align_endsfree` (the true positional optimum) achieves.
+struct Case {
+    name: &'static str,
+    pattern: &'static str,
+    text: &'static str,
+    optimal: i32,
+    note: &'static str,
+}
+
+/// The two committed dada2-rs reproducers (see `nwalign.rs::wfa_endsfree_known_divergence`).
+fn cases() -> Vec<Case> {
+    // Mode 1: text is the pattern minus its leading base; the optimum credits a
+    // free leading end-gap so all 51 shared columns score as matches (51*5=255).
+    // The pure-Rust WFA mis-places it as a penalized internal gap (255-8=247).
+    let p1 = "AACAGCGCAAACCAACTCGCTAGCTAGCAAAATCTTGTGTTTCTGCCTAGCG"; // 52 nt
+    let t1 = "ACAGCGCAAACCAACTCGCTAGCTAGCAAAATCTTGTGTTTCTGCCTAGCG"; //  51 nt (p1[1..])
+    vec![
+        Case {
+            name: "leading-gap (Mode 1)",
+            pattern: p1,
+            text: t1,
+            optimal: 255,
+            note: "optimum = free leading end-gap; pure-Rust WFA = 247 (internal gap)",
+        },
+        // The #102 minimal reproducer from the upstream issue thread.
+        // Optimal ends-free: GCGG / -CG-  (the CG aligns to the interior GG-run
+        // boundary with free ends). WFA prefers -GCGG / CG-- (suboptimal).
+        Case {
+            name: "#102 minimal (GCGG/CG)",
+            pattern: "GCGG",
+            text: "CG",
+            // 2 matches (C,G) under the optimal placement, no mismatch; free
+            // end-gaps unpenalized → 2*5 = 10 in DADA2 scoring.
+            optimal: 2 * MATCH,
+            note: "upstream #102 thread's 4-base reproducer",
+        },
+    ]
+}
+
+/// Ends-free alignment score of `pattern` vs `text` under (MATCH, MISMATCH, GAP)
+/// as computed by the **C++ WFA2-lib** via FFI.
+///
+/// STUB — returns None until the FFI binding is wired (see README). When wired,
+/// configure WFA2-lib with: gap-affine, match = +5 (or the Eizenga-equivalent),
+/// mismatch = 4, gap-opening = 0, gap-extension = 8, and free ends on both ends
+/// of both sequences; then convert its CIGAR to the DADA2 score and return it.
+fn wfa_endsfree_score(_pattern: &str, _text: &str) -> Option<i32> {
+    None
+}
+
+fn main() {
+    println!(
+        "WFA2-lib ends-free oracle (match={MATCH}, mismatch={MISMATCH}, gap={GAP})\n\
+         purpose: does the C++ lib give the OPTIMAL ends-free score with match != 0? (#102)\n"
+    );
+
+    let mut wired = true;
+    for c in cases() {
+        print!(
+            "[{}]  pattern={} ({} nt)  text={} ({} nt)  optimal={}",
+            c.name,
+            c.pattern,
+            c.pattern.len(),
+            c.text,
+            c.text.len(),
+            c.optimal
+        );
+        match wfa_endsfree_score(c.pattern, c.text) {
+            Some(s) if s == c.optimal => println!("  ->  C++ WFA={s}  PASS (fix is real)"),
+            Some(s) => println!("  ->  C++ WFA={s}  FAIL (still suboptimal upstream)"),
+            None => {
+                wired = false;
+                println!("  ->  C++ WFA=<stub>  ({})", c.note);
+            }
+        }
+    }
+
+    if !wired {
+        eprintln!(
+            "\nSTUB: wfa_endsfree_score is not wired to the C++ WFA2-lib yet.\n\
+             Add an FFI binding (README) and record the WFA2-lib commit tested.\n\
+             PASS on every case => port the match-aware ends-free termination into\n\
+             the pure-Rust wfa2lib-rs::termination.rs. Any FAIL => #102 unfixed upstream."
+        );
+        std::process::exit(2);
+    }
+}