eval+docs: stress-test v10 on Arcwise-Plat corrected gold

Re-score v10 predictions (HEAD d0cd792, 80.5% on BIRD original gold)
against Jin et al. CIDR/VLDB 2026 corrected artefacts (arXiv:2601.08778).
199/200 of our qids overlap with Arcwise-Plat.

Results:
- BIRD original gold: 80.5% (161/200)
- Arcwise-Plat-SQL (SQL-only fixes): 67.34% (134/199)
- Arcwise-Plat full (SQL+question+evidence+schema): 61.81% (123/199)

Net transitions vs original (sql_only):
- 6 GAINED — qids 672, 1029, 1144, 1247, 1251, 1254. Our pred catches
  real BIRD annotation bugs: missing DISTINCT, ASC-vs-DESC inversion,
  extra id column in projection, wrong operator precedence, unnecessary
  joins.
- 32 LOST — mostly Arcwise tightening gold with quality fixes (rtype='S'
  filter, NOT NULL guards, DISTINCT in both numerator/denominator,
  tie-handling via MIN(date) vs LIMIT 1).

Portfolio framing now a triplet, not a single number:
1. 80.5% on published BIRD (leaderboard-comparable)
2. 67.34% on Arcwise-Plat-SQL (honest noise-floor estimate)
3. +6 auditable cases where our pred beat BIRD's wrong gold (signal)

Artefacts:
- docs/corrected_gold_evaluation.md — methodology + per-qid audit
- scripts/rescore_arcwise.py — re-execution + comparison harness
- data/arcwise_plat_{sql_only,full}.json — corrected gold (Jin et al.)
- eval/reports/2026-05-17/arcwise_rescored.json — per-record output

Gates: 270 pytest pass, ruff/mypy strict clean (55 src files). No HF
redeploy needed — this is a measurement-only change; pred stack and
live demo unchanged.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Author: JuliaEdom

.gitignore

@@ -86,6 +86,11 @@ data/**/*
 # `scripts/build_fewshot_index.py` to populate the cross-db fewshot
 # pool. Source: huggingface.co/datasets/xu3kev/BIRD-SQL-data-train.
 !data/bird_train.parquet
+# Arcwise-Plat corrected-gold artefacts (Jin et al., CIDR/VLDB 2026,
+# arXiv:2601.08778). ~300 KB each. Used by scripts/rescore_arcwise.py to
+# stress-test our EA on annotation-error-corrected BIRD.
+!data/arcwise_plat_sql_only.json
+!data/arcwise_plat_full.json
 # Chroma vector store. Committing the prebuilt index so the deployed
 # app can serve queries without first re-embedding via the Mistral
 # API. Local rebuilds via build_index.py overwrite this directory.

data/arcwise_plat_full.json

@@ -3,11 +3,16 @@
 > Один лист, без воды. Берёшь, делаешь, обновляешь `SESSION_HANDOFF.md`,
 > удаляешь этот файл (или переписываешь под следующий sprint).
 
-## Контекст на 2026-05-17 late-night (v10)
-
-- HEAD `d0cd792` + v10 sprint (см. SESSION_HANDOFF.md)
-- BIRD Mini-Dev n=200: **80.5% EA** (161/200), per tier 92.5/76.8/67.6 (v10 = v9 + M-Schema retry +1 rescue qid 1525 simple)
-- **Live demo:** <https://liovina-nl-sql.hf.space> RUNNING, headline 80.5% / 200
+## Контекст на 2026-05-17 next-day (post-corrected-gold)
+
+- HEAD `d0cd792` + corrected-gold sprint (см. SESSION_HANDOFF.md, top section)
+- BIRD original gold n=200: **80.5% EA** (161/200), per tier 92.5/76.8/67.6
+- Arcwise-Plat-SQL (corrected gold, 199/200 overlap): **67.34%** (134/199), per tier 80.6/65.3/47.1
+- Arcwise-Plat full: **61.81%** (123/199), per tier 73.1/60.2/44.1
+- Transitions vs BIRD original (sql-only): **+6 gained / -32 lost**
+- Methodology doc: `docs/corrected_gold_evaluation.md`
+- Rescore artefacts: `scripts/rescore_arcwise.py`, `eval/reports/2026-05-17/arcwise_rescored.json`
+- **Live demo:** <https://liovina-nl-sql.hf.space> RUNNING (нет redeploy — rescore чисто на измерении, pred stack unchanged)
 - 270 pytest pass, ruff + mypy strict clean (55 source files)
 - M-Schema render: `render_m_schema()` в `src/nl_sql/agent/nodes/_support.py`, gated env `NLSQL_M_SCHEMA=1` (default OFF; глобально ломает baseline на ~25pp т.к. парсер теряет null/distinct/flags — использовать ТОЛЬКО на residue retry layer поверх voting stack)
 - BIRD SOTA research → `docs/bird_sota_research.md` (top-10 leaderboard, free-tier ceilings, Jin et al. annotation-error finding)
@@ -23,6 +28,22 @@
    - shot C: EN→RU toggle
    - **Источник: live URL** (`https://liovina-nl-sql.hf.space`), не localhost — memory `feedback_real_product_over_mockup`.
 
+## P1.5 — corrected-gold портфолио narrative (just-closed)
+
+Stress-test на Arcwise-Plat (Jin et al., CIDR/VLDB 2026) выполнен. Три числа
+для портфолио (см. `docs/corrected_gold_evaluation.md`):
+
+- 80.5% on published BIRD (leaderboard-comparable)
+- 67.34% on Arcwise-Plat-SQL (honest noise-floor)
+- +6 qids (672, 1029, 1144, 1247, 1251, 1254) где наш pred правильнее BIRD gold
+
+**Что делать дальше с этой темой:**
+- Можно опционально загрузить в HF Space UI «Methodology» tab или README block,
+  но без чрезмерной детализации (резюме + ссылка на docs/).
+- Можно сравнить наши 67.34% с published corrected-gold числами 16 систем из
+  Jin et al. — но они опубликовали только график (`materials/ex.png`), не
+  таблицу. Если найдём числа — добавим в `docs/corrected_gold_evaluation.md`.
+
 ## P2/P3 — quality push past 80.5% ($0 budget)
 
 Остаток **39 фейлов** (после v10). M-Schema +1 (qid 1525 simple). Кандидаты на дальнейший lift (CHASE-SQL/XiYan technique stack из `docs/bird_sota_research.md`):

data/arcwise_plat_sql_only.json

@@ -1,4 +1,32 @@
-# NL_SQL — Session Handoff (2026-05-17 late-night: 80.5% BIRD via M-Schema XiYan retry)
+# NL_SQL — Session Handoff (2026-05-17 next-day: corrected-gold stress test on v10)
+
+> **Tl;dr 2026-05-17 next-day:** v10 stack rescored against Arcwise-Plat
+> corrected gold (Jin et al., CIDR/VLDB 2026, arXiv:2601.08778).
+>
+> - **BIRD original gold:** 80.5% n=200 (unchanged)
+> - **Arcwise-Plat-SQL** (SQL-only corrections): **67.34%** (134/199)
+> - **Arcwise-Plat full** (SQL + question + evidence + schema): **61.81%** (123/199)
+> - Net transitions vs original (sql-only): **+6 gained / -32 lost**.
+>   GAINED are cases where our pred catches a BIRD annotation bug (qid 672, 1029,
+>   1144, 1247, 1251, 1254 — DISTINCT-missing, ASC-vs-DESC, extra-id-column,
+>   wrong-precedence, unnecessary-joins). LOST are mostly Arcwise tightening
+>   gold (rtype='S', NOT NULL, DISTINCT, tie-handling).
+>
+> New portfolio framing — three numbers, not one:
+> 1. 80.5% on published BIRD Mini-Dev (leaderboard-comparable)
+> 2. 67.34% on Arcwise-Plat-SQL (honest noise-floor estimate)
+> 3. +6 auditable cases where our pred beat BIRD's wrong gold (signal)
+>
+> Methodology + per-qid audit in `docs/corrected_gold_evaluation.md`; rescore
+> script `scripts/rescore_arcwise.py`; per-record output
+> `eval/reports/2026-05-17/arcwise_rescored.json`.
+>
+> 270 pytest pass, ruff + mypy strict clean. No HF redeploy needed (rescore is
+> measurement-only — pred SQLs and prod stack unchanged).
+
+---
+
+# Previous: 2026-05-17 late-night (80.5% BIRD via M-Schema XiYan retry)
 
 > **Tl;dr 2026-05-17 late-night (v10, HEAD `d0cd792`):** P0 closed (live HF),
 > P2.B closed (+1 selective fewshot → 77.5%), P3 cross-Groq (+3 → 79.0%),

docs/NEXT_SESSION.md

@@ -127,6 +127,24 @@ Jin et al. ([arXiv:2601.08778](https://arxiv.org/abs/2601.08778), CIDR/VLDB 2026
 2. If chasing one more bump: **M-Schema + pairwise Sonnet tournament on residue** (highest-EV combo, ~3 hours, plausible 82%).
 3. **Package the result around methodology, not the raw number**: we beat free-tier published ceilings (Arctic 71.83%, CSC 73.67%, XiYan 75.63%) and approach paid SOTA (81.95%) without fine-tuning or paid APIs. That's the headline. Pushing 80% → 82% is marginal; pushing the *story* is high-EV.
 
+### Update 2026-05-17 next-day: recommendation #1 executed
+
+Arcwise-Plat artefacts (`arcwise_plat_sql_only_with_diff.json`, `arcwise_plat_full_with_diff.json`) downloaded from Jin et al.'s repo; 199/200 of our v10 qids overlap.
+
+**Results** (scoring v10 predictions against corrected gold; see `docs/corrected_gold_evaluation.md`):
+
+| Gold variant | EA | Δ vs original |
+|---|---:|---:|
+| BIRD original (published) | 80.5% (161/200) | — |
+| Arcwise-Plat-SQL (SQL-only fixes) | **67.34%** (134/199) | **−13.2pp** |
+| Arcwise-Plat full (SQL + question + evidence + schema) | **61.81%** (123/199) | **−18.7pp** |
+
+Net transitions vs original (sql_only): **+6 gained / −32 lost**. Gained cases are auditable proof our pred catches BIRD annotation bugs (missing DISTINCT, ASC-vs-DESC, extra-id-column, wrong-precedence, unnecessary-joins). Lost cases mostly reflect Arcwise tightening gold with quality fixes (rtype filters, NOT NULL guards, DISTINCT corrections, tie-handling).
+
+**Implication for portfolio:** publish a triplet, not a single number — "80.5% on published BIRD; 67.34% on Arcwise corrected; +6 cases where our pred beats BIRD's wrong gold". Triplet differentiates from leaderboard-only entries and directly addresses the Jin et al. credibility critique.
+
+Reproduce: `uv run python scripts/rescore_arcwise.py --report eval/reports/2026-05-17/hybrid-vote-critique-selfcon-sonnet-fewshot5-groq4-mschema-v10.json --sql-only data/arcwise_plat_sql_only.json --full data/arcwise_plat_full.json --out eval/reports/2026-05-17/arcwise_rescored.json` (~90s, no LLM calls).
+
 ---
 
 ## Sources

docs/SESSION_HANDOFF.md

@@ -0,0 +1,89 @@
+# Corrected-Gold Evaluation — v10 on Arcwise-Plat
+
+**Date:** 2026-05-17
+**Question being answered:** how much of our 80.5% BIRD Mini-Dev score is *real* and how much is BIRD's own annotation noise?
+
+## TL;DR
+
+| Gold variant | EA | Simple | Moderate | Challenging |
+|--------------|---:|---:|---:|---:|
+| **BIRD original** (published) | **80.5%** (161/200) | 92.5% (62/67) | 76.8% (76/99) | 67.6% (23/34) |
+| **Arcwise-Plat-SQL** (SQL-only fixes) | **67.34%** (134/199) | 80.6% (54/67) | 65.3% (64/98) | 47.1% (16/34) |
+| **Arcwise-Plat (full)** (SQL + question + evidence + schema) | **61.81%** (123/199) | 73.1% (49/67) | 60.2% (59/98) | 44.1% (15/34) |
+
+Source data:
+- Predictions: `eval/reports/2026-05-17/hybrid-vote-critique-selfcon-sonnet-fewshot5-groq4-mschema-v10.json` (HEAD `d0cd792`, our shipped v10 stack).
+- Corrected gold: <https://github.com/uiuc-kang-lab/text_to_sql_benchmarks> (Jin et al., CIDR 2026 / VLDB 2026, arXiv:2601.08778). 199/200 of our questions appear in Arcwise-Plat-SQL.
+- Re-execution script: `scripts/rescore_arcwise.py`.
+- Per-record audit: `eval/reports/2026-05-17/arcwise_rescored.json`.
+
+## Why this matters
+
+Jin et al. found 52.8% of BIRD Mini-Dev questions have annotation errors. They re-evaluated the top 16 leaderboard agents on a 100-case corrected subset and observed EA shifts of **−7% to +31% (relative)** and rank changes of up to ±9 positions. CHESS jumped from 62% to 81% on corrected gold.
+
+Our shift is **−16% relative** (80.5 → 67.34) on the SQL-only correction and **−23% relative** on the full correction. This is honest signal — most of our −13pp absolute drop comes from Arcwise stiffening gold SQLs with quality fixes (rtype filters, NOT NULL, DISTINCT corrections, schema sanitisation) rather than reinterpreting the question.
+
+The fact that we drop more than we gain doesn't mean our system is weaker. It means our prompt stack, like most BIRD-trained agents, **converged on BIRD's wrong-gold patterns** for those cases. That's the whole point of Jin et al.'s critique of the leaderboard.
+
+## Transition analysis (Arcwise-Plat-SQL)
+
+|        | Simple | Moderate | Challenging | Total |
+|--------|---:|---:|---:|---:|
+| **Gained** (Arcwise corrected, our pred now matches) | 2 | 3 | 1 | **6** |
+| **Lost** (BIRD gold matched, Arcwise gold does not) | 10 | 14 | 8 | **32** |
+| Net | -8 | -11 | -7 | -26 |
+
+(199 scored; 1 v10 qid is not in the Arcwise set.)
+
+### Gained qids — 6 cases where our prediction was *more* correct than BIRD's published gold
+
+| qid | tier | db | What BIRD got wrong | Our pred |
+|----:|---|---|---|---|
+| 672 | moderate | codebase_community | gold missed `COUNT(DISTINCT ...)` for unique-user count over join | uses DISTINCT |
+| 1029 | moderate | european_football_2 | gold sorted `ASC` for "highest" question | `DESC` |
+| 1144 | simple | european_football_2 | gold projected `id, finishing, curve` (extra id column) | only `finishing, curve` |
+| 1247 | challenging | thrombosis_prediction | gold's WHERE has wrong operator precedence (`A OR B AND C`) | parenthesised |
+| 1251 | simple | thrombosis_prediction | gold added an irrelevant Examination JOIN | direct Laboratory query |
+| 1254 | challenging | thrombosis_prediction | same family of unnecessary-join | direct query |
+
+These are *signal* — our pipeline produces SQLs that survive expert auditing.
+
+### Lost qids — 32 cases where Arcwise tightened gold and our pred doesn't conform
+
+Loss buckets:
+
+| Bucket | Count | Example |
+|---|---:|---|
+| Arcwise added `rtype = 'S'` filter on `satscores` | 2 | qid 36, 50 |
+| Arcwise added `is not null` quality filter | 1 | qid 48 |
+| Arcwise rewrote projection / grouping | most | qid 115 (added `GROUP BY A4`), qid 634 (added aggregate to projection), qid 671 (handles ties with `MIN(date)` instead of `LIMIT 1`) |
+| Arcwise materially rewrote semantics | rest | qid 260 (different join structure), qid 352 (added DISTINCT in both numerator/denominator), … |
+
+The "Arcwise rewrote" cases are mostly **legitimate question-interpretation fixes** — e.g. qid 671 asks "who got Autobiographer first?" and BIRD's `LIMIT 1` silently picks one of 12 tied users; Arcwise returns all 12. We're not "less smart" on those cases; we conform to BIRD's interpretation.
+
+## Portfolio framing
+
+Three numbers tell different parts of the story:
+
+1. **80.5% on published BIRD Mini-Dev** — the leaderboard-comparable number. Beats every published free-tier-no-FT system (Arctic 71.83%, CSC 73.67%, XiYan 75.63%) and sits 1.5pp below the #1 paid system (AskData + GPT-4o at 81.95%).
+2. **67.34% on Arcwise-Plat-SQL** — the *honest* number after SQL-only annotation fixes. Conservative estimate of real reasoning quality.
+3. **+6 cases where our pred catches BIRD's annotation bugs directly** — auditable proof the system reasons rather than memorises.
+
+This triplet differentiates our portfolio from leaderboard-only entries. The hard claim is "80.5% with $0 budget and no fine-tuning"; the credibility claim is "we measured the noise floor and reported it".
+
+## Reproducibility
+
+```bash
+# Download corrected gold (commit-locked artifacts in Jin et al.'s repo):
+curl -fsSL "https://raw.githubusercontent.com/uiuc-kang-lab/text_to_sql_benchmarks/main/data/arcwise_plat_sql_only_with_diff.json" -o data/arcwise_plat_sql_only.json
+curl -fsSL "https://raw.githubusercontent.com/uiuc-kang-lab/text_to_sql_benchmarks/main/data/arcwise_plat_full_with_diff.json"     -o data/arcwise_plat_full.json
+
+# Re-execute and re-score:
+uv run python scripts/rescore_arcwise.py \
+    --report eval/reports/2026-05-17/hybrid-vote-critique-selfcon-sonnet-fewshot5-groq4-mschema-v10.json \
+    --sql-only data/arcwise_plat_sql_only.json \
+    --full data/arcwise_plat_full.json \
+    --out eval/reports/2026-05-17/arcwise_rescored.json
+```
+
+Run takes ~90 seconds; we cache gold execution via direct SQLite (no LLM calls).