Assess impact of swtbench non-test patch stripping on historical runs

[juanmichelini]

Context

PR #711 changes benchmarks/swtbench/eval_infer.py to strip non-test diffs from model_patch during post-processing (keeping only files under tests/, test/, testing/, or matching test_*.py / *_test.py / conftest.py).

This is a behavior change at evaluation time: any historical SWT-bench output.jsonl re-evaluated under the new code will produce different output.swtbench.jsonl and different scores than what was originally reported. Existing output.swtbench.jsonl files on disk are not modified, but published numbers and the benchmark monitor are no longer directly comparable to future runs.

Goal

Quantify and document the impact of this change on historical SWT-bench runs, so we know:

1. How much the headline resolved-rate moves for each past run (and per model/config).
2. Whether previously-resolved instances regress (i.e. cases where the source-code diff was actually needed for the test patch to apply / pass).
3. Whether the change is monotonic improvement in practice, or there are pockets of regression that argue for a smarter strip (e.g. intersect with gold test_patch file set per instance — noted as a follow-up in DRAFT: swtbench: strip non-test files from model_patch in post-processing (option 2 of #708) #711).

Suggested approach

• Pick a representative set of past runs from the benchmark monitor (mix of models and dates; include the litellm_proxy-openrouter-qwen-qwen3-coder-next/24103435463 run referenced in DRAFT: swtbench: strip non-test files from model_patch in post-processing (option 2 of #708) #711).
• For each run, re-run benchmarks/swtbench/eval_infer.py on the existing output.jsonl with the new code and diff the resolved set against the originally reported set.
• Tabulate: total resolved before/after, newly resolved, newly regressed, net delta. Cross-tab by original patch shape (only-test / mixed / only-source / empty) — the same buckets used in DRAFT: swtbench: strip non-test files from model_patch in post-processing (option 2 of #708) #711.
• Decide whether the benchmark monitor needs a one-shot historical re-eval pass to keep numbers consistent, and whether any leaderboard-facing artifacts need annotation.

Related

• PR DRAFT: swtbench: strip non-test files from model_patch in post-processing (option 2 of #708) #711 (the change itself)
• Issue swtbench: qwen3-coder-next score is artificially low — agent writes source-code fix alongside the test (78% of patches) #708 (option 2 of which DRAFT: swtbench: strip non-test files from model_patch in post-processing (option 2 of #708) #711 implements)

This issue was created by an AI agent (OpenHands) on behalf of @juanmichelini, per their request on PR #711.