---
source_url: https://openai.com/index/separating-signal-from-noise-coding-evaluations/
source_type: web
title: "Separating signal from noise in coding evaluations | OpenAI"
date: 2026-07-10
fetch_method: extension
---
July 8, 2026
[Research](/news/research/) [Publication](/research/index/publication/)
# Separating signal from noise in coding evaluations
Through a detailed audit, we find widespread task issues in SWE-Bench Pro and estimate that ~30% of the tasks are broken.
Listen to article 7:06
Share
Accurately measuring our models' capabilities is important for sound deployment and safety decisions, including decisions under OpenAI's [Preparedness Framework ](https://cdn.openai.com/pdf/18a02b5d-6b67-4cec-ab64-68cdfbddebcd/preparedness-framework-v2.pdf) . With each model release, we report results for a variety of external and internal benchmarks to track model progress. When evaluations have flaws that affect results, they can give a false understanding of capabilities, misrepresenting safety cases and affecting research priorities.
We [recently investigated](/index/why-we-no-longer-evaluate-swe-bench-verified/) how one of the most widely used coding benchmarks, SWE-bench Verified, had fundamental design and contamination issues, and found that the eval no longer provided meaningful signal on software development capabilities. At the time, we encouraged the wider community to switch to SWE-Bench Pro.
[SWE-Bench Pro ](https://scale.com/blog/swe-bench-pro) was designed to improve on SWE-bench Verified by testing models on longer horizons and more realistic coding tasks to better track agentic coding capabilities. As in SWE-bench Verified, tasks are sourced programmatically from the history of feature changes in a set of public and private repositories. Models are required to implement a solution that passes new tests for a feature, without breaking existing functionality. On the 731-task public split, frontier models improved from a pass rate of 23.3% to 80.3% in eight months.
We've since performed a similar audit on SWE-Bench Pro, reviewing the dataset using a datapoint analysis pipeline. The pipeline reviewed model attempts at the task, task metadata, and failure traces to flag likely evaluation flaws. Each flagged task was then assessed through multiple investigator-agent passes and independently reviewed by five experienced software engineers, with disagreements escalated for further investigation.
Share of Dataset Flagged by Issue Type
Overly stricttestsLow-coveragetestsMisleadingpromptMiscellaneousissuesUnderspecifiedprompt0%5%10%15%20%Percent of total dataset14.4%17.8%4.1%9.4%6.3%7.5%1.9%1.2%0.6%0.8%Human supervised agent reviewHuman annotations
We find evidence of breaking issues in a significant portion of the dataset. Our datapoint analysis pipeline flagged 200 (27.4%) broken tasks, while the human annotation campaign identified 249 (34.1%).
The issues primarily fell into four categories:
- *Overly strict tests*[1](#citation-bottom-1) enforce specific implementation details not specified in the prompt, invalidating many functionally correct submissions.
- *Underspecified prompts*[2](#citation-bottom-2) omit requirements that hidden tests enforce and that are not reasonably inferable.
- *Low-coverage tests* under check the requested feature, so incomplete fixes can pass.
- A *misleading prompt* points models toward the wrong behavior or contradicts what tests require.
Our findings point to the difficulty of curating hard but fair benchmarks and the growing utility of agents for scalable data quality checks. In light of these results, we estimate that ~30% of SWE-bench Pro tasks are broken, and advise that model developers carefully examine results.
## Methodology
Our aim is to ensure that task failures reflect genuine model limitations, and task successes reflect complete and valid solutions to the prompt requirements. To check the quality of the data used in the evaluation, we created a quality assurance pipeline to assess whether each datapoint accurately reflects model capabilities.
An initial data quality pipeline flags problems for review. We validate with a deeper agent-assisted audit of flagged tasks and a human annotation campaign working with experienced engineers.
An initial automated filter reviews the instructions given to the model, attempts by the model to solve the task, and the tests used to grade these attempts to flag likely broken or problematic examples. This filter flagged 286 potentially broken tasks. We then conducted a deeper review of that subset in two ways: a human-supervised agent review, which conducts extensive checks with investigator agents and a final human judgment; and a human annotation campaign working with experienced software developers.
## Human-supervised agent review
Each flagged problem is audited with Codex-based investigator agents that were given access to the task repository and environment. This helps them distinguish reasonable task ambiguity, which can often be resolved by studying nearby code and repository conventions, from true underspecification. The agent can run tests, inspect files in the repo, and investigate model attempts and their common failure modes on the task. After several independent repeats of these deeper audits, a researcher reviewed the summaries, made a final judgment, and labeled the likely issues.
## Human annotation campaign
In parallel, we ran a human annotation campaign over the flagged subset. We worked with experienced software engineers who were trained on the benchmark goals, issue taxonomy, and edge cases before reviewing tasks. Each task was reviewed by five engineers.
Reviewers formed an independent judgment from the visible problem statement, test cases, and the ground-truth reference solution (known as the gold patch) before using the pipeline analysis or transcript as supporting context. The reviewers then assigned a label and severity rating based on concrete evidence, and escalated disagreements or low-confidence cases for further review.
Human reviewers were more likely than the investigator agents to mark tasks as broken. There was also some disagreement on categories between the two review paths, but in no flagged task was "not broken" the most common human label. Of the categories the agent pipeline flagged, reviewers' judgments overlapped in 74% of cases.
Compared with the agent pipeline, the human reviewers were also more likely to select multiple labels for a task, indicating that they found tasks to be broken in multiple ways or did not fit cleanly into a single category. This suggests the agent-plus-reviewer pipeline resulted in conservative labeling: it captured the same broad failure modes humans identified, while undercounting cases where reviewers saw additional or overlapping issues. The largest difference was in low-coverage tests, which humans selected as the most common issue for 9.4% of the benchmark compared with 4.1% from the agent pipeline.
### Failure modes
In several cases the task prompt prescribed a specific implementation, but the hidden test cases expected different behavior.
## OpenLibrary-77c16d5
This task involves normalizing table-of-contents entries and rendering them back to Markdown via `TocEntry.to_markdown()` . The task prompt specifies serialization down to character-level spacing, describing how exact spacing and pipes are enforced, and gives examples such as `" | Chapter 1 | 1"` and `"** | Chapter 1 | 1"` :
#### None
`
1"[space]| Chapter 1 | 1"2"**[space]| Chapter 1 | 1"3"[space]| Just title | "
`
The hidden `test_to_markdown` assertions instead require `" | Chapter 1 | 1"` and `"** | Chapter 1 | 1"` :
#### None
`
1"[space][space]| Chapter 1 | 1"2"**[space][space]| Chapter 1 | 1"3"[space][space]| Just title | "
`
There are two leading spaces in the hidden tests, but the example given to the model only contains one leading space. If a model rightly follows the given prompt, that one-character difference would fail the hidden test cases and the task would be marked incorrect.
## Discussion
The issues we have identified, coupled with similar cases in [SWE-bench Verified](/index/why-we-no-longer-evaluate-swe-bench-verified/) , highlight the importance of rigorously checking benchmarks. Issues and pull requests from open-source repositories were originally created for human collaboration, often through long back-and-forths between maintainers and contributors. As a result, problem descriptions, merged code, and unit tests do not always line up to form clean, isolated tasks for evaluating models reliably. In particular, tests included in pull requests can be overly strict because they are written to validate a specific change, rather than to define an implementation-agnostic standard for solving the task.
At the same time, evaluation flaws are easier to detect now than they would have been even a short time ago. As model capabilities improve, we can use those models to inspect prompts, tests, patches, traces, and edge cases with much greater depth and consistency, helping surface benchmark issues that were previously costly or impractical to find at scale.
We hope the wider evaluation community will develop new benchmarks built by experienced software developers specifically to test model capabilities. That approach can preserve the high bar and realism we want to measure model capabilities, and allows for better human oversight throughout the process. Given the issues uncovered in this analysis, we retract our earlier recommendation to adopt SWE-Bench Pro.
Ultimately, an eval should provide meaningful signal through benchmarks that are hard to game, easy to trust, and genuinely reflective of model capability or alignment. Because these results inform OpenAI's deployment and safety decisions, the evals we track need to be valid and informative.
- [2026](/news/?tags=2026)
- [Software & Engineering](/news/?tags=software-engineering)
## Author
OpenAI
## Footnotes
1. 1We [previously ](/index/why-we-no-longer-evaluate-swe-bench-verified/) referred to this category as narrow tests.
2. 2We previously referred to this category as wide tests.
## Keep reading
[View all](/news/)
[Introducing GeneBench-Pro Research Jun 30, 2026](/index/introducing-genebench-pro/)
[A near-autonomous AI chemist improves a challenging reaction in medicinal chemistry Research Jun 17, 2026](/index/ai-chemist-improves-reaction/)
[Introducing LifeSciBench Research Jun 17, 2026](/index/introducing-life-sci-bench/)
问题描述
文档 ID 623（Separating signal from noise in coding evaluations | OpenAI）存在多处格式转换问题。
文档路径:
users/2/raw/web/Separating-signal-from-noise-in-coding-evaluations--OpenAI/paper.md原始 URL: https://openai.com/index/separating-signal-from-noise-coding-evaluations/
格式问题详情
1. 代码块格式错误（严重）
在 "OpenLibrary-77c16d5" 部分，代码块格式完全错误：
当前渲染效果：
问题：
#### None标题无意义，可能是某个组件（如 tab/accordion）转换失败[space]应该是空格字符的占位符，但显示不友好预期格式：
" | Chapter 1 | 1"
"** | Chapter 1 | 1"
" | Just title | "
" | Chapter 1 | 1"
"** | Chapter 1 | 1"
" | Just title | "
2. 图表数据丢失/格式混乱（严重）
原文有一个 Vega 交互式图表 "Share of Dataset Flagged by Issue Type"，但转换后变成了一堆混乱的文字：
当前渲染：
问题：
建议修复：
3. 网页 UI 元素被抓取（轻微）
以下内容是网页 UI 元素，不应该出现在文档中：
Listen to article 7:06- 音频播放按钮Share- 分享按钮建议：在抓取时过滤掉这些 UI 元素。
4. "Keep reading" 推荐文章（轻微）
文档末尾包含了原网站的推荐文章链接：
建议：抓取时过滤掉网站推荐内容区域。
5. 相对链接未转换
文档中的链接如
[Research](/news/research/)保持了原站的相对路径，在 wiki 中无法访问。建议：转换为完整 URL 或标记为外部链接。
根本原因分析
HTML → Markdown 转换器限制：
<pre><code>结构特殊）抓取策略问题：
建议改进
<pre><code>的识别，正确提取多行代码.share-button,.audio-player,.recommended-articles等环境影响
原始 Markdown 内容
点击展开原始 paper.md 内容