SWE-bench
SWE-bench is a benchmark for large language models and coding agents.
≈ 2 min read
Names in other languages
No names in other languages are held for this entry yet.
Article bodies are English and are not machine-translated. A name in another language is not a translation of the article.
SWE-bench is a benchmark for large language models and coding agents. Its 2,294 tasks were taken from issues that were actually filed, and actually fixed, in 12 Python repositories: astropy, django, flask, matplotlib, pylint, pytest, requests, scikit-learn, seaborn, sphinx, sympy and xarray. Django alone supplies 850 of them, sympy another 386.
The system being tested is shown the repository as it stood before the fix, plus the text of the issue, and has to produce a patch. It never sees the tests that will judge the result.
When the benchmark was introduced in October 2023 by a group at Princeton University and the University of Chicago, under the title "Can Language Models Resolve Real-World GitHub Issues?", the strongest model they tried, Claude 2, fixed only 1.96% of the tasks. In August 2024, OpenAI released SWE-bench Verified, a human-validated subset of SWE-bench.
In February 2026, the leading published result on SWE-bench Verified was 80.9%. In a post titled "Why SWE-bench Verified no longer measures frontier coding capabilities", OpenAI said it would stop quoting the figure at all.
The models being scored had read the repositories the tasks come from, and in the sample OpenAI examined every frontier model it tried could reproduce wording from the problem statements or from the developers' own fix. OpenAI recommended switching to SWE-bench Pro, another variant.
Text derived from https://en.wikipedia.org/w/index.php?oldid=1377784329 under CC BY-SA 4.0; modifications have been made.