Lesson plan
SWE-bench
SWE-bench is a benchmark for large language models and coding agents. Its 2,294 tasks were taken from issues that were actually filed, and actually fixed, in 12 Python repositories: astropy, django, flask, matplotlib, pylint, pytest, requests, scikit-learn, seaborn, sphinx, sympy and xarray. Django alone supplies 850 of them, sympy another 386. The system being tested is shown the repository as it stood before the fix, plus the text of the issue, and has to produce a patch. It never sees the tests that will judge the result. When the benchmark was introduced in October 2023 by a group at Princeton University and the University of Chicago, under the title "Can Language Models Resolve Real-World GitHub Issues?", the strongest model they tried, Claude 2, fixed only 1.96% of the tasks. In August 2024, OpenAI released SWE-bench Verified, a human-validated subset of SWE-bench. In February 2026, the leading published result on SWE-bench Verified was 80.9%. In a post titled "Why SWE-bench Verified no longer measures frontier coding capabilities", OpenAI said it would stop quoting the figure at all. The models being scored had read the repositories the tasks come from, and in the sample OpenAI examined every frontier model it tried could reproduce wording from the problem statements or from the developers' own fix. OpenAI recommended switching to SWE-bench Pro, another variant.
Article summary
SWE-bench is a benchmark for large language models and coding agents. Its 2,294 tasks were taken from issues that were actually filed, and actually fixed, in 12 Python repositories: astropy, django, flask, matplotlib, pylint, pytest, requests, scikit-learn, seaborn, sphinx, sympy and xarray. Django alone supplies 850 of them, sympy another 386. The system being tested is shown the repository as it stood before the fix, plus the text of the issue, and has to produce a patch. It never sees the tests that will judge the result. When the benchmark was introduced in October 2023 by a group at Princeton University and the University of Chicago, under the title "Can Language Models Resolve Real-World GitHub Issues?", the strongest model they tried, Claude 2, fixed only 1.96% of the tasks. In August 2024, OpenAI released SWE-bench Verified, a human-validated subset of SWE-bench. In February 2026, the leading published result on SWE-bench Verified was 80.9%. In a post titled "Why SWE-bench Verified no longer measures frontier coding capabilities", OpenAI said it would stop quoting the figure at all. The models being scored had read the repositories the tasks come from, and in the sample OpenAI examined every frontier model it tried could reproduce wording from the problem statements or from the developers' own fix. OpenAI recommended switching to SWE-bench Pro, another variant.
Key claims
No claims recorded.
Discussion questions
- Summarise the article in your own words. Which single claim would you verify first, and why?
- Which claim in this article is most likely to be revised as the sources update? How would you check its current status?
- If two sources disagreed about one of these facts, what would you need in order to trust one over the other?
Bibliography
No sources recorded.
Key claims and bibliography are generated from the same tracked claims and sources as the article page.