SWE-Lancer IC SWE - OpenAI pass rate
Code · 2025-11-19
Share of SWE-Lancer individual-contributor (IC SWE) freelance engineering tasks a model solves, graded by end-to-end Playwright tests, as reported by OpenAI (pass@1, one attempt per task). OpenAI's system cards run the Diamond split on the July 2025 dataset update; release posts do not always restate the split, so this stays diagnostic. Higher is better.
Top models (higher is better)
| Model | Score |
|---|---|
| GPT-5.3-Codex | 81.4 |
| GPT-5.1-Codex-Max | 80.0 |
| GPT-5.2-Codex | 76.0 |
| GPT-5.2 | 74.6 |
| GPT-5.1 | 69.7 |
| GPT-5-Codex | 67.0 |
| GPT-5.1-Codex | 66.3 |
| GPT-5 | 55.0 |
| GPT-4.5 | 17.4 |
| GPT-4.1 | 14.4 |
| GPT-4.1 Mini | 13.1 |
| GPT-4o (2024-11-20) | 12.4 |
| o1 (2024-12-17) | 9.7 |
| o3-mini | 7.4 |
| GPT-4o Mini | 4.8 |
| GPT-4.1 Nano | 3.7 |