Aider Polyglot
Code · 2024-12-21
Aider's Polyglot benchmark contains 225 difficult Exercism tasks across C++, Go, Java, JavaScript, Python, and Rust. Edit format is a run-level harness setting recorded in ModelConfig.
Top models (higher is better)
| Model | Score |
|---|---|
| GPT-5 | 88.0 |
| o3-pro | 84.9 |
| Gemini 2.5 Pro | 83.1 |
| Grok 4 | 79.6 |
| Gemini 2.5 Pro Preview 05-06 | 76.9 |
| o3 | 76.9 |
| DeepSeek-V3.2-Exp | 74.2 |
| Gemini 2.5 Pro Preview 03-25 | 72.9 |
| Opus 4 | 72.0 |
| Gemini 2.5 Pro Experimental 03-25 | 68.6 |
| Claude 3.7 Sonnet | 64.9 |
| o1 (2024-12-17) | 61.7 |
| Sonnet 4 | 61.3 |
| o3-mini | 60.4 |
| Qwen3-235B-A22B | 59.6 |
| Kimi K2 Instruct | 59.1 |
| DeepSeek-R1 | 56.9 |
| Gemini 2.5 Flash | 56.7 |
| DeepSeek-V3-0324 | 55.1 |
| Grok 3 (API) | 53.3 |
| GPT-4.1 | 52.4 |
| Claude 3.5 Sonnet (Oct 2024) | 51.6 |
| Grok 3 Mini | 49.3 |
| Gemini 2.5 Flash Preview 04-17 | 47.1 |
| GPT-4.5 | 44.9 |
| gpt-oss-120b | 41.8 |
| Qwen3 32B | 40.0 |
| Gemini Exp-1206 | 38.2 |
| Gemini 2.0 Pro Experimental 02-05 | 35.6 |
| o1-mini | 32.9 |
| GPT-4.1 Mini | 32.4 |
| Haiku 3.5 | 28.0 |
| Gemini 2.5 Flash-Lite | 27.1 |
| GPT-4o (2024-08-06) | 23.1 |
| Gemini 2.0 Flash Experimental | 22.2 |
| Qwen2.5-Max | 21.8 |
| QwQ-32B | 20.9 |
| Gemini 2.0 Flash Thinking Experimental 01-21 | 18.2 |
| GPT-4o (2024-11-20) | 18.2 |
| DeepSeek-V2.5 | 17.8 |