Ruby LLM Benchmarks — Program Fixer
Code · 2025-08-01
Ruby LLM Benchmarks Program Fixer evaluates one-shot repairs of four broken Ruby programs: a calendar, parking garage, school library, and vending machine. Each task score is 90% test-suite success plus 10% RuboCop quality; the headline score is the equal-weight mean of the four task scores, with a missing task contributing zero.
Top models (higher is better)
| Model | Score |
|---|---|
| Claude Opus 5 | 79.8 |
| GPT-6 Sol | 79.0 |
| GLM-5.3 Flash | 78.7 |
| GPT-5.6 Terra Pro | 78.1 |
| GPT-5.5 | 77.6 |
| Claude Opus 5.5 | 77.5 |
| Claude Fable 5 | 77.2 |
| Opus 4.7 | 77.1 |
| Opus 4.6 | 76.4 |
| Sonnet 4.6 | 76.3 |
| Claude Fable 5.1 | 76.0 |
| GLM-5 | 75.6 |
| GPT-5.6 Sol Pro | 75.3 |
| Opus 4.8 | 75.2 |
| GPT-5.6 Sol | 75.2 |
| GPT-6 Astra | 75.1 |
| GPT-6.1 Sol | 75.0 |
| GPT-5.4 | 74.8 |
| GLM-5.2 | 74.2 |
| Kimi K2.6 | 74.2 |
| Muse Spark 1.3 | 74.1 |
| Qwen3.8-Max | 74.0 |
| GPT-5.6 Terra | 74.0 |
| Claude Sonnet 5.5 | 73.8 |
| GPT-5.6 Luna | 73.3 |
| GPT-5.6 Luna Pro | 73.1 |
| Sonnet 4 | 72.6 |
| DeepSeek-V4-Pro | 72.5 |
| Sonnet 4.5 | 72.3 |
| GPT-5.2 Instant (source unspecified) | 72.1 |
| GLM-5.3 | 72.1 |
| Opus 4.5 | 71.6 |
| Opus 4.1 | 71.3 |
| Gemma 4 26B A4B IT | 71.0 |
| Gemma 4 31B IT | 70.9 |
| Opus 4 | 70.8 |
| GPT-4.1 | 70.8 |
| Sonnet 5 | 70.3 |
| GPT-5.3 Instant (source unspecified) | 70.2 |
| Grok 4.5 | 70.0 |