OTelBench
Code · 2026-01-18
Quesma's OTelBench asks a coding agent in a Linux terminal to add OpenTelemetry instrumentation to small services of about 300 lines in 11 languages: C++, Go, Java, JavaScript, .NET, PHP, Python, Ruby, Rust, Erlang and Swift. Its 23 tasks range from a single service to microservices and distributed context propagation, and tests check the emitted telemetry, including span names, parent-child relationships, trace IDs and propagation, not just that the code builds. Each model runs three attempts per task in Harbor, and the score is the share of its 69 runs that pass.
Top models (higher is better)
| Model | Score |
|---|---|
| Opus 4.5 | 29.0 |
| GPT-5.2 | 26.1 |
| Sonnet 4.5 | 21.7 |
| Gemini 3 Flash Preview | 18.8 |
| Gemini 3 Pro Preview | 15.9 |
| GPT-5.2-Codex | 15.9 |
| GPT-5.1 | 14.5 |
| GLM-4.7 | 13.0 |
| DeepSeek-V3.2 | 11.6 |
| GPT-5.1-Codex-Max | 11.6 |
| Kimi K2 Thinking | 7.2 |
| Haiku 4.5 | 5.8 |
| Grok 4 | 4.3 |
| Grok 4.1 Fast | 2.9 |