WitnessBench v1.0
Games · 2026-09-26
Fleet AI's first WitnessBench release, introduced in the WITNESS paper (arXiv 2609.32208): an agent must discover the hidden rules of 17 private, unseen line-puzzle games in the style of The Witness, 20 levels each, by playing them at 5 seeds through the paper's canonical text harness (an ASCII board, a rule-hypothesis memory, up to 16 moves per reply, 300 actions per level and 1,000 LLM calls per game, no code execution), and an unsolved level ends the run. The headline RHAE-L5, taken from ARC-AGI-3, scores each solved level among the first five by min((human reference actions / agent actions)^2, 1.15), weights level i by i, caps each game at its weighted completion fraction, and averages over the 17 games and 5 seeds. It is superseded by, and not comparable with, v2.0 (26 games, 3 seeds, harness v2, OAE-L20 against the shortest solution).
Top models (higher is better)
| Model | Score |
|---|---|
| Claude Fable 5 | 36.4 |
| Claude Opus 5 | 30.6 |
| GPT-6 Astra Pro | 29.8 |
| GPT-6 Astra | 27.6 |
| Kimi K3 | 24.6 |
| GPT-5.6 Sol | 23.0 |
| Opus 4.8 | 22.8 |
| Muse Spark 1.2 | 14.9 |
| Grok 4.6 | 12.9 |
| DeepSeek V4 Pro 0813 | 7.1 |
| Qwen3.8 2.4T A95B | 6.9 |
| Qwen3.8 27B | 6.4 |
| GLM-5.2 | 5.6 |
| GLM-5.3 | 4.7 |
| Qwen3.5 35B A3B | 2.8 |
| Muse Glimmer 30B | 2.7 |
| dots3-note Preview | 2.6 |
| Qwen3.5-9B | 1.0 |