← All benchmarks

WitnessBench v2.0

Games

Fleet AI's WitnessBench tests whether an agent can discover the hidden rules of a game it has never seen by playing it. Version 2.0 has 26 private, unseen line-puzzle games in the style of The Witness, 20 levels each: the agent reads the board as text, keeps a list of rule hypotheses and writes plans of moves, and is never told the rules. Every model plays each game from level 1 at 3 seeds through the same harness v2, with at most 300 actions per level and 1,000 reflection calls per episode, and an episode ends at the first level it does not beat. OAE-L20 scores each beaten level by (shortest/actions)^2 x 100 against the fewest actions the game engine accepts and an unbeaten level 0, weights level i by i over all 20 levels, and averages the game scores over the 26 games and 3 seeds. It is not comparable with v1.0 (17 games, 5 seeds, harness v1, RHAE-L5 scored against human play).

Top models (higher is better)