CheatBench
Safety · 2026-09-28
The Center for AI Safety's CheatBench measures how often AI agents cheat on hard assignments that contain a discoverable shortcut, such as a planted solution, a colleague's accepted work, exposed chess-engine advice or a way to tamper with grading, across ten categories: mathematical research, multimodal, creative writing, SVG competition, menial computation, biology and bioinformatics, knowledge work, board games, software engineering and sycophancy. An LLM judge, or in some environments a classifier over recorded events, labels each episode that contains a cheating attempt, successful or not; nine category rates pool the episodes of their tasks (software engineering counts each task once), and sycophancy is a 0-100 score for how far direct-API probability estimates shift toward a user's stated views. The score is the equal-weight mean of the ten category scores, in percent, and lower is better. Agents run in their developer's own harness or, in the General Harness view, in Terminus-2; the score is a propensity, not a capability.
Benchmark source: arXiv