Benchmarks
347 sources · 1,258 models × 3,038 benchmarks · 158,912 scoresMercor's enterprise-risk study: 12 expert-built agent tasks in law, banking and consulting, such as deleting files from a litigation folder, where the requested action is unsafe because of something only the workspace files reveal, such as a preservation letter or a user's missing permission. The score is the share of 36 runs (three per task) in which the agent saved the harmful change, even if it warned the user; lower is better. Runs use Mercor's own harness with a one-line system prompt at each API's highest reasoning setting, and a panel of three LLM judges grades by majority with expert review. The same tasks with the risk stated in the request, and benign control versions, are recorded separately.
No signal or difficulty estimate. This benchmark category is excluded from the capability model and its validation studies.
| # | Model | Setting | Score | Source |
|---|---|---|---|---|
| 1 | Claude Opus 5.5 | max · Mercor enterprise-risk harness; fresh copy of a simulated law, banking or consulting workspace per run and the same tools for every model; one-line system prompt | 22.2 | |
| 2 | Claude Fable 5.1 | max · Mercor enterprise-risk harness; fresh copy of a simulated law, banking or consulting workspace per run and the same tools for every model; one-line system prompt | 77.8 | |
| 3 | GPT-6 Astra | max · Mercor enterprise-risk harness; fresh copy of a simulated law, banking or consulting workspace per run and the same tools for every model; one-line system prompt | 80.6 | |
| 4 | GPT-6.1 Sol | max · Mercor enterprise-risk harness; fresh copy of a simulated law, banking or consulting workspace per run and the same tools for every model; one-line system prompt | 83.3 |