Benchmarks
347 sources · 1,258 models × 3,038 benchmarks · 158,912 scoresDeepnote's Agent Initiative Benchmark tests whether an AI teammate knows when to help without being asked. The agent sits in the Slack of PageMantle, a synthetic company with six years of records across 11 connected services (CRM, product analytics, billing, a warehouse, an issue tracker, docs and Slack), and faces 50 workplace situations across eight task types: it reads the conversation, may investigate with search and read tools, including SQL, and then posts, replies, sends a direct message or stays silent, within 180 seconds of model response time. Some situations call for silence. Two blinded LLM evaluators score each run from 0 to 10, and the score is their mean averaged over the 50 situations; a run that times out when help was needed is a miss. Every model runs at high reasoning effort with the same instructions and tools.
606th strongest of 798 fitted benchmarks
Signal-to-noise +3.43
442nd hardest of 798 fitted benchmarks
Capability needed ≲ 146.5
| # | Model | Setting | Score | Source |
|---|---|---|---|---|
| 1 | GPT-6 Astra | high · Deepnote Agent Initiative Benchmark harness; Slack plus search and read tools across 11 simulated PageMantle services; 180-second model-time budget | 9.0 | |
| 2 | GPT-6.1 Sol | high · Deepnote Agent Initiative Benchmark harness; Slack plus search and read tools across 11 simulated PageMantle services; 180-second model-time budget | 8.7 | |
| 3 | Claude Fable 5.2 | high · Deepnote Agent Initiative Benchmark harness; Slack plus search and read tools across 11 simulated PageMantle services; 180-second model-time budget | 8.2 | |
| 4 | Claude Opus 5.5 | high · Deepnote Agent Initiative Benchmark harness; Slack plus search and read tools across 11 simulated PageMantle services; 180-second model-time budget | 7.8 | |
| 5 | Claude Fable 5.1 | high · Deepnote Agent Initiative Benchmark harness; Slack plus search and read tools across 11 simulated PageMantle services; 180-second model-time budget | 7.5 | |
| 6 | Claude Sonnet 5.5 | high · Deepnote Agent Initiative Benchmark harness; Slack plus search and read tools across 11 simulated PageMantle services; 180-second model-time budget | 6.7 | |
| 7 | GPT-6 Luna | high · Deepnote Agent Initiative Benchmark harness; Slack plus search and read tools across 11 simulated PageMantle services; 180-second model-time budget | 6.4 | |
| 8 | Claude Haiku 5.5 | high · Deepnote Agent Initiative Benchmark harness; Slack plus search and read tools across 11 simulated PageMantle services; 180-second model-time budget | 6.0 | |
| 9 | Gemini 4 Argon | high · Deepnote Agent Initiative Benchmark harness; Slack plus search and read tools across 11 simulated PageMantle services; 180-second model-time budget | 5.6 | |
| 10 | Grok 4.7 | high · Deepnote Agent Initiative Benchmark harness; Slack plus search and read tools across 11 simulated PageMantle services; 180-second model-time budget | 4.5 |
Benchmarks outside its family whose model rankings track it most reliably (Spearman ρ over at least 10 shared models).