PI-AutomationBench direct mode
Agents · 2026-11-03
Partial score of PI-AutomationBench in 'mode: direct', which registers each task's connected tools directly with the harness instead of leaving agents to discover them through search and execute tools. Prime Intellect's chart reports it with Prime Agent. A separate benchmark because the tool mode changes the task: DeepSeek V4.1-Flash 1017 gains 5 partial points and 18 strict points from it.
Benchmark source: Primeintellect
Top models (higher is better)
| Model | Score | Source |
|---|---|---|
| DeepSeek V4.1 Flash 1017 | 96.8 | Prime Intellect, Learnings from Auditing Public Evaluations · Prime Intellect |
| Claude Opus 5.5 | 92.8 | Prime Intellect, Learnings from Auditing Public Evaluations · Prime Intellect |