← All benchmarks

PI-AutomationBench

Agents · 2026-11-03

Prime Intellect's repaired fork of the public set of Zapier's AutomationBench (https://github.com/xeophon/AutomationBench-PI/tree/858718b19e5d66c74451432783d34ee64a550a15, fixes merged through November 2, 2026). Actions describe their request bodies in JSON Schema rather than prose, and most tasks and graders were reworded or relaxed where a requirement could not be reasonably inferred. This is the partial score in its 'mode: Zapier', where agents discover actions through search and execute tools. The November 3 post calls partial scoring close to saturation and states neither the task count nor the agent harness. Scores run higher than on the private held-out set and aren't directly comparable with Zapier's own runs, which use api_fetch, or with the upstream public set.

Benchmark source: Primeintellect

Top models (higher is better)