← All benchmarks

ITSMBench

Agents · 2026-08-08

ITSMBench, from the IT service-management vendor Atomicwork and the eval lab New Measure, tests whether an agent can resolve 89 L2 and L3 IT service-desk tickets end to end, across device, network, access, security-operations and compliance work. Each task runs in a containerized enterprise of 42 mocked systems (about 1,800 database tables and over 2,000 REST endpoints, among them Okta, ServiceNow, Jira Service Management, Slack and CrowdStrike) whose APIs the agent must find through a mix of Skill.md files, MCP and plain REST, and it passes only when all of its 10 to 30 expert-written deterministic checks over the final database state confirm the right judgment, a complete fix and no unsafe changes; there is no LLM judge. The score is pass@1 in percent, the expected one-trial accuracy over repeated trials, and every model on the board runs in the open-source Pi harness. It is unrelated to the 53-task vibrantlabsai/itsm-bench dataset on Harbor Hub.

Benchmark source: Ressl

Top models (higher is better)