CloudOpsBench v1.0
Agents
An open benchmark of 45 AWS operations tasks such as decommissioning, repairs and access changes. An agent gets a shell with the AWS CLI, boto3 and Terraform for 40 minutes on the Vera emulator, four runs per task, and a checker reads the final state. The resolution rate leaves evaluation errors out. It stays diagnostic: no publisher is named, and the repository says 42 of the tasks haven't been run with oracle or negative controls.
Benchmark source: Cloudopsbench
Top models (higher is better)
| Model | Score | Source |
|---|---|---|
| Claude Fable 5.1 | 60.6 | CloudOpsBench v1.0: claude-code+claude-fable-5-1+max · CloudOpsBench |
| GPT-6 Astra | 58.9 | CloudOpsBench v1.0: codex+gpt-6-astra+max · CloudOpsBench |
| Gemini 4 Argon | 57.5 | CloudOpsBench v1.0: gemini-cli+gemini-4-argon+high · CloudOpsBench |
| Grok 4.7 | 56.5 | CloudOpsBench v1.0: cursor-cli+grok-4.7+high · CloudOpsBench |
| Claude Opus 5.5 | 47.8 | CloudOpsBench v1.0: terminus-2+claude-opus-5-5+high · CloudOpsBench |
| Gemini 3.8 Flash | 42.8 | CloudOpsBench v1.0: terminus-2+gemini-3.8-flash+high · CloudOpsBench |
| Grok 4.6 | 39.7 | CloudOpsBench v1.0: cursor-cli+grok-4.6+high · CloudOpsBench |
| DeepSeek V4.1 Flash | 21.8 | CloudOpsBench v1.0: terminus-2+deepseek-flash+high · CloudOpsBench |
| Kimi K3.1 Pro | 19.1 | CloudOpsBench v1.0: terminus-2+kimi-k3.1-pro+high · CloudOpsBench |