← All benchmarks

HANDBOOK.md Agents v1.1

Professional Work

Surge's HANDBOOK.md Agents board, which the page tags v1.1: long-context agentic instruction following in enterprise RL environments with company handbooks, internal tools and external MCP servers across five domains. The page's only changelog line, undated, reads "Cleaned up rubrics and judging protocol and recalculated scores". It names no metric, task set or run count and doesn't say what changed in the judging. The launch post and paper still describe the earlier protocol: strict pass@1 with deterministic grading and four runs on each task, over 65 tasks in the paper. Every earlier value fits whole passes over those 260 runs, but 10 of the 14 v1.1 values don't, so the item count stays unknown and these may not be the same measure. Kept apart from the original board because the runs it kept rose by as much as 16.7 points (Kimi K3 Max, 11.9 to 28.6).

Benchmark source: Surgehq

Top models (higher is better)