Long-Horizon Terminal-Bench 1.0 (mean reward)
Code · 2026-07-09
Long-Horizon Terminal-Bench (LHTB) 1.0, from Tencent HY LLM Frontier with academic co-authors, is 46 long-horizon tasks in Docker containers across nine categories, from games and puzzles to scientific computing, research reproduction, reverse engineering and professional workflows, each typically needing hundreds of terminal actions. An agent gets one attempt per task under a 90-minute budget in Terminus 2 on Harbor; on 30 tasks the harness resumes the agent after a failed hidden-verifier check until the budget runs out. Hidden verifiers rebuild the result from the agent's artifacts and award partial credit from 0 to 1, and the score is the mean reward over the 46 tasks, a trial with no recorded reward counting as 0, in percent. It is a separate task set and metric from the tbench.ai Terminal-Bench boards.
Benchmark source: arXiv