matplotgym
Code · 2026-10-29
Tuva's matplotgym: 668 tasks from 265 papers in 11 fields, each asking an agent to turn a paper's released raw data into the figure its authors drew. The agent gets the data, a schema file, a one-line claim and an empty make_figure.py, and works in a plain-text harness (one bash, python, write or done action per turn, native tool calling off, 14 turns, 120 seconds per command). The rerun script's figure is compared by code, with no judge model, against the authors' own plotting code run on the same data, on six weighted decisions (data selection, encoding, overlays, panel structure, axis scales, series coverage). Each task scores 0-1 with partial credit; the score is the mean over three attempts per task and all 668 tasks, x 100. A 171-task first release went up on October 20, 2026 with different scores.
Benchmark source: Tuva