TasteVal (compute multiplier)
Science · 2026-10-05
P-Zero Research's TasteVal measures the experimental research taste of AI models on 8 hidden, open-ended AI R&D tasks, among them curating pretraining data, pretraining and fine-tuning language models, preference modeling and robustness to adversarial prompts. The model under evaluation is the Researcher: it designs experiments and interprets their results while a fixed Coder agent (Claude Opus 4.8 in Claude Code) implements and runs them on one H100, until 40 GPU-hours or 120 wall-clock hours are used. The score is the compute multiplier against the best of at least two human experts per task: the expert's serial GPU-hours divided by the model's to reach the weaker run's final test score, a geometric mean over six seeds per task and then over tasks, so 1.0 matches the experts and 2.0 reaches their score with half the experimental compute.
Benchmark source: arXiv