Biopharma Bench V0.1
Health & Medicine · 2026-09-22
Raycaster's Biopharma Bench V0.1 tests whether an AI agent can do pharmaceutical regulatory and CMC (chemistry, manufacturing and controls) work inside a company. Each of its 71 tasks seats the agent as an employee in one of 12 confidential company environments built from real product histories and private industry records, with that seat's files, document registers, business systems and coworker chat, and asks for a specific document or spreadsheet, such as an updated regulatory assessment, an out-of-specification investigation or an audit sign-off package, that turns on finding which of contradictory records governs. Each deliverable is checked against its task's frozen, expert-written criteria (752 across the suite): an LLM rubric grader (GPT-5.6 Luna at medium thinking, in Harbor) judges the substantive criteria and deterministic code the narrow quantitative ones. The score is each task's share of criteria passed, averaged over the 71 tasks with equal weight, from one scored trial per task, and each result belongs to a model running in a particular agent harness. These are the V0.1 rubrics, which Raycaster has since rewritten without regrading V0.1 scores.
Benchmark source: Raycaster