← All benchmarks

CliniCARE-Bench

Health & Medicine · 2026-08-07

Scale AI's CliniCARE-Bench, with Emory, Vanderbilt, UC Santa Cruz and Stanford co-authors, tests retrospective clinical audit over MIMIC-IV v3.1 records. Its 25 clinician-validated scenarios are each instantiated for 30 patients, giving 750 cases; for each, an agent investigates the record through governed MCP tools for retrieval, computation and a policy corpus, with web search and fetch disabled, and returns Yes, No, Indeterminate: Lack of Data or Indeterminate: Medically Ambiguous. Reference verdicts are the majority of three agent configurations (Claude Code with Opus 4.8, Codex with GPT-5.5 and Gemini CLI with Gemini 3.1 Pro), with the 11 three-way splits adjudicated by clinicians; on a 75-case sample the reference matched clinicians on 64 of the 70 cases with a defined label. The score is the paper's four-class verdict accuracy over all 750 cases, one attempt per case, with vendor models on their own agent CLI (Claude Code, Codex or Gemini CLI) and open-weight models on opencode; defect-free accuracy and the process, grounding and calibration metrics are not imported.

Top models (higher is better)