PubMedQA (SecureBio)
Health & Medicine
SecureBio's own runs of PubMedQA on all 1,000 expert-labelled PQA-L questions, which isn't the 500-question test split of pubmedqa. Evaluated in Inspect with zero-shot prompting, usually over 10 runs. SecureBio doesn't state how refusals enter accuracy. Refusals reach only 0.2% here, and on other tests of the dashboard printed accuracy exceeds what counting refusals as wrong allows, so accuracy appears to cover answered questions only and each model's denominator differs. These scores aren't comparable with other runs of the test, which is why this is a diagnostic.
Benchmark source: Securebio