← All benchmarks

SWE-bench Multimodal

Code · 2024-10-04

SWE-bench Multimodal evaluates visual software issue resolution. The original paper releases 617 tasks from 17 JavaScript libraries; model reports may use internal harnesses and modified development subsets, so their generic observations do not establish one evaluation denominator. The exact 480-task Vals v1 protocol is the preferred primary within this family; generic results remain components without assigning them the Vals task count.

Top models (higher is better)