← All benchmarks

MultiFileTest - Java correctness rate

Code · 2025-02-10

The Java track of MultiFileTest (first released as ProjectTest), from the University of Illinois Chicago, Salesforce Research and Scale AI: the model writes unit tests for a whole multi-file Java project supplied in the prompt, run with JUnit and JaCoCo, zero-shot at temperature 0 and scored as generated. The score is the correctness rate: for each project, the share of generated unit tests that pass (zero when the suite does not compile or run), averaged over the projects. The paper says it built 20 projects per language and its executability rates move in steps of 5 points, but its Java dataset table lists 19 projects, so no item count is asserted.

Top models (higher is better)