← All benchmarks

MultiFileTest - Python correctness rate

Code · 2025-02-10

MultiFileTest (first released as ProjectTest), from the University of Illinois Chicago, Salesforce Research and Scale AI, asks a model to write unit tests for a whole multi-file GitHub project supplied in the prompt (2 to 15 files, under 1,600 lines); this track has 20 Python projects run with pytest. Generation is zero-shot at temperature 0 and the tests are scored as generated, the paper's vanilla scenario. The score is the correctness rate: for each project, the share of generated unit tests that pass (zero when the suite does not run), averaged over the 20 projects; executability, coverage and the manual- and self-fixing scenarios are not imported.

Top models (higher is better)