Longtail-SWE

Name: Longtail-SWE results
Creator: Village

Longtail-SWE (Nebius, the SWE-rebench team) asks coding agents to resolve real GitHub issues in 13 programming languages that existing issue-resolution benchmarks barely cover: Clojure, Dart, Elixir, Erlang, F#, Haskell, Julia, Kotlin, Lua, OCaml, R, Scala and Swift. Its 202 tasks from 129 repositories each come from an issue and its merged pull request, with a Docker environment and tests; a solver-judge-repairer pipeline revised descriptions and tests that rejected valid solutions or accepted wrong ones, and tasks that Qwen 3.8 27B solved in all three calibration attempts were generally dropped. An attempt succeeds only when every fail-to-pass and pass-to-pass test passes. Pass@1, the headline, is the success rate over three attempts per task. Scores are for a model in the authors' SWE-agent-like scaffold, not for the model alone.

Models

Top score

81.5

Leader

Claude Opus 5.5

Updated

Nov 6, 2026

Resolution

—

Saturation

—

No signal or difficulty estimate. Too few models have been measured on this benchmark to estimate its difficulty.

Organizations

7 dated models · 3 frontier modelsClosed weightsOpen weights

Show projectionFrontier only

Data: Nebiusatlas.village.dev

#	Model	Setting	Score	Source
1	Claude Opus 5.5	high · SWE-rebench swelike scaffold; 1M context	81.5	Longtail-SWE paper Table 1 (Opus 5.5 (high))
2	GPT-6.1 Sol	high · SWE-rebench swelike scaffold; 1M context	78.4	Longtail-SWE paper Table 1 (GPT 6.1 Sol (high))
3	GPT-5.6 Sol	high · SWE-rebench swelike scaffold; 1M context	75.7	Longtail-SWE paper Table 1 (GPT 5.6 Sol (high))
4	GPT-5.6 Luna	x-high · SWE-rebench swelike scaffold; 1M context	71.5	Longtail-SWE paper Table 1 (GPT 5.6 Luna (xhigh))
5	Kimi K3	max · SWE-rebench swelike scaffold; 262K context	63.7	Longtail-SWE paper Table 1 (Kimi K3 (max))
6	GLM-5.3 Flash	not reported · SWE-rebench swelike scaffold; 262K context	48.2	Longtail-SWE paper Table 1 (GLM-5.3-Flash (max))
7	Qwen3.6 27B	thinking · SWE-rebench swelike scaffold; 262K context	16.7	Longtail-SWE paper Table 1 (Qwen 3.6 27B (thinking))