Benchmarks
347 sources · 1,258 models × 3,038 benchmarks · 158,912 scoresPMPP-Hard (v0.1.1, released on GitHub by the developer Sinatras and packaged in Prime Intellect's RL Residency environments) asks a coding agent to write and optimize 69 GPU kernels, 60 in CUDA, 8 in Triton and 1 in CUTLASS, for NVIDIA Blackwell sm_120 under Verifiers 0.2.0. The agent never sees reference implementations or authoritative graders; only its submitted source is copied into a clean scoring sandbox, where a task passes when it builds, passes the authoritative correctness test and, on the 48 performance-gated tasks, runs within 1.25 times the reference's median time over five interleaved timing pairs with matching output digests, under KernelGuard source and timing checks. The score is the share of the 69 tasks passed with one accepted rollout per task; the release campaign's harness, reasoning setting and rollout limit (1,500 or 4,500 seconds) vary by model.
333rd strongest of 798 fitted benchmarks
Signal-to-noise +7.07
272nd hardest of 798 fitted benchmarks
Capability needed 159.1
| # | Model | Setting | Score | Source |
|---|---|---|---|---|
| 1 | Kimi K3 | max · Kimi Code | 71.0 | |
| 2 | Opus 4.8 | medium · Codex | 62.3 | |
| 3 | GPT-5.6 Sol | x-high · Codex | 59.4 | |
| 4 | Sonnet 5 | high · Claude Code | 52.2 | |
| 5 | GPT-5.5 | medium · Codex | 52.2 | |
| 6 | GLM-5.2 | medium · Codex | 39.1 | |
| 7 | DeepSeek-V4-Pro | unknown · Codex; source effort medium, native value unspecified | 34.8 | |
| 8 | DeepSeek-V4-Flash-0731 | high · Codex | 33.3 | |
| 9 | GPT-5.6 Terra | medium · Codex | 31.9 | |
| 10 | GPT-5.6 Luna | medium · Codex | 23.2 |
Benchmarks outside its family whose model rankings track it most reliably (Spearman ρ over at least 10 shared models).