Benchmarks
347 sources · 1,258 models × 3,038 benchmarks · 158,912 scoresThinking Machines Lab's adaptation of PMPP-Hard's 69 GPU kernel tasks, reported with Inkling 2 Small's launch. Each model runs at its maximum available reasoning setting in the Pi harness with a shared context-compaction policy, four independent attempts per task and a 10-hour budget; Thinking Machines fixed task and grading issues it found, and each submitted solution gets two independent LLM audits, with a third review on disagreement. The score is the area under the mean tasks-solved curve over 0 to 10 hours, in task-hours (maximum 690), so it rewards solving early as well as solving at all. Diagnostic: the fixes and audits are unpublished and the protocol differs from the v0.1.1 release's.
No signal or difficulty estimate. Held out of the fit by design.
Not enough dated models to plot (1 of 1 have a release date).
| # | Model | Setting | Score | Source |
|---|---|---|---|---|
| 1 | Inkling 2 Small | 0.99 · pi | 494 |