Concept Synth INDUCTION Challenge64 Holdout
General QA
Post-selection diagnostic for direct Round-1 Challenge64 formulas against the original (v1) holdout sidecar: train-correct responses verified correct on all available generated holdout worlds, divided by all 64 tasks. The v1 generator left one Challenge64 task without holdout worlds and others with fewer than five. Tasks without verified holdout success receive no credit, including unavailable holdout worlds; missing outcomes remain unknown in the underlying records. Holdout results are never used for prompting or response selection. Superseded on the live leaderboard by the holdouts-v2 diagnostic.
Top models (higher is better)
| Model | Score |
|---|---|
| GPT-6 Astra | 85.9 |
| GPT-6.1 Sol | 65.6 |
| Claude Opus 5.5 | 54.7 |
| Claude Sonnet 5.5 | 40.6 |
| GPT-6 Sol | 37.5 |
| Gemini 4 Argon | 37.5 |
| GPT-5.6 Sol | 29.7 |
| Claude Fable 5.1 | 29.7 |
| Claude Opus 5 | 26.6 |
| GPT-5.6 Terra | 25.0 |
| Claude Fable 5 | 23.4 |
| Grok 4.6 | 23.4 |
| Muse Spark 1.1 | 21.9 |
| Grok 4.7 | 18.8 |
| GPT-6 Luna | 18.8 |
| Grok 4 | 17.2 |
| GPT-5.4 | 14.1 |
| Gemini 3.7 Flash | 14.1 |
| Grok 4.5 | 12.5 |
| Gemini 3.8 Flash | 12.5 |
| GPT-5.6 Luna | 10.9 |
| Muse Spark 1.2 | 10.9 |
| Gemini 3.5 Flash | 9.4 |
| DeepSeek V4 Pro 0813 | 7.8 |
| Grok 4.1 Fast | 7.8 |
| Kimi K3 | 7.8 |
| Opus 4.6 | 6.3 |
| DeepSeek-V4-Pro | 6.3 |
| DeepSeek V4.1 Flash | 4.7 |
| Gemini 3.6 Flash | 4.7 |
| Kimi K2.7 Code | 4.7 |
| Opus 4.8 | 4.7 |
| Kimi K2.6 | 4.7 |
| GPT-5.2 | 3.1 |
| Sonnet 5 | 3.1 |
| Grok 4.3 | 3.1 |
| Gemini 3 Pro Preview | 1.6 |
| DeepSeek-V3.2 | 1.6 |
| Gemini 3.1 Pro Preview | 0.0 |
| Opus 4.5 | 0.0 |