ToolSandbox
Agents · 2024-08-08
Apple's ToolSandbox evaluates conversational tool use with stateful tool execution, implicit state dependencies between tools and an LLM-simulated user for on-policy multi-turn dialogue. Each of its 1,032 test scenarios is scored by matching the trajectory against milestone and minefield annotations, giving a 0-1 similarity that drops to zero on a minefield; the headline averages these similarities on a 0-100 scale.
Top models (higher is better)
| Model | Score |
|---|---|
| LFM2.5-24B-A2B | 80.7 |
| LFM3-1.7B | 80.4 |
| LFM2.5-2.6B | 78.0 |
| Qwen3.5-9B | 76.4 |
| Qwen3.5-4B | 75.4 |
| Nemotron 3.5 Lightning 30B A3B (precision unspecified) | 75.1 |
| Qwen3.6 35B A3B | 75.0 |
| Gemma 4 26B A4B IT | 72.1 |
| LFM2.5-VL-3B | 59.5 |