BFCL v2 (AST summary)
Agents · 2024-08-19
The Berkeley Function-Calling Leaderboard v2, which added live user-contributed functions and queries to the v1 test categories, scored with its "AST Summary" metric: the macro average of abstract-syntax-tree checks of the generated calls (overall_ast_summary/macro_avg/valid). Executable and relevance categories are not part of this summary, so it is kept separate from the v1 overall accuracy and from BFCL v4.
Top models (higher is better)
| Model | Score |
|---|---|
| Llama 3.1 405B Instruct | 81.1 |
| Llama 3.1 70B Instruct | 77.5 |
| Llama 3.3 70B Instruct | 77.3 |
| Llama 3.2 3B Instruct | 67.0 |
| Phi-3.5 Mini Instruct | 58.4 |
| Gemma 2 2B IT | 27.4 |
| Llama 3.2 1B Instruct | 25.7 |