BFCL v2 Live

Name: BFCL v2 Live
Published: 2024-08-19

Agents · 2024-08-19

The Live portion of the Berkeley Function-Calling Leaderboard v2: user-contributed, enterprise-style function-calling prompts (simple, multiple, parallel and parallel-multiple calls plus relevance and irrelevance detection) scored by AST matching, reported as the overall live accuracy. It is a version- and subset-specific BFCL score, kept as a diagnostic rather than pooled with other BFCL versions. Higher is better.

Top models (higher is better)

Model	Score
Llama 3.1 Nemotron Ultra 253B V1	74.1
Llama 3.3 Nemotron Super 49B V1	73.7
QwQ-32B	71.6
Llama 3.1 Nemotron Nano 4B V1.1	67.9
Mistral-NeMo-Minitron 8B Instruct	67.6
Nemotron-H 4B Instruct 128K	65.9
DeepSeek-R1-Distill-Llama-70B	65.5
Phi-4-mini-instruct	61.6
Llama 3.3 70B Instruct	60.4
Qwen2.5 3B Instruct	59.1
Llama 3.1 405B Instruct	58.7
Llama 3.2 3B Instruct	49.6
Mistral NeMo Instruct 2407	47.9
Llama 3.1 8B Instruct	44.3
DeepSeek R1 Distill Llama 8B	37.8