Nexus Function Calling

Name: Nexus Function Calling
Published: 2023-12-04

Agents · 2023-12-04

The Nexus function-calling benchmark from Nexusflow (NexusRaven) evaluates zero-shot API use across nine tasks with single, nested and parallel calls, using the open-source NexusRaven prompts and evaluation function. Scores are macro-averaged accuracy across the tasks.

Top models (higher is better)

Model	Score
Llama 3.1 405B Instruct	58.7
Llama 3.1 70B Instruct	56.7
GPT-4o (source unspecified)	56.1
GPT-4 0125 Preview	50.3
Mixtral 8x22B Instruct	48.5
Llama 3 70B Instruct	47.8
Claude 3.5 Sonnet (June 2024)	45.7
Llama 3.1 8B Instruct	38.5
Llama 3.2 3B Instruct	34.3
Gemma 2 9B IT	30.0
Phi-3.5 Mini Instruct	26.1
Gemma 2 2B IT	21.0
Llama 3 8B Instruct	18.1
Llama 3.2 1B Instruct	13.5