Gorilla APIBench

Name: Gorilla APIBench
Published: 2023-05-24

Agents · 2023-05-24

APIBench, from the Gorilla paper, asks a model to write an API call for a natural-language request over Hugging Face, Torch Hub and TensorFlow Hub models. A call counts as correct when its AST matches an API in the reference database with the ground-truth domain. Scores are 0-shot call accuracy.

Top models (higher is better)

Model	Score
Claude 3.5 Sonnet (June 2024)	60.0
GPT-4o (source unspecified)	41.4
Llama 3.1 405B Instruct	35.3
Llama 3.1 70B Instruct	29.7
Mixtral 8x22B Instruct	26.0
GPT-4 0125 Preview	22.5
Llama 3 70B Instruct	14.7
Gemma 2 9B IT	11.6
Llama 3.1 8B Instruct	8.2
Llama 3 8B Instruct	1.7