API-Bank (Meta corrected protocol)
Agents · 2024-07-23
API-Bank evaluates whether a model can make correct API calls within tool-use dialogues. This is the protocol Meta used for Llama 3 and later: 0-shot with a custom prompt and parser, ground-truth answers that Meta corrected and completed, and a scoring rule that applies exact match to keyword arguments with a unique answer and ROUGE to free-text arguments. Scores are function-calling accuracy and are not comparable with the original API-Bank metric.