Agentic Tool Use (Chat)

Name: Agentic Tool Use (Chat)
Published: 2024-09-19

Agents · 2024-09-19

Score on ToolComp-Chat, which tests dependent tool use with Python and Google Search.

Model	Score
o3-mini	63.5
Gemini 2.5 Pro Preview 03-25	62.4
O1 Pro	61.4
DeepSeek-R1	60.9
o1 (source unspecified)	60.4
DeepSeek-V3	58.5
Gemini 2.0 Pro Experimental 02-05	57.9
Gemini 2.0 Flash Thinking Experimental 01-21	57.4
GPT-4o (2024-08-06)	56.9
GPT-4.5	56.3
Claude 3.7 Sonnet	56.3
Claude 3.5 Sonnet (June 2024)	56.1
o1 Preview	55.1
DeepSeek-V3-0324	54.3
Gemini 2.0 Flash Experimental	53.3
GPT-4 0125 Preview	53.0
Gemini 1.5 Pro Experimental 0827	51.3
Gemini 2.0 Flash-Lite Preview 02-05	50.3
GPT-4o (2024-05-13)	49.5
Nova Pro	49.2
Opus 3	48.5
Gemini 2.0 Flash	48.2
Nova Lite	41.7
Nova Micro	41.0
Claude 3 Sonnet	40.4
Mistral Large 2 (Instruct 2407)	40.4
Llama 3.1 405B Instruct	40.1
GPT-4 (source unspecified)	37.9
Gemini 1.5 Pro (May 2024)	35.5
Llama 3.1 70B Instruct	33.5
GPT-4o Mini	32.8
C4AI Command R+	20.2
Llama 3.1 8B Instruct	6.1