OSWorld (self-reported, A11y tree)

Name: OSWorld (self-reported, A11y tree)
Published: 2024-04-11

Agents · 2024-04-11

Self-reported OSWorld success rate with accessibility-tree observation.

Model	Score
Claude 3.5 Sonnet (June 2024)	22.5
GPT-4o (source unspecified)	16.6
GPT-4 0125 Preview	12.2
GPT-4 Turbo	10.8
Gemini 1.5 Pro (source unspecified)	10.3
Qwen Max (source-unspecified mainline)	6.9
GPT-4o Mini	6.2
Qwen VL Max 0809	4.3
Mixtral 8x7B	3.0
LLaVA-OneVision 72B	2.7
Gemini 1.0 Pro	2.4
Llama 3 70B Instruct	1.6