Gemini 1.5 Pro (May 2024)
Google · 2024-05-14
Gemini 1.5 Pro (May 2024) is the updated checkpoint label used in Google DeepMind's Gemini 1.5 technical report; Google announced quality improvements to Gemini 1.5 Pro at I/O on May 14, 2024, and the report says the May version followed multiple pre-training and post-training iterations. Because the report provides no machine identifier that unambiguously maps this checkpoint to a released API version, Atlas retains it as a benchmark entry separate from the February version.
Benchmark scores
| Benchmark | Score |
|---|---|
| Agentic Tool Use (Chat) | 35.5 |
| BBH (BIG-Bench Hard) | 89.2 |
| GSM8K | 90.8 |
| HellaSwag (Unspecified Scoring Protocol) | 93.3 |
| HumanEval | 84.1 |
| MATH | 67.7 |
| MathVista (mini) | 63.9 |
| MGSM | 87.5 |
| MMLU | 85.9 |
| MMMU | 62.2 |