Gemini 1.0 Ultra
Google · 2024-02-15
Gemini 1.0 Ultra was Google's hosted text model for complex instruction following, coding, reasoning, and multilingual tasks. The gemini-ultra endpoint became generally available to allowlisted Vertex AI customers on February 15, 2024, with an 8,192-token maximum input and a 2,048-token maximum output.
Benchmark scores
| Benchmark | Score |
|---|---|
| Artificial Analysis Coding Index | 17.6 |
| BBH (BIG-Bench Hard) | 83.6 |
| Capability | 117.1 |
| GSM8K | 88.9 |
| GSM8K (Maj1@32) | 94.4 |
| HellaSwag (Unspecified Scoring Protocol) | 87.8 |
| HumanEval | 74.4 |
| MATH | 53.2 |
| MathVista (mini) | 53.0 |
| MGSM | 79.0 |
| MMLU (uncertainty-routed CoT@32) | 90.0 |
| MMMU | 59.4 |
| MMMU (Maj1@32) | 62.4 |