Vibe-Eval (Gemini judge)
Multimodal · 2024-05-03
Vibe-Eval is Reka's open set of 269 image-and-prompt pairs with expert-written reference answers, 100 of them a hard subset, testing visual understanding in chat models. Each response is rated 1 to 5 against the reference and the mean is rescaled to 0-100. This identity holds Google's results, which use a Gemini model as the grader in place of the paper's Reka Core grader, so they are kept apart from Reka-judged scores; higher is better.