Grok‑2 mini

Released

Grok‑2 mini is an AI model released by SpaceXSI on Tuesday, Aug 13 2024, the same day as Grok‑2. Benchmark results (shown below) cover HumanEval, MATH, MMLU-Pro, MMLU, MathVista, and MMMU.

Benchmarks

Coding

HumanEvalFunction synthesis — 164 small Python problems: the AI is given a function's description and has to write the working function. This was the coding benchmark of the GPT-3.5 and GPT-4 era, before the field moved to fixing real bugs in real repositories. Higher is better.
85.7%
#22 of 43

Reasoning & science

MATHReasoning — accuracy on the original competition mathematics test set
73%
#9 of 25
MMLUGeneral knowledge — A 57-subject multiple-choice exam — history, law, medicine, maths — that was the standard measure of how much a model knows from 2020 until roughly 2024, when frontier scores crowded into the high 80s and labs moved on to harder tests. The scores here were published years apart under different testing setups, so read them as a historical record rather than a like-for-like ranking. Higher is better.
86.2%
#19 of 46

Knowledge work

MMLU-ProKnowledge — accuracy on the ten-option MMLU-Pro benchmark
72%
#9 of 14

Multimodal

MathVistaMultimodal reasoning — mathematical reasoning about visual information
68.1%
#6 of 14
MMMUMultimodal — Tests the AI on understanding images and text together across many college subjects. Higher is better.
63.2%
#30 of 43

Compare Grok‑2 mini with

Grok‑2 mini

About

Grok-2 mini entered beta on August 13, 2024 alongside Grok-2. xAI positioned the smaller model as a balance between response speed and answer quality, with access through Grok on X and enterprise API availability planned separately. It was a named sibling in the model family, giving users a lighter alternative to the larger Grok-2.

Its launch evaluations covered both text and visual understanding, including 72.0% on MMLU-Pro, 85.7% on HumanEval and 63.2% on MMMU. Those figures came from the mini model's own column in xAI's comparison, with HumanEval reported as pass@1. They should be distinguished from Grok-2's scores and from the separate image-generation model integrated into the Grok product.

Frequently asked questions

Grok‑2 mini was released by SpaceXSI on Tuesday, Aug 13 2024.

All SpaceXSI releases

22 tracked

2023

1 release