Compare AI models

Specifications
Parameters
1B/4B/12B/27B—
API pricing
Cheapest input
$0.04DeepInfra—
Cheapest output
$0.08DeepInfra—
$15.00DeepInfra · bf16—
Benchmarks
HumanEval
87.8%85.7%
MATH
89%73%
MMLU-Pro
67.5%72%
MMLU
76.9%86.2%
MathVista
67.6%68.1%
MMMU
64.9%63.2%
Overview
CompanyGoogleSpaceXSI
Release dateMar 12 2025Aug 13 2024
AccessOpen WeightClosed
Model detailsView modelView model

Frequently asked questions

Gemma 3 and Grok‑2 mini are evenly matched across the 6 benchmarks they both report. Grok‑2 mini shipped 211 days before Gemma 3, so benchmark comparisons should account for the intervening progress.

Gemma 3 is open weight, while Grok‑2 mini is closed.

On HumanEval, Gemma 3 leads at 87.8% vs Grok‑2 mini at 85.7%. On MATH, Gemma 3 leads at 89% vs Grok‑2 mini at 73%. On MMLU-Pro, Grok‑2 mini leads at 72% vs Gemma 3 at 67.5%. On MMLU, Grok‑2 mini leads at 86.2% vs Gemma 3 at 76.9%. On MathVista, Grok‑2 mini leads at 68.1% vs Gemma 3 at 67.6%. On MMMU, Gemma 3 leads at 64.9% vs Grok‑2 mini at 63.2%.