Compare AI models

Specifications
Parameters
—8B
Context window
32k8k
Benchmarks
HumanEval
74.4%36.22%
HellaSwag
87.8%83.03%
MMLU
83.7%69.51%
GSM8K
88.9%58.45%
Overview
CompanyGoogleNVIDIA
Release dateFeb 8 2024Aug 21 2024
AccessClosedOpen Weight
Model detailsView modelView model

Frequently asked questions

Gemini 1.0 Ultra leads Mistral NeMo Minitron 8B on 4 of the 4 benchmarks they both report (HumanEval, HellaSwag, MMLU, GSM8K). Gemini 1.0 Ultra shipped 195 days before Mistral NeMo Minitron 8B, so benchmark comparisons should account for the intervening progress.

Context windows are 32k (Gemini 1.0 Ultra) vs 8k (Mistral NeMo Minitron 8B). Gemini 1.0 Ultra is closed, while Mistral NeMo Minitron 8B is open weight.

On HumanEval, Gemini 1.0 Ultra leads at 74.4% vs Mistral NeMo Minitron 8B at 36.22%. On HellaSwag, Gemini 1.0 Ultra leads at 87.8% vs Mistral NeMo Minitron 8B at 83.03%. On MMLU, Gemini 1.0 Ultra leads at 83.7% vs Mistral NeMo Minitron 8B at 69.51%. On GSM8K, Gemini 1.0 Ultra leads at 88.9% vs Mistral NeMo Minitron 8B at 58.45%.