Compare AI models

Specifications
Parameters
—8B
Context window
1M8k
Benchmarks
HumanEval
71.9%36.22%
HellaSwag
92.5%83.03%
MMLU
81.9%69.51%
GSM8K
91.7%58.45%
Overview
CompanyGoogleNVIDIA
Release dateFeb 15 2024Aug 21 2024
AccessClosedOpen Weight
Model detailsView modelView model

Frequently asked questions

Gemini 1.5 Pro leads Mistral NeMo Minitron 8B on 4 of the 4 benchmarks they both report (HumanEval, HellaSwag, MMLU, GSM8K). Gemini 1.5 Pro shipped 188 days before Mistral NeMo Minitron 8B, so benchmark comparisons should account for the intervening progress.

Context windows are 1M (Gemini 1.5 Pro) vs 8k (Mistral NeMo Minitron 8B). Gemini 1.5 Pro is closed, while Mistral NeMo Minitron 8B is open weight.

On HumanEval, Gemini 1.5 Pro leads at 71.9% vs Mistral NeMo Minitron 8B at 36.22%. On HellaSwag, Gemini 1.5 Pro leads at 92.5% vs Mistral NeMo Minitron 8B at 83.03%. On MMLU, Gemini 1.5 Pro leads at 81.9% vs Mistral NeMo Minitron 8B at 69.51%. On GSM8K, Gemini 1.5 Pro leads at 91.7% vs Mistral NeMo Minitron 8B at 58.45%.