Compare AI models

Specifications
Parameters
—8B
Context window
1M8k
Benchmarks
HumanEval
74.3%36.22%
HellaSwag
86.5%83.03%
MMLU
78.9%69.51%
GSM8K
86.2%58.45%
Overview
CompanyGoogleNVIDIA
Release dateMay 14 2024Aug 21 2024
AccessClosedOpen Weight
Model detailsView modelView model

Frequently asked questions

Gemini 1.5 Flash leads Mistral NeMo Minitron 8B on 4 of the 4 benchmarks they both report (HumanEval, HellaSwag, MMLU, GSM8K). Gemini 1.5 Flash shipped 99 days before Mistral NeMo Minitron 8B, so benchmark comparisons should account for the intervening progress.

Context windows are 1M (Gemini 1.5 Flash) vs 8k (Mistral NeMo Minitron 8B). Gemini 1.5 Flash is closed, while Mistral NeMo Minitron 8B is open weight.

On HumanEval, Gemini 1.5 Flash leads at 74.3% vs Mistral NeMo Minitron 8B at 36.22%. On HellaSwag, Gemini 1.5 Flash leads at 86.5% vs Mistral NeMo Minitron 8B at 83.03%. On MMLU, Gemini 1.5 Flash leads at 78.9% vs Mistral NeMo Minitron 8B at 69.51%. On GSM8K, Gemini 1.5 Flash leads at 86.2% vs Mistral NeMo Minitron 8B at 58.45%.