Compare AI models

Specifications
Parameters
—51B
Context window
32k128k
Benchmarks
HellaSwag
87.8%85.58%
MMLU
83.7%80.2%
GSM8K
88.9%91.43%
Overview
CompanyGoogleNVIDIA
Release dateFeb 8 2024Sep 23 2024
AccessClosedOpen Weight
Model detailsView modelView model

Frequently asked questions

Gemini 1.0 Ultra leads Llama 3.1 Nemotron 51B on 2 of the 3 benchmarks they both report (HellaSwag, MMLU, GSM8K). Gemini 1.0 Ultra shipped 228 days before Llama 3.1 Nemotron 51B, so benchmark comparisons should account for the intervening progress.

Context windows are 32k (Gemini 1.0 Ultra) vs 128k (Llama 3.1 Nemotron 51B). Gemini 1.0 Ultra is closed, while Llama 3.1 Nemotron 51B is open weight.

On HellaSwag, Gemini 1.0 Ultra leads at 87.8% vs Llama 3.1 Nemotron 51B at 85.58%. On MMLU, Gemini 1.0 Ultra leads at 83.7% vs Llama 3.1 Nemotron 51B at 80.2%. On GSM8K, Llama 3.1 Nemotron 51B leads at 91.43% vs Gemini 1.0 Ultra at 88.9%.