Compare AI models

Specifications
Parameters
—51B
Context window
1M128k
Benchmarks
HellaSwag
92.5%85.58%
MMLU
81.9%80.2%
GSM8K
91.7%91.43%
Overview
CompanyGoogleNVIDIA
Release dateFeb 15 2024Sep 23 2024
AccessClosedOpen Weight
Model detailsView modelView model

Frequently asked questions

Gemini 1.5 Pro leads Llama 3.1 Nemotron 51B on 3 of the 3 benchmarks they both report (HellaSwag, MMLU, GSM8K). Gemini 1.5 Pro shipped 221 days before Llama 3.1 Nemotron 51B, so benchmark comparisons should account for the intervening progress.

Context windows are 1M (Gemini 1.5 Pro) vs 128k (Llama 3.1 Nemotron 51B). Gemini 1.5 Pro is closed, while Llama 3.1 Nemotron 51B is open weight.

On HellaSwag, Gemini 1.5 Pro leads at 92.5% vs Llama 3.1 Nemotron 51B at 85.58%. On MMLU, Gemini 1.5 Pro leads at 81.9% vs Llama 3.1 Nemotron 51B at 80.2%. On GSM8K, Gemini 1.5 Pro leads at 91.7% vs Llama 3.1 Nemotron 51B at 91.43%.