Compare AI models

Specifications
Parameters
—51B
Context window
32k128k
Benchmarks
HellaSwag
84.7%85.58%
MMLU
71.8%80.2%
GSM8K
77.9%91.43%
Overview
CompanyGoogleNVIDIA
Release dateDec 13 2023Sep 23 2024
AccessClosedOpen Weight
Model detailsView modelView model

Frequently asked questions

Llama 3.1 Nemotron 51B leads Gemini 1.0 Pro on 3 of the 3 benchmarks they both report (HellaSwag, MMLU, GSM8K). Gemini 1.0 Pro shipped 285 days before Llama 3.1 Nemotron 51B, so benchmark comparisons should account for the intervening progress.

Context windows are 32k (Gemini 1.0 Pro) vs 128k (Llama 3.1 Nemotron 51B). Gemini 1.0 Pro is closed, while Llama 3.1 Nemotron 51B is open weight.

On HellaSwag, Llama 3.1 Nemotron 51B leads at 85.58% vs Gemini 1.0 Pro at 84.7%. On MMLU, Llama 3.1 Nemotron 51B leads at 80.2% vs Gemini 1.0 Pro at 71.8%. On GSM8K, Llama 3.1 Nemotron 51B leads at 91.43% vs Gemini 1.0 Pro at 77.9%.