Compare AI models

Specifications
Parameters
—51B
Context window
1M128k
Benchmarks
HellaSwag
86.5%85.58%
MMLU
78.9%80.2%
GSM8K
86.2%91.43%
Overview
CompanyGoogleNVIDIA
Release dateMay 14 2024Sep 23 2024
AccessClosedOpen Weight
Model detailsView modelView model

Frequently asked questions

Llama 3.1 Nemotron 51B leads Gemini 1.5 Flash on 2 of the 3 benchmarks they both report (HellaSwag, MMLU, GSM8K). Gemini 1.5 Flash shipped 132 days before Llama 3.1 Nemotron 51B, so benchmark comparisons should account for the intervening progress.

Context windows are 1M (Gemini 1.5 Flash) vs 128k (Llama 3.1 Nemotron 51B). Gemini 1.5 Flash is closed, while Llama 3.1 Nemotron 51B is open weight.

On HellaSwag, Gemini 1.5 Flash leads at 86.5% vs Llama 3.1 Nemotron 51B at 85.58%. On MMLU, Llama 3.1 Nemotron 51B leads at 80.2% vs Gemini 1.5 Flash at 78.9%. On GSM8K, Llama 3.1 Nemotron 51B leads at 91.43% vs Gemini 1.5 Flash at 86.2%.