Compare AI models

Specifications
Parameters
—6B
Context window
32k32k
Benchmarks
HumanEval
67.7%9.8%
MATH
32.6%6.9%
HellaSwag
84.7%57%
MMLU
71.8%45.2%
GSM8K
77.9%25.9%
Overview
CompanyGoogleZ.ai
Release dateDec 13 2023Jun 25 2023
AccessClosedOpen Weight
Model detailsView modelView model

Frequently asked questions

Gemini 1.0 Pro leads ChatGLM2-6B on 5 of the 5 benchmarks they both report. ChatGLM2-6B shipped 171 days before Gemini 1.0 Pro, so benchmark comparisons should account for the intervening progress.

Context windows are 32k (Gemini 1.0 Pro) vs 32k (ChatGLM2-6B). Gemini 1.0 Pro is closed, while ChatGLM2-6B is open weight.

On HumanEval, Gemini 1.0 Pro leads at 67.7% vs ChatGLM2-6B at 9.8%. On MATH, Gemini 1.0 Pro leads at 32.6% vs ChatGLM2-6B at 6.9%. On HellaSwag, Gemini 1.0 Pro leads at 84.7% vs ChatGLM2-6B at 57%. On MMLU, Gemini 1.0 Pro leads at 71.8% vs ChatGLM2-6B at 45.2%. On GSM8K, Gemini 1.0 Pro leads at 77.9% vs ChatGLM2-6B at 25.9%.