Which AI model scores highest on ARC-Challenge?

LLaMA 3.1 by Meta holds the best ARC-Challenge result among tracked models, at 96.9% (released Jul 23 2024). Higher scores are better on this benchmark.

What are the top 5 models on ARC-Challenge?

1. LLaMA 3.1 (Meta) — 96.9%; 2. GPT-4 (OpenAI) — 96.3%; 3. Gemma 2 (Google) — 71.4%; 4. LLaMA 2 (Meta) — 57.4%; 5. LLaMA 1 (Meta) — 56%.

How many models have a published ARC-Challenge score?

6 tracked models have a published ARC-Challenge score. Scores are collected from published lab reports and benchmark sources, with recorded sources credited alongside the results. We do not re-run the evaluations.

What is the best open model on ARC-Challenge?

Gemma 2 by Google is the highest-ranked model with downloadable weights on ARC-Challenge, scoring 71.4% at rank 3 overall.

Reasoning

ARC-Challenge

multiple-choice science questions from the AI2 Reasoning Challenge; unrelated to ARC-AGI

Rankings

Higher is better

ARC-Challenge — frequently asked questions

: multiple-choice science questions from the AI2 Reasoning Challenge; unrelated to ARC-AGI
: LLaMA 3.1 by Meta holds the best ARC-Challenge result among tracked models, at 96.9% (released Jul 23 2024). Higher scores are better on this benchmark.
: 1. LLaMA 3.1 (Meta) — 96.9%; 2. GPT-4 (OpenAI) — 96.3%; 3. Gemma 2 (Google) — 71.4%; 4. LLaMA 2 (Meta) — 57.4%; 5. LLaMA 1 (Meta) — 56%.
: 6 tracked models have a published ARC-Challenge score. Scores are collected from published lab reports and benchmark sources, with recorded sources credited alongside the results. We do not re-run the evaluations.
: Gemma 2 by Google is the highest-ranked model with downloadable weights on ARC-Challenge, scoring 71.4% at rank 3 overall.