Which AI model scores highest on Okapi Multilingual MMLU?

Claude 3 Opus by Anthropic holds the best Okapi Multilingual MMLU result among tracked models, at 79.1% (released Mar 4 2024). Higher scores are better on this benchmark.

What are the top 5 models on Okapi Multilingual MMLU?

1. Claude 3 Opus (Anthropic) — 79.1%; 2. Claude 3 Sonnet (Anthropic) — 69%; 3. Claude 3 Haiku (Anthropic) — 65.2%; 4. Claude 2.1 (Anthropic) — 63.4%; 5. Claude 2 (Anthropic) — 63.1%.

How many models have a published Okapi Multilingual MMLU score?

6 tracked models have a published Okapi Multilingual MMLU score. Scores are collected from published lab reports and benchmark sources, with recorded sources credited alongside the results. We do not re-run the evaluations.

Multilingual knowledge

Okapi Multilingual MMLU

five-shot accuracy on the Okapi translations of MMLU used in the Claude 3 model card

Rankings

Higher is better

Okapi Multilingual MMLU — frequently asked questions

: five-shot accuracy on the Okapi translations of MMLU used in the Claude 3 model card
: Claude 3 Opus by Anthropic holds the best Okapi Multilingual MMLU result among tracked models, at 79.1% (released Mar 4 2024). Higher scores are better on this benchmark.
: 1. Claude 3 Opus (Anthropic) — 79.1%; 2. Claude 3 Sonnet (Anthropic) — 69%; 3. Claude 3 Haiku (Anthropic) — 65.2%; 4. Claude 2.1 (Anthropic) — 63.4%; 5. Claude 2 (Anthropic) — 63.1%.
: 6 tracked models have a published Okapi Multilingual MMLU score. Scores are collected from published lab reports and benchmark sources, with recorded sources credited alongside the results. We do not re-run the evaluations.