# HumanEval — AI model rankings

164 small Python problems: the AI is given a function's description and has to write the working function. This was the coding benchmark of the GPT-3.5 and GPT-4 era, before the field moved to fixing real bugs in real repositories. Higher is better.

3 tracked models have a published HumanEval score. Higher is better. Scores are as published at each model's release.

## Ranking

| Rank | Model | Developer | Score | Released |
| --- | --- | --- | --- | --- |
| 1 | Claude 2 | Anthropic | 71.2% | Jul 11 2023 |
| 2 | GPT-4 | OpenAI | 67% | Mar 14 2023 |
| 3 | Claude Instant 1.2 | Anthropic | 58.7% | Aug 9 2023 |


---

Canonical page: https://aireleasetracker.com/benchmark/humaneval
Full dataset: https://aireleasetracker.com/llms-full.txt · JSON: https://aireleasetracker.com/models.json
Source: AI Release Tracker (https://aireleasetracker.com). Benchmark scores are the figures published by the releasing lab at launch.