HumanEval
codingPending Verification
Evaluates code generation capabilities by asking models to complete Python functions based on docstrings and function signatures.
Published: 2021
Score Range: 0-100
Top Score: 92.4
HumanEval Leaderboard
| Rank | Model | Provider | Score | Parameters | Released | Type |
|---|---|---|---|---|---|---|
| 1 | o1-mini | OpenAI | 92.4 | 2024-09-12 | Text | |
| 2 | o1-preview | OpenAI | 92.4 | 2024-09-12 | Text | |
| 3 | Claude 3.5 Sonnet | Anthropic | 92 | 2024-06-20 | Multimodal | |
| 4 | GPT-4o | OpenAI | 90.2 | 2024-05-13 | Multimodal | |
| 5 | Gemini Diffusion | 89.6 | 2025-05-20 | Text | ||
| 6 | Grok-2 | xAI | 88.4 | Unknown | 2024-08-13 | Multimodal |
| 7 | Claude 3.5 Haiku | Anthropic | 88.1 | 2024-10-22 | Multimodal | |
| 8 | Kimi K2 | Moonshot AI | 85.7 | 1T total, 32B activated | 2025-07-11 | Text |
| 9 | Grok-2 mini | xAI | 85.7 | Unknown | 2024-08-13 | Multimodal |
| 10 | Claude 3 Opus | Anthropic | 84.9 | 2024-03-04 | Multimodal |
About HumanEval
Methodology
HumanEval evaluates model performance using a standardized scoring methodology. Scores are reported on a scale of 0 to 100, where higher scores indicate better performance. For detailed information about the scoring system and methodology, please refer to the original paper.
Publication
This benchmark was published in 2021.Technical Paper