Tests models on their ability to write code in multiple programming languages.

Aider-Polyglot

American Invitational Mathematics Examination (AIME) 2024 problems.

AIME-2024

American Invitational Mathematics Examination (AIME) 2025 problems.

AIME-2025

American Invitational Mathematics Examination (AIME) problems test advanced mathematical problem-solving.

AIME

AI2 Reasoning Challenge (ARC) tests reasoning through grade-school science questions.

The first comprehensive evaluation of LLMs' function calling capabilities, testing various forms including parallel and multiple function calls across diverse programming languages. Evaluates models on execution accuracy and ability to withhold function selection when appropriate.

Berkeley Function-Calling Leaderboard

Beyond the Imitation Game Benchmark (BIG-bench) is a collaborative benchmark of 204 diverse tasks.

BIG-bench

BIRD (BIg Bench for LaRge-scale Database Grounded Text-to-SQL Evaluation) is a cross-domain dataset that examines the impact of extensive database contents on text-to-SQL parsing. It contains over 12,751 unique question-SQL pairs, 95 big databases with a total size of 33.4 GB, and covers more than 37 professional domains including blockchain, hockey, healthcare, and education.

BIRD-SQL

Evaluates models on their ability to browse and interact with web interfaces.

BrowseComp

Tests reasoning on challenging problems from arXiv papers across multiple scientific domains.

CharXiv-Reasoning

Evaluates models on competitive programming problems from the Codeforces platform.

Codeforces

CoVoST 2 is an open, large-scale multilingual speech-to-text translation (ST) dataset developed to advance research in both high- and low-resource language settings. It includes translations from 21 languages into English and from English into 15 languages, making it the largest publicly available corpus in terms of language coverage and volume (2880 hours, 78K speakers). Built on Mozilla's Common Voice, CoVoST 2 extends its predecessor by incorporating rigorous quality control (language model perplexity, LASER scores, length heuristics) and improved data utilization via revised train/dev/test splits. The dataset supports automatic speech recognition (ASR), machine translation (MT), and end-to-end ST, with baseline models implemented in fairseq. Extensive evaluations show that multilingual training substantially boosts performance on low-resource language pairs.

CoVoST 2

Evaluates models on their ability to solve cybersecurity challenges across various domains including cryptography, web exploitation, reverse engineering, binary exploitation, and forensics.

Cybersecurity CTF

Discrete Reasoning Over Paragraphs (DROP) requires models to resolve references in a passage and perform discrete operations.

DROP

EgoSchema is a very long-form video question-answering dataset and benchmark for evaluating long video understanding capabilities of vision and language systems. Derived from Ego4D, it consists of over 5000 human-curated multiple-choice question-answer pairs spanning more than 250 hours of real video data covering a broad range of natural human activity and behavior. Each question requires selecting the correct answer from five options based on a three-minute-long video clip. EgoSchema introduces temporal certificate sets to measure intrinsic temporal understanding length, showing 5.7x longer temporal lengths than the second closest dataset and 10-100x longer than other video understanding datasets. Current state-of-the-art video and language models achieve less than 33% accuracy (random is 20%) while humans achieve about 76%, highlighting significant gaps in long-term video understanding capabilities.

EgoSchema

Evaluates models on their ability to ground responses in factual information.

FACTS-Grounding

A balanced collection of culturally sensitive and culturally agnostic MMLU tasks designed for efficient evaluation of multilingual models in 15 languages (including English).

Global-MMLU-Lite

A multilingual evaluation set spanning 42 languages that combines machine translations for MMLU questions along with professional translations and crowd-sourced post-edits. Includes cultural sensitivity annotations classifying questions as Culturally Sensitive (CS) or Culturally Agnostic (CA).

Global-MMLU

Graduate-level Problems in Quantitative Analysis (GPQA) evaluates advanced reasoning on graduate-level physics and mathematics problems.

GPQA

Grade School Math 8K (GSM8K) consists of 8.5K high-quality grade school math word problems.

GSM8K

Tests common sense natural language inference through completion of scenarios.

HellaSwag

Google's internal holdout set of competition math problems

HiddenMath

Evaluates code generation capabilities by asking models to complete Python functions based on docstrings and function signatures.

HumanEval

A challenging benchmark of novel problems designed to test the limits of AI capabilities.

Humanitys-Last-Exam

A contamination-limited benchmark with frequently-updated questions from recent sources, scoring answers automatically against objective ground-truth values. Covers math, coding, reasoning, language, instruction following, and data analysis tasks.

LiveBench

Evaluates models on their ability to solve coding problems in real-time.

LiveCodeBench-v5

A sample of 500 diverse problems from the MATH benchmark, spanning topics like probability, algebra, trigonometry, and geometry. The questions test a model's ability to apply mathematical principles, execute complex calculations, and communicate solutions clearly. Models are prompted to present final answers in boxed LaTeX format, and evaluated using parsing logic from the PRM800K dataset grader. Most models are evaluated with temperature set to 0, except for reasoning models that require specific temperature settings.

MATH 500

A dataset of 12,500 challenging competition mathematics problems requiring multi-step reasoning.

MATH

Evaluates mathematical reasoning in visual contexts, combining vision and mathematical problem-solving.

MathVista

Multilingual Grade School Math (MGSM) extends GSM8K to 10 languages.

MGSM

MMLU-Pro is an enhanced benchmark with over 12,000 challenging questions across 14 domains including Biology, Business, Chemistry, Computer Science, Economics, Engineering, Health, History, Law, Math, Philosophy, Physics, Psychology, and Others. It features 10 answer choices per question (vs. 4 in MMLU) and focuses on complex reasoning tasks.

MMLU-Pro

Massive Multitask Language Understanding (MMLU) tests knowledge across 57 subjects including mathematics, history, law, and more.

MMLU

Massive Multi-discipline Multimodal Understanding (MMMU) evaluates multimodal understanding across 30 disciplines.

MMMU

MRCR (Multi-Round Coreference Resolution) is part of the Michelangelo benchmark suite that evaluates long-context reasoning in LLMs. It tests models' ability to track identities and references in adversarial conversation histories up to 1M tokens. Unlike retrieval-based benchmarks, MRCR requires synthesis, reasoning, and contextual understanding over extended contexts. The benchmark is synthetically generated to avoid pretraining contamination and provides unambiguous scoring metrics. MRCR is recommended as a superior alternative to traditional retrieval benchmarks for evaluating long-context reasoning capabilities.

Michelangelo Long-Context Reasoning (128k)

Michelangelo Long-Context Reasoning (1M)

Multi-IF evaluates LLMs on multi-turn and multilingual instruction following across 8 languages, with 4,501 conversations of three turns each. It reveals performance degradation with each additional turn and challenges with non-Latin scripts.

Multi-IF

A multi-domain challenge set created by Scale AI to test models across diverse tasks.

Scale-MultiChallenge

A benchmark of simple but precise questions to test factual knowledge and reasoning.

SimpleQA

Software Engineering Benchmark (SWE-bench) evaluates models on real-world software engineering tasks.

SWE-bench

Evaluates models on real-world freelance software engineering tasks.

SWE-Lancer

Tool Augmented Understanding Benchmark (TAU-bench) evaluates models on their ability to use tools.

TAU-bench

Evaluates models on their ability to use terminal commands to solve system administration tasks.

Terminal-bench

Measures a model's tendency to reproduce falsehoods commonly believed by humans.

TruthfulQA

Evaluates models on their ability to understand and generate content with specific vibes or styles.

Vibe-Eval

Video-MME is the first comprehensive Multi-Modal Evaluation benchmark for assessing Multi-modal Large Language Models (MLLMs) in video analysis. It features four key distinctions: 1) Diversity across 6 primary visual domains with 30 subfields; 2) Duration spanning short-, medium-, and long-term videos ranging from 11 seconds to 1 hour; 3) Breadth in data modalities including video frames, subtitles, and audio; and 4) Quality annotations from expert annotators. The benchmark comprises 900 manually selected videos totaling 254 hours, with 2,700 question-answer pairs. Evaluations of state-of-the-art MLLMs revealed Gemini 1.5 Pro as the best-performing commercial model, significantly outperforming open-source alternatives, while highlighting the need for improvements in handling longer sequences and multi-modal data.

Video-MME

WMT24 is the 2024 edition of the Workshop on Machine Translation, which provides a ranking of General Machine Translation systems and Large Language Models. The evaluation includes both automatic metrics and human evaluation, with the human evaluation considered the official ranking as it is superior to automatic metrics. The benchmark serves as a comprehensive evaluation platform for comparing the performance of various translation systems, including traditional MT systems and newer LLM-based approaches. The preliminary results are provided to participants to assist in their system submissions, while the final official ranking is determined through human evaluation.

WMT24

A sparse Mixture-of-Experts model (8 experts) with a total of 46.7B parameters (12.9B active per token). Outperformed dense models like Llama 2 70B and GPT-3.5 on many benchmarks at significantly lower compute cost.

Mixtral 8×7B

Specifications

Benchmark Scores

Advanced Specifications

Capabilities & Limitations

Related Models

Mistral Large (123B)

Pixtral 123B

Mistral 7B