LLM Benchmarks

AI2 Reasoning Challenge (ARC) tests reasoning through grade-school science questions.

Advanced competitive programming benchmark for evaluating large language models on algorithmic problem-solving and code generation.

The FACTS Grounding Leaderboard evaluates LLMs' ability to generate factually accurate long-form responses grounded in provided context documents up to 32k tokens.

Graduate-level Problems in Quantitative Analysis (GPQA) evaluates advanced reasoning on graduate-level physics and mathematics problems.

Tests common sense natural language inference through completion of scenarios.

Evaluates code generation capabilities by asking models to complete Python functions based on docstrings and function signatures.

Long-Context Frontiers benchmark evaluating long-context language models on real-world tasks requiring context up to 128k tokens, including retrieval, RAG, SQL-like reasoning, and many-shot in-context learning across text, visual, and audio modalities.

A dataset of 12,500 challenging competition mathematics problems requiring multi-step reasoning.

Massive Multitask Language Understanding (MMLU) tests knowledge across 57 subjects including mathematics, history, law, and more.

A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI with 11.5K college-level questions across 6 disciplines and 30 subjects.

A benchmark of over 1,400 freelance software engineering tasks from Upwork, valued at $1 million USD total in real-world payouts. Evaluates frontier LLMs on both individual contributor tasks and software engineering management decisions.