LOFT (128k)
Long-Context Frontiers benchmark evaluating long-context language models on real-world tasks requiring context up to 128k tokens, including retrieval, RAG, SQL-like reasoning, and many-shot in-context learning across text, visual, and audio modalities.
LOFT (128k) Leaderboard
| Rank | Model | Provider | Score | Parameters | Released | Type |
|---|---|---|---|---|---|---|
| 1 | Grok 3 | xAI | 83.3 | Unknown (multi-trillion estimated) | 2025-02-19 | Multimodal |
| 2 | Grok 3 Mini | xAI | 83.1 | Unknown | 2025-02-19 | Multimodal |
About LOFT (128k)
Description
## Overview LOFT (Long-Context Frontiers) is a comprehensive benchmark designed to evaluate long-context language models (LCLMs) on paradigm-shifting tasks that leverage their ability to process millions of tokens. Unlike traditional benchmarks that rely on synthetic tasks, LOFT focuses on real-world applications where LCLMs can potentially replace specialized systems and complex pipelines. ## Key Features - **Dynamic Scaling**: Supports context lengths from 32k to 1M+ tokens with automatic scaling capabilities - **Multi-Modal**: Covers text, visual, and audio modalities across 35 datasets - **Real-World Tasks**: Six core task categories that test practical applications - **Corpus-in-Context (CiC) Prompting**: Novel prompting approach for long-context evaluation ## Task Categories ### 1. Text Retrieval Evaluates LCLMs' ability to retrieve relevant information from large text corpora without specialized dual-encoder models. Includes datasets from BEIR benchmark covering argument retrieval (ArguAna), fact checking (FEVER), question answering (FIQA, MS MARCO, NQ), duplicate detection (Quora), citation prediction (SciFact), and multi-hop reasoning (HotPotQA, MuSiQue, QAMPARI, QUEST). ### 2. Visual Retrieval Tests text-to-image retrieval (Flickr30k, MS COCO), text-to-video retrieval (MSR-VTT), and image-text retrieval (OVEN) capabilities, comparing against specialized models like CLIP. ### 3. Audio Retrieval Evaluates multilingual audio retrieval across five languages (English, Hindi, Chinese, Spanish, French) using the FLEURS dataset, testing against dual-encoder models. ### 4. Retrieval-Augmented Generation (RAG) Assesses LCLMs' ability to reason over entire corpora directly, eliminating traditional RAG pipeline complexity. Uses subsets of retrieval datasets with phrase-level answer annotations. ### 5. SQL-like Reasoning Tests compositional reasoning over entire databases represented as text, using Spider (single-turn) and SParC (multi-turn) datasets. Evaluates natural language database querying without formal SQL conversion. ### 6. Many-Shot In-Context Learning Explores scaling from traditional few-shot to hundreds/thousands of examples using Big-Bench Hard (BBH) and LongICLBench (LIB) datasets for classification tasks. ## Evaluation Methodology **Corpus-in-Context (CiC) Prompting** is introduced as a novel approach that: - Provides task-specific instructions - Formats entire corpora with unique identifiers - Includes grounded few-shot examples with chain-of-thought reasoning - Maintains consistent query formatting ## Key Findings ### Performance Highlights - **Text Retrieval**: Gemini 1.5 Pro achieves 77% average performance, matching specialized Gecko retriever at 128k context - **Visual Retrieval**: Gemini 1.5 Pro (83% average) outperforms CLIP (71% average) across all visual benchmarks - **Audio Retrieval**: Perfect 100% performance on FLEURS datasets, matching/exceeding PaLM 2 DE - **RAG**: LCLMs excel at multi-hop reasoning but struggle with multi-target tasks requiring comprehensive passage ranking - **SQL**: Significant gap remains (38% vs 65% for specialized models), indicating room for improvement in compositional reasoning - **Many-Shot ICL**: Performance varies by task complexity, with knowledge-intensive tasks benefiting more from additional examples ### Positional Analysis Reveals attention degradation in later corpus sections, but strategic placement of few-shot examples can mitigate this "lost in the middle" effect. ## Technical Details - **Context Lengths**: 32k, 128k, and 1M token versions - **Evaluation Scale**: Up to 100 test queries per dataset with 5 few-shot and 10 development queries - **Metrics**: Task-specific (Recall@1 for retrieval, exact match for RAG, accuracy for SQL/ICL) - **Models Evaluated**: Gemini 1.5 Pro, GPT-4o, Claude 3 Opus ## Significance LOFT demonstrates that LCLMs can rival specialized systems across multiple domains without task-specific training, while revealing areas for improvement in long-context reasoning. The benchmark's scalable design ensures continued relevance as context windows expand to billions of tokens.
Methodology
LOFT (128k) evaluates model performance using a standardized scoring methodology. Scores are reported on a scale of 0 to 100, where higher scores indicate better performance. For detailed information about the scoring system and methodology, please refer to the original paper.
Publication
This benchmark was published in 2024.Technical Paper