SuperAPI AI Directory
Partner with us中文AI newsFavorites
DirectoryNews and tutorials

FutureX - A Dynamic Real-Time Evaluation Benchmark Jointly Launched by ByteDance, Fudan University, and Other Universities

Author: AI小集 · Original source: AI工具集

What Is FutureX

FutureX is a dynamic real-time evaluation benchmark jointly released by research teams from ByteDance, Fudan University, Stanford University, and Princeton University, specifically designed for future prediction tasks of LLM agents. It collects future event questions in real time from 195 high-quality websites through a semi-automated pipeline, and automatically obtains the true results for scoring after the events are resolved, effectively avoiding data contamination. FutureX covers multiple domains such as politics, economics, finance, sports, and entertainment, and includes various types of questions such as single-choice, multiple-choice, open-ended ranking, and numerical prediction, divided into four difficulty levels, comprehensively evaluating the reasoning and prediction capabilities of LLM agents.

FutureX

Main Features of FutureX

  • Dynamic Real-Time Updates: FutureX can collect future event questions in real time and automatically obtain the true results for scoring after the events are resolved, ensuring the timeliness and dynamism of the evaluation.
  • Avoiding Data Contamination: By focusing on future event prediction, FutureX ensures that the answers have not yet occurred when the agent makes predictions, avoiding data contamination and ensuring the fairness of the evaluation.
  • Simulating Real-World Challenges: FutureX places LLM agents in real-world information flows and requires them to predict future events, which demands advanced cognitive skills such as information gathering, data synthesis, probability weighing, and causal reasoning.
  • Large-Scale Cross-Domain Coverage: FutureX collects questions from 195 high-quality websites, covering multiple domains such as politics, economics, finance, sports, and entertainment, providing a comprehensive evaluation environment.
  • Automated Evaluation Process: The evaluation process of FutureX is fully automated, automatically updating questions, collecting answers, and conducting objective scoring every day, improving the efficiency and scalability of the evaluation.
  • Multiple Question Types and Difficulty Levels: FutureX includes various types of questions such as single-choice, multiple-choice, open-ended ranking, and numerical prediction, and is divided into four difficulty levels, comprehensively evaluating the capabilities of LLM agents.
  • Advancing the Development of LLM Agents: FutureX provides a dynamic, contamination-free evaluation standard for LLM agents, driving them toward the level of professional human analysts and promoting performance improvements in complex reasoning and prediction tasks.

Core Advantages of FutureX

  • Design Principles: FutureX aims to provide a dynamic, comprehensive, and data-contamination-free evaluation, simulating real-world challenges and assessing the core intelligence of LLM agents.
  • No Data Contamination: By focusing on future event prediction, FutureX ensures that the answers have not yet occurred when the agent makes predictions, thereby avoiding data contamination.
  • Simulating Real-World Challenges: FutureX places agents in real-world information flows and requires them to predict future events, which demands advanced cognitive skills such as information gathering, data synthesis, probability weighing, and causal reasoning.
  • Large-Scale and Cross-Domain Coverage: FutureX collects questions from 195 high-quality websites through a semi-automated pipeline, covering multiple domains such as politics, economics, finance, sports, and entertainment.
  • Dynamic and Automated Evaluation Process: FutureX automatically updates questions, collects answers, and conducts objective scoring every day, ensuring the timeliness, objectivity, and scalability of the evaluation.

Construction Process of FutureX

  • Website Collection and Screening: Use the AIME agent to collect a large number of relevant website URLs, screen out high-quality websites through LLM and manual review, and ultimately determine 195 as the event database.
  • Event template generation: Create event templates for each website; these templates can generate events adapted to different times based on variables.
  • Daily event planning: Generate prediction questions from the event database every day, including manipulation of events (such as adding random options) and filtering (removing harmful, subjective, or overly simple events).
  • Agent prediction and evaluation: Trigger the agent model every day to make predictions on new events, and automatically obtain the true results for scoring after the events are resolved.
  • Continuous updates and maintenance: Update the event database every day, remove events with unavailable results, and add new events to ensure the benchmark's dynamism and timeliness.

Data characteristics of FutureX

  • Real-time: FutureX's data is updated in real time, collecting future event questions from 195 high-quality websites every day to ensure that the evaluation content stays in sync with current information.
  • Diversity: The data covers multiple fields such as politics, economics, finance, sports, and entertainment, and includes multiple types of questions such as single-choice, multiple-choice, open-ended ranking, and numerical prediction.
  • No contamination: It focuses on future event prediction, ensuring that the answers have not yet occurred when the agent makes predictions, avoiding data contamination and ensuring the fairness of the evaluation.
  • Dynamic: FutureX's events and answers are updated dynamically, and the event database adds new events or removes unavailable events according to the actual situation, maintaining the vitality of the data.
  • Challenging: Through event filtering and difficulty-level classification, FutureX ensures the challenge of the questions, comprehensively evaluating the capabilities of LLM agents from simple multiple-choice questions to complex open-ended questions.
  • Large-scale: FutureX is currently the largest and most diverse real-time future prediction benchmark, generating about 500 events per week and providing abundant evaluation samples.
  • Reliability: Through strict data screening and manual review, the reliability and quality of data sources are ensured, providing a trustworthy foundation for evaluation.

FutureX project address

  • arXiv technical paper：https://arxiv.org/pdf/2508.11987

Experimental results of FutureX

  • Overall results: Grok-4 and Gemini-2.5-flash Deep Research perform best on the most difficult tasks, while basic LLMs perform well on simple tasks.
  • Results at different difficulty levels: As task difficulty increases, model performance drops significantly, especially at Level 4 (super agent level), where models struggle the most.
  • Results in different domains: Different models show different strengths in different domains; for example, GPT models perform better in cryptocurrency and technology domains, while DouBao-Seed1.6-Thinking stands out in finance and economics.
  • Factor analysis: Through linear regression analysis, it was found that difficulty level, domain, and model name have a significant impact on performance.
  • Case studies: Includes comparisons between LLM agents and Wall Street financial analysts, the impact of fake websites on agents, and evaluation of real-time search capabilities.

Application scenarios of FutureX

  • Financial domain: FutureX can be used to evaluate the predictive capabilities of LLM agents regarding future events such as stock prices and economic indicators, helping financial institutions screen high-performance analytical agents.
  • Policy Making: Provide policymakers with reliable intelligent agent evaluation tools to help them assess the potential impacts of different policies.
  • Business Decision-Making: Help enterprises evaluate market trends and consumer behavior, providing support for business decisions.
  • Technology Trend Analysis: Predict technological development and innovation trends, providing a basis for decision-making for technology companies and investors.
  • Sports Event Prediction: Predict sports competition outcomes and athlete performance, providing reference for sports betting and event organizers.
  • Entertainment Industry: Predict the popularity and box office revenue of entertainment products such as movies and music, providing support for decision-making in the entertainment industry.