ForeSci: Evaluating LLM Agents for Forward-Looking AI Research Judgment Corresponding author.
Qiuyu Tian 1,2 Haojie Yin 3 Yingce Xia 2 Youyong Kong 1 Zequn Liu 2 1 Southeast University, Nanjing, China 2 Beijing Zhongguancun Academy, Beijing, China 3 Duke Kunshan University, Kunshan, China · 2026
Read ForeSci: Evaluating LLM Agents for Forward-Looking AI Research Judgment Corresponding author. by Qiuyu Tian 1,2 Haojie Yin 3 Yingce Xia 2 Youyong Kong 1 Zequn Liu 2 1 Southeast University, Nanjing, China 2 Beijing Zhongguancun Academy, Beijing, China 3 Duke Kunshan University, Kunshan, China (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-00644.txt.
Introduction
Tian and co-authors present ForeSci, a 2026 test of how well AI agents judge future research paths. The test asks which bottleneck to attack or which line of work to back, before proof exists. It holds 500 tasks from four fast-moving AI fields and four kinds of choice.
Each task ships with a knowledge base cut off at a fixed date. Later papers stay hidden and serve only to check the answers. The team tests plain language models, a hybrid search setup, and three research agents. Agents give clearer traces of their proof than search does. But their picks beat plain models only by a small, uneven margin. The authors find a key flaw: agents cite the right proof yet pick the wrong target. Checks that tie each choice back to its proof cut this drift.
Catalog record
| CREATOR | Qiuyu Tian 1,2 Haojie Yin 3 Yingce Xia 2 Youyong Kong 1 Zequn Liu 2 1 Southeast University, Nanjing, China 2 Beijing Zhongguancun Academy, Beijing, China 3 Duke Kunshan University, Kunshan, China |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 90 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.00644
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ForeSci: Evaluating LLM Agents for Forward-Looking AI Research Judgment Corresponding author. Qiuyu Tian 1,2 Haojie Yin 3 Yingce Xia 2 Youyong Kong 1 Zequn Liu 2 1 Southeast University, Nanjing, China 2 Beijing Zhongguancun Academy, Beijing, China 3 Duke Kunshan University, Kunshan, China Abstract AI research often requires decisions before future evidence exists: which bottleneck to attack, which direction to pursue, or where a project should be positioned. We introduce ForeSci, a temporally controlled benchmark for evaluating whether LLM agents can make such forward-looking research judgements from historical evidence. ForeSci contains 500 tasks across four fast-moving AI domains and four decision families. Each task is paired with a cutoff-aligned offline knowledge base; post-cutoff papers are hidden during generation and used only for validation. To avoid random future-event prediction, tasks are derived from pre-cutoff taxonomy branches and evidence signals, and answer-generation backbones are selected to precede the task cutoffs. We evaluate native LLMs, Hybrid RAG, and three research-agent adaptations across four backbones. Results show that explicit evidence organization improves traceability and factual support, but gains depend strongly on the decision family. Diagnostics reveal a recurring evidence-decision decoupling: agents may cite relevant evidence while forecasting the wrong research object. ForeSci turns forward-looking AI research judgement into a controlled benchmark for evaluating research agents as decision-making systems. 1 Introduction AI research moves on a timescale where today’s frontier becomes tomorrow’s baseline. The value of a research decision ( e.g. , which bottleneck to attack, which direction is worth a six-month commitment) often lies in anticipating where the field is going. As autonomous research agents are increasingly deployed for ideation, planning, and scientific workflow execution ( 26 ; 23 ; 37 ; 44 ; 12 ; 5 ; 28 ; 41 ) , they are being asked to participate in this forward-looking decision layer. Whether current LLM agents can make defensible, evidence-grounded research judgements about an as-yet-unwritten future is therefore a central open question. Existing benchmarks do not fully answer this question. Prior work has mostly evaluated whether AI systems can answer questions over papers, synthesize literature ( 21 ; 40 ; 22 ) , use tools ( 45 ; 33 ) , execute research workflows ( 5 ; 28 ; 41 ) , or generate components of future papers, such as related work, contribution content, citations, and impact ( 2 ) . None of these tasks asks whether an agent can produce an open-ended research decision, such as picking a bottleneck, ranking a research agenda, or selecting a venue, using only the evidence available at a specific historical moment. Building such a benchmark raises two challenges. First, the evidence boundary must be enforceable. Post-cutoff papers should not appear in retrieval or in the backbone’s training data. Otherwise, a system may rely on hindsight rather than foresight ( 50 ; 46 ; 24 ; 2 ; 42 ) . Second, the tasks must be historically inferable. They should be grounded in signals available before the cutoff, rather than in arbitrary future events or design choices. A foresight benchmark must therefore govern both what a system can see and what it is fair to ask. Figure 1: Representative ForeSci task examples across the four decision families: direction forecasting, bottleneck–opportunity discovery, strategic research planning, and venue-aware research positioning. To address these challenges, we introduce ForeSci, a temporally controlled benchmark for forward-looking AI research judgement. It contains 500 tasks across four fast-moving AI domains and four decision families (Figure 1 ). Each task pairs a public question with a cutoff-aligned offline knowledge base, while post-cutoff evidence is hidden until evaluation. Tasks are constructed from pre-cutoff taxonomy branches, node-level evidence records, and method-evolution signals, ensuring that each decision is historically inferable but not directly answerable from future leakage. Each answer is evaluated by four complementary signals: factual support following the atomic-fact ( 29 ) , future-target alignment ( 42 ) , evidence traceability, and reviewer persuasiveness motivated by peer-review reliability analyses ( 9 ) . We evaluate a native LLM, Hybrid RAG, and three offline-adapted research-agent systems across four LLM backbones. To avoid data leakage, all systems operate within the same historical knowledge base and all LLM backbones are trained before the time cutoff. Results show that agent-style methods improve evidence traceability and factuality, but the strongest method differs by decision family. A diagnostic audit further reveals an evidence-decision decoupling: agents can cite relevant pre-cutoff evidence while forecasting the wrong object, mis-assigning causal roles, or selecting the wrong intervention. Beyond retrospective evaluation, we demonstrate that the same construction pipeline supports fully prospective forecasting, enabling continued evaluation of research agents as new literature emerges. Our key contributions include: • A temporally-controlled benchmark with 500 tasks across four AI domains and four decision families, paired with cutoff-aligned offline knowledge bases and pre-cutoff backbones; the same pipeline supports fully prospective forecasting beyond retrospective evaluation • A multi-signal evaluation protocol separating factuality, future-target alignment, evidence traceability, and reviewer persuasiveness, validated against human experts. • A systematic evaluation and diagnostic audit of LLM research agents showing that agent-style methods improve traceability and factuality task-conditionally, and identifying a previously-unstudied failure mode—evidence-decision decoupling. 2 Related Work 2.1 Autonomous Research Agents AI-for-science systems have moved from local literature QA toward agentic workflows that retrieve, synthesize, ideate, and execute parts of the research loop 26 ; 10 . PaperQA-style systems ( 21 ) , Chain-of-Ideas ( 23 ) , AI-Researcher ( 37 ) , AI Scientist ( 44 ) , Intern-Atlas 43 and recent agentic AI-for-science workflows ( 12 ) illustrate this shift toward autonomous research assistance. These systems typically emphasize the mechanics of retrieval, synthesis, tool selection, or experiment execution. In practical research use, however, the same systems are also asked to decide which evidence matters and where the field is moving. As these agents are increasingly deployed for ideation and planning, they are implicitly asked to make research decisions. Yet whether they can do so from evidence available at a specific historical moment remains an open question. ForeSci targets this decision layer rather than paper search or summary. 2.2 Benchmarks for Autonomous Research Existing benchmarks for autonomous research mainly focus on scientific reasoning 27 ; 4 ; 3 ; 25 ; 16 , artifacts—literature-grounded question answering ( 40 ; 21 ) , machine-learning research workflows ( 5 ; 28 ) , and paper-based agent arenas ( 41 ) . These benchmarks measure retrieval, tool use, synthesis, or execution. Compared to them, ForeSci instead asks systems to make prospective research decisions rather than recover accessible answers or execute known workflows. A few recent works begin to evaluate higher-order research capabilities beyond idea generation or workflow execution. Some focus on the novelty 35 ; 34 , taste 39 , impact 17 ; 51 , and future alignment 42 of agent-generated ideas. PreScience ( 2 ) moves further by predicting the components of future papers. These benchmarks connect generated research artifacts with later evidence, but their targets are often artifact-level properties such as idea quality, paper components, or citation-related outcomes. ForeSci instead treats the model output as a decision object: a ranked plan, a bottleneck diagnosis, a forecasted direction, or a venue recommendation. Although these works leverage future papers or citation signals as evaluation references, ForeSci focuses on