Learning to Predict Future-Aligned Research Proposals with Language Models

Heng Wang, Pengcheng Jiang, Jiashuo Sun, Zhiyi Shi · 2026

Read Learning to Predict Future-Aligned Research Proposals with Language Models by Heng Wang, Pengcheng Jiang, Jiashuo Sun, Zhiyi Shi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-27146.txt.

Introduction

Wang and co-authors train language models to propose research that the future actually pursues. Judging proposal quality is hard because novelty resists automatic scoring. Their trick reframes generation as scientific forecasting with a time cutoff. Models see a question and papers before the cutoff, then draft a structured proposal. A Future Alignment Score checks the draft against papers published later. Training used 21,835 paper occurrences with synthesized reasoning traces. Future-aligned tuning raised the score by up to 10.6 percent. Two generated proposals were even implemented with gains.

Catalog record

CREATORHeng Wang, Pengcheng Jiang, Jiashuo Sun, Zhiyi Shi
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE87299
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.27146

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Learning to Predict Future-Aligned Research Proposals with Language Models Heng Wang, Pengcheng Jiang, Jiashuo Sun, Zhiyi Shi Abstract Large language models (LLMs) are increasingly used to assist ideation in research, but evaluating the quality of LLM-generated research proposals remains difficult: novelty and soundness are hard to measure automatically, and large-scale human evaluation is costly. We propose a verifiable alternative by reframing proposal generation as a time-sliced scientific forecasting problem. Given a research question and inspiring papers available before a cutoff time, the model generates a structured proposal and is evaluated by whether it anticipates research directions that appear in papers published after the time. We operationalize this objective with the Future Alignment Score (FAS), computed via retrieval and LLM-based semantic scoring against a held-out future corpus. To train models, we build a time-consistent dataset of 17,771 papers from targets and their pre-cutoff citations, and synthesize reasoning traces that teach gap identification and inspiration borrowing. Across Llama-3.1 and Qwen2.5 models, future-aligned tuning improves future alignment over unaligned baselines (up to +10.6% overall FAS), and domain-expert human evaluation corroborates improved proposal quality. Finally, we demonstrate practical impact by implementing two model-generated proposals with a code agent, obtaining 4.17% accuracy gain on MATH from a new prompting strategy and consistent improvements for a novel model-merging method. Figure 1: Given inspiring papers S and a research question q available before a cutoff time t_{C} , the model generates a proposal \tilde{P} . We evaluate whether the proposal anticipates future human research directions by comparing it against papers published after t_{C} using retrieval and LLM-based semantic alignment. 1 Introduction Large language models (LLMs) are increasingly used to assist researchers in idea exploration, literature synthesis, and research proposal drafting ( 30 ; 27 ) . Recent systems can identify potential research gaps ( 15 ) , estimate or generate experimental outcomes ( 34 ) , and even produce end-to-end papers and code ( 16 ; 40 ; 41 ) , raising the possibility that LLMs could meaningfully accelerate scientific discovery. However, it remains unclear how to train and compare such systems at scale because there is no reliable automated objective for assessing the quality of generated research proposals. This difficulty stems from the nature of proposal generation itself. Unlike question answering or summarization, research proposal generation has no single ground-truth output. Key attributes of a strong proposal—such as novelty, technical soundness, feasibility, and long-term impact—are subjective and often only verifiable retrospectively. As a result, automatic metrics, including LLM-based judges, are only weakly aligned with these properties, while human evaluation is costly, inconsistent, and difficult to scale ( 26 ) . Consequently, it remains unclear what learning objective should guide models toward generating genuinely useful proposals rather than merely fluent and internally consistent text. To address this challenge, we reframe proposal generation as a time-sliced scientific forecasting problem (Figure 1 ). Instead of directly optimizing subjective notions such as novelty or creativity, we ask a verifiable question: can a model generate proposals that anticipate research directions later pursued by human researchers? Concretely, given a research question q and inspiring papers S available before a cutoff time t_{C} , the model generates a structured proposal and is evaluated by its semantic alignment with papers published after t_{C} . If a proposal aligns strongly with future publications, it suggests that the model has captured research trajectories that later proved scientifically meaningful, rather than merely producing arbitrary creative text. We therefore treat future alignment as a scalable surrogate for proposal quality. We operationalize this idea with the Future Alignment Score (FAS) , defined as the maximum semantic alignment between a generated proposal and top retrieved papers from a held-out future corpus, computed via retrieval and LLM-based semantic scoring. By turning proposal generation into a forecasting task in semantic space, this formulation provides a scalable automatic evaluation signal grounded in publication timelines. To enable learning under this objective, we construct large-scale time-consistent supervision from published papers. For each target paper, we extract a leakage-controlled research question and a set of inspiring citations, then synthesize a structured proposal that simulates a plausible pre-publication research plan. Beyond final proposal text, the model must also learn how to transform prior work into a new research direction by identifying gaps in the literature and repurposing useful ideas, which is a necessary process in human research. To capture this process, we synthesize citation-grounded reasoning traces and introduce stepwise reasoning supervision that decomposes proposal generation into stages of problem formulation, method design, and experimental planning. Experiments show that future-aligned supervision consistently improves future alignment over baselines, and ablation studies demonstrate that stepwise reasoning supervision yields additional gains. We further validate this framework beyond FAS: domain-expert human evaluation shows that Stepwise CoT proposals are preferred over prompting baselines across soundness, excitement, and overall assessment, while remaining competitive with human-derived proposals. Finally, to test whether generated proposals translate into executable research, we implement two proposals produced by our best model in an LLM-assisted coding environment, obtaining measurable improvements compared to baselines. Our contributions can be summarized as: • We introduce a time-sliced formulation of research proposal generation that evaluates proposals by their semantic match to future published papers, providing a scalable automatic signal grounded in future publications. • We construct time-consistent supervision from published papers and their pre-cutoff citations, producing a dataset of 17,771 machine learning papers and synthesizing structured proposal targets and reasoning traces that capture gap analysis and inspiration borrowing. • Across multiple LLM families, supervised fine-tuning substantially improves future alignment, and domain-expert human evaluation corroborates improvements in perceived proposal quality; we further demonstrate practical impact by implementing two model-generated proposals. 2 Method Research proposal generation is difficult to evaluate and supervise at scale. We address this with a future-aligned learning framework (Figure 2 ) built on three key ideas: (1) future alignment as a verifiable surrogate for proposal quality, (2) time-consistent supervision synthesized from historical papers without future leakage, and (3) citation-grounded stepwise reasoning that decomposes proposal generation into staged scientific planning. 2.1 Future-Aligned Proposal Prediction We formulate research proposal generation as a time-sliced learning problem over scientific papers. Let \mathcal{C} denote a time-stamped corpus, where each paper p has publication time t_{p} . For a target paper Y published after cutoff time t_{C} , papers with t_{p}\leq t_{C} are treated as observable context, while papers with t_{p}>t_{C} form a held-out future corpus. For each target paper Y , we construct an input instance (q,S) , where q is a research question derived from Y and S is a set of inspiring papers selected from its references. Given (q,S) , the model generates a structured proposal \hat{P} : (q,S)\rightarrow\hat{P} . Rather than reconstructing Y , the goal is to generate a proposal that anticipates research directions later realized in the future literature. We represent \hat{P} using a structured schema consisting of Research Question , Hypothesis , Proposed Method , Novelty Claims , and Experimental Details , enabling both complete proposal generation and fine-grained evaluation. 2.2 Verifiable Learning Objective Directly optimizing subjective notions such as novelty or creativity is difficult to scale, so we use future alignment as a verifiable surrogate objective. Let \mathcal{C}_{\text{future}}=\{p\in\mathcal{C}\mid t_{p}>t_{C}\} denote the future corpus. Given a generated proposal \hat{P} , we first

Read the complete text