GUIDE: Towards Scalable Advising for Research Ideas
Yaowenqi Liu, BingXu Meng, Rui Pan, Jerry Huang, Tong ZhangUniversity of Illinois Urbana-Champaign {yl140, bingxum2, ruip4, jerry8, tozhang}@illinois.edu · 2025
Read GUIDE: Towards Scalable Advising for Research Ideas by Yaowenqi Liu, BingXu Meng, Rui Pan, Jerry Huang, Tong ZhangUniversity of Illinois Urbana-Champaign {yl140, bingxum2, ruip4, jerry8, tozhang}@illinois.edu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-08870.txt.
Introduction
Yaowenqi Liu and co-authors build GUIDE, a system for advising on research ideas at scale, in their 2025 arXiv paper. AI now generates hypotheses and designs experiments across many fields, but scalable, high-quality feedback to refine those ideas is scarce. The team explored what makes an advising system strong, from model size to reasoning structure.
A relatively small model, paired with a compressed literature database and a structured reasoning framework, beat a powerful general model on acceptance rates for its top-ranked ICLR submissions. When restricted to high-confidence calls, its acceptance rate on a test set topped ninety percent. The system could raise the quality of automated science.
Catalog record
| CREATOR | Yaowenqi Liu, BingXu Meng, Rui Pan, Jerry Huang, Tong ZhangUniversity of Illinois Urbana-Champaign {yl140, bingxum2, ruip4, jerry8, tozhang}@illinois.edu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 58 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-08870
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
GUIDE: Towards Scalable Advising for Research Ideas Yaowenqi Liu, BingXu Meng, Rui Pan, Jerry Huang, Tong ZhangUniversity of Illinois Urbana-Champaign {yl140, bingxum2, ruip4, jerry8, tozhang}@illinois.edu Abstract The field of AI research is advancing at an unprecedented pace, enabling automated hypothesis generation and experimental design across diverse domains such as biology, mathematics, and artificial intelligence. Despite these advancements, there remains a significant gap in the availability of scalable advising systems capable of providing high-quality, well-reasoned feedback to refine proposed hypotheses and experimental designs. To address this challenge, we explore key factors that underlie the development of robust advising systems, including model size, context length, confidence estimation, and structured reasoning processes. Our findings reveal that a relatively small model, when equipped with a well-compressed literature database and a structured reasoning framework, can outperform powerful general-purpose language models such as Deepseek-R1 in terms of acceptance rates for self-ranked top-30% submissions to ICLR 2025. Moreover, when limited to high-confidence predictions, our system achieves an acceptance rate exceeding 90% on the ICLR 2025 test set, underscoring its potential to significantly enhance the quality and efficiency of hypothesis generation and experimental design. The code is released at https://github.com/HowardLiu0830/GUIDE-Research-Idea-Evaluation . * * footnotetext: Equal Contribution. 1 Introduction Large Language Models (LLMs) have demonstrated remarkable progress in tasks ranging from text generation to code synthesis ( Achiam et al. 2023 ) . Recently, their application to academic research assistance —especially in providing feedback on scientific writing and research ideas—has garnered increasing attention. Systems such as Google’s Co-Scientist ( Gottweis et al. 2025 ) exemplify a broader shift toward agentic LLMs capable of collaborating with human researchers to improve scientific workflows, from hypothesis generation to peer review ( Jin et al. 2024 ; Tan et al. 2024 ) . This emerging capability holds significant promise for accelerating scientific discovery and democratizing research access. Among these agentic tasks, one particularly impactful yet underexplored area is the development of LLM-based advising agents , models designed to provide detailed, constructive, and hallucination-free suggestions for academic papers. The goal of these systems is to emulate human advisors by identifying strengths and weaknesses in submissions, suggesting actionable improvements, and assigning quantitative evaluations. However, existing LLMs often struggle with review fidelity: they may produce inflated ratings, fail to identify methodological flaws, or hallucinate evaluations not grounded in the text ( Ye et al. 2024 ; Yu et al. 2024 ; Yan 2024 ) . These limitations stem from a lack of fine-grained supervision, domain-specific alignment, and proper advising-style training data. System Retrieval-Augmented Modular Summarization Rubric-Guided Alignment MetaGen ( Bhatia et al. 2020 ) ✗ ✗ ✓ MReD ( Shen et al. 2021 ) ✗ ✗ ✗ ReviewRobot ( Wang et al. 2020 ) ✓ ✗ ✗ Reviewer2 ( Gao et al. 2024 ) ✗ ✗ ✗ CycleResearcher ( Weng et al. 2024 ) ✗ ✗ ✓ GUIDE ✓ ✓ ✓ Table 1: Comparison of LLM-based peer review systems. While some systems (e.g., CycleResearcher) are a part of broader end-to-end scientific agents, this comparison focuses specifically on their review capabilities. To address these challenges, we propose a novel and scalable framework for generating reliable, constructive, and expert-aligned suggestions. Our system is built upon a compressed knowledge base of paper summaries and metadata, distilled from full-text scientific papers in the field of machine learning, which enables efficient and accurate retrieval through a retrieval-augmented generation (RAG) pipeline. Before hypothesis verification, the system retrieves dozens of relevant papers to provide rich external context. Furthermore, to ensure that our models produce high-quality feedback, we introduce a rubric-guided alignment strategy that instructs LLMs to follow and apply evaluation criteria akin to those used in major natural language processing (NLP) conferences. However, even with clear rubrics and guidelines, LLMs still exhibit the tendency to produce overly favorable and superficial revision suggestions. To address this issue, Reward rAnked FineTuning (RAFT; Dong et al. 2023 ) is used to align an open-source LLM with expert review criteria and domain-specific literature. This alignment enables our model to generate detailed, rubric-grounded feedback, with a particular emphasis on methodological rigor and experimental soundness—areas often neglected by existing systems. The combination of aforementioned techniques gives rise to our advising system: G uidelines (Rubrics), U nderstanding (Summarized), I nformation Retrieval (RAG), D irection (Advising Improvement with RLHF), and E xplanation (LLM reasoning), or GUIDE in short. As shown in Table 1 , compared to previous systems, GUIDE is the only one to incorporate all three components of retrieval augmentation, modular summarization, and rubric-guided alignment, highlighting its unique design and broader capability compared to prior approaches. To evaluate GUIDE, we conduct a controlled experiment using the ICLR 2016–2024 paper submissions dataset, demonstrating the systematic improvements of our methods by predicting the acceptances of ICLR 2025 submissions. Specifically, we adopt the metrics of Top-5% Precision, Top-30% Precision, and Recall, which respectively evaluate the system’s ability to identify high-quality papers, acceptable papers, and retrieve good papers. Additionally, a customized reward model based on rank classification is employed to accelerate the system’s intermediate alignment. Empirical results show that our system fine-tuned on the Qwen-2.5-7B-Instruct backbone model, outperforms large general-purpose language models in terms of rating alignment with actual acceptance. Moreover, rubric-guided prompting focusing on novelty and significance reduces hallucinated content and leads to more grounded, constructive feedback. Our contributions are summarized as follows: • End-to-end hypothesis advisor : We introduce an LLM-based system GUIDE that provides actionable suggestions for both research ideas and experimental design . Our advising system, GUIDE-7B, outperforms large general-purpose LLMs such as GPT-4o-mini ( Achiam et al. 2023 ) and DeepSeek-R1 ( Guo et al. 2025 ) in terms of Top-30% precision—a metric that measures the acceptance rate of the top 30% of papers, as rated based on the suggested strengths and weaknesses. • Scalable advising with modular summarization : We show that summarizing different sections of the literature separately effectively mitigates the limited context-length issue in idea advising scenarios, allowing more relevant content to be retrieved and compared for advising. In particular, the abstract and methodology sections are shown to be the most important for evaluating the quality of a paper. • Rubric-guided alignment : We demonstrate that integrating rubric-based instruction significantly enhances the reviewer expertise of LLMs and improves evaluation usefulness. 2 Related Works Hypothesis Discovery in Scientific Research. Recent progress in large language models has enabled their integration into early-stage scientific workflows, particularly in hypothesis generation and ideation ( Zhou et al. 2022 ; Ruan et al. 2024 ; Singhal et al. 2025 ) . While these models have demonstrated promise in producing plausible hypotheses Yao et al. 2023 ; Tu et al. 2024 ; Ruan et al. 2024 , significantly less attention has been devoted to hypothesis verification , the task of evaluating whether hypotheses are substantiated, methodologically sound, and experimentally grounded ( Yang et al. 2022 ; Qiu et al. 2023 ) . Our work advances this understudied area by proposing a rubric-guided framework that evaluates scientific claims in a manner aligned with human peer reviewers. Jones 2025 ; Ifargan et al. 2025 ; Swanson et al. 2024 ; Saab et al. 2024 ; Taylor et al. 2022 have proposed retrieval-augmented generation (RAG; Lewis et al. 2020 ) techniques to improve LLMs’ access to