ScholarCopilot: Training Large Language Models for Academic Writing with Accurate Citations

Yubo Wang, Xueguang Ma, Ping Nie, Huaye Zeng, Zhiheng Lyu, Yuxuan Zhang, Benjamin Schneider, Yi Lu, Xiang Yue, Wenhu Chen · 2025

Read ScholarCopilot: Training Large Language Models for Academic Writing with Accurate Citations by Yubo Wang, Xueguang Ma, Ping Nie, Huaye Zeng, Zhiheng Lyu, Yuxuan Zhang, Benjamin Schneider, Yi Lu, Xiang Yue, Wenhu Chen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-00824.txt.

Introduction

Yubo Wang and co-authors build a tool to help write research papers with sound sources in 2025. Good papers need clear text and exact links to past work. Many writing tools add sources at the end, which can lead to wrong or weak links. Their tool, called ScholarCopilot, decides while it writes when it needs a source.

At that point the tool searches a store of past papers and feeds the best matches back in. It was trained on a large set of papers from an open archive. Tests show it finds the right source far more often than older search tools. Human judges also rated its drafts and its source choices above a well known chat tool.

Catalog record

CREATORYubo Wang, Xueguang Ma, Ping Nie, Huaye Zeng, Zhiheng Lyu, Yuxuan Zhang, Benjamin Schneider, Yi Lu, Xiang Yue, Wenhu Chen
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE49 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-00824

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ScholarCopilot: Training Large Language Models for Academic Writing with Accurate Citations Yubo Wang, Xueguang Ma, Ping Nie, Huaye Zeng, Zhiheng Lyu, Yuxuan Zhang, Benjamin Schneider, Yi Lu, Xiang Yue, Wenhu Chen Abstract Academic writing requires both coherent text generation and precise citation of relevant literature. Although recent Retrieval-Augmented Generation (RAG) systems have significantly improved factual accuracy in general-purpose text generation, their ability to support professional academic writing remains limited. In this work, we introduce ScholarCopilot, a unified framework designed to enhance existing large language models for generating professional academic articles with accurate and contextually relevant citations. ScholarCopilot dynamically determines when to retrieve scholarly references by generating a retrieval token [RET], which is then used to query a citation database. The retrieved references are fed into the model to augment the generation process. We jointly optimize both the generation and citation tasks within a single framework to improve efficiency. Our model is built upon Qwen-2.5-7B and trained on 500K papers from arXiv. It achieves a top-1 retrieval accuracy of 40.1% on our evaluation dataset, outperforming baselines such as E5-Mistral-7B-Instruct (15.0%) and BM25 (9.8%). On a dataset of 1,000 academic writing samples, ScholarCopilot scores 16.2/25 in generation quality—measured across relevance, coherence, academic rigor, completeness, and innovation—significantly surpassing all existing models, including much larger ones like the Retrieval-Augmented Qwen2.5-72B-Instruct. Human studies further demonstrate that ScholarCopilot, despite being a 7B model, significantly outperforms ChatGPT, achieving 100% preference in citation quality and over 70% in overall usefulness. 0 0 footnotetext: \dagger Core Contributors 1 1 footnotetext: \dagger Project website: https://tiger-ai-lab.github.io/ScholarCopilot/ Figure 1 : Comparison of traditional Retrieval-Augmented Generation (RAG) systems and our proposed ScholarCopilot . Traditional RAG systems (left) separately perform retrieval and generation, leading to representation misalignment. In contrast, ScholarCopilot (right) dynamically generates retrieval tokens ( [RET] ) during text generation for integrated and context-aware reference retrieval. 1 Introduction Academic writing is a knowledge-intensive task that requires both structured content generation and accurate citation of relevant literature. While large language models (LLMs) such as GPT-4 ( Achiam et al., 2023 ) , Deepseek-v3 ( Liu et al., 2024 ) , and Qwen2.5 ( Yang et al., 2024 ) can generate fluent academic-style text, they frequently hallucinate citations, undermining their reliability for research writing ( Huang et al., 2025 ; Tonmoy et al., 2024 ) . Recent advanced retrieval-augmented generation (RAG) ( Lewis et al., 2020 ; Shi et al., 2023 ) systems address this issue by retrieving relevant references from external knowledge bases to enhance factual consistency and reduce hallucinations. These approaches typically follow a first-retrieve-then-generate pipeline, as illustrated in Figure 2 (left), where retrieval is conducted independently prior to generation. However, this pipeline neglects the evolving generation context, making it difficult to dynamically adjust retrieval decisions based on the changing information needs during writing. For instance, when generating an introduction mentioning GPT-4, traditional approaches cannot adaptively retrieve GPT-4-related references precisely when needed, since retrieval decisions are predetermined without awareness of the specific generation context. Consequently, these methods suffer from three key limitations: (1) separate optimization of retrieval and generation models leads to misalignment in query intent; (2) predetermined retrieval decisions lack flexibility and context-awareness; (3) static pipeline limits user control over the generation of content and citation needs. To overcome these limitations, we propose ScholarCopilot , an agentic RAG framework tailored for assisting academic paper writing that seamlessly integrates text generation and citation retrieval in a unified, iterative manner, as illustrated in Figure 2 (right). Instead of relying on separate retrieval and generation stages, ScholarCopilot dynamically determines when retrieval is necessary by generating special retrieval tokens ( [RET] ) based on the evolving generation context. Upon generating these tokens, ScholarCopilot pauses the generation process, retrieves relevant scholarly references, and integrates their content (abstracts or key excepts) directly back into subsequent generation steps. The dense representations of these retrieval tokens are optimized via contrastive learning, enabling efficient similarity search. Additionally, ScholarCopilot allows optional user refinement and citation triggering during the iterative process, providing flexibility to integrate human domain expertise for further improving generation quality. This unified, iterative approach enhances citation accuracy, improves content coherence, and maintains efficiency without additional overhead. We evaluate ScholarCopilot extensively on academic writing tasks, focusing on generation quality, retrieval accuracy, and overall user experience . Our model achieves 40.1% top-1 retrieval accuracy , significantly surpassing baselines such as E5-Mistral-7B-Instruct (15.0%) ( Wang et al., 2023 ) and BM25 (9.8%) ( Robertson et al., 2009 ) , with consistent performance gains across all Top-K thresholds. In terms of generation quality, ScholarCopilot scores 16.2/25 on a 1000 samples dataset with LLM-as-judge across five dimensions (relevance, coherence, academic rigor, completeness, and innovation), substantially outperforming larger models such as Qwen-2.5-7B-Instruct (13.9) and Qwen-2.5-72B-Instruct (15.8) . A comprehensive user study with 10 experienced academic writers further confirms ScholarCopilot’s effectiveness, particularly highlighting its citation accuracy (100% positive ratings) and overall usefulness (70% positive ratings) compared to ChatGPT. Our main contributions are summarized as follows: • A unified generation-retrieval model that effectively integrates retrieval into the generative process, enabling seamless citation retrieval while reducing inference overhead and improving citation accuracy and relevance. • A comprehensive evaluation framework that assesses both retrieval accuracy and academic text quality along five critical dimensions: content relevance, logical coherence, academic rigor, information completeness, and scholarly innovation. • A large-scale training dataset consisting of 500k computer science papers from arXiv with comprehensive citation networks (33 matched citations per paper on average), facilitating robust learning of academic writing patterns and citation-aware scholarly practices. Figure 2 : Comparison between traditional Retrieval-Augmented Generation (RAG) methods (left) and ScholarCopilot (right). Traditional RAG follows a static retrieval-then-generation pipeline, retrieving references independently before generation. ScholarCopilot dynamically interleaves retrieval and generation by producing retrieval tokens ( [RET] ) based on current context, enabling context-aware citation retrieval and optional user refinement. 2 Related Work 2.1 Dense Retrieval Recent studies have demonstrated that dense retrieval methods using pretrained language models to encode text into dense vectors outperform traditional lexical retrievers like TF-IDF and BM25. Following the introduction of DPR ( Karpukhin et al., 2020 ) , several approaches have been proposed to improve dense retrieval through advanced training strategies (e.g., ANCE ( Xiong et al., 2021 ) , Condenser ( Gao & Callan, 2021 ) ), data augmentation techniques (e.g., BGE ( Xiao et al., 2023 ) , GTE ( Li et al., 2023 ) , DRAMA ( Ma et al., 2025 ) ), and by leveraging large language models as backbones (e.g., LLM2Vec ( BehnamGhader et al., 2024 ) , RepLlama ( Ma et al., 2024 ) , Mistral-E5 ( Wang et al., 2023 ) ). Today, commercial embedding models (e.g., OpenAI ( Neelakantan et al., 2022 ) , GeminiEmbed ( Lee et al., 2025 ) ) are widely used in real-world retrieval systems. However, most existing methods are designed for single-turn retrieval with short queries for retrieval, which is not well suited for citation suggestions where paper contexts are used to retrieve the next relevant citation. 2.2 Retrieval Augmented Generation In the era of LLM, the Retrieval Augmented Generation (RAG) paradigm integrates a retrieval model for the generation model, allowing the generation model to have access to external knowledge, improving the generation’s correctness and factuality for downstream tasks, such as

Read the complete text