Think&Cite : Improving Attributed Text Generation with Self-Guided Tree Search and Progress Reward Modeling
Junyi Li, Hwee Tou Ng · 2024
Read Think&Cite : Improving Attributed Text Generation with Self-Guided Tree Search and Progress Reward Modeling by Junyi Li, Hwee Tou Ng (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-14860.txt.
Introduction
Junyi Li and Hwee Tou Ng improve cited answer generation in a 2024 paper. Language models often state falsehoods with confidence, so attributed generation makes them back claims with evidence. The authors treat the writing of a cited answer as a step-by-step reasoning problem joined with search.
Their framework, Think&Cite, searches over draft paths with a tree search that the model itself guides by reflecting on partial states. A progress reward model scores each state on two fronts: how far the text has come and how well it is supported. Across three data sets, the method beat prior approaches by clear margins. The authors argue that self-guided search plus honest progress scoring yields more trustworthy cited text.
Catalog record
| CREATOR | Junyi Li, Hwee Tou Ng |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 45 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-14860
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Think&Cite : Improving Attributed Text Generation with Self-Guided Tree Search and Progress Reward Modeling Junyi Li, Hwee Tou Ng Abstract Despite their outstanding capabilities, large language models (LLMs) are prone to hallucination and producing factually incorrect information. This challenge has spurred efforts in attributed text generation, which prompts LLMs to generate content with supporting evidence. In this paper, we propose a novel framework, called Think&Cite , and formulate attributed text generation as a multi-step reasoning problem integrated with search. Specifically, we propose Self-Guided Monte Carlo Tree Search (SG-MCTS), which capitalizes on the self-reflection capability of LLMs to reflect on the intermediate states of MCTS for guiding the tree expansion process. To provide reliable and comprehensive feedback, we introduce Progress Reward Models to measure the progress of tree search from the root to the current state from two aspects, i.e., generation and attribution progress. We conduct extensive experiments on three datasets and the results show that our approach significantly outperforms baseline approaches. 1 Introduction Large language models (LLMs) ( Zhao et al. 2023 ) have achieved outstanding performance on many natural language processing tasks. Despite the advances, LLMs often generate responses that contain hallucination and inaccurate information ( Ji et al. 2023 ; Huang et al. 2023 ; Zhang et al. 2023 ) . This issue undermines their reliability, and more importantly, hurts users’ trust in LLMs. To improve the reliability of LLMs, a new paradigm for generation, attributed text generation , is proposed, such that LLMs generate responses with in-text citations that provide evidence for any statement ( Gao et al. 2023b ) , as shown in Figure 1 . Most existing work ( Slobodkin et al. 2024 ; Sun et al. 2024 ; Fierro et al. 2024 ) simply prompts LLMs to provide citations while generating texts. Besides, other work ( Li et al. 2024 ; Huang et al. 2024 ) attempts to fine-tune LLMs on massive supervised training data that contains texts with annotated citations. Despite these recent efforts, it remains an open challenge to develop LLMs capable of learning to generate faithful content with reliable references. First, existing approaches adopt an auto-regressive generation paradigm that can be characterized as “System 1”, a mode of thinking which is fast and instinctive, but less accurate ( Kahneman 2011 ) . Thus, any intermediate generation errors (e.g., false statements or erroneous citations) can potentially lead to incorrect final responses. Inspired by research on complex reasoning ( Zhang et al. 2024 ; Wang et al. 2024 ) , we aim to develop models in the “System 2” mode for attribution to external evidence, requiring more in-depth, deliberative, and logical thinking ( Kahneman 2011 ) . Second, attributed text generation often involves long text generation. Liu et al. 2023 find that long-form responses from existing LLMs usually contain unsupported statements and inaccurate citations. We argue that the absence of explicit generation planning in previous work hinders advances in such systems. Figure 1: Given a question, the model generates texts by citing passages from a corpus as supporting evidence. In this paper, we propose Think&Cite , a novel framework integrating search algorithms into attributed text generation. We conceptualize the generation task as a multi-step reasoning problem, where the model generates a sentence in each step through an iterative think-verbalize-cite paradigm. To enhance this generation process, we propose Self-Guided Monte Carlo Tree Search (SG-MCTS), which extends the classic MCTS with two innovations. First, our approach leverages the self-reflection capability of LLMs to reflect on the intermediate states of MCTS in real time, so as to guide the tree expansion process and proactively avoid inadequate reasoning paths. This is different from prior work which mainly reflected on the final outcome or complete trajectory. Second, we propose Progress Reward Models (PRM) to measure the progress of tree search from the root to the current state from two aspects, i.e., generation progress and attribution progress. In contrast to only evaluating single steps, the progress-based reward model can provide reliable and comprehensive evaluation to guide the MCTS search process. To the best of our knowledge, we are the first to apply tree search algorithms to the task of attributed text generation. We conduct extensive experiments on three datasets to verify the effectiveness of our approach. The results show that our model significantly outperforms previous prompting-based and fine-tuning baselines. 2 Related Work Attributed Text Generation. Large language models (LLMs) have been used in attributed text generation due to their outstanding language generation capabilities ( Gao et al. 2023b ; Huang et al. 2024 ; Sun et al. 2024 ; Li et al. 2024 ; Slobodkin et al. 2024 ) . The work on LLMs for attributed text generation can be broadly categorized into two types. The first type involves fine-tuning LLMs with preference learning ( Li et al. 2024 ) and reinforcement learning ( Huang et al. 2024 ) , which teach LLMs to generate supportive and relevant citations to achieve higher rewards. However, this approach depends on human labor to curate high-quality datasets with annotated in-text citations. Another line of work directly instructs LLMs to generate attributed texts with appropriate prompts by attribute-then-generate planning ( Slobodkin et al. 2024 ) , or employing external verifiers to guide generation ( Sun et al. 2024 ) . However, this approach generates texts and citations in an auto-regressive manner, where any inaccurate intermediate generation can easily lead to failure in the subsequent process. In contrast, our approach proposes self-guided tree search with progressive reward to consider multiple paths. LLMs with Tree Search. Integrating tree search algorithms with LLMs has attracted significant attention. Recent studies have investigated the use of tree search methods to enhance the performance of LLMs during inference ( Zhang et al. 2024 ; Wang et al. 2024 ; Ye and Ng 2024 ) . Sutton 2019 highlights the superiority of scaling in both learning and search, over other approaches. Empirical evidence further demonstrates that scaling inference-time computation can significantly improve LLM performance without requiring additional training ( Brown et al. 2024 ; Snell et al. 2024 ) . A* search ( Hart et al. 1968 ) and Monte Carlo Tree Search (MCTS) ( Browne et al. 2012 ) are employed as planning techniques to improve the performance of LLMs in solving complex reasoning problems. These search algorithms have been widely applied in reinforcement learning ( Silver et al. 2017 ) and many real-world applications such as AlphaGo ( Silver et al. 2016 ) . Our work is the first to apply tree search algorithms (i.e., Monte Carlo Tree Search) to solve the task of attributed text generation. Moreover, we propose self-guided MCTS that relies on the reflection capability of LLMs to improve tree expansion. 3 Problem Formulation Figure 2: Overall framework of our proposed Think&Cite approach. Our proposed framework aims to have a pre-trained LLM \mathcal{M}_{\theta} generate responses with in-text citations that serve as evidence for the output content, referred to as attributed text generation ( Slobodkin et al. 2024 ; Gao et al. 2023a ) . Formally, given an input question \bm{x} and a corpus of text passages \mathcal{D} , the model \mathcal{M}_{\theta} is required to generate a response \bm{y}=\langle y_{1},...,y_{T}\rangle consisting of T sentences where each