Teaching LLMs to Refine with Tools

Dian Yu, Yuheng Zhang, Jiahao Xu, Tian Liang, Linfeng Song · 2024

Read Teaching LLMs to Refine with Tools by Dian Yu, Yuheng Zhang, Jiahao Xu, Tian Liang, Linfeng Song (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-16871.txt.

Introduction

Dian Yu and seven co-authors present CaP, a way to teach large language models to fix their own chain of thought replies with help from outside tools. Models can already polish replies from feedback, but most fixes stay in the same thought format. Their work came out in twenty twenty-four.

Sticking to one format can leave errors in place. CaP uses tools to rework thought chains made by the same model or by others. Training runs in two stages, first guided fine tuning, then preference tuning with variants of a known method. The authors stress that the second stage is key. They also weigh sampling plans for mixing thought and tools at run time.

Catalog record

CREATORDian Yu, Yuheng Zhang, Jiahao Xu, Tian Liang, Linfeng Song
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-16871

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Teaching LLMs to Refine with Tools Dian Yu, Yuheng Zhang, Jiahao Xu, Tian Liang, Linfeng Song Abstract Large language models (LLMs) can refine their responses based on feedback, enabling self-improvement through iterative training or test-time refinement. However, existing methods predominantly focus on refinement within the same reasoning format, which may lead to non-correcting behaviors. We propose CaP, a novel approach that uses external tools to refine chain-of-thought (CoT) responses generated by the same or other LLMs. CaP employs a two-stage training process: supervised fine-tuning followed by preference optimization with DPO variants. Our observations highlight the critical role of preference optimization in enabling effective refinement. Additionally, we compare several sampling strategies to leverage CoT and tools at inference time. Experimental results demonstrate CaP’s potential for effective cross-reasoning refinement and efficient inference. 1 Introduction Figure 1 : Overview of refining CoT solutions with PoT solutions during alignment and inference. It has been shown that large language models (LLMs) can exhibit certain capabilities of refining their responses based on feedback ( Saunders et al., 2022 ; Madaan et al., 2024 ; Qu et al., 2024 ; Kumar et al., 2024 ) , thereby enabling self-improvement of LLMs via iterative training or at test-time. Recent studies show that simply prompting LLMs can hardly achieve effective refinement even when the reference answer is provided along with the previous attempt, and thus further alignment such as supervised fine-tuning and preference optimization is still essential ( Qu et al., 2024 ; Kumar et al., 2024 ) . Most studies focus on response-level refinement, likely because obtaining large-scale, high-quality process-level supervision at a low annotation or computation cost ( Lightman et al., 2023 ; Wang et al., 2023b ) remains an open research question. A key step is constructing response pairs, consisting of a previous attempt and its refined version. To scale this process without incurring high annotation costs, previous studies often sample responses from LLMs in a multi-turn fashion, followed by answer verification and filtering: reference answers and a prior failed attempt are provided to guide the generation of a successful attempt ( Zelikman et al., 2022 ) . Alternatively, correct and incorrect responses can be paired to simulate the multi-turn process ( Welleck et al., 2022 ; Snell et al., 2024 ) . However, these studies focus on scenarios where responses follow the same reasoning format, such as step-by-step reasoning in natural language (CoT ( Wei et al., 2022 ) ) or solving problems through executable code (PoT ( Chen et al., 2022 ) ). Leveraging different types of reasoning ( Gou et al., 2024 ; Yue et al., 2024 ) has proven effective, particularly for improving LLMs’ performance in mathematical reasoning. Responses of different reasoning types for the same question naturally serve as alternative solutions, given the inherent differences between programming language and natural language. However, the problem of refining responses across distinct types of reasoning remains underexplored. As a preliminary investigation, we focus on the question: Can we teach LLMs to refine CoT responses using tools? We propose an approach called CaP , which leverages external tools to effectively refine CoT responses, regardless of whether CoT responses are generated by the same LLM, a stronger one, or a weaker one. In turn, CoT responses can also serve as context or draft information to facilitate the generation of PoT solutions. To develop CaP, we train LLMs through a two-stage process: supervised fine-tuning (SFT) followed by preference optimization using DPO ( Rafailov et al., 2024 ) as shown in Figure 1 . Our observations indicate that SFT alone is insufficient to instill effective refinement behavior; the subsequent preference optimization stage is essential for enabling this capability (left subfigure in Figure 2 ). When applying the same paradigm as CaP but using CoT for refinement, we observe negative results, aligning with prior studies ( Kumar et al., 2024 ; Ye et al., 2024 ) , which reveals that LLMs trained to refine CoT with CoT are likely to exhibit non-correcting behaviors. Figure 2 : Left: CaP performance using greedy decoding based on different sources of CoT responses. Right: Average accuracy of BoN and BoNBoN sampling strategies on out-of-distribution math tasks. Finally, we explore the benefits of CaP during inference. Compared to the PoT/CoT-only variants, our CaP model consistently achieves superior Best-of-N (BoN) performance within the same sample budget, and it further narrows the gap between one-sample and BoN performance (Figure 2 ), an essential step toward reducing the computational overhead during inference ( Sessa et al., 2024 ) . Recognizing the importance of CoT response correctness for refinement performance, we compare several sampling strategies and introduce a new sampling strategy called BoNBoN , which allocates half of the budget to CoT sampling and then uses the resulting BoN CoT sample as the initial attempt to generate N refined samples in PoT for BoN selection. The positive results suggest a promising direction for studying adaptive allocation of inference-time computation by leveraging diverse reasoning steps or solutions for efficient inference. 2 Method 2.1 Best-of-N Sampling from Teacher Models For complex tasks such as mathematical reasoning, evaluating the correctness of a response based solely on the generated rationale remains a significant challenge ( Daheim et al., 2024 ) . Therefore, to construct high-quality training data, we focus on a setting where a reference answer – usually concise and lacking any form of rationale – is provided to assist the evaluation. We apply a reference-based critic model, trained with next-token prediction to evaluate and score each CoT response y_{\text{cot}} or each PoT response y_{\text{pot}} and its corresponding execution result y_{\text{exec}} , given a question q and its ground truth reference answer y_{\text{ref}} . We use the aforementioned critic model to generate Yes or No critics and regard the likelihood that a solution is positive or negative as the confidence score for a certain critic (e.g., p_{\theta_{\text{ref}}}(\text{Yes}\mid q,y_{\text{pot}},y_{\text{exec}},y_{\text{ref}}) and p_{\theta_{\text{ref}}}(\text{No}\mid q,y_{\text{cot}},y_{\text{ref}}) ), inspired by previous studies on reference-free/reference-based outcome/process reward modeling ( Lightman et al., 2023 ; Zhang et al., 2024 ; Tian et al., 2024 ) . While more detailed critics that pointing out errors can undoubtedly enhance both evaluation and refinement, they heavily depend on human-annotated data ( McAleese et al., 2024 ) , which we leave for future work. Supposing that we have two teacher policy models \pi_{\text{cot}} and \pi_{\text{pot}} , we use Best-of-N sampling strategy to build SFT and preference optimization training data. We draw N_{1} CoT samples from \pi_{\text{cot}} and N_{2} PoT samples from \pi_{\text{pot}} . We use the highest-scoring responses as the winning and losing responses ( y_{\text{cot}}^{+} , y_{\text{cot}}^{-} , y_{\text{pot}}^{+} , y_{\text{pot}}^{-} ) in later stages, to reduce noise that may be introduced by an imperfect critic model. Note that a question may have only positive (+) or negative (-) responses, depending on the evaluations of the critic model. Instead of prompting LLMs in a multi-turn manner, we explore pairing CoT responses with positive PoT responses to simulate a multi-turn process. This is inspired by previous studies on CoT-CoT refinement ( Welleck et al., 2022 ; Snell et al., 2024 ) . Refining a CoT response using another CoT response, even when reference answers are available, remains an open challenge for LLMs ( Zelikman

Read the complete text