LLM2 : Let Large Language Models Harness System 2 Reasoning
Cheng Yang Chufan Shi Siheng Li Bo Shui Yujiu Yang Wai Lam · 2024
Read LLM2 : Let Large Language Models Harness System 2 Reasoning by Cheng Yang Chufan Shi Siheng Li Bo Shui Yujiu Yang Wai Lam (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-20372.txt.
Description
Abstract Large language models (LLMs) have exhibited impressive capabilities across a myriad of tasks, yet they occasionally yield undesirable outputs. We posit that these limitations are rooted in the foundational autoregressive architecture of LLMs, which inherently lacks mechanisms for differentiating between desirable and undesirable results.
Catalog record
| CREATOR | Cheng Yang Chufan Shi Siheng Li Bo Shui Yujiu Yang Wai Lam |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 28 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-20372
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
LLM2 : Let Large Language Models Harness System 2 Reasoning Thanks: The work described in this paper is partially supported by a grant from the Research Grant Council of the Hong Kong Special Administrative Region, China (Project Code: 14200620). Cheng Yang Chufan Shi Siheng Li Bo Shui Yujiu Yang Wai Lam Abstract Large language models (LLMs) have exhibited impressive capabilities across a myriad of tasks, yet they occasionally yield undesirable outputs. We posit that these limitations are rooted in the foundational autoregressive architecture of LLMs, which inherently lacks mechanisms for differentiating between desirable and undesirable results. Drawing inspiration from the dual-process theory of human cognition, we introduce LLM2, a novel framework that combines an LLM (System 1) with a process-based verifier (System 2). Within LLM2, the LLM is responsible for generating plausible candidates, while the verifier provides timely process-based feedback to distinguish desirable and undesirable outputs. The verifier is trained with a pairwise comparison loss on synthetic process-supervision data generated through our token quality exploration strategy. Empirical results on mathematical reasoning benchmarks substantiate the efficacy of LLM2, exemplified by an accuracy enhancement from 50.3 to 57.8 (+7.5) for Llama3-1B on GSM8K. Furthermore, when combined with self-consistency, LLM2 achieves additional improvements, boosting major@20 accuracy from 56.2 to 70.2 (+14.0) 1 1 1 Code is available at https://github.com/yc1999/LLM2 . . 1 Introduction Figure 1: An illustration of the training and inference stages of LLM2. The training stage includes (a) synthetic process-supervision data collection and (b) the optimization of a process-based verifier. The inference stage involves (c) a dual-process LLM for generation. Large language models ( Brown et al. 2020 ; Chowdhery et al. 2023 ; OpenAI 2023 ) have exhibited remarkable abilities across various tasks that span general assistance ( OpenAI 2022 ) , coding ( Chen et al. 2021 ) , vision ( Alayrac et al. 2022 ) and more. However, they still occasionally produce undesirable outputs in many scenarios, e.g., reasoning and planning ( Mialon et al. 2023 ; Hu and Shu 2023 ) , factual consistency ( Min et al. 2023 ) , and human value alignment ( Bai et al. 2022 ) , etc. We hypothesize these deficiencies stem from the fundamental design of LLMs. Specifically, the next-token prediction objective optimizes LLMs to maximize the probability of human-generated strings empirically, with no explicit mechanism to distinguish between desirable and undesirable outputs. During the inference stage, LLMs autoregressively generate outputs token-by-token in a single pass, with no awareness of their errors. This procedure is reminiscent of System 1 in the dual-process theory, which postulates that thinking and reasoning are underpinned by two distinct cognitive systems Stanovich and West 2000 ; Evans 2003 ; Kahneman 2011 . System 1 operates automatically and subconsciously, guided by instinct and experience. In contrast, System 2, thought to be unique to humans, is more controlled and rational, enabling deliberate thinking for difficult tasks, especially when System 1 may make mistakes ( Sloman 1996 ) . In this paper, we introduce LLM2 , which aims to empower LLMs with System 2 reasoning. As shown in Figure 1 , LLM2 integrates an LLM (System 1) with a process-based verifier (System 2). During inference, the LLM generates multiple candidates at each time step, and the verifier provides timely feedback on each candidate. By efficiently exploring the generation space based on the verifier’s feedback, LLM2 ultimately identifies more effective outputs. During the training stage, the process-based verifier is optimized with a pairwise comparison loss to distinguish between desirable and undesirable tokens. To obtain informative token pairs data for process-supervision, we propose a token quality exploration strategy that generates synthetic data based on the potential impact of tokens on the generated text. We evaluate LLM2 on two representative mathematical reasoning datasets: GSM8K Cobbe et al. 2021 and MATH Hendrycks et al. 2021 . With the integration of System 2 reasoning, LLM2 achieves substantial performance improvement across Llama3 models ranging from 1B to 8B parameters. For instance, compared to the vanilla Llama3-1B, LLM2 significantly improves accuracy from 50.3 to 57.8 (+7.5) on GSM8K, and from 24.2 to 28.8 (+4.6) on MATH. Combining LLM2 with self-consistency further boosts the model’s performance, enhancing major@20 accuracy from 56.2 to 70.2 (+14.0) on GSM8K. Further analysis of the utilization of self-generated answers underscores the effectiveness and promising potential of synthetic process-supervision data. 2 Method 2.1 Dual-process LLM We aim to build a dual-process LLM (i.e., LLM2), where an LLM serves as System 1 for giving plausible proposals and a verifier functions as System 2 for deliberate thinking to refine and prevent mistakes introduced by System 1. Specifically, we formalize this procedure as: \log\pi^{*}(x_{t}|x_{<t})\propto\log\pi(x_{t}|x_{<t})+\beta s(x_{<t},x_{t}), (1) where \pi and \pi^{*} represent the policies of the LLM and dual-process LLM, respectively. The verifier steers \pi during decoding based on the process score s(x_{<t},x_{t}) , with \beta controlling the strength. For computational efficiency, we focus verification on the most probable tokens at each time step. Therefore, we filter out low probability tokens using an adaptive plausibility constraint ( Li et al. 2022 ) : \mathcal{V}_{t}=\{v\in\mathcal{V}:\mathbf{z}_{t}[v]\geq\log\alpha+\max_{w}\mathbf{z}_{t}[w]\}, (2) where \mathbf{z}_{t} represents the logits of \pi , \mathcal{V} is the vocabulary and \mathcal{V}_{t}\subset\mathcal{V} denotes the token set filtered with the hyperparameter \alpha\in[0,1] at time step t . Therefore, the logits of \pi^{*} at time step t , denoted as \mathbf{z}^{*}_{t} , are computed as: \mathbf{z}^{*}_{t}[v]=\begin{cases}\mathbf{z}_{t}[v]+\beta s(x_{<t},v)&\text{if }v\in\mathcal{V}_{t},\\ -\infty&\text{otherwise}.\end{cases} (3) The probability distribution \pi^{*}(x_{t}|x_{<t})=\text{softmax}(\mathbf{z}^{*}_{t}) . This formulation allows \pi^{*} to integrates seamlessly with various decoding strategies, depending on the use case. Figure 2: Results of LLM2 and other baselines’ performance on GSM8K and MATH with Llama3 series. 2.2 Process-based Verifier We initialize the verifier from an LLM, replacing the unembedding head with a linear head to produce scalar scores. Given a dataset \mathcal{D}=\big\{x^{i}\big\}_{i=1}^{N} , we synthesize process-supervision \mathcal{D}_{p}(x)=\big\{x_{<t},x_{t}^{+},x_{t}^{-}\big\}_{t=1}^{T} for each instance x , where x_{t}^{+} is more appropriate than x_{t}^{-} . Accordingly, the training dataset for the verifier is \mathcal{D}_{s}=\big\{x^{i},\mathcal{D}_{p}(x^{i})\big\}_{i=1}^{N} . We train the verifier with a pairwise comparison loss ( Ouyang et al. 2022 ) : \displaystyle\mathcal{L} \displaystyle(s_{\theta},\mathcal{D}_{s})=-\mathbb{E}_{\big(x,\mathcal{D}_{p}(x)\big)\sim D_{s}} \displaystyle\sum_{t=1}^{T}\left[\log\sigma\big(s_{\theta}(x_{<t},x_{t}^{+})-s_{\theta}(x_{<t},x_{t}^{-})\big)\right]. (4) 2.3 Synthetic Process-supervision We aim to create \mathcal{D}_{p}(x)=\big\{x_{<t},x_{t}^{+},x_{t}^{-}\big\}_{t=1}^{T} for each instance x . In particular, we use the ground-truth token x_{t} as x_{t}^{+} , which is desirable to be correct. Regarding x_{t}^{-} , our goal is to select tokens that express the undesirable failure modes of LLMs, e.g., reasoning errors, hallucinations and misalignment with human values. Then, through learning to distinguish between x_{t}^{+} and x_{t}^{-} , the verifier can discern desirable and undesirable behaviors. To create x_{t}^{-} , one can sample tokens from the distributions predicted by LLMs. However, LLMs may assign a high probability to alternative correct tokens, which leads to false x_{t}^{-} and confuses the training of the verifier. To alleviate this issue, we introduce a token quality exploration strategy for sampling x_{t}^{-} . Specifically, the token quality exploration strategy evaluates the quality of individual tokens based on their potential impact on the generated text. This strategy involves three key steps: Continuation Generation For each candidate token v\in\mathcal{V}\setminus\{x_{t}^{+}\} at time step t , we use the LLM to generate N continuations \{c_{j}\}_{j=1}^{N} , each starting with x_{<t} concatenated with v . Quality Assessment We evaluate the quality of each continuation based on the correctness of all decoded