Chain-of-Thought in Large Language Models: Decoding, Projection, and Activation

Hao Yang, Qinghua Zhao, Lei Li · 2024

Read Chain-of-Thought in Large Language Models: Decoding, Projection, and Activation by Hao Yang, Qinghua Zhao, Lei Li (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-03944.txt.

Introduction

Hao Yang, Qinghua Zhao, and Lei Li looked inside chain-of-thought prompting in 2024, the technique where a language model reasons step by step before answering. It plainly works, and many studies tune its surface features, but its internal mechanics were murky. The authors probe three stages of the process: how tokens are decoded, how internal representations are projected, and which neurons activate, comparing chain-of-thought against plain prompting throughout.

Three findings stand out. The model imitates the format of the examples it was shown while genuinely integrating the question's content, rather than merely copying. During generation its token scores fluctuate more, yet the final distribution over next tokens ends up more concentrated, as if the reasoning commits it. And in the last layers, chain-of-thought switches on a broader set of neurons than standard prompts, consistent with wider knowledge being retrieved for the answer.

Catalog record

CREATORHao Yang, Qinghua Zhao, Lei Li
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE52 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-03944

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Chain-of-Thought in Large Language Models: Decoding, Projection, and Activation Hao Yang, Qinghua Zhao, Lei Li Abstract Chain-of-Thought prompting has significantly enhanced the reasoning capabilities of large language models, with numerous studies exploring factors influencing its performance. However, the underlying mechanisms remain poorly understood. To further demystify the operational principles, this work examines three key aspects: decoding, projection, and activation, aiming to elucidate the changes that occur within models when employing Chain-of-Thought. Our findings reveal that LLMs effectively imitate exemplar formats while integrating them with their understanding of the question, exhibiting fluctuations in token logits during generation but ultimately producing a more concentrated logits distribution, and activating a broader set of neurons in the final layers, indicating more extensive knowledge retrieval compared to standard prompts. Our code and data will be publicly avialable when the paper is accepted. 1 Introduction Large language models (LLMs) have demonstrated remarkable performance across various tasks, yet their efficacy remains suboptimal in certain reasoning tasks ( Wei et al. 2022 ) . This underscores the urgent need to understand and optimize LLMs in complex reasoning scenarios. The Chain-of-Thought (CoT) prompting has emerged as a promising solution, significantly enhancing LLMs performance in these challenging tasks ( Wei et al. 2022 ; Zhang et al. 2024 ) . Recent research has extended CoT to more complex structures such as trees and graphs ( Yao et al. 2024 ; Besta et al. 2024 ) , while other studies have explored various factors affecting CoT performance, including the number of reasoning steps, difficulty, length, and correctness of reasoning processes ( Webson and Pavlick 2021 ; Chia et al. 2023 ; Wang et al. 2023 ; Schaeffer et al. 2023 ; Jin et al. 2024 ; Prabhakar et al. 2024 ) . Several hypotheses are proposed regarding the underlying principles of CoT’s functionality. Schaeffer et al. 2023 suggest that CoT enhances the reasoning capabilities of LLMs, while Madaan et al. 2023 argue that CoT reduces task difficulty, making tasks more comprehensible to models. Saparov and He 2023 propose that models mimic its format, filling in answers based on the provided exemplar. Merrill and Sabharwal 2024 offer a theoretical analysis of how CoT improves the computational power of models. Although these hypotheses await further experimental validation, they provide valuable insights for exploring the fundamental mechanisms of CoT. Despite these advancements, the specific changes that occur both internally and externally within the model when employing CoT remain unclear. This paper aims to address this knowledge gap by examining CoT mechanisms from three perspectives: the decoding phase, the projection phase, and neuron activation. We seek to answer the following questions: 1. Does the large model merely mimic the patterns found in CoT exemplars? 2. What changes occur in the model’s projection space (e.g., logits) when using CoT prompt as opposed to the standard prompt? 3. Does CoT enable models’ deeper and broader utilization of knowledge acquired during the pre-training phase? To answer these questions, we employ a multi-faceted approach: • In the decoding phase, we analyze CoT-generated text, focusing on four test points: time, action, location, and number. We also conduct a transfer test to evaluate CoT’s performance across different datasets. • In the projection phase, we examine horizontal and vertical changes in projected logits, analyzing logits values and probability distributions at each time step. • We compare the range, intensity, and dynamic evolution of neuron activations under standard and CoT-style prompts. Our experiments yield insightful results that contribute to a deeper understanding of CoT mechanisms in large language models. 2 Related Work Recent years have witnessed remarkable progress in LLMs, with models such as GPT-4 ( Achiam et al. 2023 ) and LLaMA ( Meta 2024 ) demonstrating exceptional capabilities across a wide range of tasks. The introduction of CoT has further elevated their performance, particularly in complex reasoning tasks including arithmetic, commonsense, and symbolic reasoning ( Wei et al. 2022 ) . The success of CoT has spurred a wave of research aimed at extending and refining this approach. Notable contributions include the development of self-consistency ( Wang et al. ) , Boosting of Thoughts ( Chen et al. ) , and least-to-most prompting ( Zhou et al. ) . These methods have been complemented by zero-shot CoT ( Kojima et al. 2022a ) , which eliminates the need for task-specific examples, and more specialized techniques such as Program-of-Thought ( Bi et al. 2024 ) , Mixture-of-Thoughts ( Li and Qiu 2023 ) , and Contrastive Chain-of-Thought ( Zhang et al. 2024 ) . The versatility of CoT has been further demonstrated through its application in enhancing performance across diverse tasks ( Tanneru et al. 2024 ) . Research on CoT has uncovered key factors influencing its performance. Studies show that maintaining a constant number of reasoning steps, even with errors, does not hinder performance as long as logical coherence is maintained ( Jin et al. 2024 ; Chia et al. 2023 ; Wang et al. 2023 ; Schaeffer et al. 2023 ; Webson and Pavlick 2021 ; Prabhakar et al. 2024 ) . While intermediate steps in CoT do not directly improve problem-solving, they enhance task comprehension ( Madaan et al. 2023 ) . CoT also stabilizes significance scores ( Wu et al. 2023 ) , and diverse examples complement each other to improve reasoning ( Ye et al. 2023 ) . Understanding why CoT is effective remains limited, but recent research has proposed several hypotheses based on experimental findings. Madaan et al. 2023 use counterfactual methods to modify exemplar content, observing that while intermediate steps in CoT may not enhance task-solving abilities, they improve task comprehension. Saparov and He 2023 ; Chia et al. 2023 ; Wang et al. 2023 ; Wang and Zhou 2024 suggest reasoning abilities are acquired during pre-training, with exemplars guiding response generation. Madaan and Yazdanbakhsh 2022 posit that intermediate steps act as templates for the model to fill in answers rather than aiding task-solving. Nowak et al. 2024 ; Merrill and Sabharwal 2024 argue CoT boosts computational power. Gudibande et al. 2023 find that while imitation improves style and adherence to instructions, it does not enhance factuality, coding, or problem-solving. Rai and Yao 2024 use GPT-4 to study neuronal activation patterns, explaining the importance of equations, textual explanations, and exemplar diversity, as well as why incorrect reasoning does not necessarily impair performance. Our paper contributes to the growing body of research investigating the underlying mechanisms of CoT prompting. Unlike existing work, we approach this question from three distinct perspectives: decoding, pre-decoding steps, and neuronal activation, specifically exploring whether the model mimics exemplars during decoding, how the model’s projection space changes with CoT, and whether CoT facilitates deeper knowledge exploration, aiming to provide a more comprehensive understanding of how CoT enhances the reasoning capabilities of language models. 3 Methodology 3.1 Evaluation Tasks Following the methodologies outlined by Chia et al. 2023 ; Madaan et al. 2023 ; Jin et al. 2024 , we conducted experiments on reasoning tasks, specifically focusing on arithmetic reasoning, commonsense reasoning, and symbolic reasoning. (a) AQuA (b) GSM8K (c) SVAMP (d) Bamboogle (e) StrategyQA (f) Date (g) Sports (h) Coin Flip (i) Last Letter Figure 1: Statistical analysis of test points matches

Read the complete text