The Curse of CoT : On the Limitations of Chain-of-Thought in In-Context Learning

Tianshi Zheng, Yixiang Chen, Chengxi Li, Chunyang Li, Qing Zong, Haochen Shi, Baixuan Xu, Yangqiu Song, Ginny Y. Wong, Simon See · 2025

Read The Curse of CoT : On the Limitations of Chain-of-Thought in In-Context Learning by Tianshi Zheng, Yixiang Chen, Chengxi Li, Chunyang Li, Qing Zong, Haochen Shi, Baixuan Xu, Yangqiu Song, Ginny Y. Wong, Simon See (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-05081.txt.

Introduction

Tianshi Zheng and co-authors report a surprising failure of chain-of-thought prompting, the trick of making a model reason step by step, in a 2025 paper. Across sixteen top language models and nine pattern-based learning tasks, step-by-step reasoning consistently lost to answering directly. The gap held across model sizes and task difficulty. Even long-reasoning models, stars at abstract problems, could not shake it.

The team's analysis blames a hybrid mechanism. Explicit reasoning falters because models struggle to infer the hidden pattern from examples. Implicit reasoning, which often lands the right answer quietly, gets disrupted when long rationales push the examples farther away. Noise from the weak explicit channel then poisons the result. The authors say the field's faith in universal step-by-step prompting needs rethinking.

Catalog record

CREATORTianshi Zheng, Yixiang Chen, Chengxi Li, Chunyang Li, Qing Zong, Haochen Shi, Baixuan Xu, Yangqiu Song, Ginny Y. Wong, Simon See
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE63 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-05081

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

The Curse of CoT : On the Limitations of Chain-of-Thought in In-Context Learning Tianshi Zheng, Yixiang Chen, Chengxi Li, Chunyang Li, Qing Zong, Haochen Shi, Baixuan Xu, Yangqiu Song, Ginny Y. Wong, Simon See Abstract Chain-of-Thought (CoT) prompting has been widely recognized for its ability to enhance reasoning capabilities in large language models (LLMs) through the generation of explicit explanatory rationales. However, our study reveals a surprising contradiction to this prevailing perspective. Through extensive experiments involving 16 state-of-the-art LLMs and nine diverse pattern-based in-context learning (ICL) datasets, we demonstrate that CoT and its reasoning variants consistently underperform direct answering across varying model scales and benchmark complexities. To systematically investigate this unexpected phenomenon, we designed extensive experiments to validate several hypothetical explanations. Our analysis uncovers a fundamental explicit-implicit duality driving CoT’s performance in pattern-based ICL: while explicit reasoning falters due to LLMs’ struggles to infer underlying patterns from demonstrations, implicit reasoning—disrupted by the increased contextual distance of CoT rationales—often compensates, delivering correct answers despite flawed rationales. This duality explains CoT’s relative underperformance, as noise from weak explicit inference undermines the process, even as implicit mechanisms partially salvage outcomes. Notably, even long-CoT reasoning models, which excel in abstract and symbolic reasoning, fail to fully overcome these limitations despite higher computational costs. Our findings challenge existing assumptions regarding the universal efficacy of CoT, yielding novel insights into its limitations and guiding future research toward more nuanced and effective reasoning methodologies for LLMs. Figure 1: (a) Performance comparison across nine pattern-based ICL benchmarks, averaged over 16 LLMs. (b) Performance gaps with varying numbers of demonstrations. 1 Introduction Chain-of-Thought (CoT) prompting ( Wei et al. 2022 ) has emerged as a pivotal technique in advancing modern large language models (LLMs). By encouraging models to generate explanatory rationales (i.e., intermediate reasoning steps) prior to producing the final answer, CoT significantly improves the reasoning capabilities of LLMs, enabling them to achieve more accurate and interpretable outcomes. Extensive evidence has demonstrated that CoT is particularly effective in tasks involving mathematical, symbolic, or code-based data, and also leads to substantial improvements in general natural language reasoning and factual reasoning ( Sprague et al. 2024 ; Zheng et al. 2024 ; Yu et al. 2024 ) . Building upon the foundation of CoT, numerous advanced reasoning frameworks—such as ReAct ( Yao et al. 2023b ) , Tree-of-Thought (ToT) ( Yao et al. 2023a ) , and Graph-of-Thought (GoT) ( Besta et al. 2024 ) —have been proposed to facilitate problem-solving in more sophisticated scenarios. Furthermore, the emerged ability of generating long-CoT reasoning steps has become a driving factor behind advanced reasoning models such as OpenAI o1 ( OpenAI 2024 ) , o3-mini ( OpenAI 2025 ) , and Deepseek-R1 ( DeepSeek-AI et al. 2025 ) . Beyond empirical improvements, recent theoretical analyses also indicate that CoT enables transformers to perform inherently serial computations and thus overcome their intrinsic limitations in parallel computation ( Li et al. 2024 ) . Despite the well-established effectiveness of CoT, several studies have also explored its limitations. For instance, Ye & Durrett 2022 conducted experiments on earlier LLMs such as GPT-3 ( Brown et al. 2020 ) and OPT ( Zhang et al. 2022 ) , demonstrating that these models may generate unreliable explanations in few-shot textual reasoning scenarios. Additionally, Stechly et al. 2025 highlighted CoT’s reliance on problem-specific prompts and its limited scalability in planning tasks. Furthermore, Zhang et al. 2025 showed that although CoT effectively improves performance, it still faces inherent limitations stemming from the complexity involved in navigating the prompt and answer spaces. Nonetheless, CoT remains widely recognized in current LLM literature as a broadly effective approach to LLM problem-solving, consistently outperforming direct answering. In this paper, we reveal a strikingly counterintuitive finding: Chain-of-Thought prompting unexpectedly degrades LLM performances in certain problem-solving contexts. We investigate in-context learning (ICL) tasks, in which LLMs learn to predict the output of a test instance by extrapolating beyond demonstrations in the form of input-output pairs. Specifically, our analysis focuses on pattern-based ICL benchmarks where the relationships (e.g., patterns, rules, functions) between inputs and outputs are explicitly definable. Through extensive experiments 1 1 1 Code and data: https://github.com/HKUST-KnowComp/CoT-ICL-Eval involving 16 modern LLMs and 9 diverse ICL benchmarks (spanning textual, numerical, and symbolic data), we demonstrate that CoT and its reasoning variants (e.g., ToT, ReAct) consistently underperform direct answering by a significant margin (Figure 1 a). Furthermore, we observe that this performance gap widens as the number of in-context demonstrations increases (Figure 1 b). Our findings challenge the prevailing assumption that CoT is universally effective across various reasoning tasks. To systematically investigate the underlying causes of this unexpected ”curse” effect, we formulate and evaluate three core hypotheses through extensive tailored experiments: • Hypothesis 1. CoT increases the contextual distance between demonstrations and answers, disrupting the few-shot learning structure and thereby degrading performance. • Hypothesis 2. LLMs struggle to infer patterns from demonstration pairs under CoT. • Hypothesis 3. LLMs falter in applying inferred patterns to test instances under CoT. The experimental results empirically validate Hypotheses 1 and 2, providing valuable insights into the limitations of Chain-of-Thought prompting in in-context learning scenarios. Interestingly, we observed that LLMs employing CoT often achieve correct answers even when the inferred patterns are incorrect. This observation suggests a perspective of duality on the CoT mechanism in ICL (Hypothesis 4) : the final prediction arises from an interplay between explicit reasoning (articulated through CoT rationales) and implicit reasoning (similar to direct answering), where both processes contribute to pattern inference and execution. However, LLMs’ limited ability to infer accurate patterns explicitly (as validated by Hypothesis 2) introduces noise into the reasoning process, as flawed rationales disrupt the prediction pipeline. Compounding this issue, the increased contextual distance caused by CoT’s inserted rationales further diminishes the efficacy of implicit reasoning (as validated by Hypothesis 1). Consequently, CoT prompting underperforms direct answering, which relies exclusively on robust implicit mechanisms. Further experiments reveal that even long-CoT reasoning models—despite consuming 40× more inference tokens—achieve only comparable or inferior performance to standard LLMs using direct answering. In summary, our findings advocate for a more nuanced perspective on Chain-of-Thought prompting. Although CoT has demonstrated considerable success in enhancing the reasoning capabilities of large language models, our analysis has revealed critical limitations, especially within pattern-based, in-context learning scenarios. By providing deeper insights into the underlying mechanisms behind these limitations, we highlight that the benefits of CoT rationales are not universally applicable, emphasizing the need for adaptive and context-aware reasoning approaches. Consequently, this work contributes to a more balanced and comprehensive understanding of CoT, informing the development of more robust and flexible reasoning methodologies, and paving the way for future innovations aimed at optimizing large language model performance. 2 Preliminaries Our investigation focuses on in-context learning tasks characterized by explicitly defined input-output functions . Specifically, a consistent and verbalized pattern governs the relationship between each input-output pair within the demonstrations. In this section, we provide a formal definition of pattern-based in-context learning and describe model inference under both direct answering and Chain-of-Thought prompting. 2.1 Pattern-based in-context learning In pattern-based in-context learning, LLMs are provided with a limited number of demonstration pairs, each comprising an input and its corresponding output. These pairs adhere to an

Read the complete text