Innate Reasoning is Not Enough: In-Context Learning Enhances Reasoning Large Language Models with Less Overthinking

Yuyao Ge, Shenghua Liu, Yiwei Wang, Lingrui Mei, Lizhe Chen, Baolong Bi Xueqi Cheng · 2025

Read Innate Reasoning is Not Enough: In-Context Learning Enhances Reasoning Large Language Models with Less Overthinking by Yuyao Ge, Shenghua Liu, Yiwei Wang, Lingrui Mei, Lizhe Chen, Baolong Bi Xueqi Cheng (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-19602.txt.

Introduction

Yuyao Ge and co-authors test whether prompting still helps models that are trained to reason at length. Their 2025 paper studies reasoning language models, which think step by step with reflection and self-correction. A natural question is whether chain-of-thought prompts, a standard trick for chat models, add anything for these systems.

The team compared no prompt, a zero-shot prompt, and few-shot prompts across math tasks and model sizes from one point five to thirty-two billion parameters. Prompting helped in most settings. Large models gained most on hard problems, while small models showed the reverse pattern. Prompting also cut excess reflection sharply in some cases, by about ninety percent. Attention analysis adds detail on why.

Catalog record

CREATORYuyao Ge, Shenghua Liu, Yiwei Wang, Lingrui Mei, Lizhe Chen, Baolong Bi Xueqi Cheng
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-19602

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Innate Reasoning is Not Enough: In-Context Learning Enhances Reasoning Large Language Models with Less Overthinking Yuyao Ge, Shenghua Liu, Yiwei Wang, Lingrui Mei, Lizhe Chen, Baolong Bi Xueqi Cheng Abstract Recent advances in Large Language Models (LLMs) have introduced Reasoning Large Language Models (RLLMs), which employ extended thinking processes with reflection and self-correction capabilities, demonstrating the effectiveness of test-time scaling. RLLMs exhibit innate Chain-of-Thought (CoT) reasoning capability obtained from training, leading to a natural question: “Is CoT prompting, a popular In-Context Learning (ICL) method for chat LLMs, necessary to enhance the reasoning capability of RLLMs?” In this work, we present the first comprehensive analysis of the impacts of Zero-shot CoT and Few-shot CoT on RLLMs across mathematical reasoning tasks. We examine models ranging from 1.5B to 32B parameters, finding that contrary to concerns, CoT prompting significantly enhances RLLMs’ performance in most scenarios. Our results reveal distinct patterns: large-capacity models show minimal improvement on simple tasks but substantial gains on complex problems, while smaller models exhibit the opposite behavior. Further analysis demonstrates that CoT prompting effectively controls the distribution of the numbers of thinking tokens and reasoning steps, reducing excessive reflections by approximately 90% in some cases. Moreover, attention logits analysis reveals the RLLMs’ overfitting to reflection-related words, which is mitigated by external CoT guidance. Notably, our experiments indicate that for RLLMs, one-shot CoT consistently yields superior performance compared to Few-shot CoT approaches. Our findings provide important insights for optimizing RLLMs’ performance through appropriate prompting strategies. 1 Introduction Recent advances in Large Language Models (LLMs) have introduced Reasoning Large Language Models (RLLMs) such as OpenAI o1 ( 2024 ), DeepSeek-R1 ( 2025 ), and Qwen QwQ ( 2025 ). These models generate innate Chain-of-Thought (CoT) before answering at inference time, demonstrating the effectiveness of test-time scaling ( Muennighoff et al., 2025 ) . Innate CoT is characterized by reflection and self-correction, which can significantly enhance a model’s ability to solve complex reasoning tasks ( Kumar et al., 2024 ) . Over the past three years, as one of the most effective reasoning methods, CoT prompting ( Wei et al., 2022 ) has been widely applied to LLMs and has helped models externally establish chain-like thinking for reasoning problems. This approach has significantly improved performance across various reasoning tasks ( Zhang et al., 2022 ; Wang et al., 2022 ; Lyu et al., 2023 ; Ge et al., 2024 ) . However, researchers from DeepSeek Guo et al. (2025) have expressed concern that Few-shot CoT might actually impair the performance of reasoning LLMs, implying that Few-shot CoT may not benefit RLLMs as it does traditional LLMs. This raises a concern: “ Is CoT prompting, a popular In-Context Learning (ICL) method for chat LLMs, necessary to enhance the reasoning capability of RLLMs? ” Figure 1 : CoT prompting continues to play an important role in reasoning LLMs: (1) improving reasoning performance, (2) controlling the number of thinking tokens, (3) regulating the number of reasoning steps, and (4) mitigating overthinking. In this paper, we present the first comprehensive analysis examining the impact of Zero-shot CoT ( Kojima et al., 2022 ) & Few-shot CoT ( Wei et al., 2022 ) on RLLMs for mathematical problems. Our experiments examine models with parameter sizes ranging from 1.5B to 32B, revealing that Zero-shot CoT & Few-shot CoT significantly enhance the performance of reasoning LLMs on mathematical tasks in most cases. Notably, for large-capacity models: the improvement on simple datasets is minimal; however, the improvement on complex datasets is substantial; whereas for small-capacity models, the situation is reversed. Additionally, we find that Zero-shot CoT & Few-shot CoT play a significant role in regulating the numbers of thinking tokens and reasoning steps. To explore why direct prompting generates so many thinking tokens, we have statistically analyzed the reflection frequency of RLLM outputs. Surprisingly, on complex datasets, the average frequency of reflection per question is as high as over 800 for the 1.5B capacity model and over 400 for the 32B capacity model. It is worth noting that CoT prompting can reduce the average number of reflections by about 90% in some cases. To explore the reason for RLLM’s excessive reliance on reflection, we conducted deep analysis of the attention logits (before softmax) and attention scores (after softmax) of R1-8B and found its particular focus on words such as “Wait,” “Double-Check,” and “Ensure.” In contrast, its base model, LLaMA3.1-8B-Instruct , did not exhibit such special attention. Thus, we believe that the reason for this overthinking is the RLLM’s overfitting to reflection and self-correction mechanisms. Through external prompt guidance, such as Zero-shot CoT & Few-shot CoT, the overfitting phenomenon can be alleviated. After observing the significant impact of Zero-shot CoT & Few-shot CoT on the performance of RLLMs, we conducted experiments to test the effect of the number of shots on performance. Unlike the common trend in LLMs where more shots generally lead to better performance, we found that one-shot CoT achieved the best performance across all datasets and models. Our main contributions are as follows: • We are the first to conduct a comprehensive analysis examining the impact of Zero-shot CoT & Few-shot CoT on RLLMs for mathematical problems. • In terms of breadth, our experiments have revealed that Zero-shot CoT & Few-shot CoT plays a crucial role in controlling the distribution of the numbers of thinking tokens and reasoning steps of RLLM, as well as in suppressing overthinking. • In terms of depth, we conducted a visualization analysis of the attention logits of RLLMs and discovered the overfitting of RLLM to reflection words. Additionally, our findings indicate that one-shot CoT achieved the best performance for RLLMs. 2 Related Works 2.1 Research and Analysis on CoT Prompting Chain-of-Thought prompting, first introduced by Wei et al. (2022) , showed that providing exemplars of intermediate reasoning steps can significantly boost LLMs’ performance on complex tasks. Soon after, Kojima et al. (2022) discovered that even without any demonstrations, simply appending a prompt like “ Let’s think step by step ” enables strong Zero-shot CoT reasoning. Subsequent efforts focused on automating and refining CoT prompts. Zhang et al. (2022) proposed Auto-CoT, which automatically generates diverse reasoning chains for Few-shot prompts. In parallel, Wang et al. (2022) introduced a self-consistency decoding strategy: by sampling multiple distinct reasoning paths and selecting the most consistent final answer, they achieved striking performance gains in CoT prompting. Beyond new prompting strategies, researchers also analyzed how CoT content affects outcomes. Jin et al. (2024) found that longer reasoning sequences, even containing minor mistakes, substantially enhance LLM reasoning accuracy, whereas overly concise chains degrade it. Building upon this work, Wu et al. (2025) demonstrate that a nuanced relationship exists between CoT length and performance, identifying an optimal length that balances decomposition benefits against error accumulation based on model capability and task complexity. 2.2 Reasoning Large Language Models with Innate CoT Despite the excellent performance of CoT prompting, inherent limitations in adaptability persist; consequently, reasoning LLMs, such as OpenAI’s o1 ( 2024 ) have been introduced to generate reasoning internally, offering enhanced efficiency and broader generality. Shortly after its introduction, the research community responded with projects: Open-O1 ( 2024 ). Subsequently, Alibaba launched both Marco-O1 ( 2024 ) and QWQ-preview ( 2024

Read the complete text