StyleBench: Evaluating thinking styles in Large Language Models
Junyu Guo, Shangding Gu, Ming Jin, Costas Spanos, Javad Lavaei · 2025
Read StyleBench: Evaluating thinking styles in Large Language Models by Junyu Guo, Shangding Gu, Ming Jin, Costas Spanos, Javad Lavaei (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-20868.txt.
Introduction
More structured thinking does not always make a language model smarter; sometimes it just makes it slower. Junyu Guo and four co-authors test that trade-off with StyleBench, a 2025 benchmark of thinking styles. They compare five reasoning styles, from chain-of-thought to chain-of-draft, across five tasks.
Fifteen open-source models, ranging from tiny to very large, take the tests. Extra structure improves accuracy only in limited situations set by task demands and model size. Elsewhere it costs computation and can hurt robustness. The authors frame reasoning structure as a design choice under a capacity budget, not a fixed recipe. Their results help practitioners pick a thinking style that fits the job.
Catalog record
| CREATOR | Junyu Guo, Shangding Gu, Ming Jin, Costas Spanos, Javad Lavaei |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 56 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-20868
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
StyleBench: Evaluating thinking styles in Large Language Models Junyu Guo, Shangding Gu, Ming Jin, Costas Spanos, Javad Lavaei Abstract The effectiveness of Large Language Models (LLMs) is heavily influenced by the reasoning strategies, or styles of thought , employed in their prompts. However, the interplay between these reasoning styles, model architecture, and task type remains poorly understood. To address this, we introduce StyleBench , a comprehensive benchmark for systematically evaluating reasoning styles across diverse tasks and models. We assess five representative reasoning styles—Chain-of-Thought (CoT), Tree-of-Thought (ToT), Algorithm-of-Thought (AoT), Sketch-of-Thought (SoT), and Chain-of-Draft (CoD)—on five reasoning tasks, using 15 open-source models from major families (LLaMA, Qwen, Mistral, Gemma, GPT-OSS, Phi, and DeepSeek) ranging from 270M to 120B parameters. Our large-scale analysis reveals that no single style is universally optimal. We demonstrate that strategy efficacy is highly contingent on both model scale and task type: search-based methods (AoT, ToT) excel in open-ended problems but require large-scale models, while concise styles (SoT, CoD) achieve radical efficiency gains on well-defined tasks. Furthermore, we identify key behavioral patterns: smaller models frequently fail to follow output instructions and default to guessing, while reasoning robustness emerges as a function of scale. Our findings offer a crucial roadmap for selecting optimal reasoning strategies based on specific constraints, We open source the benchmark in https://github.com/JamesJunyuGuo/Style_Bench . 1 Introduction Large Language Models (LLMs) have demonstrated impressive capabilities across a diverse range of tasks, including mathematical reasoning, code generation, and complex question answering ( Imani et al. 2023 ; Wang & Chen 2023 ; Tan et al. 2023 ) . A key insight from prior work is that their performance on challenging problems is not merely a function of scale, but is critically dependent on the methods used to guide reasoning ( Huang & Yang 2025 ) . This has spurred the development of sophisticated prompting techniques designed to structure the model’s internal reasoning process. Notable among these are Chain-of-Thought (CoT) ( Wei et al. 2022 ) , which decomposes problems into sequential steps, and more advanced paradigms like Tree-of-Thought (ToT) ( Yao et al. 2023 ) , which explores multiple reasoning paths in parallel, and Reasonflux ( Yang et al. 2025b ) , employing high-level templates to explore potential solutions. Despite the outstanding capabilities of advanced models like GPT-4o ( Agarwal et al. 2025 ) and DeepSeek ( Guo et al. 2025 ) , their application to specialized or highly complex problems often reveals critical limitations. Performance remains highly sensitive to prompt phrasing and frequently necessitates iterative feedback to achieve robust results ( Sel et al. 2023 ) . In response, recent work has sought to automate reasoning strategy selection. For instance, Gao et al. 2024 proposed a two-stage meta-reasoning process to identify the optimal style, while Wan et al. 2025 leveraged generated instructions to guide the model. However, these approaches assume the existence of a well-understood mapping between problem types and optimal reasoning strategies—a foundation that is currently lacking. The existing literature provides an incomplete picture, as evaluations are typically limited to a single reasoning style, a narrow set of tasks, or a small selection of models. This leaves a significant gap in our understanding of how these strategies generalize across different model architectures, problem domains, and computational budgets. A critical challenge is the trade-off between reasoning depth and efficiency. For simple queries, we expect an LLM to provide a concise, direct answer without a verbose reasoning chain. For complex problems, however, a more elaborate ‘thinking’ procedure is necessary to achieve high accuracy. This raises the important issue of preventing LLMs from overthinking on simple tasks without compromising their ability to reason deeply on hard ones, see e.g., ( Chen et al. 2024 ; Fang et al. 2025 ; Sui et al. 2025 ) . An ideal reasoning strategy should be both effective and efficient, adapting its cognitive load to the complexity of the problem at hand. The gaps in generalization understanding and the need for adaptive efficiency motivate the central question of our benchmark: How do contemporary reasoning strategies perform across a diverse suite of tasks, model scales, and architectures, and which approach offers the optimal balance between performance and computational efficiency? To address these gaps, we introduce StyleBench , a rigorous and extensive benchmark for evaluating reasoning strategies in LLMs. We systematically assess several representative styles—from simple prompting to complex multi-path searches—across diverse models and tasks. Our work provides clear, empirical guidance and a practical roadmap for selecting the most effective reasoning strategy for a given application. Our contributions are summarized as follows: • Comprehensive Benchmark: We introduce a large-scale benchmarking framework that systematically evaluates five reasoning styles (Chain of Thought Wei et al. 2022 , Tree of Thought Yao et al. 2023 , Algorithm of Thought Sel et al. 2023 , Sketch of Thought Aytes et al. 2025 , Chain of Draft Xu et al. 2025 ) across five diverse tasks, including mathematical reasoning ( Cobbe et al. 2021 ) , question answering ( Liu et al. 2020 ; Talmor et al. 2018 ) , and puzzle-solving. • Extensive Model Coverage: Our evaluation encompasses 15 state-of-the-art open-source LLMs spanning major model families (Qwen Yang et al. 2025a , LlaMA Grattafiori et al. 2024 , Mistral Jiang et al. 2024 , Gemma Team et al. 2025 , GPT-OSS Agarwal et al. 2025 , DeepSeek Guo et al. 2025 ) and scales (270M to 120B parameters). • Model–Style Interaction: We demonstrate that reasoning style efficacy is highly contingent on model architecture and scale, showing that optimal strategy selection is model-dependent. • Task–Style Affinities: We identify strong correlations between task types and effective reasoning strategies. Structured multi-step reasoning (e.g., CoT) excels in mathematical tasks, while branching-based exploration (e.g., ToT, AoT) proves more effective for open-ended puzzles like Game of 24. In-context learning styles (CoD, SoT) perform best on symbolic reasoning and commonsense reasoning tasks. • Scaling Laws for Reasoning: We provide the empirical analysis of how reasoning style performance scales with model size, revealing non-trivial trade-offs between accuracy, latency and efficiency. Core Logic and Structural Framework Dataset Questions Reasoning Style Processing LLM Response Mathematical Reasoning e.g., Solve 2x + 5 = 15 Common Sense QA e.g., Why is sky blue? Logical Deduction e.g., Syllogism validation Multi-step Decision e.g., Route planning Step-by-Step Decompose equation solving into sequential steps Causal Explanation Explain light scattering mechanisms Rule Application Apply syllogism reasoning rules Path Planning Identify routes and connections 1. Subtract 5: 2x = 10 2. Divide by 2: x = 5 Answer: x = 5 Blue light scattered more by atmospheric molecules due to shorter wavelength Valid syllogism: All A→B, C→A, therefore C→B Bus Line 1 (15min) → Transfer → Subway Line 3 (20min) Figure 1: Core Logic and Structural Framework for Reasoning Style Processing 2 Related Work 2.1 Reasoning with Large Language Models Recent advances in LLM reasoning have been driven by the development of structured thinking styles . Chain-of-Thought (CoT) prompting ( Wei et al. 2022 ) demonstrated that step-by-step reasoning can substantially improve performance, particularly in mathematical and logical tasks. Building on this idea, Tree-of-Thought (ToT) ( Yao et al. 2023 ) introduced a branching exploration strategy that allows models to consider multiple reasoning paths in parallel. Subsequent paradigms