Your thoughts tell who you are: Characterize the reasoning patterns of LRMs

Yida Chen, Yuning Mao, Xianjun Yang, Suyu Ge, Shengjie Bi, Lijuan Liu, Saghar Hosseini, Liang Tan, Yixin Nie, Shaoliang Nie · 2025

Read Your thoughts tell who you are: Characterize the reasoning patterns of LRMs by Yida Chen, Yuning Mao, Xianjun Yang, Suyu Ge, Shengjie Bi, Lijuan Liu, Saghar Hosseini, Liang Tan, Yixin Nie, Shaoliang Nie (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-24147.txt.

Introduction

Reasoning models leave long traces of their steps, and those traces may carry a style as distinct as handwriting. Yida Chen and 9 co-authors built a method that lets a language model describe how two models' traces differ, in plain words. Repeating that comparison builds a readable taxonomy of thinking styles.

The taxonomy sorted traces back to their source model with high accuracy when models differed in size, family, or training aim. A case study then borrowed the largest model's style for smaller ones at test time, which lifted their science scores. Style, the results suggest, is not mere flavor: it tracks how well a model performs.

Catalog record

CREATORYida Chen, Yuning Mao, Xianjun Yang, Suyu Ge, Shengjie Bi, Lijuan Liu, Saghar Hosseini, Liang Tan, Yixin Nie, Shaoliang Nie
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE56 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-24147

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Your thoughts tell who you are: Characterize the reasoning patterns of LRMs Yida Chen, Yuning Mao, Xianjun Yang, Suyu Ge, Shengjie Bi, Lijuan Liu, Saghar Hosseini, Liang Tan, Yixin Nie, Shaoliang Nie Abstract Current comparisons of large reasoning models (LRMs) focus on macro-level statistics such as task accuracy or reasoning length. Whether different LRMs reason differently remains an open question. To address this gap, we introduce the LLM-proposed Open Taxonomy (LOT), a classification method that uses a generative language model to compare reasoning traces from two LRMs and articulate their distinctive features in words. LOT then models how these features predict the source LRM of a reasoning trace based on their empirical distributions across LRM outputs. Iterating this process over a dataset of reasoning traces yields a human-readable taxonomy that characterizes how models think. We apply LOT to compare the reasoning of 12 open-source LRMs on tasks in math, science, and coding. LOT identifies systematic differences in their thoughts, achieving 80–100% accuracy in distinguishing reasoning traces from LRMs that differ in scale, base model family, or objective domain. Beyond classification, LOT’s natural-language taxonomy provides qualitative explanations of how LRMs think differently. Finally, in a case study, we link the reasoning differences to performance: aligning the reasoning style of smaller Qwen3 models with that of the largest Qwen3 during test time improves their accuracy on GPQA by 3.3–5.7%. 1 Introduction Following the success of GPT-o1 and DeepSeek-R1, a wave of large reasoning models (LRMs) has recently become available. These models differ in training recipes and report varying benchmark performance, but far less is known about whether they also reason differently from one another. In this paper, we pose a fundamental question: can LRMs be distinguished by their reasoning patterns, and if so, what are the key distinguishing traits? A growing body of work has begun probing the reasoning style of individual LRMs, yielding insights into how artificial thinkers “think”. Marjanović et al. 2025 , for example, finds that DeepSeek-R1’s reasoning depth correlates with the human cognitive load when processing complex sentences. Bogdan et al. 2025 annotates functions of DeepSeek-R1-Distill-Qwen’s reasoning steps, showing that plan generation and re-evaluation are critical for solving math problems. However, only a few studies attempt broader comparisons across LRMs and on multiple reasoning behaviors. Gandhi et al. 2025 compares the reasoning patterns of base large language models (LLMs) and their RL fine-tuned variants, finding that the habits of base models correlate with fine-tuning gains. Along the same lines, Jiang et al. 2025 shows that LRMs differ in how they structure their reasoning steps. However, Gandhi et al. 2025 ; Jiang et al. 2025 ; Bogdan et al. 2025 adopt a deductive approach, relying on fixed, researcher-defined taxonomies of reasoning behaviors when comparing artificial thinkers. The deductive approach risks biasing analyses towards researchers’ theories and overlooking unexpected behaviors of models, such as attempting to “visualize” the chemical structure of compounds given in the question. To address this limitation, we introduce the LLM-proposed Open Taxonomy (LOT), an inductive method that identifies reasoning features distinguishing two LRMs directly from their outputs. LOT operates in three stages: (1) an LLM compares thinkings from two LRMs on the same question and highlights distinguishing reasoning traits in natural language; (2) the LLM annotates these features in reasoning traces from other questions, converting textual reasoning into vectors of features; (3) a logistic classifier is trained on these vectors to predict the source model for unseen traces. When classification fails on a new trace, LOT returns to stage 1 to propose new features observed in the failed sample. Iterating this cycle yields an open taxonomy of reasoning traits that reliably separates the thought processes of different LRMs. We apply LOT to compare and classify the reasoning traces of 12 LRMs across diverse model scales, base model families, and specialized domains. LOT achieves 80-100% accuracy in classifying the reasoning of LRMs when they differ substantially along one of the axes above. In classifying LRMs with various parameter scales, LOT outperforms few-shot prompting (by 23.8% on average), a recent automatic prompt engineer method, VML ( Xiao et al. 2025 ) (by 19.6%), and a human-defined reasoning taxonomy ( Gandhi et al. 2025 ) (by 11.7%) in accuracy. Its natural-language taxonomies also provide human-readable explanations of distinctive reasoning traits across LRMs—for example, smaller models more often fall into circular reasoning, and a code-specialized model occasionally uses Python functions to solve math problems ( section 4 ). In a case study, we further link the reasoning differences among Qwen3 models at different scales with their performance gaps on GPQA-Diamond, showing that aligning the reasoning style of smaller Qwen3 models to that of the largest Qwen3 can improve their accuracy by 3.3–5.7% ( section 5 ). In summary, our main contributions are: (1) introduce LOT, an inductive method that constructs human-readable taxonomies of reasoning features to characterize the thought processes of LRMs; (2) show that LOT accurately classifies LRMs’ reasoning across domains and outperforms existing approaches; (3) explain systematic reasoning differences in natural language; and (4) through a case study on Qwen3 models, demonstrate that these differences have a causal link to performance gaps. 2 Related Work Classification as a Probe to Illustrate the Models’ Behavioral Differences Existing studies of LRMs’ reasoning ( Gandhi et al. 2025 ; Jiang et al. 2025 ; Bogdan et al. 2025 ) rely on predefined taxonomies of behaviors, limiting their analyses to researcher-chosen categories. Recently, Sun et al. 2025 uses classification as an exploratory probe to detect differentiating output patterns of non-reasoning LLMs, such as Grok-2 and Gemini-1.5. They train neural classifiers to predict the source model of generated texts and obtain high accuracies that suggest the existence of “signatures” patterns in LLMs’ outputs. However, the features learned by neural models are not directly interpretable. Sun et al. 2025 instead infers the LLMs’ behavioral differences through post-hoc counterfactual intervention, which manipulates specific textual properties chosen by the researchers and measures the change in classification accuracy. Since the intervened properties are chosen by researchers, this feature-discovery process remains deductive and may not reflect what the classifier has learned. Can we use the classifier’s learned features to directly explain the LLMs’ behavioral differences? In this work, we use classification not only as an exploratory sensor to detect reasoning differences, but also as an explanatory tool to interpret what these differences are. To achieve this, we design a novel automatic prompt engineer algorithm that, by comparing the thought processes of LRMs, inductively generates human-readable reasoning features for classifying those processes. Automatic Prompt Engineer for Interpretable Text Classification Recent Verbalized Machine Learning (VML) ( Xiao et al. 2025 ) proposes using LLMs to generate interpretable, natural-language decision trees for text classification. In VML, the LLM receives a batch of training samples as input and updates the decision rules, expressed in natural language, based on the observed patterns. VML generates a decision tree by iterating this process. While effective for short-text tasks such as classifying word-gender associations ( Srivastava et al. 2023 ) , VML is impractical for classifying long reasoning traces, which may span tens of thousands of tokens. To accommodate the context window of existing LLMs, VML must drastically reduce its batch size, leaving updates to its

Read the complete text