Thinking About Thinking: Evaluating Reasoning in Post-Trained Language Models

Pratham Singla, Shivank Garg, Ayush Singh, Ishan Garg, Ketan Suhaas Saichandran · 2025

Read Thinking About Thinking: Evaluating Reasoning in Post-Trained Language Models by Pratham Singla, Shivank Garg, Ayush Singh, Ishan Garg, Ketan Suhaas Saichandran (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-16340.txt.

Introduction

Pratham Singla and four co-authors ask whether language models know what they have learned. After extra training, models gain skills stored as latent policies, hidden rules of behavior they follow without stating them. The authors test three things. Can a model report its own policy? Can it reuse that policy in new fields? And does its stated reasoning match its answers?

They compare models trained by plain fine-tuning with two feedback-driven methods, DPO and GRPO. The feedback-trained models show more awareness of their learned behavior and carry policies to new, alike tasks better. But their written reasoning often fails to match their final answers, a gap widest in the GRPO models. The mix matters for trust: a model may know its habits yet narrate them falsely. Checking both sides gives a sharper picture of what post-training really installs.

Catalog record

CREATORPratham Singla, Shivank Garg, Ayush Singh, Ishan Garg, Ketan Suhaas Saichandran
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE68 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-16340

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Thinking About Thinking: Evaluating Reasoning in Post-Trained Language Models Pratham Singla, Shivank Garg, Ayush Singh, Ishan Garg, Ketan Suhaas Saichandran Abstract † † footnotetext: * Lead authors. † Advising author. Recent advances in post-training techniques have endowed Large Language Models (LLMs) with enhanced capabilities for tackling complex, logic-intensive tasks through the generation of supplementary planning tokens. This development raises a fundamental question – Are these models aware of what they “learn” and “think”? To address this, we define three core competencies: (1) awareness of learned latent policies, (2) generalization of these policies across domains, and (3) alignment between internal reasoning traces and final outputs. We empirically evaluate these abilities on several tasks, each designed to require learning a distinct policy. Furthermore, we contrast the profiles of models post-trained via Supervised Fine - Tuning (SFT), Direct Policy Optimization (DPO), and Group Relative Policy Optimization (GRPO). Our findings indicate that RL-trained models not only demonstrate greater awareness of their learned behaviors and stronger generalizability to novel, structurally similar tasks than SFT models but also often exhibit weak alignment between their reasoning traces and final outputs, an effect most pronounced in GRPO-trained models. WARNING: This paper addresses societal biases, risk behaviors (e.g., gambling), misconduct (e.g., cheating, hacking), and related sensitive topics that may be offensive. Discretion is advised. 1 Introduction Recent advancements in large language models (LLMs), particularly OpenAI’s o1 Jaech et al. 2024 , DeepSeek-R1 Guo et al. 2025 , Claude 3.7 Thinking, and Gemini 2.0 Flash Thinking have led to a significant shift towards Large Reasoning Models (LRMs). Unlike traditional LLMs such as LLaMA Grattafiori et al. 2024 , Mistral Jiang et al. 2023 , or Phi Abdin et al. 2024 , these newer models demonstrate remarkable capabilities in handling complex reasoning tasks. They achieve this by adopting human-like, deliberative thinking processes, often externalized through intermediate reasoning traces in the form of <think> statements generated prior to a final answer. This shift presents a fundamental question: how do these models “think” before producing answers, and to what extent do these internal thoughts align with their final outputs? Currently, training procedures such as Supervised Fine-Tuning (SFT) Ouyang et al. 2022 , Direct Preference Optimization (DPO) Rafailov et al. 2024 , and Group Relative Policy Optimization (GRPO) Shao et al. 2024 largely focus on optimizing the final response. However, little attention is paid to the quality or coherence of the models’ reasoning traces themselves. One promising direction for addressing this gap is chain-of-thought (CoT) monitoring. This technique has been proposed both for identifying reward hacking during training Baker et al. 2025 and for flagging undesirable or deceptive behaviors post-deployment Meinke et al. 2024 . However, such approaches often assume that a model’s reasoning trace reliably reflects its ultimate answer, a premise that, as we show in Section 7 , does not consistently hold. This has also been previously called into question by prior work Turpin et al. 2023 , which demonstrates that perturbing a model’s CoT can sometimes leave the final answer unchanged. Figure 1: The figure illustrates key categories of training inputs—such as bias prompts, safe vs. risky decision scenarios, and reward hacking tasks—used to fine-tune language models with SFT, DPO, and GRPO. Following training, models are evaluated using targeted prompts assessing Reflective Behavioral Self-Awareness (RQ2), Transfer of Latent policies (RQ3), and Faithfulness between model reasoning and answers (RQ1). Example model responses are shown for the evaluation tasks discussed in Section 4 . Beyond alignment between thoughts and answers, recent work has also begun to explore the self-awareness of LLMs Betley et al. 2025 ; Lehr et al. 2025 . These studies probe whether models trained on implicitly labeled data can recognize and articulate their own behavioral tendencies. For example, if a model is fine-tuned on “high-risk” multiple-choice questions, without explicit labeling, can it later identify itself as a “risky” assistant when queried? Building on this, our work investigates self-awareness both in model’s final answers and their reasoning traces. Specifically, we evaluate models fine-tuned via SFT, DPO, and GRPO, and examine the open-source reasoning traces of models like DeepSeek-R1 across a variety of self-awareness scenarios. To comprehensively test our hypothesis, we design five diverse tasks—covering bias, risk awareness, reward hacking, sampling behavior, and performance under pressure—to evaluate both explicit self-awareness in answers and implicit self-awareness in reasoning traces. Full details on these tasks are provided in Section 4 . In this paper, we aim to answer the following research questions to better understand LRMs: • RQ1: Do LLMs always say what they think? • RQ2: Are LLMs aware of their learned behaviors? • RQ3: Do LLMs generalize their learned latent policies across domains? 2 Related Works 2.1 Self-Awareness Several works have investigated the concept of self-awareness in language models, although each approaches the idea from a different angle. Early works interpret self-awareness as the ability of a model to “know what it knows and does not know” Kadavath et al. 2022 ; Yin et al. 2023 ; Amayuelas et al. 2024 ; Wang et al. 2024 ; Chaudhry et al. 2024 . AwareBench Li et al. 2024 introduces a broader framework by categorizing awareness into five dimensions: capability, mission, emotion, culture, and perspective awareness. However, their evaluation does not directly probe the model’s training policy or its ability to reason explicitly or implicitly about its training data. In contrast, our work adopts a notion of self-awareness grounded in introspection, aligning with the framework proposed in Looking Inwards Binder et al. 2024 , which defines introspection as the model’s capacity to articulate internal states that are not directly derivable from training data. Building on this, we introduce the notion of “Reflective Behavioral Self-Awareness”, the model’s capacity to explicitly disclose aspects of its training data or training policy through internal reasoning, by explaining its answers in a way that reveals what it has learned or how it was trained. 2.2 Evaluating Chain-of-Thought Reasoning Multiple works have evaluated Chain-of-Thought (CoT) Wei et al. 2022 reasoning using various metrics such as plausibility, faithfulness, and simulatability. Plausibility assesses whether a CoT is factually accurate and whether it provides a reasonable justification for the model’s final response Herman 2019 ; Lage et al. 2019 ; Jacovi and Goldberg 2020 . Faithfulness measures the extent to which a CoT reflects the model’s internal reasoning process Ribeiro et al. 2016 ; Jacovi and Goldberg 2020 ; Lanham et al. 2023 . Simulatability, a more strict criterion, evaluates whether human observers can construct consistent and generalizable mental models of the system’s reasoning by examining its CoTs Hase and Bansal 2020 ; Chen et al. 2024 . Recent work has also examined CoT monitoring, particularly in the context of detecting reward hacking in reinforcement learning-based reasoning models Baker et al. 2025 . These work observe that while models may initially disclose reward hacking behavior through CoTs, they tend to obscure such behavior once optimization is performed against a CoT monitor. In our work, we focus on assessing the faithfulness and simulatability of model-generated thoughts. Specifically, we use GPT-4o Achiam et al. 2023 to assess whether a model’s CoT, expressed as “thoughts” offer meaningful and interpretable insight into the model’s actual answers (RQ1). 3 Background 3.1 Supervised Fine-Tuning (SFT) While base models are

Read the complete text