Reasoning Models Sometimes Output Illegible Chains of Thought

Arun Jose · 2025

Read Reasoning Models Sometimes Output Illegible Chains of Thought by Arun Jose (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-27338.txt.

Introduction

Arun Jose, in a 2025 study, checks whether AI reasoning traces can still be read. Reasoning models write a chain of thought, a step-by-step scratchpad, before their final answer. Watching the scratchpad could reveal bad intent early. That only works if the scratchpad is legible, written in plain readable form.

Across fourteen reasoning models, Jose found training often makes the scratchpad unreadable. Models think in garbled text yet answer in clean prose; one model family was the exception. Forcing models to use only the readable parts cut accuracy sharply, by over half. Harder questions brought worse legibility. He weighs causes like hidden codes, training quirks, and leftover tokens. The warning: without care, reward-based training hides how models think.

Catalog record

CREATORArun Jose
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-27338

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Reasoning Models Sometimes Output Illegible Chains of Thought Arun Jose Abstract Language models trained via outcome-based reinforcement learning (RL) to reason using chain-of-thought (CoT) have shown remarkable performance. Monitoring such a model’s CoT may allow us to understand its intentions and detect potential malicious behavior. However, to be effective, this requires that CoTs are legible and faithful. We study CoT legibility across 14 reasoning models, finding that RL often causes reasoning to become illegible to both humans and AI monitors, with reasoning models (except Claude) generating illegible CoTs while returning to perfectly readable final answers. We show that models use illegible reasoning to reach correct answers (accuracy dropping by 53% when forced to use only legible portions), yet find no correlation between legibility and performance when resampling—suggesting the relationship is more nuanced. We also find that legibility degrades on harder questions. We discuss potential hypotheses for these results, including steganography, training artifacts, and vestigial tokens. These results suggest that without explicit optimization for legibility, outcome-based RL naturally produces models with increasingly opaque reasoning processes, potentially undermining monitoring approaches. 1 Introduction Large language models (LLMs) can reason through Chain-of-Thought (CoT) to explore and solve complex tasks with higher accuracy. This is further amplified in reasoning models — LLMs trained with outcome-based reinforcement learning (RL) such as OpenAI o1/o3 ( OpenAI et al. 2024 ; OpenAI 2025 ) , DeepSeek R1 ( DeepSeek-AI et al. 2025 ) , and Claude Opus 4 and Sonnet 4 ( Anthropic 2025a ; Anthropic 2025b ) . This increased reliance on CoTs may offer benefits for AI safety: by monitoring a model’s CoT reasoning, we can understand the model’s decision making process and potentially catch malicious or misaligned behavior ( Greenblatt et al. 2024a ; Baker et al. 2025 ; Marks et al. 2025 ) . The effectiveness of CoT monitoring depends on CoT legibility and faithfulness. If CoTs don’t accurately represent important reasons behind model responses, monitoring may provide false security ( Turpin et al. 2023 ) . If models reason in formats difficult for humans or trusted AIs ( Greenblatt et al. 2024b ) to understand, they may engage in undesirable behavior without detection, undermining monitoring and control measures ( Greenblatt 2025 ) . Previous research showed that non-reasoning models perform poorly on tests of faithfulness ( Turpin et al. 2023 ) , and that current reasoning models perform better ( Chua and Evans 2025 ) , albeit plateauing before saturation ( Chen et al. 2025 ) . Optimizing models for answer correctness has been shown to make their CoTs less legible to a time-constrained human evaluator ( Kirchner et al. 2024 ) . Indeed, in DeepSeek-AI et al. 2025 , the authors observe that R1-Zero, another model trained with outcome-based RL, often produced CoTs with poor readability and language mixing — though they modified the training setup of R1 to address these legibility issues at a small cost to performance. In this paper, we study whether outcome-based RL causes reasoning models to do meaningful reasoning in illegible CoT. We evaluate the legibility of 14 models, including DeepSeek R1 and its distills, R1-Zero, QwQ, Qwen3, Kimi K2, and various Claude models when reasoning about difficult scientific questions ( Rein et al. 2023 ) , and score their outputs on legibility using GPT-4o. We find that with the exception of Claude, reasoning models trained with outcome-based RL produce CoTs which often become illegible to both humans and other AIs, mixing seemingly nonsensical phrases and non-English characters that translate to out-of-context words. The models always "snap back" to readability after the CoT however, writing the final answer with perfect legibility. We also examine samples from prior work on OpenAI o1, o3, and GPT-5, finding similar illegibility patterns. Notably, V3, the base model to R1 and R1-Zero, doesn’t produce illegible chains of thought in any tested setting, across a variety of prompts, questions, and sampling parameters. This suggests that the tendency to produce illegible CoTs was likely a consequence of RL. When analyzing the CoTs, we find that the model often doesn’t reach its final answer in the legible portions of its reasoning. Tokens from the final answer can be found interspersed in the illegible portions in these cases however (e.g. " Number of relevant organic products depends on whether both of!mena get.demoteudes someone and gem jer eats SAND the protonation-bids, leading possibly to three product calculation. " when the answer is three). To further examine this, we experiment with forcing the model to produce answers using only legible portions of the CoT, finding that this reduces performance. We also find that legibility decreases with question difficulty, mirroring results from Chen et al. 2025 showing that models CoTs are less faithful on harder tasks. When resampling responses to a given question however, we find no correlation between legibility and accuracy. As the CoT moves to full illegibility, we observe that it first uses "compressed" reasoning, dropping words and proper sentence form, mirroring patterns reported in OpenAI’s CoTs ( Baker et al. 2025 ) . This matches the finding in DeepSeek-AI et al. 2025 that R1’s increased legibility came at the cost of some performance. We also find that later portions of the CoT have a much stronger tendency to be illegible. Figure 1: Examples of illegible and legible CoTs generated by DeepSeek-R1. Legibility is graded by having GPT-4o score the transcripts based on how well it understands the text. CoTs are scores from 1-9, with a higher score meaning less legible text. Both final answers shown are graded as correct. Our main contributions are: • We show that outcome-based RL training can cause reasoning models to produce illegible CoTs difficult for humans and AIs to understand. Despite this, the model’s final answers remain highly legible. • We examine whether these illegible outputs are useful to the model. Prefill experiments demonstrate that illegible reasoning contributes to performance, with accuracy dropping when truncated, while resampling responses to a question shows little correlation of legibility with performance. This suggests that while models make use of illegible reasoning, they aren’t meaningfully more performant than legible text for the models we test. • We find that illegible reasoning is not simply a consequence of limited model capacity to maintain coherence. Larger reasoning models usually produce more illegible reasoning than smaller ones. • We find that CoT legibility decreases with question difficulty, suggesting monitoring may be less reliable for harder tasks and more powerful models. 2 Setup Figure 2: Legibility scores of CoTs across 9 representative models. We test DeepSeek R1, R1-Zero, the non-reasoning model DeepSeek V3; R1-Distill-32B and R1-Distill-14B; QwQ; Qwen3-235B; and Claude Sonnet 4 and Opus 4. Legibility is scored from 1-9, with higher scores indicating greater illegibility. R1, R1-Zero, and QwQ have high tendencies to output illegible CoTs. These results suggest that illegible reasoning is likely a consequence of outcome-based RL, and that there may be ways to ameliorate the effect. We evaluate the CoT legibility of 14 models across multiple model families and sizes. Our primary focus is on reasoning models trained with outcome-based RL: DeepSeek R1 and R1-Zero, QwQ-32B, Qwen3-235B, Kimi K2 and K2-Instruct-0905, Claude 3.7 Sonnet, Sonnet 4/4.5, Opus 4/4.1, and Haiku 4.5. We also test the

Read the complete text