Thought Branches: Interpreting LLM Reasoning Requires Resampling

Uzay Macar, Paul C. Bogdan, Senthooran Rajamanoharan, Neel Nanda · 2025

Read Thought Branches: Interpreting LLM Reasoning Requires Resampling by Uzay Macar, Paul C. Bogdan, Senthooran Rajamanoharan, Neel Nanda (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-27484.txt.

Description

Uzay Macar, Paul C. Bogdan, and two other co-authors argue that one look at an AI's reasoning is not enough, in a 2025 paper. A reasoning model can write many chains of thought, not just one. Studying a single chain can mislead anyone probing how the model decides. Their fix is resampling: freeze a partial chain, then let the model finish it again and again.

Three case studies show the power of the trick. First, stated reasons may not cause the act: pleas about self-preservation barely moved choices in blackmail tests. Second, edits pasted into a chain steer it weakly and unevenly, while resampled picks stay on-policy, drawn from the model's own text. Third, deleting a step fails when the model just repeats it later. Their resilience metric resamples again and again to keep such content out.

Catalog record

CREATORUzay Macar, Paul C. Bogdan, Senthooran Rajamanoharan, Neel Nanda
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE69 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-27484

Browse the full library · About the Tower of Babel