Reversal Invariance A symmetry argument against the arrow of chain-of-thought

Mihir Sahasrabudhe · 2025

Read Reversal Invariance A symmetry argument against the arrow of chain-of-thought by Mihir Sahasrabudhe (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-00341.txt.

Description

Mihir Sahasrabudhe points out a hidden symmetry in how language models are trained. The standard training goal, predicting the next token, gives a text and its exact reversal the same likelihood. In that sense the goal is direction-blind, even though human language and reasoning are strongly time-ordered. The symmetry helps explain a curious finding: models trained on reversed text can perform about as well as models trained on normal text.

Sahasrabudhe argues this is a real limitation, not a harmless quirk. If word order carries phonological, morphological, and causal information, a symmetric goal may miss it. He proposes judging pretraining by temporal asymmetry, and calls for losses and architectures that model the arrow of language while keeping standard modeling skill.

Catalog record

CREATORMihir Sahasrabudhe
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE24 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-00341

Browse the full library · About the Tower of Babel