RESIDUAL CONNECTIONS AND THE CAUSAL SHIFT: UNCOVERING A STRUCTURAL MISALIGNMENT IN TRANSFORMERS

Jonathan Lys, Vincent Gripon, Bastien Pasdeloup, Axel Marmoret, Lukas Mauch, Fabien Cardinaux, Ghouthi Boukli Hacene · 2026

Read RESIDUAL CONNECTIONS AND THE CAUSAL SHIFT: UNCOVERING A STRUCTURAL MISALIGNMENT IN TRANSFORMERS by Jonathan Lys, Vincent Gripon, Bastien Pasdeloup, Axel Marmoret, Lukas Mauch, Fabien Cardinaux, Ghouthi Boukli Hacene (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-14760.txt.

Description

Jonathan Lys and others find a fault line in transformers where short paths and next-word aims pull apart. Short paths tie each step's signal to the word in hand, while training aims at the next word, so a poor guide word can send wrong news onward.

The team traces this shift with decoding paths over tied word spaces and like-measures, and finds hidden signals flip from in-word ties to out-word ties deep in the net. A fix follows from that find. The paper names a quiet flaw in the build all large language models share.

Catalog record

CREATORJonathan Lys, Vincent Gripon, Bastien Pasdeloup, Axel Marmoret, Lukas Mauch, Fabien Cardinaux, Ghouthi Boukli Hacene
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE18 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-14760

Browse the full library · About the Tower of Babel