Amplify Adjacent Token Differences: Enhancing Long Chain-of-Thought Reasoning with Shift-FFN
Bingning Wang, Guang Liu, Shizhu He, Jun Zhao, Kang Liu · 2025
Read Amplify Adjacent Token Differences: Enhancing Long Chain-of-Thought Reasoning with Shift-FFN by Bingning Wang, Guang Liu, Shizhu He, Jun Zhao, Kang Liu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-17153.txt.
Description
Researchers present Shift-FFN in a 2025 paper, a fix for a flaw in long chain-of-thought reasoning by language models. Top models now solve hard tasks by thinking in long chains. When that skill is taught to smaller student models, though, training often ends in a loop: the model repeats past steps till it hits the length cap.
Tests tie that loop to how alike the states of steps next to each other are. The less they differ, the more the model loops. Shift-FFN widens the gap between such steps inside the network. The aim is long, clean chains of thought with no wasted repeats.
Catalog record
| CREATOR | Bingning Wang, Guang Liu, Shizhu He, Jun Zhao, Kang Liu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-17153