Analyzing the Power of Chain of Thought softmax through Memorization Capabilities
Lijia Yu, Xiao-Shan Gao, Lijun Zhang · 2025
Read Analyzing the Power of Chain of Thought softmax through Memorization Capabilities by Lijia Yu, Xiao-Shan Gao, Lijun Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-01190.txt.
Description
Lijia Yu, Xiao-Shan Gao, and Lijun Zhang ask a pointed question about chain of thought prompting: does it expand what transformers can reason about, across all tasks? Chain of thought clearly helps on some mathematical problems, but the authors recast reasoning as memorization, so the question becomes what chain of thought transformers can memorize.
They give a complete description of the memorization capabilities of fixed precision transformers, with and without chain of thought. For each case they give necessary and sufficient conditions for memorizing a finite reasoning data set, and matching parameter count bounds. The two conditions turn out not to imply each other. So there exist reasoning tasks where chain of thought does not enhance a transformer's power. That is a negative answer to the opening question.
Catalog record
| CREATOR | Lijia Yu, Xiao-Shan Gao, Lijun Zhang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 137 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-01190