Analyzing the Power of Chain of Thought softmax through Memorization Capabilities

Lijia Yu, Xiao-Shan Gao, Lijun Zhang · 2025

Read Analyzing the Power of Chain of Thought softmax through Memorization Capabilities by Lijia Yu, Xiao-Shan Gao, Lijun Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-01190.txt.

Description

Lijia Yu, Xiao-Shan Gao, and Lijun Zhang ask a pointed question about chain of thought prompting: does it expand what transformers can reason about, across all tasks? Chain of thought clearly helps on some mathematical problems, but the authors recast reasoning as memorization, so the question becomes what chain of thought transformers can memorize.

They give a complete description of the memorization capabilities of fixed precision transformers, with and without chain of thought. For each case they give necessary and sufficient conditions for memorizing a finite reasoning data set, and matching parameter count bounds. The two conditions turn out not to imply each other. So there exist reasoning tasks where chain of thought does not enhance a transformer's power. That is a negative answer to the opening question.

Catalog record

CREATORLijia Yu, Xiao-Shan Gao, Lijun Zhang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE137 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-01190

Browse the full library · About the Tower of Babel