An Asymptotic Theory of Chain-of-Thought in In-Context Learning
Kaito Takanami, Cengiz Pehlevan · 2026
Read An Asymptotic Theory of Chain-of-Thought in In-Context Learning by Kaito Takanami, Cengiz Pehlevan (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-03217.txt.
Description
Takanami and Pehlevan build an exact theory of chain-of-thought depth in in-context learning. Chain-of-thought makes models spell out steps at test time. How the gain scales with depth is poorly known. The team studies a solvable case: in-context weight prediction for linear regression, where test-time thought is an iterative polish of the weight estimate.
Random matrix tools, in a high-dimensional limit, give an exact formula for the test error. The error is a function of depth, pretraining data size, and context length. A sharp phase change splits three zones: exponential gain, polynomial gain with saturation, and overthinking. The team also pins how the best depth scales. Deep thought helps most when pretraining and context are rich. With thin pretraining or context, more steps can amplify error or stall. Tests on learned linear and softmax attention models match the theory.
Catalog record
| CREATOR | Kaito Takanami, Cengiz Pehlevan |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 137 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.03217