Position as Probability: Self-Supervised Transformers that Think Past Their Training for Length Extrapolation
Philip Lee · 2025
Read Position as Probability: Self-Supervised Transformers that Think Past Their Training for Length Extrapolation by Philip Lee (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-00920.txt.
Description
Philip Heejun Lee wrote this 2025 paper on helping transformers handle sequences far longer than those they trained on. Deep sequence models usually lose accuracy when test inputs run much past training length. Yet tasks such as multi-step arithmetic and step-by-step reasoning demand exactly that skill.
Lee's method, called PRISM, treats position as a probability rather than a fixed mark. It learns relative positions through a filter that updates a histogram step by step. Position doubt is kept as a blend of options instead of one hard choice. In tests, the method reached top scores on length generalization. It held up at ten times the training length on tasks that were out of reach before. The test set spans addition, multiplication, and the SCAN composition tasks.
Catalog record
| CREATOR | Philip Lee |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 86 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-00920