Position as Probability: Self-Supervised Transformers that Think Past Their Training for Length Extrapolation

Philip Lee · 2025

Read Position as Probability: Self-Supervised Transformers that Think Past Their Training for Length Extrapolation by Philip Lee (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-00920.txt.

Description

Philip Heejun Lee wrote this 2025 paper on helping transformers handle sequences far longer than those they trained on. Deep sequence models usually lose accuracy when test inputs run much past training length. Yet tasks such as multi-step arithmetic and step-by-step reasoning demand exactly that skill.

Lee's method, called PRISM, treats position as a probability rather than a fixed mark. It learns relative positions through a filter that updates a histogram step by step. Position doubt is kept as a blend of options instead of one hard choice. In tests, the method reached top scores on length generalization. It held up at ten times the training length on tasks that were out of reach before. The test set spans addition, multiplication, and the SCAN composition tasks.

Catalog record

CREATORPhilip Lee
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE86 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-00920

Browse the full library · About the Tower of Babel