Wavelet-based Positional Representation for Long Context

Yui Oka, Taku Hasegawa, Kyosuke Nishida, Kuniko Saito · 2025

Read Wavelet-based Positional Representation for Long Context by Yui Oka, Taku Hasegawa, Kyosuke Nishida, Kuniko Saito (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-02004.txt.

Description

Yui Oka, Taku Hasegawa, Kyosuke Nishida, and Kuniko Saito wrote a computing paper in two thousand twenty-five on long text in language models. Models learn where words sit only for lengths seen in training. When text runs longer than that, the sense of position breaks down.

The authors study two older ways to mark position and find limits in each. One uses a fixed scale, and the other narrows what the model can attend to. Their new method uses wavelets, which capture patterns at many scales at once, without shrinking the field of view. Tests showed gains for both short and long text, and better reach beyond training length.

Catalog record

CREATORYui Oka, Taku Hasegawa, Kyosuke Nishida, Kuniko Saito
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE70 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-02004

Browse the full library · About the Tower of Babel