Context-aware Biases for Length Extrapolation

Ali Veisi, Hamidreza Amirzadeh, Amir Mansourian · 2025

Read Context-aware Biases for Length Extrapolation by Ali Veisi, Hamidreza Amirzadeh, Amir Mansourian (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-08067.txt.

Description

Ali Veisi, Hamidreza Amirzadeh, and Amir Mansourian improve, in a 2025 preprint, how transformers handle texts longer than any they trained on. Models usually learn position from added bias signals tied to relative distance. Fixed or globally learned biases cannot adapt to what a given passage is about. The authors' method, CABLE, learns a bias per token and per attention head. The bias bends with the input's context. That flexibility targets the sharp failure of models on over-long inputs.

A mid-size GPT-2 fitted with CABLE scored better on long texts than rivals with standard encodings. Tests ran on two large public corpora. A BERT model with CABLE also did better at long-context retrieval. The gains stacked on top of the strongest existing relative encodings. The authors release their code openly. They pitch context-aware biases as a general patch for length limits in deployed models.

Catalog record

CREATORAli Veisi, Hamidreza Amirzadeh, Amir Mansourian
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-08067

Browse the full library · About the Tower of Babel