Rotary Outliers and Rotary Offset Features in Large Language Models

André Jonasson · 2025

Read Rotary Outliers and Rotary Offset Features in Large Language Models by André Jonasson (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-01832.txt.

Description

André Jonasson presents a 2025 study of strange large activations that show up in the attention layers of modern language models. These models locate words with rotary position encodings, which rotate queries and keys by their relative positions. Certain features in the rotated queries and keys, called rotary offset features, keep emerging with outsized values and are often read as outliers. Jonasson asks why they arise and where they live.

He derives bounds predicting which rotary frequencies spawn the features and how far apart the query-key angles must sit. Tests across models of many sizes and designs confirm the predictions. The features appear reliably across layers, heads, and architectures. The work gives model builders a principled account of an artifact they often just notice and work around.

Catalog record

CREATORAndré Jonasson
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE30 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-01832

Browse the full library · About the Tower of Babel