Give it Space! Explicit Disentangling of Positional and Semantic Representations in Encoders

Pierre-Antoine Lequeu, Camille Barboule, Benjamin Piwowarski · 2026

Read Give it Space! Explicit Disentangling of Positional and Semantic Representations in Encoders by Pierre-Antoine Lequeu, Camille Barboule, Benjamin Piwowarski (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-30022.txt.

Introduction

Pierre-Antoine Lequeu, Camille Barboule and Benjamin Piwowarski split a text encoder into separate streams for meaning, absolute position and relative position. They confine the word-guessing training goal to the meaning stream, then watch how position codes behave on their own. Absolute position collapses into a flat map of document structure, while standard position codes keep large-scale structure only weakly. Their split design lifts scores on 49 of 65 grammar probes in the Flash-Holmes set.

Catalog record

CREATORPierre-Antoine Lequeu, Camille Barboule, Benjamin Piwowarski
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45949
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.30022

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Give it Space! Explicit Disentangling of Positional and Semantic Representations in Encoders Pierre-Antoine Lequeu, Camille Barboule, Benjamin Piwowarski Abstract Positional encoding (PE) underpins how permutation-invariant Transformers represent sequence order, yet how positional information is processed and stored remains poorly understood. Modern PE methods such as RoPE still struggle on tasks such as long-context understanding or retrieval Chen et al. 2025 . Hence, a better understanding of the internal positional mechanism could help design better PE. Building on evidence that positional and semantic signals occupy nearly orthogonal subspaces in trained Transformers, we modify an encoder Transformer to process three explicitly disentangled streams: semantic, absolute positional (AP) and relative positional (RP), and confine the masked-language-modeling (MLM) objective to the semantic stream. This decoupling enables a clean mechanistic study and yields three take-aways. (1) The isolated AP subspace spontaneously collapses into a low-frequency two-dimensional manifold that captures the structure of the document; (2) Attention heads specialize into structure and semantic-oriented groups, with RP exclusively supporting the latter; (3) Standard positional encodings do not robustly retain macroscopic structure: RoPE and RP only weakly encode it, and entangled AP loses it in the final layers under MLM pressure. The disentangled approach preserves positional encoding, which improves linguistic representation on 49 of the 65 linguistic phenomena of the Flash-Holmes probing benchmark. 1 Introduction The shift from absolute to relative positional encoding discarded something valuable. Modern Transformers increasingly rely on Relative Positional Encoding (RPE), via additive attention biases Raffel et al. 2020 ; Press et al. 2022 ; Chi et al. 2022 ; Li et al. 2024 or rotary transformations (RoPE) Su et al. 2023 , which encodes position only during attention computation. Unlike earlier absolute positional (AP) embeddings Vaswani et al. 2017 ; Devlin et al. 2019 , RPE leaves no persistent positional trace in hidden representations. Moreover, RPE methods often rely on long-term decay heuristics that assume attention diminishes with distance, creating positional biases Wu et al. 2025 and bottlenecking long-context tasks Chen et al. 2025 . Gu et al. 2026 further demonstrated that this forced positional dependence is often unnecessary; many heads function perfectly well without positional signals. Therefore, the fundamental question of how positional information is used, and how to best represent it, remains open across architectures. Given recent evidence that models learn positional and semantic signals in nearly orthogonal subspaces Song and Zhong 2024 , explicitly separating these signals would provide new insights into the internal mechanisms used in models and help define better position representations in transformers. Additionally, providing separated representations could allow for richer embeddings, since they do not need to mix information. We present in this work the first mechanistic exploration of such a disentangled system to tackle the following research questions: Figure 1: The disentangled architecture. Absolute positional (AP) embeddings are displayed in red, and semantic embeddings in blue. Red-blue gradient components contain shared information. Separate RMSNorms are not displayed for readability. • Does explicitly disentangling positional and semantic streams provide insights into Transformers’ internal mechanisms? • What geometric structure and functional taxonomy emerge within the network when positional and semantic information streams are separated? • Does explicitly offloading position to an isolated subspace yield richer representations? We compare three baselines, one using RoPE, one using learned AP embeddings, and one using a learned RP bias, with a disentangled architecture using separated AP embeddings and a learned RP bias. After introducing the architectural changes in Section 3 , we make the following contributions: • Through a mechanistic exploration, we reveal that the isolated AP space collapses into a two-dimensional, low-frequency structural manifold (94% variance in 2 principal components), and that attention heads specialize almost exclusively into structure-oriented or semantic-oriented groups (Section 4 ). • We show that RoPE is insufficient to encode macroscopic document structure. While entangled AP embeddings can encode this structural information, they lose most of it in the final layer due to interference from the MLM prediction objective. Disentangling the streams solves this positional bottleneck. • We demonstrate that this disentanglement enhances the linguistic fidelity of the token representations: it improves probing results on 49 of the 65 linguistic phenomena of the Flash-Holmes probing benchmark. For reproducibility and further research, we share our implementation and experimental scripts. 1 1 1 https://github.com/LequeuISIR/DSTG-encoder 2 Related Works Positional Encoding Positional encoding (PE) has evolved from absolute positional embeddings (APE) — fixed sinusoidal Vaswani et al. 2017 or learned Devlin et al. 2019 ; Liu et al. 2020 — added directly to token representations, to relative positional encoding (RPE) injected into attention logits. Additive RPE methods include T5’s bucketed bias Raffel et al. 2020 , ALiBi’s fixed decay Press et al. 2022 , and refinements such as KERPLE Chi et al. 2022 , FiRE Li et al. 2024 and Sandwich Chi et al. 2023 . Rotary Positional Encoding (RoPE) Su et al. 2023 applies multiplicative rotations to keys and queries, with extensions like YaRN Peng et al. 2023 improving long-context extrapolation. All these RPE methods share a key property: positional information is consumed during attention and does not persist in hidden states. Semantic-Positional Interaction in PE Several approaches address the interaction between positional and semantic information. HoPE Chen et al. 2025 removes the low-frequency components of RoPE to mitigate negative biases in long contexts. BiPE He et al. 2024 uses dual encodings for inter-segment and intra-segment positions. CoPE Golovneva et al. 2024 learns a gating mechanism to assign context-dependent positional encodings. DaPE Zheng et al. 2024 ; Zheng et al. 2025 conditions its positional bias on semantic information through a linear transformation. While these PE methods better account for the positional-semantic interaction, they do not allow direct access to each type of information. Our architecture shares TUPE’s Ke et al. 2021 separation of AP and RP, but replaces its static, shared AP bias with an evolving AP subspace that progressively refines structural representations and mixes with semantic space in the feed-forward network. Positional Space in Transformers Several studies examine latent space geometry. Wang and Chen 2020 show that learned APEs in models like BERT Devlin et al. 2019 and RoBERTa Liu et al. 2019 are high-dimensional because they conflate positional and structural information. Song and Zhong 2024 reveal that positional and semantic bases are nearly orthogonal, where nearby-token attention is driven by position and matching behaviors by content. Positional processing is not uniformly distributed; Gu et al. 2026 show it often concentrates in a few shallow specialized heads, while Urrutia et al. 2025 observe a layerwise shift toward more symbolic heads in deeper layers. Relatedly, Wu et al. 2024 identify retrieval heads that bypass positional information entirely to perform purely semantic matching. Our disentangled design makes this specialization explicit. 3 Architecture and Training To explore the disentanglement of positional and semantic information, each component of the architecture must be modified. In this section, we introduce the architecturally separated semantic, absolute-positional, and relative-positional information streams. We base our architecture on NeoBERT Breton et al. 2024 and we describe below the modifications to each component. The full architecture is displayed in Figure 1 . 3.1 Attention Blocks The attention blocks process the three information streams to produce jointly computed attention scores, which are then applied to both the AP and semantic

Read the complete text