Sense Representations Are Inducible Interfaces

Jan Christian Blaise Cruz and Alham Fikri Aji, SEACrowd {jan.cruz,alham.fikri}@mbzuai.ac.ae · 2026

Read Sense Representations Are Inducible Interfaces by Jan Christian Blaise Cruz and Alham Fikri Aji, SEACrowd {jan.cruz,alham.fikri}@mbzuai.ac.ae (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-28669.txt.

Introduction

Jan Christian Blaise Cruz and Alham Fikri Aji introduce ACROS, which induces a sense pathway into a frozen pretrained decoder model. Sense representations, meaning decompositions, aid disambiguation, steering, and cross-lingual alignment, but require pretraining with sense structure in place. ACROS adds a gated residual addition, preserving model quality while supporting zero-shot word-sense disambiguation, low-KL steering, and cross-lingual adaptation. On SmolLM2-360M it reaches competitive disambiguation scores, recovers about ninety percent of positive steering shifts with a proxy, and adapts to four languages.

Catalog record

CREATORJan Christian Blaise Cruz and Alham Fikri Aji, SEACrowd {jan.cruz,alham.fikri}@mbzuai.ac.ae
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE72828
LICENSEhttp://creativecommons.org/licenses/by-sa/4.0/
RIGHTSCC BY SA
SOURCEarXiv

Source: https://arxiv.org/abs/2605.28669

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Sense Representations Are Inducible Interfaces Jan Christian Blaise Cruz and Alham Fikri Aji, SEACrowd {jan.cruz,alham.fikri}@mbzuai.ac.ae Abstract Sense representations (explicit, per-token meaning decompositions) are useful for disambiguation, steering, and cross-lingual alignment, but existing approaches require models to be pretrained with sense structure baked in. We introduce ACROS, which induces an explicit sense pathway into a frozen pretrained decoder LM through a gated residual addition. On SmolLM2-360M, ACROS preserves base LM quality while supporting three uses of the same induced variables: zero-shot word-sense disambiguation (64.95 F1 on Raganato ALL, competitive with the WordNet first-sense heuristic), low-KL lexical steering across 5,161 CoInCo cases where a simple non-oracle proxy recovers about 90% of positive shifts, and SENSIA cross-lingual adaptation to four languages (mean R@1 0.988, target FLORES PPL 7.94). ACROS makes sense representations an inducible interface for ordinary pretrained LMs. 1 Introduction Words carry multiple meanings, and a model that can explicitly represent, manipulate, and align those meanings gains capabilities that dense, opaque hidden states do not easily support. Sense-aware systems already exist for individual tasks: disambiguation ( 4 ; 18 ) , representation steering ( 25 ; 22 ) , and cross-lingual alignment ( 6 ) . But each is tied to its own architecture or training procedure. No single system exposes the same sense objects for measurement, intervention, and alignment. Pretrained LMs are the natural unifying framework, as they already encode rich lexical structure ( 19 ) , but keep it locked inside dense hidden states. Backpack Language Models ( 14 ) come closest to exposing it, giving each token multiple explicit sense vectors. The catch is that Backpacks must be pretrained as Backpacks. Most strong checkpoints are ordinary decoders, and as we show in Section 2 , converting one into a Backpack creates a representational bottleneck that destroys prediction quality. Figure 1: ACROS unifies measurement, intervention, and alignment in one residual sense interface designed for pretrained decoder LMs. Existing sense-based capabilities were previously tied to different architectures, with no single interface that supports all three. ACROS inserts a residual sense pathway so the same induced variables can be read, steered, and aligned. In this paper, we propose A r c hitecture-agnostic R esidual Induction o f S ense Representations (ACROS), a method that induces an explicit sense pathway into a frozen decoder LM through a gated residual addition, without modifying the original prediction path. Our contributions are: 1. A residual sense-induction method that adds a reusable sense interface to a pretrained decoder LM while preserving its prediction quality. 2. A unified evaluation showing that the same induced variables support word-sense disambiguation, low-KL lexical steering, and SENSIA cross-lingual adaptation: three roles that previously required separate systems. 3. Ablations examining architecture portability across three decoder families, the role of the explicit sense pathway relative to dense hidden-state readout, and interface capacity and scale. 2 Background and Motivation Sense representations serve three roles in NLP, each supported by separate systems. Measurement systems disambiguate words in context through gloss-informed embeddings ( 4 ; 18 ) or multi-sense distillation ( 26 ; 27 ) . Intervention methods steer model behavior by modifying internal representations ( 15 ; 8 ; 30 ; 25 ; 22 ; 31 ; 2 ; 5 ) , though recent evaluations find this can be brittle ( 28 ; 7 ) . Alignment methods use sense structure to anchor cross-lingual adaptation ( 6 ; 14 ) . However, a WSD model cannot steer generation; a steering method cannot be evaluated as WSD; neither provides sense anchors for alignment. For senses to become a general-purpose interface , all three roles must be supported by the same representations inside the same model. 2.1 The Backpack Bottleneck Backpack Language Models ( 14 ) come closest to this goal. Each token has K explicit sense vectors with learned contextualization weights, and the same machinery can be measured, edited ( 13 ) , and used for adaptation. However, Backpacks must be pretrained in that form. Most strong LM checkpoints are ordinary decoders, so the practical question is whether an existing LM can be converted into a Backpack. We test this directly by converting SmolLM2-360M ( 1 ) 1 1 1 SmolLM2 was chosen because it is text-only (which eliminates potential visually-learned confounds), English-only (which eliminates multilingual confounds relevant to later evaluation via SENSIA), and has a fully-open pretraining corpus (which prevents test-set leakage concerns). , an English-only Llama-style decoder, into converted Backpack models through continued training, distillation, and frozen-backbone controls. Table 1 tells a consistent story: every conversion variant learns non-collapsed sense vectors, but sharply degrades ordinary LM behavior. FLORES perplexity ( 11 ) jumps from 25.1 to between 196 and 357; LAMA factual recall ( 19 ) falls well below half of the original. The frozen-backbone control is especially telling: even when the original backbone and LM head are kept fixed and only the sense-side modules are trained, prediction quality still collapses. The failure is not in the training procedure; it is in the output architecture. Appendix A gives extended diagnostics across additional conversion variants. Variant PPL LAMA@1 Sense Sep. Original 25.1 0.3155 – BP + CPT 356.9 0.0695 -0.0833 BP + Distil. 196.3 0.0420 -0.1275 BP + Distil. + Freeze 236.9 0.1245 -0.0308 Table 1: Converting a pretrained LM to a Backpack model creates sense representations, but the resulting model loses the original LM’s predictive ability. PPL is English FLORES perplexity; LAMA@1 is filtered factual cloze accuracy. Sense separation (Sense Sep.) is the mean off-diagonal cosine between sense vectors, where lower or negative values indicate that slots are not collapsing onto the same direction. Why conversion fails. The bottleneck has a geometric explanation. In a converted Backpack, the pre-logit representation is a convex mixture of K sense vectors: h_{q}^{\mathrm{BP}}=\sum_{k}\alpha_{q,k}v_{q,k} , with \alpha_{q,k}\geq 0 and \sum_{k}\alpha_{q,k}=1 . This forces all prediction-relevant information through a space spanned by at most K vectors. Figure 2 shows the problem concretely: we collect 10,000 final-layer hidden states from the pretrained SmolLM2 backbone on its own pretraining corpus, FineWeb, and measure how much variance the top k singular components capture. At the canonical sense count ( K{=}32 ), only 84.9% of the variance is retained; reaching 95% requires rank 287, and 99% requires rank 697. A converted Backpack with practical K simply cannot represent the full space that the pretrained LM uses for prediction. Figure 2: Conversion fails because practical sense counts cannot span the hidden-state space the LM already uses. On SmolLM2-360M final-layer states, practical sense counts ( K{=}8 , 32, 64) retain far less variance than the ranks needed for 95% or 99% coverage, explaining why Backpack conversion harms prediction quality. This is not a formal impossibility result (Backpack sense vectors are input-dependent and could in principle span a larger space), but it explains why conversion consistently fails even under favorable conditions. The residual design of ACROS avoids this bottleneck entirely: instead of replacing the hidden state with a sense mixture, we add a sense mixture to the existing hidden state. 3 ACROS ACROS adds an explicit sense pathway to a pretrained decoder LM while leaving the original prediction path intact. The design principle is simple: rather than forcing all prediction through a sense mixture (which creates the bottleneck

Read the complete text