A Universal Vibe? Finding and Controlling Language-Agnostic Informal Register with SAEs
Uri Z. Kialy, Avi Shtarkberg, Ayal Klein · 2026
Read A Universal Vibe? Finding and Controlling Language-Agnostic Informal Register with SAEs by Uri Z. Kialy, Avi Shtarkberg, Ayal Klein (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-26236.txt.
Introduction
Kialy, Shtarkberg, and Klein ask whether language models treat slang as one shared idea or as facts tied to each tongue. They probe the Gemma-2-9B-IT model with sparse autoencoders in English, Hebrew, and Russian. Each test word can be read in a plain sense or a slang sense. They find a small shared core of features for informal tone, strongest in deep layers. Steering those features shifts formality in all three tongues, and in six more unseen ones.
Catalog record
| CREATOR | Uri Z. Kialy, Avi Shtarkberg, Ayal Klein |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52843 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.26236
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A Universal Vibe? Finding and Controlling Language-Agnostic Informal Register with SAEs Uri Z. Kialy, Avi Shtarkberg, Ayal Klein Abstract While multilingual language models successfully transfer factual and syntactic knowledge across languages, it remains unclear whether they process culture-specific pragmatic registers, such as slang, as isolated language-specific memorizations or as unified, abstract concepts. We study this by probing the internal representations of Gemma-2-9B-IT using Sparse Autoencoders (SAEs) across three typologically diverse source languages: English, Hebrew, and Russian. To definitively isolate pragmatic register processing from trivial lexical sensitivity, we introduce a novel dataset in which every target term is polysemous, appearing in both literal and informal contexts. We find that while much of the informal-register signal is distributed across language-specific features, a small but highly robust cross-linguistic core consistently emerges. This shared core forms a geometrically coherent “informal register subspace” that sharpens in the model’s deeper layers. Crucially, these shared representations are not merely correlational: activation steering with these features causally shifts output formality across all source languages and transfers zero-shot to six unseen languages spanning diverse language families and scripts. Together, these results provide the first mechanistic evidence that multilingual LLMs internalize informal register not just as surface-level heuristics, but as a portable, language-agnostic pragmatic abstraction. [ Extension = .otf, Script=Hebrew ] 1 Introduction While large language models (LLMs) have largely mastered multilingual syntax and factual recall, human communication is fundamentally driven by social signaling. When a speaker uses informal register or slang, they are not merely conveying information; they are negotiating social distance, signaling cultural belonging, and expressing casual intimacy. Because modern alignment pipelines overwhelmingly favor standard, highly edited text, LLMs suffer from a pervasive “register gap.” They frequently fail to appropriately process the dynamic, context-dependent language actually used in human interaction ( Sun et al. 2024 ; Li et al. 2024 ) , leading to miscalibrated responses and latent safety vulnerabilities where harmful intent is obfuscated by casual phrasing. Addressing this requires models calibrated not just for factual accuracy, but for deep sociolinguistic competence. This register gap exposes a fundamental, unresolved question about the architecture of multilingual representation: when an LLM encounters informal register across diverse cultures, does it process each instance as an isolated, language-specific surface form? Or does it construct a unified, language-agnostic abstraction of “casual human interaction”? Prior mechanistic interpretability work has demonstrated that models build shared cross-lingual subspaces for facts and syntax ( Conneau et al. 2020 ; Wendler et al. 2024 ) , but whether they abstract pragmatic and social concepts remains entirely unknown. We ask: Do LLMs represent informal register as a universal pragmatic concept, or as a disconnected collection of language-specific memorizations? To answer this, we probe the internal representations of Gemma-2-9B-IT using Sparse Autoencoders (SAEs). To isolate pragmatic register from the trivial confound of lexical bias (e.g., detecting informality merely due to the presence of specialized slang terms), we introduce a novel cross-lingual dataset across English, Russian, and Hebrew. Crucially, our dataset is strictly lexically controlled: every target term is polysemous, appearing in both its standard literal sense and its informal slang sense. This forces the model to resolve register exclusively from pragmatic context. What we find fundamentally challenges the view of models as mere surface-pattern matchers. By isolating latent SAE features that activate differentially across slang and literal contexts, we identify a small, geometrically coherent “informal register subspace” in the model’s latent space, shared across all three typologically diverse source languages. Crucially, this abstraction is not merely correlational. Steering with this cross-linguistic feature set systematically and causally modulates output formality. Strikingly, this single sociolinguistic intervention transfers zero-shot to six typologically distant languages—including Japanese, Thai, Amharic, and Georgian—that were completely absent from our feature extraction pipeline. This zero-shot generalization provides compelling evidence that the model has internalized informal register not as a linguistic artifact, but as a single, abstract pragmatic concept. Our main contributions are threefold. (i) Methodological: We introduce a novel, lexically-controlled dataset of polysemous terms across English, Hebrew, and Russian designed to isolate pragmatic register from any vocabulary bias. (ii) Observational: We identify a geometrically coherent, language-agnostic “informal register subspace” within the SAE latent space that sharpens in the model’s deeper layers. (iii) Causal: We demonstrate that these shared latent features causally drive generation formality. The zero-shot transfer of this control to six typologically diverse, unseen languages provides the first mechanistic evidence for a universal internal representation of pragmatic register. 2 Background The Register Gap in NLP While large language models exhibit remarkable multilingual fluency, their training and alignment are overwhelmingly anchored in standard, edited text. This creates a critical “register gap” when models encounter the dynamic, culturally situated nature of informal language ( Li et al. 2024 ) . Consequently, prior multilingual research on style has largely been restricted to supervised formal-to-informal text transfer ( Rao and Tetreault 2018 ; Briakou et al. 2021 ; Babakov et al. 2022 ) . Multilingual resources for slang remain exceptionally scarce; existing datasets are typically monolingual ( Sun et al. 2024 ) or evaluate dialectal variations within a single language rather than across diverse typologies ( Wuraola et al. 2024 ) . As recent evaluations demonstrate, even state-of-the-art models struggle to reliably deploy or comprehend informal register in context ( Wu and Sun 2025 ) . Because earlier studies relied heavily on surface-level features or black-box classifiers ( Nguyen et al. 2016 ; Brooke et al. 2010 ) , the internal mechanisms models use to resolve informal register remain obscured. Our work addresses this by mechanistically isolating these pragmatic representations across linguistic boundaries. Cross-Linguistic Model Interpretability A central debate in representation learning is whether multilingual models construct a shared, abstract “interlingua” or merely memorize parallel, language-specific surface patterns. Recent work has identified emergent cross-lingual alignment ( Conneau et al. 2020 ) and demonstrated that semantically equivalent concepts occupy shared geometric subspaces ( Wendler et al. 2024 ) , often mediated by specialized multilingual neurons ( Tang et al. 2024 ; Liu et al. 2024 ) . However, these investigations have heavily indexed on factual knowledge and syntactic structure. Much less is known about whether models develop abstract, shared circuitry for pragmatic register —a dimension of language that is inherently social and culturally idiosyncratic. While prior work has successfully identified monolingual steering directions and style-sensitive neurons ( Subramani et al. 2022 ; Rimsky et al. 2024 ; Lai et al. 2024 ) , it remains an open question whether these mechanisms transfer cross-lingually. Our results bridge this gap, examining the balance between shared and language-specific latent features to determine if models possess a universal pragmatic architecture. Sparse Autoencoders and Activation Steering To untangle these complex pragmatic representations, we utilize Sparse Autoencoders (SAEs). By decomposing dense neural activations into an overcomplete, sparse basis, SAEs effectively mitigate polysemanticity, revealing human-interpretable features that standard neuron-level analyses obscure ( Bricken et al. 2023 ; Templeton et al. 2024 ; Cunningham et al. 2023 ) . Beyond serving as observational lenses, SAEs unlock activation steering : the ability to intervene on specific latent features at inference time to prove their causal role in model behavior. This technique has successfully modulated model safety, mitigating social biases ( Durmus et al. 2024 ) and controlling refusal behaviors ( O’Brien