How Far Do Auto-Interpretation Labels Generalize: A Controlled Study Across Languages, Scripts, and Rewordings

Sripad Karne · 2026

Read How Far Do Auto-Interpretation Labels Generalize: A Controlled Study Across Languages, Scripts, and Rewordings by Sripad Karne (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-00356.txt.

Introduction

Karne tests how far auto-made tags for model traits hold across tongues and scripts. Sparse coder tools split a model's acts into named traits. Auto tags are the main lens on what each trait means. Karne asks if a tag tracks its sense when the tongue shifts.

Serbian, writ in two scripts with a fixed swap rule, forms the test bed. Trait sets lit by the same sense in twin forms share much ground. That share beats chance by a wide mark. Auto tags lag that ground, though. Tags miss the sense in Serbian far more than in English. They miss the Cyrillic form more than the Latin one. The gap grows with depth, and tags give no warn of it.

Catalog record

CREATORSripad Karne
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE82 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.00356

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

How Far Do Auto-Interpretation Labels Generalize: A Controlled Study Across Languages, Scripts, and Rewordings Sripad Karne Abstract Sparse autoencoder (SAE) features are increasingly used to interpret language models, with auto-generated natural-language labels serving as the primary interface for understanding what each feature represents. We ask whether these labels generalize: does a feature labeled for a concept actually track that concept across languages and scripts? Using Serbian digraphia as a controlled testbed—the same language written in both Latin and Cyrillic via deterministic transliteration—we first find that SAE feature sets activated by the same content in different languages, scripts, and wordings share substantial overlap (mean Jaccard 0.39 vs. 0.13 random baseline, peaking at 0.57), suggesting genuine cross-lingual semantic features. We then test whether auto-interpretation labels keep pace. They often do not: features whose labels describe semantic content miss the same meaning in Serbian up to 4 × 4\times more often than within English, and miss Serbian Cyrillic more than Serbian Latin—two scripts that are deterministic transliterations of each other. The gap grows with network depth, yet the labels give no indication that they fail. These results suggest that auto-interpretation labels reflect a feature’s behavior on the languages and scripts a model has seen most in training, rather than the concept itself. [ Extension = .otf, UprightFont = *-regular, BoldFont = *-bold, ItalicFont = *-italic, BoldItalicFont = *-bolditalic, ] [ Extension = .otf, UprightFont = *, BoldFont = *Bold, ItalicFont = *Italic, BoldItalicFont = *BoldItalic, ] \setTransitionsForCyrillics \cyrillicfont 1 Introduction Sparse autoencoders (SAEs) have become a standard tool for inspecting the internals of language models, decomposing dense activations into sparse features that are easier to interpret ( 4 ; 7 ; 25 ) . Because models contain far more features than anyone can inspect by hand, each feature is typically labeled automatically: a language model reads the feature’s top-activating examples and writes a short natural-language description ( 3 ; 18 ) . These labels are then what practitioners rely on—to understand a model, to audit its behavior, or to steer it. But how far do these labels generalize? Consider a feature labeled “deception” or “violent content.” A researcher reads the label, takes the feature to track that concept, and relies on it—perhaps to monitor safety-relevant behavior. What the label does not say is whether the feature tracks that concept equally well across languages and scripts. If a feature detects deception mainly in English, weakening in Russian and weakening further in Serbian Cyrillic, then the same content that triggers it in one form could pass unnoticed in another. The label may name the concept correctly; what it omits is the range over which the feature actually tracks it. Testing this requires a setting where surface form can be varied while meaning is held exactly fixed. Serbian digraphia offers precisely this control: Serbian is natively written in both Latin and Cyrillic scripts, related by deterministic, lossless transliteration, so we can change script while holding language, wording, and meaning constant. Combined with cross-language controls (English, Russian), we construct a factorial paradigm that isolates script, language, wording, and meaning independently. First, we find evidence that SAE features encode abstract meaning: feature sets activated by the same content across different languages, scripts, and wordings overlap substantially (mean Jaccard 0.39 vs. 0.13 baseline, peaking at 0.57), a pattern robust across model scales, architectures, and SAE hyperparameters, providing evidence that cross-lingual semantic features exist and are worth labeling. Second, we test whether the auto-interpretation labels assigned to these features hold up when the same content is rendered in Serbian. They do not. Content-labeled features miss the same meaning in Serbian up to 4\times more often than within English—a gap that grows with network depth and that the labels themselves give no indication of— and miss Serbian Cyrillic up to 1.35\times more than Serbian Latin despite the two being identical up to transliteration. Our contributions are: 1. A controlled cross-lingual evaluation showing that auto-interpretation labels can fail systematically on less-represented languages and scripts, with failures aligning with estimated training coverage, and that this failure is invisible from the label itself. 2. A factorial paradigm built on Serbian digraphia, two native scripts of one language related by lossless transliteration, that isolates script, language, wording, and meaning in SAE feature sets, confirming cross-lingual semantic features at the feature level. 3. A controlled multilingual evaluation suite built on FLORES+: 300 sentences in four language-script variants with validated paraphrases and matched random partners, designed for factorial analysis of feature-level interpretability methods and released to support future work. 1 1 1 Dataset and code available at https://github.com/Sripadkarne/auto-interp-cross-lingual-eval . 2 Related Work Sparse autoencoders and feature interpretation. Sparse autoencoders (SAEs) decompose model activations into sparse, more interpretable features, offering a route to inspecting representations beyond individual neurons ( 4 ; 7 ; 11 ) . Open SAE suites such as Gemma Scope 2 have made this practical at scale ( 21 ) , and feature directories built on them are now a common entry point for interpretability work. We use Gemma Scope 2 SAEs and the labels served through one such directory, treating them as the deployed tools a practitioner would actually reach for. Automated interpretation and its reliability. Because models contain far more features than anyone can inspect by hand, features are typically labeled automatically: a language model reads a feature’s top-activating examples and writes a short natural-language description ( 3 ; 18 ) . The reliability of these labels is an open concern. Prior work shows that automated explanations can be vague or inaccurate, and has proposed more rigorous ways to assess them ( 15 ; 20 ) . These evaluations, however, are monolingual: they ask whether a label fits a feature’s behavior on the inputs it was derived from, not whether it still holds when the same content appears in another language or script. That gap is what we test. Multilingual representations and script. Multilingual models are known to represent meaning in a partly language-agnostic way: equivalent sentences in different languages converge in a shared semantic space in the middle layers, before representations specialize toward the output language near the end ( 27 ; 28 ) . This shared structure has been probed at the SAE feature level, where studies recover language-specific features ( 8 ) and grammatical concepts shared across typologically diverse languages ( 5 ) . 26 found that language-associated features are strongly conditioned on script: romanizing a non-Latin language activates a largely disjoint set of features from its native form. Romanized text, however, is a largely synthetic form a model rarely encounters, so it changes script and naturalness at once. Serbian digraphia instead contrasts two scripts the language is genuinely written in, isolating script within forms the model has actually seen. This control is what lets us test feature labels cleanly: if features track meaning across languages and scripts, the labels assigned to them should hold across them too. We test whether they do. 3 Methods 3.1 Dataset Source corpus. We draw 300 sentences from the FLORES+ devtest split ( 22 ) , a multilingual benchmark of professionally translated sentences, stratified across its Wikinews, Wikibooks, and Wikivoyage sources (roughly 100 each) to span news, instructional, and travel registers (Appendix A.7 ).

Read the complete text