SLAM : Structural Linguistic Activation Marking for Language Models
Fabrice Harel-Canada, Amit Sahai[0.3em] [email protected]@cs.ucla.edu, [0.5em] University of California, Los Angeles · 2026
Read SLAM : Structural Linguistic Activation Marking for Language Models by Fabrice Harel-Canada, Amit Sahai[0.3em] [email protected]@cs.ucla.edu, [0.5em] University of California, Los Angeles (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-05443.txt.
Introduction
Watermarking language model text usually bends word choices, and readers pay for detection with worse text. Fabrice Harel-Canada and Amit Sahai present SLAM, a watermark that works differently. Instead of touching word frequencies, it writes the mark into the model's internal geometry at generation time.
Sparse autoencoders find internal directions tied to features like voice, tense, and clause order. Steering those directions leaves meaning and word sampling free. On two Gemma-2 models, detection was perfect, and the quality cost was far smaller than that of leading token-based watermarks, though paraphrasing that reshapes syntax can still erase the mark.
Catalog record
| CREATOR | Fabrice Harel-Canada, Amit Sahai[0.3em] [email protected]@cs.ucla.edu, [0.5em] University of California, Los Angeles |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 95 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.05443
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SLAM : Structural Linguistic Activation Marking for Language Models Fabrice Harel-Canada, Amit Sahai[0.3em] [email protected]@cs.ucla.edu, [0.5em] University of California, Los Angeles Abstract LLM watermarks must be detectable without compromising text quality, yet most existing schemes bias the next-token distribution and pay for detection with measurable quality loss. We present SLAM ( S tructural L inguistic A ctivation M arking), a novel white-box watermarking scheme that sidesteps this cost by writing the mark into structural geometry rather than token frequencies: sparse autoencoders identify residual-stream directions encoding linguistic structure (e.g., voice, tense, clause order), and we causally steer those directions at generation time, leaving lexical sampling and semantics unconstrained. On Gemma-2 2B and 9B, SLAM achieves 100 % 100\% detection accuracy with a quality cost of only 1 1 – 2 2 reward points—compared to 7.5 7.5 – 11.5 11.5 for KGW, EWD, and Unigram—with naturalness and diversity preserved at near-unwatermarked levels across both models. The trade-off is a complementary robustness profile: SLAM resists word-level edits but is vulnerable to paraphrase that restructures syntax (at a quality cost), the converse of token-distribution methods. 1 Introduction The widespread deployment of large language models has created an urgent need for reliable provenance attribution: distinguishing AI-generated text from human-authored text, and tracing generated content to its source model. Watermarking, which embeds an imperceptible but detectable signal into generated text, is a promising approach, but existing methods face a fundamental tension between detectability, quality, and robustness. The dominant paradigm ( Kirchenbauer et al. 2023 ; Kuditipudi et al. 2024 ; Zhao et al. 2024 ; Dathathri et al. 2024 ) biases the model’s next-token distribution during generation and tests for the resulting statistical regularity at detection time. These schemes are efficient and theoretically grounded, but biasing the sampling distribution comes at a measurable cost: KGW, EWD, and Unigram lose 7.5 – 11.5 reward points relative to the unwatermarked baseline on Gemma-2 PT models, and their conditional perplexity ratios collapse to 0.30 – 0.42 (repetitive output the LM finds easier to predict than its own natural generation, the signature of green-list-token recurrence; see § 4.2 ). SAEMark ( Yu et al. 2026 ) uses SAEs as a post-hoc scorer over N{=}50 generation candidates, achieving near-lossless quality but at a significant O(N\times M) generation cost. Furthermore, its reliance on semantic features leaves it vulnerable to domain shift and erasure from simple text edits. Our approach. SLAM sidesteps token-distribution biasing entirely by writing the watermark into the residual stream’s structural geometry rather than into token frequencies. Syntactic structure is encoded in specific, localized subspaces of the LLM residual stream ( Park et al. 2024 ; Hewitt and Manning 2019 ) , a fact made operational by public SAEs such as Gemma Scope ( Lieberum et al. 2024 ) . Unlike semantic features, structural features (passive voice, PP-fronting) are by construction domain-invariant: a passive-voice SAE direction fires regardless of whether the subject is a scientist or a banker. SLAM causally steers those directions during generation, biasing which structural variant the model produces (e.g., passive vs. active voice) without constraining which tokens it samples. The result is a watermark that leaves the lexical surface essentially unconstrained. On reward quality, SLAM achieves \Delta\text{Reward}={-}1.3/{-}1.9 on 2B/9B — best among all methods on 2B, and second only to SynthID on 9B (SynthID’s -0.2 comes paired with PPL ratio 0.72 , indicating repetitive text). SLAM ’s PPL ratio of 1.24/1.36 stays closest to 1 on 2B, and distinct- n , Self-BLEU, and MAUVE all remain within tight tolerance of the unwatermarked baseline at a modest \sim 1.65\times overhead. The trade-off is a complementary robustness profile: word-level edits leave SLAM intact ( 100\% post-attack True Positive Rate (TPR) on synonym substitution, deletion, and reordering) while syntax-restructuring paraphrase (e.g., DIPPER) can remove the signal at a quality cost to the attacker, the converse of token-distribution methods. Because we explicitly target provenance attribution rather than tamper-proof cryptographic authentication, this vulnerability to severe structural rewriting is a calculated, acceptable cost for near-lossless generation quality. Contributions. 1. A contrastive mining pipeline (46,579 sentence pairs spanning 104 syntactic, morphological, and discourse-level phenomena, each elicited across 5 semantic domains) that isolates structural SAE directions via composite scoring (contrastive \times purity \times cross-domain consistency). 2. PCA-bidirectional structural directions : SVD of the contrastive difference matrix yields k orthogonal structural modes per phenomenon, allowing SLAM to operate on individual SAE latents ( k{=}1 ) or composite subspaces ( k{>}1 ); bidirectional mining captures both directions of each alternation, doubling the steerable bank. This handles distributed structural representations in larger models and enables a quality–detection trade-off (Figure 4 ). 3. A complete white-box watermarking scheme with HMAC-keyed feature selection and an early-stop generation loop (cap N\!\leq\!4 candidates per prompt; the first candidate clearing the calibrated \hat{z} threshold is returned), an order of magnitude fewer LLM calls than SAEMark. End-to-end overhead is 1.7\times / 1.6\times the unwatermarked baseline on 2B/9B, with detection in one model forward pass ( \sim 420–640 ms; vs. Adaptive’s \sim 10.7 s, Table 8 ). 4. Empirical validation on Gemma 2 2B and 9B (PT) against six baselines across four families of quality metrics ( \Delta Reward; conditional PPL ratio; grammar-error rate; distinct- n , Self-BLEU, MAUVE) at matched detection rate: SLAM is the only method that does not catastrophically fail on any metric, paired with an attack-validity study establishing the complementary robustness profile (word-level immune, syntax-targeted-paraphrase-vulnerable). 2 Background 2.1 LLM watermarking Token-distribution watermarks ( Kirchenbauer et al. 2023 ; Kuditipudi et al. 2024 ; Zhao et al. 2024 ; Dathathri et al. 2024 ; Hu et al. 2024 ) bias next-token sampling (via green/red token lists, exponential minimum sampling, or entropy-aware selection) and detect the resulting statistical regularity at inference time. All produce signal in token-frequency space; we show this is disrupted by moderate paraphrase (50–64% TPR post-DIPPER for KGW/EWD/Unigram; Figure 2 ). Non-distortion as a quality benchmark. Dathathri et al. 2024 formalise the gold standard for quality preservation as K -sequence non-distortion : over K consecutive responses, the marginal distribution of any single response equals the unwatermarked distribution. SynthID provably satisfies K{=}1 non-distortion; this aggregate-distributional guarantee implies equal marginal distributions, so the measurable reward gap we observe (§ 4.2 ) likely reflects finite-sample variance or the MarkLLM tournament-sampling implementation not perfectly realising the theoretical ideal at N{=}100 prompts. KGW, EWD, and Unigram make no such guarantee and pay a substantially larger quality cost. SLAM does not prove non-distortion (steering modifies the intermediate distribution rather than the sampling distribution) but empirically approaches it on every quality axis we measure. Linguistic and syntactic watermarking. An earlier line of work embeds watermarks through lexical and syntactic rewriting: synonym substitution ( Atallah et al. 2001 ; Chang and Clark 2010 ) , masked-LM edits ( Ueoka et al. 2021 ) , and invariant syntactic features ( Yoo et al. 2023 ) . These surface-form signals are eliminated by any paraphraser not constrained to preserve exact wording. Semantic-invariant watermarks ( Liu et al. 2024a ; Hou et al. 2024 ; Ren et al. 2024 ) and post-hoc black-box marking ( Chang et al. 2024 ) seek robustness without model access. White-box functional-invariant marking ( Fernandez et al. 2024 ) and distribution-free schemes ( Giboulot and Furon