On the Persistent Effects of Lexicality in Large Language Models

Hammad Rizwan, Muhammad Umair Haider, Nishant Subramani, Mona T. Diab, A.B. Siddique, Hassan Sajjad · 2026

Read On the Persistent Effects of Lexicality in Large Language Models by Hammad Rizwan, Muhammad Umair Haider, Nishant Subramani, Mona T. Diab, A.B. Siddique, Hassan Sajjad (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-02750.txt.

Introduction

Rizwan and co-authors show that word overlap steers language model representations more than meaning does. Representations from these models feed many downstream applications. Their structure often tracks lexical overlap over semantic content. The team quantifies that lexical pull against semantic signal. Adversarial semantic stress tests supply the probes. An information theory lens frames the findings.

Lexical influence spans the full depth of models. It holds across architectures, training regimes, and objective functions. Models trained for semantic similarity are not exempt. A mid-depth region stands out. There, lexical and semantic signals degrade together. That zone is transitional, poor for surface form and meaning alike. Downstream effects are shown through summarization and model editing case studies. The work warns that representation quality cannot be assumed from task scores alone.

Catalog record

CREATORHammad Rizwan, Muhammad Umair Haider, Nishant Subramani, Mona T. Diab, A.B. Siddique, Hassan Sajjad
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE60 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.02750

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

On the Persistent Effects of Lexicality in Large Language Models Hammad Rizwan, Muhammad Umair Haider, Nishant Subramani, Mona T. Diab, A.B. Siddique, Hassan Sajjad Abstract Representations extracted from large language models (LLMs) play an important role in many downstream applications. However, the structure of these representations is often influenced by lexical overlap rather than semantic content. Our understanding of the relationship between this lexical influence and semantic content, and its implications for downstream tasks, remains limited. In this work, we investigate representations to quantify the effect of lexical overlap relative to semantic content. We consider several adversarial semantic stress tests and further connect our findings to the information theory perspective. We find that lexical influence extends across the depth of models, consistently across architectures, training regimes, and objective functions, including the models trained for semantic similarity. Moreover, we observe a mid-depth region in which both lexical and semantic signals degrade simultaneously, indicating a transitional regime where representations are poor for both surface form and meaning. We further demonstrate the effect of lexical influence on downstream uses of LLMs using summarization and model editing as a case study. 1 Introduction Natural Language Processing (NLP) has seen rapid advances with the emergence of LLMs, which now achieve strong performance across a wide variety of benchmarks and downstream tasks. Beyond generating text, these models are increasingly used as general-purpose embedding engines, providing vector representations that underpin retrieval, textual similarity, clustering, and evaluation pipelines. The widespread reliance on LLM embeddings implicitly assumes that, as we progress through model depth, representations gradually move away from local lexical structure and converge toward increasingly abstract sequence-level semantics ( 17 ; 16 ; 12 ) . In practice, this assumption is fragile, since it does not imply that lexical cues are eliminated as representations can remain highly similar whenever inputs share tokens, even when their meanings diverge ( 6 ; 38 ) . When lexical overlap influences embedding geometry, similarity estimates degrade, making it harder to retrieve or classify documents based on meaning rather than wording and weakening systems that depend on meaning-driven representations. We therefore ask how representational geometry evolves along the axes of lexical and semantic information; how training regimes, including pretraining objectives, instruction tuning, and contrastive learning, shape these geometric structures; and how lexical influence propagates to downstream uses of LLMs. We treat every layer of a model as a candidate embedding space and organize our study into four components. First , to quantify lexical influence, we use a triplet semantic-equivalence stress test consisting of an anchor, a meaning-preserving paraphrase, and a meaning-changing distractor that shares substantial lexical overlap with the anchor. This construction isolates semantic similarity failures driven by surface-form similarity and localizes the depths at which representations most often conflate lexical overlap with semantic equivalence. Second , to quantify these failures in terms of lexical and semantic signals, we pair the stress tests with two layer-wise measurements: lexical decodability and semantic fidelity. Lexical decodability is quantified by training linear token-identity probes on hidden states, measuring how much surface form remains linearly recoverable at each layer. We hypothesize that a strong token identity corresponds to a salient lexical signal that downstream methods can readily leverage, whether they rely on similarity geometry or learn additional predictors. We quantify semantic fidelity based on the representation’s performance on diverse embedding tasks. Together, these diagnostics distinguish among competing explanations for representation-similarity failures: lexical dominance, weak semantic organization, and transitional regimes in which both lexical and semantic signals are simultaneously low. Third , as layer-wise lexical and semantic geometry does not by itself characterize information processing dynamics , we relate our investigation to these dynamics using input (prompt) entropy across layers. We test whether information compression and decompression shifts correlate with the layer-wise changes in lexical and semantic structure. Fourth , we show the consequences of lexical influence in practical LLM applications, focusing on abstractive summarization evaluation and factual model editing. Our investigation leads to several central insights: Lexical influence persists across depth (§ 3 ). Adversarial stress tests across multiple models reveal that token-overlap-driven similarity errors persist throughout model depth. Although attenuated in deeper layers, the effect is neither fully eliminated nor confined to shallow representations. Adaptation regimes improve embeddings, but do not remove lexical influence (§ 3 ). Instruction tuning and metric learning improve overall embedding quality, yet our stress tests show that metric learning still fails under lexical overlap adversaries. Lexically similar but semantically mismatched pairs continue to receive inflated similarity scores, indicating that the dominant training paradigm for embedding models reduces but does not eliminate lexical bias. Lexical decodability is non-monotonic across depth (§ 4.1 ). The ability to decode exact lexical identity from token representations fluctuates across layers rather than declining monotonically. This is in line with findings by 2 , indicating that depth does not induce a clean lexical-to-semantic transition ( 47 ; 17 ; 22 ) . A mid-depth valley where lexical and semantic signals both weaken (§ 4 , § 5 ) . Across model families, intermediate layers form a valley-like region where token identity is least recoverable, and semantic performance stagnates or degrades across both raw embedding geometry and linearly probed semantic evaluations. This valley appears to align with a compression–re-expansion point in prompt information across layers; a similar valley shows up under full attention even when representation entropy stays roughly constant with depth, indicating that the effect is not simply an entropy transition. Practical implications (§ 6 ). In embedding-based pipelines, lexical overlap can miscalibrate similarity signals, degrading performance. In summarization evaluation, common reference-based metrics systematically reward reference wording, favoring surface overlap over semantic preservation. In weight-space model editing, updates generalize along surface-form similarity, producing correlated shifts for token-overlapping distractors, compromising edit locality. 2 Experimental Setup and Preliminaries We use a common set of datasets and model families to ensure consistency across experiments. Datasets. To probe lexical influence, we use two adversarial benchmarks, CounterFact ( 29 ) and SugarCrepe++ (SCPP) ( 6 ) . Relative to CounterFact, which primarily perturbs entities, SCPP offers a more targeted benchmark for lexical-influence tests, as it introduces systematic shifts in attributes and relations. We additionally use CounterFact for model editing. Dataset details and reference samples are provided in Appendix D.1 . To measure lexical decodability, token-identity probes are trained on WikiText ( 30 ) , and to evaluate semantic fidelity, the MTEB benchmark ( 32 ) is utilized. Models. We consider three training paradigms: pretrained , instruction-tuned , and metric learning trained embedding models, spanning multiple model families. 1 1 1 Llama 3.2 ( 10 ) and Gemma 3 ( 46 ) in both pretrained and instruction-tuned variants, along with embedding models including Qwen-3 ( 55 ) (Qwen3-Embedding-8B) and KaLM ( 56 ) (KaLM-Embedding-Gemma3-12B-2511). Note: KaLM modifies the Gemma3 architecture to use full attention. 3 Measuring Lexical Influence In the context of learned representations, we define lexical influence as a semantic failure case in which an anchor prompt is embedded closer to a lexically overlapping distractor than to its meaning-preserving paraphrase, violating the intended triplet ordering. We measure this effect on a dataset of triplets \mathcal{D}=\{(a,p,d)\} , where a is an anchor, p a meaning-preserving paraphrase, and d a lexical distractor. Dataset usage

Read the complete text