Do Language Models Encode Semantic Relations? Probing and Sparse Feature Analysis
2026
Read Do Language Models Encode Semantic Relations? Probing and Sparse Feature Analysis (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-17624.txt.
Introduction
Diera and Scherp probe how language models store semantic relations between concepts. They test synonymy, antonymy, hypernymy, and hyponymy in Pythia-70M, GPT-2, and Llama 3.1 8B. Linear probes pair with sparse autoencoders and activation patching to locate the signals. Hypernymy spreads out and resists removal, while hyponymy sits in compact, fragile features, they find. Signals peak in mid-layers, mostly past the residual stream. Antonymy proved easiest and synonymy hardest.
Catalog record
| YEAR | 2026 |
|---|---|
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40941 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.17624
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Do Language Models Encode Semantic Relations? Probing and Sparse Feature Analysis Abstract Understanding whether large language models (LLMs) capture structured meaning requires examining how they represent concept relationships. In this work, we study three models of increasing scale: Pythia-70M, GPT-2, and Llama 3.1 8B, focusing on four semantic relations: synonymy, antonymy, hypernymy, and hyponymy. We combine linear probing with mechanistic interpretability techniques, including sparse autoencoders (SAE) and activation patching, to identify where these relations are encoded and how specific features contribute to their representation. Our results reveal a directional asymmetry in hierarchical relations: hypernymy is encoded redundantly and resists suppression, while hyponymy relies on compact features that are more easily disrupted by ablation. More broadly, relation signals are diffuse but exhibit stable profiles: they peak in the mid-layers and are stronger in post-residual/MLP pathways than in attention. Difficulty is consistent across models (antonymy easiest, synonymy hardest). Probe-level causality is capacity-dependent: on Llama 3.1, SAE-guided patching reliably shifts these signals, whereas on smaller models the shifts are weak or unstable. Our results clarify where and how reliably semantic relations are represented inside LLMs, and provide a reproducible framework for relating sparse features to probe-level causal evidence. Keywords: semantic relations, interpretability, language models Do Language Models Encode Semantic Relations? Probing and Sparse Feature Analysis Andor Diera, Ansgar Scherp Ulm University Ulm, Germany {firstname.lastname}@uni-ulm.de Abstract content † † footnotetext: The source code and datasets are available at: github.com/drndr/semantic_relation_interpret 1. Introduction Understanding the internal mechanisms by which large language models (LLMs) process and represent knowledge is fundamental to advancing both theoretical insights and practical applications in natural language processing (NLP). Interpretability research addresses this need by providing powerful frameworks for dissecting internal representations, moving beyond black-box evaluations to examine how specific activation patterns emerge in the models Sharkey et al. 2025 . This deeper understanding serves two main objectives: identifying hidden biases embedded within learned representations and developing more precise methods for controlling model behavior and outputs Liu et al. 2024 ; Soo et al. 2025 . While existing interpretability research has made significant progress in identifying individual concepts and feature representations within LLMs Cunningham et al. 2023 ; Gurnee et al. 2023 , the investigation of semantic relationships between these concepts remains largely unexplored. Certain relationships, such as synonymy and antonymy, which test whether models can distinguish similarity from opposition in meaning, and hierarchical relations like hypernymy and hyponymy, are particularly diagnostic of whether models develop structured conceptual understanding or rely on surface-level associations. Existing research suggests that LLMs still face substantial challenges in accurately predicting or encoding such relationships Roussinov et al. 2023 ; Moskvoretskii et al. 2024 ; Cao et al. 2024 . This limitation underscores a broader gap in our understanding of how (or if) LLMs encode relational structure, emphasizing the need for interpretability research to analyze the semantic relationships embedded within model representations. In this paper, we ask whether semantic relation-signals exist in LLMs, where they are localized across layers/components, and how strongly they can be manipulated. To address these questions, we focus on four semantic relations: synonymy, antonymy, hypernymy, and hyponymy. We examine the Pythia-70M, GPT-2-124M, and Llama 3.1-8B models, using linear probing, sparse autoencoders (SAEs), and activation patching. Our results suggest there are no unique “relation layers”; depth profiles are diffuse at middle layers and peak locations are unstable, yet in Llama 3.1 SAE patching reliably induces probe-level causal shifts. Relations differ in profile, spanning clear gaps in probe accuracy, intervention sensitivity, and the relative weight of sufficiency versus necessity. Antonymy is reliably captured by probes and can be causally influenced by sparse features. Synonymy is the hardest to decode and is the least affected by activation patching, indicating a more diffuse encoding. Hierarchical relations diverge: hypernymy is easy to strengthen but hard to suppress; hyponymy strengthens and collapses readily. Although our claims are limited to probe-level causality, the findings still reveal manipulability, stable difficulty ordering, and a bias in hierarchical relations. The main contributions of our paper: • A systematic comparison of how LLMs encode four semantic relations (synonymy, antonymy, hypernymy, and hyponymy) across scales from 70M to 8B parameters. • An integrated probing and sparse feature analysis framework for identifying relation-specific activations and assessing their probe-level causal influence. • Results show that relational information concentrates in post-residual and MLP components but remains diffusely distributed across mid-layers, with broad and uncertain depth peaks. • SAE-guided activation patching isolates relation-specific features and demonstrates probe-level necessity/sufficiency, including asymmetric directionality in hierarchical relationships. 2. Related Work Machine learning interpretability is often approached through two main strategies: concept-based interpretability and reverse engineering Sharkey et al. 2025 . While both aim to make the inner workings of complex models more transparent, they differ in methodology and focus. Concept-based approaches center on aligning internal representations with human-understandable constructs, whereas reverse engineering seeks to uncover the underlying computational mechanisms without relying on predefined concepts. 2.1. Concept-based Interpretability A common strategy for identifying human-interpretable concepts within a model is to train a simple classifier (often referred to as a probe) to detect their presence in intermediate representations. In the context of LLMs, probing has been used to assess encoded linguistic properties Conneau et al. 2018 ; Hewitt and Manning 2019 , factual knowledge Peng et al. 2022 ; Gurnee and Tegmark 2024 , and concept relations Aspillaga et al. 2021 . An alternative approach represents concepts as vectors in the activation space, which enables controlled interventions by shifting activations along specific conceptual directions Bolukbasi et al. 2016 ; Chanin et al. 2024 . Complementary to these methods, concept bottleneck models introduce an explicit intermediate layer of human-defined concepts, enabling models to make predictions through interpretable concept activations Sun et al. 2024 . Despite their appeal, concept-based methods face several limitations. Probes and concept vectors capture correlations rather than true causality Belinkov 2022 , while concept bottlenecks may constrain model capacity or fail to generalize when concept sets are incomplete Yuksekgonul et al. 2023 . In general, concept-based interpretability methods impose predefined conceptual frameworks rather than uncovering the emergent features that a model has learned Sharkey et al. 2025 . 2.2. Mechanistic Interpretability In contrast to concept-based approaches, mechanistic interpretability aims to reverse-engineer the internal computation of models by analyzing circuits, neurons, and attention patterns Sharkey et al. 2025 . In LLMs, this approach seeks to reveal not only what linguistic or factual information is present, but also how such information is processed, represented, and transformed across layers. Among the earliest demonstrations of mechanistic interpretability in transformer-based language models was the discovery of induction heads (attention heads that perform token copying), providing a functional explanation for how LLMs generalize patterns through in-context learning Elhage et al. 2021 . Subsequent research has uncovered a range of interpretable behaviors in LLMs, including mechanisms for object identification Wang et al. 2023 , factual recall Nanda et al. 2023 ; Geva et al. 2023 , and arithmetic functions Stolfo et al. 2023 ; Quirke et al. 2024 . A key challenge in mechanistic interpretability is the polysemanticity of individual neurons, where a single neuron often activates for multiple unrelated features, making it difficult to assign consistent functional roles Elhage et al. 2022 . One promising direction for mitigating this