Bridging Compositional and Distributional Semantics: A Survey on Latent Semantic Geometry via AutoEncoder

Yingji Zhang, Danilo S. Carvalho, André Freitas · 2025

Read Bridging Compositional and Distributional Semantics: A Survey on Latent Semantic Geometry via AutoEncoder by Yingji Zhang, Danilo S. Carvalho, André Freitas (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-20083.txt.

Introduction

Yingji Zhang, Danilo Carvalho, and André Freitas survey ways to give language model internals a clearer semantic structure in a 2025 paper. Today's models store meaning as patterns of numbers that work well but resist interpretation. Symbolic, compositional accounts of meaning are clear but rigid. The survey bridges the two.

It reviews three autoencoder designs that compress text into latent codes: the variational, the vector-quantised, and the sparse autoencoder. Each shapes its inner space differently, with consequences for interpretability and control. The authors compare those geometries against what compositional semantics needs. Their map of the field aims to help builders design models whose inner spaces humans can read and steer.

Catalog record

CREATORYingji Zhang, Danilo S. Carvalho, André Freitas
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-20083

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Bridging Compositional and Distributional Semantics: A Survey on Latent Semantic Geometry via AutoEncoder Yingji Zhang, Danilo S. Carvalho, André Freitas Abstract Integrating compositional and symbolic properties into current distributional semantic spaces can enhance the interpretability, controllability, compositionality, and generalisation capabilities of Transformer-based auto-regressive language models (LMs). In this survey, we offer a novel perspective on latent space geometry through the lens of compositional semantics, a direction we refer to as semantic representation learning . This direction enables a bridge between symbolic and distributional semantics, helping to mitigate the gap between them. We review and compare three mainstream autoencoder architectures— Variational AutoEncoder (VAE), Vector Quantised VAE (VQVAE), and Sparse AutoEncoder (SAE)—and examine the distinctive latent geometries they induce in relation to semantic structure and interpretability. 1 Introduction Representation Learning. Representation learning encompasses a broad set of techniques aimed at automatically discovering useful features and structures from complex, high-dimensional data. The core objective is to transform raw inputs into compact, informative representations that capture the underlying geometry, patterns, and semantic relationships within the data Bengio et al. 2013 ; Alshammari et al. 2025 . By learning to encode the essential factors of variation, representation learning enables models to generalise more effectively and supports interpretability and modularity across a wide range of domains, including vision Radford et al. 2021 , speech Liao et al. 2024 , language Yang et al. 2025a , and multimodal learning Yang et al. 2025b ; Liu et al. 2025 . Compositional Semantics. In the field of natural language learning and generation, improving the geometric structure of semantic features within the representation space can enhance the controllability and interpretability of language models (LMs), which has been widely explored in the task, such as style-transfer Liu et al. 2023 ; Gu et al. 2023 ; Gu et al. 2022 ; Hu and Li 2021 ; Vasilakes et al. 2022 . Those studies provide a quasi-symbolic, localised, compositional generation control to informal, less-rigid semantic features, such as sentiment and topic. Recently, a line of studies investigated this kind of control over formal semantic features Zhang et al. 2024b ; Zhang et al. 2025a ; Zhang et al. 2024e ; Zhang et al. 2024c ; Zhang et al. 2024d ; Zhang et al. 2025b . Formal/symbolic semantics, such as Montague Semantics Dowty et al. 2012 , Semantic Role Labelling Palmer et al. 2010 , and Argument Structure Theory Jackendoff 1992 , which provides a canonical, granular, and rigid representation of meaning in natural language, has been a central focus of philosophical and linguistic inquiry for thousands of years. Its goal is to systematically capture the logical structure of linguistic expressions and to model how complex meanings arise from simpler parts. All these studies provide valuable insights to bridge the compositional and distributional semantics. Language Models. Over the last few years, Transformer-based auto-regressive LMs with distributional semantic space have emerged as powerful tools that offer a flexible and scalable foundation for addressing a broad and diverse spectrum of natural language tasks Touvron et al. 2023 . However, despite their empirical success, they often operate as opaque black boxes with limited insight into the underlying semantic mechanisms they use. This raises an important question: can we develop language representations or models that incorporate the properties of symbolic, structural, and compositional semantics? Addressing this question yields several important advantages: Interpretability and Controllability: Models grounded in symbolic and structured representations are inherently more controllable. For example, in the sentence: animals require oxygen for survival , the words are functionally combined into sentence semantics: \lambda x(\text{animals}(x)\rightarrow\text{require}(x,\text{oxygen})) where x is the variable of any entity within a logical structure. In this case, we can localise the sentence semantics by replacing x with birds , etc. This localised process has been applied in Natural Logic, which manipulates the surface form of sentences during inference Bowman et al. 2015 ; Angeli and Manning 2014 ; MacCartney and Manning 2007 . Safety and Trustworthiness: As language models are increasingly deployed in high-stakes domains, ensuring their safety and trustworthiness becomes critical. Systems that incorporate formal semantic structures are more likely to produce outputs that are logically coherent and consistent with intended meanings. Symbolic reasoning frameworks can help verify that a system adheres to specified rules or constraints, reducing the likelihood of hallucinations, contradictions, or harmful outputs. This alignment with formal systems of logic and meaning contributes to the reliability and accountability of AI systems Chen et al. 2024 . Composition and Generalisation: One of the enduring strengths of compositional semantics is its ability to handle compositionality—the principle that the meaning of a complex expression is determined by the meanings of its parts and the rules used to combine them. Embedding this principle into language models enhances their ability to generalise to novel combinations and linguistic constructions, beyond those seen during training. In the Math Reasoning task Meadows et al. 2024 , for example, the model is tasked with predicting the output of a mathematical transformation given a specified operation and an input formula during training. However, current LMs often fail to generalise effectively to variable or length out-of-distribution (OOD) inputs Zhang et al. 2024e . Structured semantic representations—such as syntax trees that explicitly encode the function, constant, and variable—can provide a more robust framework for generalisation, enabling models to maintain compositional integrity even in the face of lexical variation. Our Scope. To integrate these potential advantages into current LM, this survey provides a systemic review of studies that focus on explicitly embedding symbolic, compositional, and semantic properties into the latent space of LMs - a direction we term semantic representation learning . We seek to bridge the gap between symbolic semantics and neural representations, facilitating more interpretable and controllable to LMs’ generation. The survey is organised as follows: (a) Section 2 frames the scope of the survey, stating a definition of semantic representation learning; (b) Section 3 reviews the main data resources for tasks requiring compositional semantics; (c) Section 4 provides a detailed classification of the main architectural patterns and approaches proposed for semantic representation learning; (d) Section 5 describes quantitative and qualitative metrics for the evaluation of semantic representation space; (e) Section 6 outlines open challenges and promising directions for future research. Expression Compositional Semantic Tasks Datasets & Description Definitions Definition Modelling(vec2def) Noraset et al. 2016 , Yang et al. 2019 , Mickus et al. 2022 Definition Modelling(word2def) Noraset et al. 2016 , WordNet , Wiktionary , Wikipedia Reverse Dictionary(def2vec) Semeval-2022 Task 1: CODWOE Mickus et al. 2022 Explanations QA with explanation graph WorldTree Jansen et al. 2018 QA with entailment tree EntailmentBank Dalvi et al. 2021 Syllogistic NLI Rule-based Explanatory NLI corpus Zhang et al. 2024d commonsense QA Zebra-KB Molfese et al. 2024 … explanations corpora survey Thayaparan et al. 2020 Others Sentiment Style-transfer Yelp reviews Shen et al. 2017 , Yahoo answers Yang et al. 2017 IMDb movie review Diao et al. 2014 ; Maas et al. 2011 ; Li et al. 2018 Stanford Sentiment Treebank Socher et al. 2013 Formality Style-transfer GYAFC Rao and Tetreault 2018 Codeswitching Style-transfer LinCE Aguilar et al. 2020 Topic Style-transfer AGNews dataset Zhang et al. 2015 , DBpedia Zhang et al. 2015 Toxic Style-transfer Jigsaw Toxic Comment Disentanglement dSentences … style-transfer corpora survey

Read the complete text