Probability Signature: Bridging Data Semantics and Embedding Structure in Language Models

Junjie Yao, Zhi-Qin John Xu · 2025

Read Probability Signature: Bridging Data Semantics and Embedding Structure in Language Models by Junjie Yao, Zhi-Qin John Xu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-20124.txt.

Introduction

Junjie Yao and Zhi-Qin John Xu wrote this 2025 arXiv paper on how language models organize meaning. Inside these models, each token is a point in an embedding space. Points for digits, for example, line up in their natural order. Why such structure forms has stayed unclear.

The authors propose probability signatures, summaries of how tokens appear together in data. These signatures, they argue, mirror the semantic links between tokens. In small models trained on a composite addition task, the signatures shaped the embedding layout strongly. Theory on gradient flow backed up the experiments. The team then trained a Qwen model on parts of the Pile corpus. There too, signatures aligned closely with the learned embedding structure, above all for strong pairwise links.

Catalog record

CREATORJunjie Yao, Zhi-Qin John Xu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE62 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-20124

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Probability Signature: Bridging Data Semantics and Embedding Structure in Language Models Junjie Yao, Zhi-Qin John Xu Abstract The embedding space of language models is widely believed to capture the semantic relationships; for instance, embeddings of digits often exhibit an ordered structure that corresponds to their natural sequence. However, the mechanisms driving the formation of such structures remain poorly understood. In this work, we interpret the embedding structures via the data distribution. We propose a set of probability signatures that reflect the semantic relationships among tokens. Through experiments on the composite addition tasks using the linear model and feedforward network, combined with theoretical analysis of gradient flow dynamics, we reveal that these probability signatures significantly influence the embedding structures. We further generalize our analysis to large language models (LLMs) by training the Qwen2.5 architecture on the subsets of the Pile corpus. Our results show that the probability signatures are faithfully aligned with the embedding structures, particularly in capturing strong pairwise similarities among embeddings. Our work uncovers the mechanism of how data distribution guides the formation of embedding structures, establishing a novel understanding of the relationship between embedding organization and semantic patterns. 1 Introduction In recent years, deep neural network-based large language models (LLMs) have demonstrated remarkable performance ( Comanici et al. 2025 ; OpenAI et al. 2024 ; DeepSeek-AI et al. 2025 ) . The development of these models has largely followed what Richard Sutton termed “the bitter lesson”–that the most effective approach to improving AI performance has historically been to leverage greater computational resources, larger models, and more data, rather than incorporating human knowledge or specialized architectures ( Sutton 2019 ) . This trend has been formalized through scaling laws, which quantify the relationship between model performance and factors such as model size, dataset size, and computational budget through power law relationships ( Kaplan et al. 2020 ) . While these scaling laws provide valuable quantitative predictions for model performance, they also reveal a concerning limitation: the power law relationship suggests that achieving further significant improvements may require prohibitively large increases in model and data size, making continued scaling increasingly impractical and resource-intensive. One promising approach to address these limitations is to develop a deeper understanding of the underlying mechanisms that drive transformer models’ success in natural language processing (NLP). The No Free Lunch theorem establishes that no single algorithm can perform optimally across all problem domains, highlighting the fundamental importance of understanding both the characteristics of the data and the properties of the algorithms that process it ( Wolpert & Macready 1997 ) . Recent research has made significant progress in uncovering key properties of deep learning models, including the edge of stability phenomenon ( Wu et al. 2018 ; Cohen et al. 2021 ) , frequency principle ( Xu et al. 2020 ; Xu et al. 2025a ) , attention patterns ( Elhage et al. 2021 ; Olsson et al. 2022 ; Bhojanapalli et al. 2020 ) , parameter distribution properties ( Kovaleva et al. 2021 ; Dar et al. 2023 ) , condensation phenomenon ( Luo et al. 2021 ; Xu et al. 2025b ) , and embedding structure ( Cai et al. 2021 ) . There has also been some investigation into data characteristics—such as the power-law decay of correlations between elements (like pixels) as a function of their distance ( Ruderman 1994 ) . However, there is a significant gap in understanding how these two fundamental aspects—algorithmic properties and data characteristics—interact to produce the remarkable performance. The embedding space, which acts as the encoder of the language tokens, therefore provides an ideal entry point for investigating how algorithmic properties and data characteristics interact. Ideally, the structure of embeddings should reflect the semantic relationships among tokens. A concrete example involves digits such as 0,1,2,\dots , which possess a natural ordering. Their embedding vectors accordingly display an ordered structure consistent with this numerical sequence, reflecting basic reasoning capabilities in mathematical tasks ( Mikolov et al. 2013b ; Ethayarajh et al. 2019 ; Zhang et al. 2024 ; Yao et al. 2025 ) . However, the cause of the consistency between embedding structures and semantic structures is still an open question, and the driving factors of the embedding structure are still not well characterized. In this work, we identify a set of probability signatures that encode the semantic information in the data into the structure of the embedding space of language models. Such probability signatures are constructed based on label distribution, input distribution, input/output co-occurrence distribution, etc, that systematically capture inherent token-level relationships and reflect semantic structures. This result is achieved via utilizing an embedding-based model with gradient flow analysis of embedding vectors and unembedding vectors for well-designed variable-controlled experiments. We further extend our findings to LLMs with realistic corpora, such as the Qwen2.5 architecture ( Team 2024 ) and subsets of the Pile dataset ( Gao et al. 2020 ; Biderman et al. 2022 ) . The analysis approach with the controlled experiments offers a promising methodology to uncover more and more probability signatures that can bridge the data semantics and embedding structure in language models. 2 Related Work Parameter analysis in LLMs Investigating the underlying parameter properties in LLMs is crucial for understanding the foundation of models. Some works focus on the specific modules in models. Elhage et al. 2021 ; Olsson et al. 2022 uncover mechanisms such as induction heads from the attention module. Bhojanapalli et al. 2020 reveals the rank-collapse phenomenon of the attention matrix. Geva et al. 2021 ; Geva et al. 2022 ; Dai et al. 2022 investigates the characteristics and functions of the FFN in LLMs. Additionally, analysis of a single neuron has been widely employed in mechanism interpretation, particularly in circuits analysis Hanna et al. 2023 ; Wang et al. 2023 ; Hanna et al. 2024 ; Wang et al. 2025 , sparse autoencoders (SAE) Huben et al. 2024 ; Bricken et al. 2023 , transcoders Dunefsky et al. 2024 , and cross-layer transcoders (CLT) Ameisen et al. 2025 . There are also some studies investigating the global properties of all parameters. Dar et al. 2023 ; Katz et al. 2024 introduce a framework for interpreting all parameters of Transformer models by projecting them into the embedding space. Kovaleva et al. 2021 ; Yu et al. 2025 provide an analysis of the parameter distribution, demonstrating the significance of these outliers. In this work, we will focus on the embedding space, explaining the formation of its structure from both experimental and theoretical perspectives. Embedding structure and representation learning Since the introduction of static word embeddings by Mikolov et al. 2013a ; Pennington et al. 2014 and the adoption of contextualized embeddings ( Devlin et al. 2019 ; Peters et al. 2018 ) , significant attention has been devoted to analyzing embedding properties. Gao et al. 2019 ; Ethayarajh 2019 ; Timkey & van Schijndel 2021 explore the anisotropy of embedding space, while Cai et al. 2021 show that embeddings exhibit isotropy within clusters. Liu et al. 2022 offers insights into grokking by emphasizing the role of well-organized embedding structures. Zhang et al. 2024 establishes a connection between embedding structure and model generalization, and Yao et al. 2025 provides an analysis of

Read the complete text