Enhancing Lexicon-Based Text Embeddings with Large Language Models
Yibin Lei, Tao Shen, Yu Cao, Andrew Yates · 2025
Read Enhancing Lexicon-Based Text Embeddings with Large Language Models by Yibin Lei, Tao Shen, Yu Cao, Andrew Yates (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-09749.txt.
Introduction
Yibin Lei, Tao Shen, Yu Cao, and Andrew Yates present a new kind of text embedding in a 2025 paper. Embeddings turn text into number lists so computers can search and match meaning. Most are dense lists with no readable parts, but lexicon-based versions tie each number to a word, which keeps them readable and fast to search.
Their system, LENS, is the first of its kind built from a large language model. It groups near-duplicate words into clusters so each number stands for one word family. On a major benchmark, LENS beat dense embeddings while staying compact. Its size can also be trimmed without special training tricks. Blending LENS with a dense model gave the best search scores in the study.
Catalog record
| CREATOR | Yibin Lei, Tao Shen, Yu Cao, Andrew Yates |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 49 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-09749
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Enhancing Lexicon-Based Text Embeddings with Large Language Models Yibin Lei, Tao Shen, Yu Cao, Andrew Yates Abstract Recent large language models (LLMs) have demonstrated exceptional performance on general-purpose text embedding tasks. While dense embeddings have dominated related research, we introduce the first lexicon-based embeddings (LENS) leveraging LLMs that achieve competitive performance on these tasks. LENS consolidates the vocabulary space through token embedding clustering to handle the issue of token redundancy in LLM vocabularies. To further improve performance, we investigate bidirectional attention and various pooling strategies. Specifically, LENS simplifies lexical matching with redundant vocabularies by assigning each dimension to a specific token cluster, where semantically similar tokens are grouped together. Extensive experiments demonstrate that LENS outperforms dense embeddings on the Massive Text Embedding Benchmark (MTEB), delivering compact representations with dimensionality comparable to dense counterparts. Furthermore, LENS inherently supports efficient embedding dimension pruning without any specialized objectives like Matryoshka Representation Learning. Notably, combining LENS with dense embeddings achieves state-of-the-art performance on the retrieval subset of MTEB (i.e., BEIR). 1 1 1 Our code and models are available at https://github.com/Yibin-Lei/LENS . 1 Introduction Text embeddings are vector representations of text that power a wide range of applications, including retrieval, question answering, semantic textual similarity, and clustering. Recent advances in LLMs have shown that a single model can generate embeddings excelling across diverse tasks, highlighting their versatility ( Wang et al., 2023 ; Li et al., 2024 ; BehnamGhader et al., 2024 ; Lei et al., 2024 ; Muennighoff et al., 2024 ; Meng et al., 2024 ; Lee et al., 2024a ) . Figure 1: The redundancy and noise in LLM tokenizers, as well as the absence of bidirectional dependencies in causal LLMs motivate LENS. While dense embeddings that encode texts into low-dimensional, real-valued latent semantic spaces dominate recent research, lexicon-based embeddings ( Formal et al., 2021a ; Formal et al., 2021b ; Shen et al., 2023a ; Lassance et al., 2024 ) offer distinctive advantages. These high-dimensional representations, where each dimension corresponds to a specific token of the vocabulary, align more closely with the pre-training objectives of language models due to their shared use of the vocabulary space and the language modeling head ( Shen et al., 2023a ) . Recent studies have demonstrated that lexicon-based embeddings can surpass their dense counterparts, utilizing masked language models under specific control ( Déjean et al., 2023 ) . Additionally, lexicon-based embeddings can offer better transparency, providing clearer insights into the representations via the weight of each token. Moreover, the combination of dense and lexicon-based embeddings has also been shown to be promising in prior studies, as they effectively complement each other Lin (2021) ; Shen et al. (2023b) ; Gao et al. (2021) . Despite these benefits, lexicon-based embeddings remain underexplored beyond retrieval tasks. To effectively apply them to other tasks, it is essential to address the challenges posed by LLMs, as shown in Fig. 1 . The first issue is the inherent redundancy of LLM vocabularies. Since most modern tokenizers rely on subword tokenization (e.g., ‘‘education’’ is split into ‘‘edu’’ and ‘‘cation’’ ), it fragments the entire vocabulary space ( Soler et al., 2024 ) . More importantly, semantically equivalent tokens can appear in multiple forms in the tokenizer (e.g., ‘‘what’’ , ‘‘What’’ , ‘‘ what’’ and ‘‘review’’ , ‘‘reviews’’ ), introducing inconsistencies and difficulties in lexicon matching (e.g., the choice of whether ‘‘what’’ and/or ‘‘ what’’ appears in the lexical representation should be consistent along with their associated weights). Consequently, recent studies indicate that replacing the original tokenization of BM25 ( Robertson et al., 1995 ) with the XLM-R tokenizer ( Conneau et al., 2020 ) can lead to a significant performance drop due to the noisier vocabulary ( Chen et al., 2024 ) . The second challenge is that LLMs typically employ unidirectional attention during pre-training, where tokens only attend to preceding tokens. This limitation prevents each token from fully leveraging the surrounding context, which is crucial as lexicon-based embeddings are derived from the outputs of all tokens. To address these challenges, we first explore the potential of LLMs generating embeddings where each dimension corresponds to a token cluster instead of the traditional single token, with each cluster grouping tokens that share similar meanings or stem from the same lexeme. To achieve this, we utilize a simple yet effective approach that directly clusters the token embeddings and leverages the centroids of these clusters as the new token embeddings for the language modeling head. As shown in Table 4 , the resulting clusters naturally group tokens with similar meanings, forming more coherent and compact embeddings. These cluster-based embeddings can achieve the equivalent feature size as dense embeddings from the same LLM (e.g., 4,000d), which is much smaller than previous lexicon-based embeddings. Such a property i) facilitates the integration of LENS into existing dense frameworks like FAISS, reducing the need for sparsity constraints, and also ii) eliminates computational overhead in tasks such as clustering and classification, where inverted indices are not a natural fit. Given recent studies highlighting the significant impact of attention mechanisms and pooling strategies on dense embeddings ( Li et al., 2024 ; Muennighoff et al., 2024 ; BehnamGhader et al., 2024 ; Lee et al., 2024a ) , we incorporate variants of these two factors in our framework to examine how they affect lexicon-based embeddings. Contrary to prior findings ( Li et al., 2024 ) , which suggest that preserving the original architecture of LLMs typically yields optimal performance for dense embeddings, our results indicate that bidirectional attention is critical for achieving superior performance with lexicon-based embeddings. Built on these techniques, we introduce LENS, a framework designed to generate low-dimensional lexicon-based embeddings that achieve impressive results across a variety of tasks. Specifically, our experiments demonstrate that LENS outperforms dense embeddings on the Massive Text Embedding Benchmark (MTEB) ( Muennighoff et al., 2023 ) , achieving state-of-the-art (SOTA) zero-shot performance among models trained exclusively on public data, as of December 1, 2024. These results demonstrate that sparse lexicon-based embeddings can perform competitively with dense embeddings across a diverse range of tasks. Qualitative examples also illustrate that LENS produces grounded and meaningful representations. Our analysis also shows that LENS, even when using 2000 clusters, still outperforms embeddings that leverage the original vocabulary space. Efficiency can be further improved through Top-K pruning: by retaining only the top 256 of 4000 dimensions at inference time, LENS maintains high performance with minimal degradation. This sparsification requires no specialized training (e.g., Matryoshka Representation Learning Kusupati et al. (2022) ) and arises naturally from its lexicon-based design. Finally, combining LENS with dense embeddings achieves SOTA performance on the retrieval subset of MTEB (specifically, BEIR). 2 Related Work Lexicon-Based Embeddings. Lexicon-based embeddings assign each dimension of the embedding vector to a specific token in the vocabulary. With advances in masked language models, recent studies have demonstrated that lexicon-based embeddings ( Mallia et al., 2021 ; Lin and Ma, 2021 ; Zhuang and Zuccon, 2021 ; Formal et al., 2021a ; Formal et al., 2021b ; Shen et al., 2023a ; Nguyen et al., 2023a ; Lassance et al., 2024 ; Nguyen et al., 2023b ; Nguyen et al., 2024 ; Lupart et al., 2024 ) can deliver