CoRECT: A Framework for Evaluating Embedding Compression Techniques at Scale

Laura Caspari, Michael Dinzinger, Kanishka Ghosh Dastidar, Christofer Fellicious, Jelena Mitrović, Michael Granitzer · 2025

Read CoRECT: A Framework for Evaluating Embedding Compression Techniques at Scale by Laura Caspari, Michael Dinzinger, Kanishka Ghosh Dastidar, Christofer Fellicious, Jelena Mitrović, Michael Granitzer (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-19340.txt.

Introduction

L Caspari and five co-authors present CoRECT. It is a frame for testing embedding compression at realistic scale. Dense retrieval systems answer searches well. But they need huge memory for their indices. Compression can shrink those indices a lot. The loss in ranking quality can stay small.

Older studies often ignore how hard the corpus is. Corpus size and document length both matter. CoRECT tests compression under controlled complexity. A newly curated dataset backs the tests. Eight compression types are benchmarked. No single winner emerges everywhere. Search builders can pick with eyes open.

Catalog record

CREATORLaura Caspari, Michael Dinzinger, Kanishka Ghosh Dastidar, Christofer Fellicious, Jelena Mitrović, Michael Granitzer
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE31 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-19340

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CoRECT: A Framework for Evaluating Embedding Compression Techniques at Scale Laura Caspari, Michael Dinzinger, Kanishka Ghosh Dastidar, Christofer Fellicious, Jelena Mitrović, Michael Granitzer Abstract Dense retrieval systems have proven to be effective across various benchmarks, but require substantial memory to store large search indices. Recent advances in embedding compression show that index sizes can be greatly reduced with minimal loss in ranking quality. However, existing studies often overlook the role of corpus complexity – a critical factor, as recent work shows that both corpus size and document length strongly affect dense retrieval performance. In this paper, we introduce CoRECT ( Co ntrolled R etrieval E valuation of C ompression T echniques), a framework for large-scale evaluation of embedding compression methods, supported by a newly curated dataset collection. To demonstrate its utility, we benchmark eight representative types of compression methods. Notably, we show that non-learned compression achieves substantial index size reduction, even on up to 100M passages, with statistically insignificant performance loss. However, selecting the optimal compression method remains challenging, as performance varies across models. Such variability highlights the necessity of CoRECT to enable consistent comparison and informed selection of compression methods. All code, data, and results are available on GitHub 1 1 1 https://github.com/padas-lab-de/CoRECT and HuggingFace. 2 2 2 https://huggingface.co/datasets/PaDaS-Lab/CoRE 1 Introduction The success and wide-spread adoption of large language models (LLMs) has caused a shift in the field of information retrieval, moving from sparse, keyword-based matching to dense retrieval methods. Although effective, such systems require large amounts of memory to store embedding-based indices, which can easily surpass the size of the underlying data. To overcome memory-related scalability constraints, researchers have developed a range of embedding compression techniques that achieve significant size reductions with little loss in retrieval quality [ 14 , 17 , 22 , 23 , 24 , 31 , 33 , 34 , 35 , 37 ] . While these methods show strong results on individual corpora, their systematic evaluation across a broader range of models and datasets is still lacking. In addition, they overlook how retrieval performance scales with corpus complexity, which may cause an overestimation of effectiveness on large and heterogeneous collections [ 19 , 26 ] , particularly when vector compression limits embedding expressiveness. Figure 1 : The CoRECT framework. In this work, we critically re-examine recent claims regarding the effectiveness of embedding compression methods, including vector quantization, dimensionality reduction, and Learning-To-Hash (LTH) techniques. We introduce CoRECT , a framework for systematically evaluating and comparing compression methods across diverse datasets. Our main objective is to investigate how these techniques perform under varying levels of corpus complexity – an aspect often overlooked in prior studies. To this end, we operationalize corpus complexity in two ways: (1) by scaling the corpus size from 10K to 100M passages and up to 10M documents, and (2) by comparing performance across both passage-level and document-level retrieval tasks (see Figure 1 ). To evaluate performance along these dimensions, we introduce a newly curated dataset collection named Controlled Retrieval Evaluation (CoRE) , constructed through targeted subsampling of MS MARCO v2 3 3 3 https://microsoft.github.io/msmarco/TREC-Deep-Learning.html and comprising 76 human-judged TREC DL queries. CoRE serves as the primary evaluation dataset in CoRECT. To emphasize the robustness and generalizability of our findings, we further complement CoRE with datasets from BeIR [ 29 ] . Finally, to demonstrate the utility of CoRECT, we apply the framework to a diverse and representative set of embedding compression methods. As such, our main contributions are as follows: 1. The Controlled Retrieval Evaluation of Compression Techniques (CoRECT) framework, designed to systematically evaluate compression methods. To ensure robustness, we further extend our evaluation by including the domain-diverse corpora of the BeIR benchmark [ 29 ] . 2. The dataset collection CoRE, the cornerstone of CoRECT, is used to assess the effect of corpus complexity on retrieval performance. 3. We apply CoRECT on four different models using eight representative types of compression techniques in over 40 combinations and find that choosing the wrong compression method can significantly impact retrieval performance. Furthermore, we show that there is no single best method that fits all models. 2 Related Work The following section reviews embedding compression methods, their evaluation, and related research on the influence of corpus complexity on performance. Compression Methods. The compression techniques discussed below fall into three broad categories. The first group reduces the precision of embeddings by representing each dimension with a lower number of bits, in the following called Scalar and Binary Quantization (SBQ). This encompasses type-casting floating points, i.e. to float8 [ 21 ] , or mapping them to discrete integer values, i.e. uint8 . Its simplicity makes SBQ a popular compression approach supported by various libraries and vector databases, such as FAISS [ 8 ] , OpenSearch 4 4 4 https://opensearch.org/ or Milvus 5 5 5 https://milvus.io/ , as well as recent embedding models through quantization-aware training [ 2 , 12 ] . The second group encompasses approaches for dimensionality reduction which remove certain dimensions from the embedding vector. The most common method in the context of data analysis is Principal Component Analysis (PCA), which can also be used to compress embedding vectors. More recently, Matryoshka Representation Learning (MRL) [ 17 , 35 ] , which is integrated into model training, has found wide-spread adoption in contemporary embedding models [ 2 , 12 , 18 , 36 , 39 ] . The third and last group consists of regular hashing and Learning-to-Hash (LTH) methods. A simple baseline is Locality-Sensitive Hashing (LSH), whose numerous proposed variants and extensions make it a well-studied compression methods [ 3 , 6 , 7 , 11 , 13 ] . Regarding LTH methods, Product Quantization (PQ) [ 15 ] has emerged as a popular method with its own proposed expansions like Joint optimization of query encoding and Product Quantization (JPQ) [ 37 ] or Distill-VQ [ 33 ] . Yamada et al. [ 34 ] introduce the Binary Passage Retriever (BPR) to binarize embeddings generated by the Dense Passage Retriever [ 16 ] . Thakur et al. [ 28 ] enhance prior work by introducing domain adaptation modules, enabling methods like BPR and JPQ to perform effectively in zero-shot retrieval settings. Evaluation of Compression Methods. Due to their popularity, SBQ methods have received growing attention in both academic studies and industry reports [ 14 , 22 , 23 , 24 , 31 ] , demonstrating that high compression ratios can be achieved at minimal performance loss. Papers proposing new compression techniques tend to emphasize improvements over previous approaches [ 28 , 33 , 34 , 37 ] , like improving over standard PQ. However, most of these evaluations are limited to a small number of datasets and methods and do not consider scalability. To the best of our knowledge, the only thorough study was conducted by Zhang et al. [ 38 ] , who evaluate compression methods in the context of deep learning recommendation models. Additionally, they compare a small set of methods in a Retrieval Augmented Generation (RAG) setting on a single dataset, using exact match as a metric. However, due to their focus on recommendation, the chosen evaluation metrics and models are not representative of those commonly used in retrieval.

Read the complete text