Statistical Foundations of DIME: Risk Estimation for Practical Index Selection
Giulio D’Erasmo, Cesare Campagnano, Antonio Mallia, Pierpaolo Brutti, Nicola Tonellotto, Fabrizio Silvestri · 2026
Read Statistical Foundations of DIME: Risk Estimation for Practical Index Selection by Giulio D’Erasmo, Cesare Campagnano, Antonio Mallia, Pierpaolo Brutti, Nicola Tonellotto, Fabrizio Silvestri (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-05649.txt.
Introduction
Giulio D'Erasmo and 5 co-authors give, in a 2026 paper, a statistical foundation for choosing which embedding dimensions to keep during search. Dense embeddings power modern information retrieval, yet many of their dimensions are noisy or redundant. The DIME method scores dimension importance per query, but originally needed a costly grid search to fix one size for all queries.
The authors replace that search with a statistically grounded criterion applied at inference time. Each query gets its own optimal set of dimensions. Tests reportedly matched the original method's effectiveness while cutting embedding size by about half on average across models and datasets. Smaller embeddings mean cheaper storage and faster retrieval for real-world search systems.
Catalog record
| CREATOR | Giulio D’Erasmo, Cesare Campagnano, Antonio Mallia, Pierpaolo Brutti, Nicola Tonellotto, Fabrizio Silvestri |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 31 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-05649
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Statistical Foundations of DIME: Risk Estimation for Practical Index Selection Giulio D’Erasmo, Cesare Campagnano, Antonio Mallia, Pierpaolo Brutti, Nicola Tonellotto, Fabrizio Silvestri Abstract High-dimensional dense embeddings have become central to modern Information Retrieval, but many dimensions are noisy or redundant. Recently proposed DIME (Dimension IMportance Estimation), provides query-dependent scores to identify informative components of embeddings. DIME relies on a costly grid search to select a priori a dimensionality for all the query corpus’s embeddings. Our work provides a statistically grounded criterion that directly identifies the optimal set of dimensions for each query at inference time. Experiments confirm achieving parity of effectiveness and reduces embedding size by an average of ∼ 50 % \sim 50\% across different models and datasets at inference time 1 1 1 Code available at https://github.com/giulio-derasmo/RDIME . 1 Introduction In Information Retrieval (IR) Dense Retrieval (DIR) represents a significant advancement that is driven by the use of high-dimensional embedding spaces to represent queries and documents. DIR models, for example 7 and 14 , have demonstrated a competitive trade-off between effectiveness and efficiency in numerous benchmarks. However, observations have indicated that not all dimensions are equally useful, and some can even be harmful 15 ; 18 ; 24 . Research in this area has explored compressing encoder embeddings by projecting them onto a lower-dimensional manifold. For instance, one can use Principal Component Analysis (PCA) or autoencoders designed to learn and represent such a manifold 16 ; 22 ; 1 ; 4 . Although these approaches reduce retrieval performance, they do contribute to improved efficiency. In contrast, 11 theorise the existence of a query-dependent lower-dimensional manifold within the original embedding space, where retrieval performance also increases. Their work introduces Dimension Importance Estimation (DIME), a method that scores each dimension by fusing the embeddings of the query and a relevant document through element-wise multiplication. These scores can then be used to rank dimensions, retaining only those most likely to contribute to effective retrieval. One major drawback of this approach is that the exact number of dimensions to retain cannot be fixed in advance, and previous work often resorts to evaluating a grid of candidate dimensionalities or simply keeping a fixed percentage of dimensions. The current study addresses this limitation. In Section 3 , we prove that DIME estimates the squared latent signal, that is, the true information need underlying the query, which the embedding aims to capture. Unlike prior approaches, which do not specify the number of dimensions in advance, our Risk Dimension Importance Estimation (RDIME) provides a theoretically grounded criterion for identifying the dimensions to be retained. We further generalize DIME using a kernel-based formulation, which allows for rigorous weighted contributions from relevant documents and thereby improves the estimator’s robustness. We validate this criterion in Section 5 . 2 Background DIME In recent years, 11 conjectured the Manifold Clustering Hypothesis, which posits that high-dimensional embeddings lie in a query-dependent low-dimensional manifold where retrieval is more effective. To discover this manifold, Dimension Importance Estimation (DIME) assigns importance scores to each embedding dimension using a query-dependent function \bm{u}_{q} . The importance scores should estimate the contribution of each dimension to retrieving relevant documents for the given query \bm{q} . They propose different methods for estimating the importance of dimensions in dense retrieval systems. These methods are based on reinforcing the query signal with a specific document \bm{p}\!\in\!\mathbb{R}^{p} as: \bm{u}_{q}=\bm{q}\odot\bm{p}. Two examples are PRF and LLM DIMEs. The first one, inspired by 21 Pseudo-Relevance Feedback (PRF), assumes that the top M documents \bm{d}_{1},\ldots,\bm{d}_{M} retrieved by a similarity measure such as BM25 20 are likely relevant to the query. We shall refer to them as pseudo-relevant documents. Their centroid is used to reinforce the original signal. The second one, instead, uses a synthetic document generated by a Large Language Model (LLM). Although PRF can enhance the retrieval effectiveness of IR systems, it suffers from several limitations. 9 address this by designing a scoring function that incorporates irrelevant feedback. In parallel work, 3 propose a different weighting scheme to avoid the non-tunable parameter k . Top- k Thresholding Following DIME, a Top- k thresholding strategy is employed to select the most informative dimensions. Specifically, given the importance scores \bm{u}_{q} , S=\operatorname{Top-}k(\bm{u}_{q})\subseteq\{1,\ldots,p\} (1) where S contains the indices of the k dimensions with highest importance score in \bm{u}_{q} . However, k must be fixed globally for all queries and cannot be tuned using a validation set, limiting the method’s adaptability. Modulation Estimators The recovery of true signals from noisy observations is a central problem in sparse and redundant representations theory 2 ; 10 . One approach to address this challenge is the modulation estimators framework. In this framework, a random vector \bm{X}\!\in\!\mathbb{R}^{p} represents the noisy observation, where the dimensions X_{i} are independent with \mathbb{E}[X_{i}]=\xi_{i} and \text{Var}(X_{i})=\sigma^{2} for every i\in\{1,\ldots,p\} . Given a modulator function (f_{i})_{i=1}^{p}\in[0,1]^{p} , the goal is to produce a linear estimator with components f_{i}X_{i} for the corresponding true signal components \xi_{i} , such that the \ell^{2} risk, \mathbb{E}\big[\sum_{i=1}^{p}(\xi_{i}-f_{i}X_{i})^{2}\big] , is minimised. Under the assumption that only a few dimensions carry signal information, 8 proposed the hard-threshold estimator, defined component-wise as: f_{i}X_{i}=\mathbf{1}_{S}(X_{i})X_{i},\quad S=\{x\in\mathbb{R}\ :\ |x|>\lambda\} for some threshold \lambda>0 . We shall show that DIME can be interpreted as a modulation estimator of the query’s latent signal, which represents a user’s information need. 3 Method In this section, we formalize the problem of estimating latent information needs from noisy query embeddings. We introduce a hard thresholding approach that retains only the most informative components, providing a practical strategy for denoising queries. 3.1 IR for Modulation Estimators Let \bm{\theta}\!\in\!\mathbb{R}^{p} denote the query’s latent signal, or in other words, the user’s information need. The query embedding \bm{q}\!\in\!\mathbb{R}^{p} , which is obtained through an encoder, can be modeled as \bm{q}=\bm{\theta}+\varepsilon\bm{z},\quad\bm{z}\sim\mathcal{N}(0,I_{p}), where \varepsilon controls the noise level and \bm{z} is standard Gaussian noise. The query embedding is then interpreted as a noisy sample of the underlying information need. Theorem 1 (Hard Thresholding Estimator) . Given the noisy embedding \bm{q} , the optimal hard-threshold estimator \hat{\bm{\theta}}(S^{\star}) of the latent signal \bm{\theta} under \ell_{2} risk is given by: \hat{\theta}_{i}(S^{\star})=q_{i}\mathbf{1}_{S^{\star}}(i)\quad\forall i\in\{1,\ldots,p\}, where S^{\star}=\{i\in\{1,\ldots,p\}\,|\,\theta^{2}_{i}>\varepsilon^{2}\}. (2) Proof. We want to find the best subset S^{\star} that minimises the \ell_{2} risk function: \displaystyle S^{\star} \displaystyle=\underset{S}{\mathrm{argmin}}\sum_{i=1}^{p}\mathbb{E}\big[\hat{\theta}_{i}(S)-\theta_{i}\big]^{2} \displaystyle=\underset{S}{\mathrm{argmin}}\Big\{\sum_{i\in S}\mathbb{E}\big[q_{i}-\theta_{i}\big]^{2}+\sum_{i\notin S}\mathbb{E}\big[\theta_{i}^{2}\big]\Big\} \displaystyle=\underset{S}{\mathrm{argmin}}\Big\{|S|\varepsilon^{2}+\sum_{i\notin S}\theta_{i}^{2}\Big\} Each coordinate i contributes \varepsilon^{2} to the risk if retained, and \theta_{i}^{2} if discarded. Therefore, to minimise the risk, we should include i in S^{\star} if and only if \theta^{2}_{i}>\varepsilon^{2} , which yields ( 2 ). ∎ In order to be applied, Theorem 2 requires knowledge of the set S^{\star} defined in Eq. 2 , which depends on the unknown true signal \bm{\theta} . We show in the next section that DIME serves as an estimator of \bm{\theta}^{2} and it can then be directly applied in Eq. 2 . 3.2 DIME as an Estimator of Squared Latent Signal Let the query \bm{q}\!\in\!\mathbb{R}^{p} and \mathcal{D}^{M}=\{\bm{d}^{(i)}\}_{i=1}^{M}\subset\mathbb{R}^{p} its top- M (pseudo) relevant documents. According to the Probability Ranking Principle 19 , an IR system should rank documents in decreasing order of relevance to the query. Here, we interpret relevance as inversely related to variance: highly relevant documents are modeled as having lower noise. In comparison, less relevant documents have higher noise. Formally, we model each document