LLM Sparsity Prior for Robust Feature Selection
Caleb Skinner, Yihan Guo, Meng Li · 2026
Read LLM Sparsity Prior for Robust Feature Selection by Caleb Skinner, Yihan Guo, Meng Li (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-23102.txt.
Introduction
Skinner, Guo, and Li propose the LLM Sparsity Prior, a robust way to use language model weights for variable choice. LLM-Lasso methods degrade when generated weights are off. The authors build a framework for rating weight quality across regimes. Their prior folds those weights into inclusion odds of Spike-and-Slab models via two knobs for global sparsity and weight concentration. Upper-level priors let the model discount misleading weights on the fly. On a private acute kidney injury dataset it improved forecasts and found clinical features baselines missed.
Catalog record
| CREATOR | Caleb Skinner, Yihan Guo, Meng Li |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 46334 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.23102
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
LLM Sparsity Prior for Robust Feature Selection Caleb Skinner, Yihan Guo, Meng Li Abstract Large language models (LLMs) offer a scalable mechanism to elicit domain-informed prior information for high-dimensional variable selection. However, existing methods such as LLM-Lasso are sensitive to weight quality, with performance degrading substantially when LLM-generated weights are inaccurate. To address this challenge, we first introduce a framework for quantifying the quality of LLM-generated weights, enabling rigorous evaluation of LLM-informed methods across varying weight regimes. We then propose the LLM Sparsity Prior (LSP) , which integrates LLM-generated weights into the prior inclusion probabilities of Spike-and-Slab and Spike-and-Slab Lasso models via two interpretable hyperparameters governing global sparsity and weight concentration. Hierarchical hyperpriors on these parameters allow the model to dynamically discount uninformative or misleading weights, improving robustness without sacrificing gains when weights are accurate. Finally, we develop principled prompt engineering strategies and validate the method on a private medical dataset studying Acute Kidney Injury. LSP improves prediction accuracy and identifies clinically relevant features missed by the baselines, with robustness to prompt variation and particular effectiveness in low-data regimes. 1 Introduction The integration of large language models (LLMs) into statistical pipelines has emerged as a significant area of research in machine learning ( 40 ; 27 ) . This trend is driven by the success of pre-trained LLMs across a wide variety of domains and tasks ( 5 ; 26 ; 22 ) . These models excel at encoding domain knowledge and applying it to complex reasoning tasks ( 34 ; 19 ; 31 ) . They are increasingly leveraged to conduct traditional machine learning tasks, including prediction ( 25 ; 33 ; 10 ) , regression ( 13 ) , and statistical inference ( 27 ) . Feature selection is a task particularly well-suited for LLM-informed modeling, especially in the context of low-data-regimes where signal is sparse and standard machine learning models may struggle ( 9 ; 40 ) . For example, 9 extract a prior distribution from natural language descriptions to enhance feature selection in low-data regimes, 18 demonstrate impressive accuracy using only the feature set and a description of the prediction task, and 35 combine classical statistical methods with LLM queries in an iterative approach. The LLM-Lasso ( 40 ) is the first framework to integrate LLM queries and statistical methodologies into a unified feature selection procedure. In this approach, LLM queries generate feature importance weights that encode contextual information derived from feature metadata. These weights are incorporated into the Lasso penalty term, with the degree of reliance on the weights controlled via cross-validation. However, the variance inherent in cross validation-based hyperparameter selection can lead to an over-dependence on the LLM-generated feature weights. These weights directly modify the objective function, and we demonstrate that inaccurate or “poor” weights can result in substantial performance degradation. This paper makes three contributions to the growing field of LLM-assisted machine learning. First, we adapt established feature importance evaluation metrics to quantify the quality of the LLM-generated weights, providing a principled diagnostic tool for analyzing the performance of LLM-informed methods across varying weight quality regimes. Second, we introduce the LLM Sparsity Prior (LSP), a novel framework integrating feature importance weights into the prior inclusion probabilities of Spike-and-Slab and Spike-and-Slab Lasso models. The method utilizes two interpretable hyperparameters: a sparsity parameter controlling the baseline inclusion probability and a concentration parameter regulating weight influence. We propose hierarchical hyperpriors for both parameters that ensure LSP’s robustness to poor weights, while preserving efficient posterior estimation. Simulation studies confirm that LSP remains robust under uninformative or misleading weights, while significantly outperforming baselines when weights are accurate. Third, we develop principled prompt engineering strategies for querying weights and evaluate the generated weights on a private medical dataset studying Acute Kidney Injury. LSP demonstrates robustness to deviations from the recommended prompt engineering strategy and to the natural stochasticity of LLMs. We find that the proposed method improves the performance of both the Spike-and-Slab and Spike-and-Slab Lasso, particularly in low-data regimes. The remainder of this paper is organized as follows. Section 2 details the weight quality framework. Section 3 introduces the LLM Sparsity Prior, hierarchical priors, and posterior estimation strategies. Section 4 presents the simulation results, followed by an application to Acute Kidney Injury (AKI) in Section 5 . We conclude the paper in Section 6 . 2 Measuring Weight Quality Given the proliferation of available LLMs and vast design space of prompting strategies, the variety of generated feature weights is large. LLMs are susceptible to hallucinations and sensitive to prompt variations, thus the derived weights may not accurately capture the ground truth. To assess the performance and robustness of LLM-informed methods to various weights, it becomes necessary to quantify the quality of these weights prior to their integration into statistical pipelines. These quality measurements facilitate rigorous comparison between LLM generation mechanisms and are essential for establishing the statistical properties of LLM-informed methods. We require a weight-quality metric that monotonically increases as weight quality improves and has the precision to permit straightforward comparisons. Let w\in\mathbbm{R}^{p}_{>0} denote a generated weight vector encoding information about p features, where \text{max}(w)>\text{min}(w) . In practice, the elements w_{j} often take values from a discrete set of positive integers, where a larger magnitude suggests a higher probability of inclusion for feature j . We define \gamma^{*}\in\{0,1\}^{p} as the ground truth feature inclusion vector such that \gamma^{*}=1 denotes an active feature in the underlying model and \gamma^{*}=0 denotes an inactive one. Drawing on established feature importance evaluation practices ( 15 ; 7 ) , we adapt two standard metrics to this setting: \phi_{\ell_{1}},\phi_{\text{pairwise}}:\{0,1\}^{p}\times\mathbbm{R}^{p}_{>0}\to[0,1] . For each technique, as \phi\to 1 , the weight quality approaches perfect alignment, whereas \phi\to 0 implies the weights directly oppose the ground truth. The \ell_{1} weight agreement is defined \phi_{\ell_{1}}(\gamma^{*},w)=1-\frac{1}{p}\bigg\|\frac{w-\text{min}(w)}{\text{max}(w)-\text{min}(w)}-\gamma^{*}\bigg\|_{1}. (1) This metric computes the \ell_{1} distance between the ground truth and the min-max scaled weight vector. Notably, a value of \phi_{\ell_{1}}(\gamma^{*},w)\approx 0.5 indicates that the weights contain no material information regarding \gamma^{*} , equivalent to random guessing. The pairwise weight agreement measures the frequency with which the relative ordering of elements in w aligns with the strict ordering of \gamma^{*} . We define the pairwise indicator matrices for w and \gamma^{*} as M_{i,j}^{\gamma^{*}}=\mathbbm{1}(\gamma_{i}^{*}>\gamma_{j}^{*})\quad\quad\quad M_{i,j}^{w}=\mathbbm{1}(w_{i}>w_{j}). (2) The pairwise weight agreement is defined as the normalized sum of disagreements over index pairs: \phi_{\text{pairwise}}(\gamma^{*},w)=1-\frac{\sum_{i\neq j}|M_{i,j}^{\gamma^{*}}-M_{i,j}^{w}|}{p(p-1)}. (3) Note that \phi_{\text{pairwise}} is related to Kendall’s \tau rank correlation, adapted here to the binary ground truth. In applied settings, evaluating LLM-generated weights prior to deploying LLM-informed methods could guide the selection of specific models or the refinement of prompt strategies. Because the true inclusion vector \gamma^{*} is unknown, we propose using a standard statistical estimate of the inclusion vector, denoted \hat{\gamma} , to construct a natural plug-in estimator: \hat{\phi}=\phi(\hat{\gamma},w) . However, in high-dimensional or low-data regimes, \hat{\gamma} likely serves as a poor estimator, limiting the reliability of inferential conclusions. For this reason, we interpret \hat{\phi} as a measure of the empirical alignment between the weights and the data, rather than a measure of alignment with the true inclusion vector. 3 LLM Sparsity Prior for Spike-and-Slab Bayesian prior distributions offer a principled mechanism for integrating LLM-generated weights into statistical models ( 9 ) . Ideally, a prior encapsulates