What Language Models Know But Don’t Say: Non-Generative Prior Extraction for Generalization
Sara Rezaeimanesh, Mohammad M. Ghassemi · 2026
Read What Language Models Know But Don’t Say: Non-Generative Prior Extraction for Generalization by Sara Rezaeimanesh, Mohammad M. Ghassemi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-17609.txt.
Introduction
Sara Rezaeimanesh and Mohammad M. Ghassemi draw hidden knowledge out of language models. In medicine and finance, labeled data is costly and scarce. Small training sets then fail on real populations. Language models hold vast prior knowledge from years of research text. The authors extract that knowledge without generation. Their non-generative prior extraction feeds small-data models.
A good prior can lift a weak dataset to useful skill. It avoids the risks of free text generation. Hospitals and banks could gain models they can afford. The method respects data limits that rule those fields. It treats the big model as a library, not an oracle. That stance may widen safe use.
Catalog record
| CREATOR | Sara Rezaeimanesh, Mohammad M. Ghassemi |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 57 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-17609
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
What Language Models Know But Don’t Say: Non-Generative Prior Extraction for Generalization Sara Rezaeimanesh, Mohammad M. Ghassemi Abstract In domains like medicine and finance, large-scale labeled data is costly and often unavailable, leading to models trained on small datasets that struggle to generalize to real-world populations. Large language models contain extensive knowledge from years of research across these domains. We propose LoID (Logit-Informed Distributions), a deterministic method for extracting informative prior distributions for Bayesian logistic regression by directly accessing their token-level predictions. Rather than relying on generated text, we probe the model’s confidence in opposing semantic directions (positive vs. negative impact) through carefully constructed sentences. By measuring how consistently the LLM favors one direction across diverse phrasings, we extract the strength and reliability of the model’s belief about each feature’s influence. We evaluate LoID on ten real-world tabular datasets under synthetic out-of-distribution (OOD) settings characterized by covariate shift, where the training data represents only a subset of the population. We compare our approach against (1) standard uninformative priors, (2) AutoElicit, a recent method that prompts LLMs to generate expert-style priors via text completions, (3) LLMProcesses, a method that uses LLMs to generate numerical predictions through in-context learning by formatting tabular data as natural language prompts and (4) an oracle-style upper bound derived from fitting logistic regression on the full dataset. We assess performance using Area Under the Curve (AUC). Across datasets, LoID significantly improves performance over logistic regression trained on OOD data, recovering up to 59% of the performance gap relative to the oracle model. LoID outperforms AutoElicit and LLMProcessesc on 8 out of 10 datasets, while providing a reproducible and computationally efficient mechanism for integrating LLM knowledge into Bayesian inference. 1 Introduction Machine learning models used in critical domains like medicine and finance often lack access to large, well-balanced, and comprehensively labeled datasets. Clinical datasets may cover only a single hospital or demographic, while credit-risk tables can be skewed toward historical lending patterns. For example, 25 benchmarked several deep learning models on clinical prediction tasks—including mortality and length of stay—using the MIMIC-III intensive care electronic health record dataset from a single hospital site. As a consequence, models trained on small, non-representative samples frequently fail to generalize when confronted with real-world populations that differ from their training distribution 5 ; 20 ; 2 . Bayesian methods offer a principled way to encode prior knowledge and mitigate such brittleness, but manually eliciting high-quality priors at scale remains a longstanding challenge 24 . LLMs, trained on vast corpora of scientific literature and domain-specific data 6 , offer a promising new source of priors for downstream modeling. Recent efforts have explored LLMs both as zero-shot and few-shot predictors for tabular classification 11 ; 28 , and as in-context learners capable of producing joint probabilistic predictions 30 . Other work has investigated the integration of LLMs with classical statistical methods: LLM-FE 1 uses LLMs to guide feature engineering, LLM-Lasso 34 for feature selection, and LLMCorr 35 for post-hoc model correction. AutoElicit 9 introduced a technique to query LLMs for Gaussian priors, but its reliance on generative sampling introduces run-to-run variability. Parallel to this, out-of-distribution (OOD) generalization has gained increasing attention, as real-world deployment often involves data that differs from the development environment. Meta-learning approaches like TabPFN 13 and Drift-Resilient TabPFN 12 address OOD robustness through synthetic task generation and causal modeling. Others have proposed logit-based calibration strategies such as UE-NL 14 , or LLM-guided data augmentation methods like Summary Boosting 22 and LLMOverTab 16 to cover underrepresented regions of the feature space. However, despite this momentum, few works have systematically explored how to extract priors from the internal structure of LLMs (e.g. hidden states, logits, etc.) for tabular tasks under distribution shift. We address this gap with LoID (Logit-Informed Distributions) , a deterministic framework that elicits informative priors for Bayesian logistic regression directly from an LLM’s token-level logits. Instead of relying on generated completions or examples, LoID probes the LLM with semantically paired sentences highlighting positive and negative feature-target relationships. The logit difference between the relational tokens quantifies directional belief, and the variance across paraphrased prompts yields an uncertainty estimate—together forming Normal priors over model coefficients. These priors integrate into Bayesian workflows, without any fine-tuning or in-context learning. We evaluate LoID on ten publicly available tabular datasets that are widely used benchmarks and have appeared in numerous prior studies, ensuring comparability with existing methods. We compare LoID against uninformative Bayesian inference and LLMProcesses. To provide an upper bound, we additionally train and evaluate a logistic regression model on the entire dataset, offering insight into performance when full population data is available. Across extensive out-of-distribution splits, LoID outperforms existing approaches in predictive accuracy (AUC). By deterministically extracting domain knowledge encoded in large language models, LoID enables robust and reproducible inference. In summary, our contributions include: • Logit-based deterministic prior elicitation. LoID is the first method to extract Bayesian priors directly from the token-level logits of large language models, eliminating the variability associated with text generation and prompt sampling. • Baseline comparison across ten OOD datasets. We evaluate LoID on diverse OOD tabular datasets from healthcare, finance, and social science. Our experiments demonstrate that LoID often outperforms existing approaches across most settings. 2 Related Work 2.1 LLMs for Tabular Data Modeling The integration of LLMs into tabular data modeling has garnered attention, particularly in addressing challenges such as limited data availability, complex feature interactions, and the necessity for domain-specific feature engineering. Several studies have explored the use of LLMs as standalone predictors for tabular data. 11 introduce TabLLM, a framework that applies large language models to zero-shot and few-shot classification of tabular data by converting rows into natural-language strings, enabling effective classification even with minimal labeled examples. 28 propose LLMProcesses, a framework that enables LLMs to produce probabilistic regression outputs by conditioning on both numerical data and natural language descriptions of prior knowledge. 30 present JOLT, a method that leverages the in-context learning capabilities of LLMs to define joint probabilistic predictions over tabular data. Beyond standalone applications, recent research has focused on leveraging LLMs to enhance classical statistical models. For instance, 1 introduced LLM-FE, a framework that combines evolutionary search with the reasoning capabilities of LLMs to iteratively generate and refine feature transformations for tabular data. Similarly, LLM-Lasso 34 is a domain-informed approach to feature selection and regularization by leveraging large language models to guide the selection process. 9 proposed AutoElicit, a method that interacts with LLMs to extract priors for Gaussian-based probabilistic models; however, this approach primarily focuses on Gaussian priors and lacks flexibility in handling other priors that may be more suitable for different data characteristics. In a different vein, 35 presented LLMCorr, a training-free framework where LLMs act as post-hoc correctors to propose corrections for the predictions of arbitrary machine learning models. 21 investigated the reasoning capabilities of LLMs over tabular data and proposed combining symbolic and textual reasoning to improve performance on table-based question answering tasks. 2.2 Out-of-Distribution Generalization OOD generalization is a fundamental challenge in real-world applications of tabular machine learning. Models trained on limited or biased datasets often fail to generalize when deployed on test data that differ from the training distribution. A variety of approaches have been proposed to