Extracting Probabilistic Knowledge from Large Language Models for Bayesian Network Parameterization

Aliakbar Nafar, Kristen Brent Venable, Zijun Cui, Parisa Kordjamshidi · 2025

Read Extracting Probabilistic Knowledge from Large Language Models for Bayesian Network Parameterization by Aliakbar Nafar, Kristen Brent Venable, Zijun Cui, Parisa Kordjamshidi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-15918.txt.

Introduction

Aliakbar Nafar and co-authors test whether large language models can supply odds for Bayesian networks in a paper from 2025. Such networks model how events link, but they need expert odds for each link. Experts are scarce, so the team asks if a model's stored facts can stand in.

The authors asked models for the chance of one event given another. They tried the method on eighty public networks, from health care to finance. The model odds beat simple baselines such as random or flat guesses. The work hints that chat models can help set up networks fast. Careful checks against real data are still needed.

Catalog record

CREATORAliakbar Nafar, Kristen Brent Venable, Zijun Cui, Parisa Kordjamshidi
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-15918

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Extracting Probabilistic Knowledge from Large Language Models for Bayesian Network Parameterization Aliakbar Nafar, Kristen Brent Venable, Zijun Cui, Parisa Kordjamshidi Abstract Large Language Models (LLMs) have demonstrated potential as factual knowledge bases; however, their capability to generate probabilistic knowledge about real-world events remains understudied. This paper investigates using probabilistic knowledge inherent in LLMs to derive probability estimates for statements concerning events and their interrelationships captured via a Bayesian Network (BN). Using LLMs in this context allows for the parameterization of BNs, enabling probabilistic modeling within specific domains. Experiments on eighty publicly available Bayesian Networks, from healthcare to finance, demonstrate that querying LLMs about the conditional probabilities of events provides meaningful results when compared to baselines, including random and uniform distributions, as well as approaches based on next-token generation probabilities. We explore how these LLM-derived distributions can serve as expert priors to refine distributions extracted from minimal data, significantly reducing systematic biases. Overall, this work introduces a promising strategy for automatically constructing Bayesian Networks by combining probabilistic knowledge extracted from LLMs with small amounts of real-world data. Additionally, we evaluate several prompting strategies for eliciting probabilistic knowledge from LLMs and establish the first comprehensive baseline for assessing LLM performance in extracting probabilistic knowledge. 1 Introduction The potential of language models as sources for extracting factual knowledge has been demonstrated in several studies ( Petroni et al. 2019 ; Roberts et al. 2020 ; AlKhamissi et al. 2022 ; Zhao et al. 2025 ) . However, it remains unclear whether LLMs possess the ability to generate meaningful probabilistic estimates for events and their relationships based on their internal knowledge. We refer to an LLM’s capability to assign a specific probability to an uncertain proposition by utilizing its internal knowledge as probabilistic estimation . This contrasts with providing a confidence score for a concrete answer ( Xiong et al. 2024 ) or solving a probabilistic problem that has a known numerical solution, a process referred to as probabilistic inference ( Nafar et al. 2024b ) . For instance, consider the question, “What is the probability that a person who smokes cigarettes will develop cancer in their lifetime?” There is no definitive correct answer to infer directly from the question itself; however, a medical expert familiar with the literature might approximate 20%. Similarly, we expect an adept language model to produce a similar estimate. Here, we evaluate the probabilistic estimation capabilities of LLMs such as GPT-4o ( OpenAI et al. 2024 ) , Gemini Pro 1.5 ( Team et al. 2024 ) , and Claude 3.5 ( Anthropic 2024 ) , and utilize their internal knowledge to construct domain-specific Bayesian networks (BNs). BNs are a powerful method for representing uncertainty and dependencies between events. The reliability of inference in BNs hinges on the accuracy of conditional probability table (CPT) entries, which are typically obtained by collecting data or eliciting expert judgments. These approaches can be expensive or unattainable in domains where data is scarce ( You et al. 2019 ; Longato et al. 2023 ) or expert time is limited ( Das 2008 ; Xiaoguang et al. 2019 ) . By extracting probabilistic knowledge of the LLMs, we explore whether they can serve the role of experts for parameterizing BNs. We assume the dependency structure within the BN is given, leaving the task of extracting such dependencies using LLMs for future research. Nonetheless, preliminary results indicate that LLMs show promise in accurately identifying network dependencies ( Babakov et al. 2025 ) . Given the structure of a BN, LLMs are required to predict a probability distribution for each node, conditioned on its parent nodes. In this work, we focus on discrete variables , so we query the LLM separately for each state of a node, prompting it to directly generate a numerical output representing the conditional probability given its parent nodes. After collecting these raw numerical outputs from the LLM, a normalization step is performed to ensure valid probability distributions. In domains where only a small dataset is available, initial distributions estimated by LLMs can potentially function as expert-derived prior probabilities. We test this hypothesis by adjusting the LLM predictions with a small number of data samples, effectively applying a partial calibration to the model’s initial estimates. We demonstrate that this approach not only enhances the predictions of the LLMs but also surpasses purely data-driven methods and provides a superior prior compared to a uniform prior baseline, which is typically employed in the absence of additional information. Furthermore, our experiments indicate that even when the number of data samples is large, incorporating LLM priors improves the performance. These findings highlight the promise of leveraging LLMs as expert knowledge sources for probabilistic estimation across various real-world domains. In summary, our contributions are as follows: 1) Comprehensive Evaluation of LLM Probabilistic Estimation with Real-World BNs. We provide the first large-scale empirical analyses of how LLMs quantify probabilities. We investigate differences in LLM accuracy across diverse BN domains and varying levels of conditional complexity, highlighting the strengths and limitations of LLMs as probabilistic knowledge bases. 2) LLM Predictions as Expert-Driven Priors. We demonstrate that the LLM predicted probabilities can serve as expert-derived priors and combining them with sampled data improves accuracy and sample efficiency compared to purely data-driven methods or using a uniform prior baseline. 3) Automatic LLM-to-BN Pipeline. We introduce an automated procedure for parameterizing real-world BNs given the network structure 1 1 1 The code and analytical tools are available at . . 2 Related Work Parameter Learning. Within the field of Bayesian Networks, parameter learning has historically revolved around data-driven approaches ( Ji et al. 2015 ) or expert elicitation ( Mendes 2014 ) . Data-driven methods compute frequencies for each configuration of parent states, estimating conditional probabilities via frequentist or Bayesian methods. Expert elicitation is commonly used in domains where data is limited or unobserved for some variables ( Woodberry et al. 2005 ; Constantinou et al. 2016 ) . This work aims to see whether LLM-based probability queries can fill a similar role to experts. Zero-shot Regression. Using LLMs for regression in a zero-shot setting is an emerging field, and work in this area is limited. Following Vacareanu et al. 2024 , which shows that LLMs are capable regressors in a few-shot setting, Nafar et al. 2025 tests the regression capability of LLMs in a zero-shot setting (using internal knowledge) for realistic questions such as estimating the medical insurance cost based on age. However, they don’t use any probability estimation. Paruchuri et al. 2024 asks LLMs to calculate the probabilities for a range of values in a given distribution, but their dataset is limited to only 12 questions, and the queries are elementary with no conditional variables. Confidence Elicitation. Confidence elicitation in LLMs has been studied in classification tasks, where a confidence score ranging from 0 to 1 is assigned to a discrete class label. Among these, Kadavath et al. 2022 treat the models as a white box and use their token probabilities to assess confidence. However, token likelihood merely indicates the model’s uncertainty about the next token ( Kuhn et al. 2023 ) , rather than the confidence in the text itself. Consequently, Xiong et al. 2024 ; Yang et al.

Read the complete text