Do LLMs Act Like Rational Agents? Measuring Belief Coherence in Probabilistic Decision Making Khurram Yamin, Jingjing Tang, Santiago Cortes-Gomez, Amit Sharma, Eric Horvitz, Bryan Wilder Abstract Large language models (LLMs) are increasingly deployed as agents in high-stakes domains where optimal actions depend on both uncertainty about the world and consideration of utilities of different outcomes, yet their decision logic remains difficult to interpret. We study whether LLMs are rational utility maximizers with coherent beliefs and stable preferences. We consider behaviors of models for diagnosis challenge problems. The results provide insights about the relationship of LLM inferences to ideal Bayesian utility maximization for elicited probabilities and observed actions. Our approach provides falsifiable conditions under which the reported probabilities cannot correspond to the true beliefs of any rational agent. We apply this methodology to multiple medical diagnostic domains with evaluations across several LLMs. We discuss implications of the results and directions forward for uses of LLMs in guiding high-stakes decisions. 1 Introduction LLMs are increasingly used to support high-stakes decisions under uncertainty, such as medical diagnosis, where good actions require reasoning about a patient’s latent state and trading off costs and benefits. To understand and improve these recommendations, a common strategy is to elicit the model’s probabilities over relevant unknowns (e.g., disease likelihoods) from the available evidence. For instance, developers may wonder whether failures to seek additional information ( 29 ; 24 ) stem from overconfident beliefs. However, it is unclear whether stated probabilities reflect the model’s “true” beliefs: an elicited probability could track an internal epistemic state, or it could be a superficial linguistic output only weakly linked to the computations that drive choices ( 36 ; 49 ; 28 ) . Making precise what it means for an LLM to hold a belief – formally, a subjective probability—and how to test it remains challenging. Several recent studies, focus on testing the accuracy and calibration of expressed probabilities ( 47 ; 48 ; 10 ) . While important, these questions are distinct. On the one hand, a decision maker can hold inaccurate or miscalibrated beliefs in good faith. On the other hand, LLMs could express beliefs that fail such conditions for many reasons, even if they internally “know better.” For example, biases introduced during finetuning could lead models to verbalize miscalibrated probabilities. We propose a novel strategy for validating elicited probability judgments from LLMs that centers on jointly eliciting beliefs and decisions on tasks where action quality depends on the relevant probabilities. E.g., in medical diagnosis, we ask models both for the probability that a patient has a condition and for a diagnostic action (e.g., yes/no/abstain). We then test whether the elicited probabilities could be the subjective beliefs of any rational agent for that decision problem. In the axiomatic tradition of 34 , rational action under uncertainty corresponds to expected-utility maximization with respect to a belief over states and a utility function over outcomes. Building on Bayesian decision theory, we treat beliefs as subjective probabilities while remaining agnostic about utilities ( 31 ) , deriving testable implications that link reported beliefs to choices; violations imply the joint pattern of elicited beliefs and actions cannot be rationalized by any Bayesian utility maximizer. This yields a minimalist but falsifiable evaluation strategy: rather than asserting that a reported probability reflects the model’s “true belief,” we identify conditions under which it cannot be the belief of a rational decision-maker. Violation rates then quantify how well utility maximization (relative to elicited beliefs) explains behavior. In this sense, we propose a decision-oriented means to sidestep defining what a model “really” believes: when the goal is ultimately to understand and improve model behavior , elicited beliefs are a useful measure of internal state if they induce a self-consistent description of how the model makes decisions. Our empirical strategy is fully black-box, using only elicited probabilities and observed choices, and is complementary to white-box approaches that analyze internal activations. Finally, compared to prior work that tests probabilistic self-consistency (e.g., adherence to probability axioms), our approach yields richer, decision-linked implications. Finally, we empirically instantiate this framework across multiple LLM families on clinically grounded diagnostic tasks spanning four medical settings. Beyond conventional calibration and discrimination, we evaluate whether elicited diagnostic beliefs are action-consistent, providing a sharper test of whether reported uncertainty is mechanistically meaningful for high-stakes decision support. This complements current efforts toward uncertainty-aware clinical LLMs that explicitly model diagnostic uncertainty and its explanation ( 51 ) . 2 Related Work Belief elicitation from humans: A long literature in psychology and economics seeks to elicit true subjective probabilities, often using monetary incentives to encourage truthful reporting. Yet reviews ( 43 ; 7 ) emphasize that validating belief elicitation is difficult and arguably ill-posed: beliefs are unobservable and may be shaped by elicitation itself, so researchers frequently evaluate methods by how well elicited beliefs predict actions. Our approach is similar in spirit, but LLMs pose different constraints. Human studies typically assume preferences (e.g., more money is better) to design proper scoring rules ( 41 ; 17 ) or test best responses in games ( 35 ; 39 ) . For LLMs, assuming a specific utility is strong and hard to justify, and credibly incentivizing model in simulation is difficult. Accordingly, we remain agnostic about the model’s utility. Finally, whereas belief elicitation for humans can “contaminate” subsequent actions ( 9 ; 6 ) , we avoid this for LLMs by eliciting beliefs and actions in separate context windows. Belief elicitation from LLMs: With the advent of large language models there has been renewed interest in characterizing LLM beliefs. 21 provide a philosophical account, and our work can been seen as operationalizing their “use” criterion. Other work has studied how to elicit priors from LLMs ( 53 ) and whether LLMs express beliefs that are consistent with probabilistic axioms ( 52 ; 15 ) . Our work introduces a new set of tools to test coherence of beliefs in comparison to decisions. Perhaps the most related is work by 36 , who test whether LLMs make bets on events which go in the same direction as their beliefs. However, they do not introduce a formal framework for testing when beliefs and actions should correspond in a specific way. Further afield, an emerging literature tests whether LLMs outputs are consistent with various cognitive biases seen in humans ( 13 ; 5 ; 8 ) but this work does not focus on validity of elicited beliefs. Mechanistic interpretability: A central motivation for our study is the concern that LLMs may know more than what they tell us in verbalized probabilities. Mechanistic interpretability uses access to the models’ internal representations to test such questions ( 44 ; 4 ) . For example, researchers train probes which predict distinctions like true vs false claims ( 3 ; 30 ) from activations. We view mechanistic interpretability as complementary to our work in two respects: (1) our tests are fully black-box and rely only on model outputs (useful for closed models), and (2) many mechanistic approaches (e.g., linear probes) require supervised labels, injecting external information and complicating a direct interpretation of their outputs as beliefs of the model. Our framework is unsupervised and output-only, and we view multiple toolkits as valuable given the difficulty of formally defining “belief.” Medical applications: A growing body of work evaluates large language models in clinically consequential settings, emphasizing that downstream quality depends on decision policies rather than static accuracy alone ( 46 ; 16 ; 20 ; 50 ) . Clinical evaluations further stress that models may lack reliable metacognitive awareness of their own limitations, making naive self-reported confidence insufficient for safe triage ( 18 ; 20 ) . Evaluations have also stressed difficulties in seeking information when necessary to make a diagnosis, perhaps reflecting incorrect probabilistic reasoning ( 24 ; 27 ) . Our aim is to provide a principled framework within which to approach such questions. 3 Methodology 3.1 Decision-Theoretic Framework We formalize the decision problem as follows. An environment generates an unknown state of the world \theta\sim P^{\star}(\theta) and observations x\sim P^{\star}(x\mid\theta) . The induced ground-truth posterior under the environment is P^{\star}(\theta\mid x) . After observing x , a decision maker forms its own (potentially misspecified) subjective posterior belief P_{S}(\theta\mid x) , which need not equal P^{\star}(\theta\mid x) . In the classic framework of expected utility maximization, the decision maker then selects an action a\in\mathcal{A} to maximize expected utility: a(x)\;=\;\argmax_{a\in\mathcal{A}}\;\sum_{\theta\in\Theta}P_{S}(\theta|x)u(a,\theta). (1) Equivalently, letting \ell(a,\theta)=-u(a,\theta) denote a loss function, we can minimize the objective. This optimization requires two components: (1) beliefs about the unknown state—captured here by the subjective posterior P_{S}(\theta\mid x) —and (2) preferences over outcomes—captured here by the utility u (or, equivalently, the loss \ell ). Our goal is to test whether an agent’s reported beliefs are consistent with their acting as a utility maximizer in order to tell whether it is possible for the reported beliefs to play the role of true subjective probabilities in rational decision making. However, testing for consistency with expected utility maximization is by itself too strict – agents may fail to exactly maximize their utility function for a variety of reasons, or their utility on a given instance may include idiosyncratic preferences that we do not observe. Accordingly, we base our empirical strategy on the random utility model which forms the foundation for econometric work on human behavior ( 31 ) . Random utility maximizers augment Equation 1 with an additional random noise term which generates variation in their decision making: Definition 3.1 (Random Utility Model) . An agent is a random utility maximizer if it selects actions according to a(x)\in\argmax_{a\in\mathcal{A}}\sum_{\theta\in\Theta}\left(P_{S}(\theta|x)u(a,\theta)\right)+\varepsilon_{a}, (2) where \{\varepsilon_{a}\}_{a\in\mathcal{A}} are action-specific random variables which satisfy \{\varepsilon_{a}\}_{a\in\mathcal{A}}\perp(x,\theta,P_{S}(\theta\mid x)) . The randomness in \epsilon induces (from the perspective of the experimenter) randomness in the choices of the agent. For example, if \epsilon follows a logistic distribution (the most common specification in the literature), we obtain \displaystyle\Pr(a(x)=a)\propto\exp\left(\sum_{\theta\in\Theta}P_{S}(\theta|x)u(a,\theta)\right). (3) That is, from the experimenter’s perspective, the agent takes a softmax of their utility function. While this is one common distributional model that we include for intuition, our results will not require specific parametric assumptions on \epsilon . A final common modification to the standard utility-maximization framework is to allow agents to express preferences over risk that are not compatible with strict expected utility maximization. For example, humans are often observed to act in risk-averse ways, or to express greater sensitivity to losses than gains. Some work has argued that LLMs exhibit similar phenomena ( 23 ; 38 ; 22 ) . Prospect theory ( 26 ) is the classical modification of expected utility maximization to account for such behavior. In prospect theory, the agent has a monotone probability weighting function w:[0,1]\to[0,1] and calculates losses for each action with respect to w(P_{S}) instead of P_{S} , maximizing \sum_{\theta\in\Theta}w(P_{S}(\theta|x))u(a,\theta) . Combining this with random shocks to account for variation in decision making ( 11 ) , we obtain Definition 3.2 (PT-RUM) . An agent is a prospect-theoretic random utility maximizer if it selects actions according to a(x)\in\argmax_{a\in\mathcal{A}}\Big(\sum_{\theta\in\Theta}w(P_{S}(\theta|x))u(a,\theta)+\varepsilon_{a}\Big), (4) where w is monotone increasing and \{\varepsilon_{a}\}_{a\in\mathcal{A}}\perp(x,\theta,P_{S}(\theta\mid x)) . Our empirical strategy will allow for the possibility that agents act in a risk-averse manner consistent with Definition 3.2 , in order to separate whether it is possible for elicited beliefs to represent subjective probabilities from whether the agent is risk-averse. 3.2 Testable Implications for Belief Elicitation A central objective of this paper is to test whether it is possible for a model’s verbalized probability beliefs to reflect subjective belief states that actually drive its choices in the manner prescribed by rational decision making. We will refer to the probability judgment elicited from a model by some method (e.g., prompting) as P_{E}(\theta\mid x) . Our goal is to test whether it is possible for a rational decision maker to hold those judgments as true subjective beliefs, P_{S}(\theta|x)=P_{E}(\theta\mid x) , and still take the same actions as the model. We will construct a framework to test this in the context of a specific decision problem, where we assume that the experimenter either knows the true joint distribution over (x,\theta) or has a sample from this distribution. Our framework gathers data in two steps. First, on a sample of values of x , we elicit P_{E}(\theta\mid x) using whatever method we wish to test the validity of. In our experiments, we prompt the model to report a probability distribution P_{E}(\theta\mid x) in natural language ( 37 ) . Specifically, for each context x in our evaluation set, we query the model to provide probability estimates for each possible state. For the tasks that we consider, the state space is binary ( \theta\in\{0,1\} indicating absence or presence of a condition), so we elicit p(x):=P_{E}(\theta=1\mid x) directly. The model is instructed to provide a numerical probability estimate. However, our methods are applicable to evaluate any elicitation method. We select natural-language probability estimates as our experimental focus because of their simplicity and the wide literature investigating them. As a notational note, we will often use p(x):=P_{E}(\theta=1\mid x) or p^{*}(x):=P^{*}(\theta=1\mid x) in shorthand to refer to the binary case, we use P_{E}(\theta\mid x) or P(\theta\mid x) when results generalize beyond the binary case. Second, we prompt the model with the same x in a separate context and ask it to take an action a . This generates a sample of the form (x_{i},P_{E}(\theta=1\mid x_{i}),a_{i},\theta_{i}) where we observe for each x_{i} the model’s elicited belief, its action, and the true state (which was not revealed to the model). Third, we apply a set of statistical tests which test properties that the sample (x_{i},P_{E}(\theta=1\mid x_{i}),a_{i},\theta_{i}) should satisfy if the elicited beliefs and actions indeed reflect rational decision making relative to the expressed beliefs. Violations of these properties quantify the extent to which elicited beliefs depart from useful interpretation as a true subjective probability. We next detail these tests. Conditional independence of actions and outcomes: The key intuition behind this test is that, once we condition on model’s subjective belief, the realized outcome \theta provides no additional information that could affect the choice: P_{S} is a sufficient statistic for the impact of the distribution of future events on the agent’s choices. On the other hand, if a rational agent’s actions have excess correlation with the true state given their elicited P_{E} , the agent must have information about \theta that was not reported. Proposition 3.3 (Conditional Independence under Truthful Reporting) . For any agent satisfying Definition 3.1 or 3.2 , their subjective beliefs and actions satisfy a\perp\theta\mid P_{S}(\theta\mid x) . Conversely, if for a set of elicited beliefs P_{E} , a\not\perp\theta\mid P_{E}(\theta|x) , then there is no agent satisfying Definition 3.1 or 3.2 (under any utility function) who holds subjective belief P_{S}=P_{E} . For intuition, fix x . Under Definition 3.2 , the expected loss depends on x only through w(P_{E}(\theta\mid x)) , so Eq. ( 4 ) implies a(x_{i})=g(w(P_{E}(\theta\mid x)),\varepsilon) for some function g . Since \varepsilon\perp(x_{i},\theta,P_{E}(\theta\mid x)) , we obtain a\perp\!\!\!\perp\theta\mid w(P_{E}(\theta\mid x)) and specifically a\perp\!\!\!\perp\theta\mid P_{E}(\theta\mid x) . Remark 3.4 . Observing a\perp\!\!\!\perp\theta\mid P_{E}(\theta\mid x) does not imply truthful reporting by a rational agent. E.g.  consider an agent who takes a uniformly random action independent of x and reports a (independent) uniformly random value in [0,1] as p_{E}(x) . This satisfies a\perp\theta\mid P_{E}(\theta|x) without being a utility-maximizing agent with subjective belief p_{S}=p_{E} . In order to empirically test this condition, we first conduct a direct conditional-independence (CI) test of the null hypothesis A\perp\!\!\!\perp\theta\mid w(P_{E}(\theta\mid x)) via the nonparametric CMI (Conditional Mutual Information) Test for H_{0}:I(A;\theta\mid p)=0 . While a binary reject/fail-to-reject CI decision is useful, it can be hard to interpret how large a violation is in practice. Therefore, we additionally quantify the degree of CI violation via an out-of-sample predictive-performance comparison. Concretely, we train two predictive models for the agent’s action a and compare nested feature sets. Model 1 conditions on the elicited belief p_{E}(x) , while Model 2 conditions on (p_{E}(x),\theta) . Under the conditional-independence null, adding \theta should not improve prediction once we condition on p_{E} . We use cross-validated log-loss and report percent improvement as the relative reduction in log-loss from including \theta . Significance is assessed by a bootstrap CI. To check whether violations are driven by omitted context, we then repeat the test while conditioning on an explicit evidence vector x : Model 1( x ) uses (p_{E}(x),x) and Model 2( x ) uses (p_{E}(x),x,\theta) . Persistent gains from \theta then indicate that p_{E}(x) is not decision-sufficient even after controlling for observed context. Monotone pairwise choice probabilities. In this section, we specialize to binary classification problems with state \theta\in\{0,1\} (as in all of the empirical examples we consider). The intuition behind this test is that, while we do not know the agent’s utilities, the difference in their utilities for different actions is a monotone function of p . For example, an expected utility maximizer evaluates the difference in utility between two actions a_{1} and a_{2} as \displaystyle\Delta_{a_{1},a_{2}}(p_{S}(x))=\,\, \displaystyle p_{S}(x)\cdot(u(a_{1},1)-u(a_{2},1)) \displaystyle+(1-p_{S}(x))\cdot(u(a_{1},0)-u(a_{2},0)) which is a linear function of p_{S}(x) (and hence monotone). Monotonicity of pairwise differences in utility translates into monotonicity in choice probabilities under an additional restriction: Definition 3.5 (IIA) . Let \Pr(a(x)=a|\mathcal{C}) denote the probability that an agent chooses action a when presented with choices \mathcal{C}\subseteq\mathcal{A} . The agent satisfies independence of irrelevant alternatives if for any actions a_{1},a_{2}\in\mathcal{A} and any \mathcal{C} containing a_{1},a_{2} , \frac{\Pr(a(x)=a_{1}|\mathcal{C})}{\Pr(a(x)=a_{2}|\mathcal{C})}=\frac{\Pr(a(x)=a_{1}|\mathcal{A})}{\Pr(a(x)=a_{2}|\mathcal{A})} . Essentially, IIA requires that the relative probability of the agent choosing a over b does not depend on the presence of other items. It is satisfied for example by the logit model in Equation 3 . Note that it is also automatically satisfied for decision problems with two actions. Under this condition, we obtain a testable condition that the relative choice probabilities between two actions are monotone in p_{S}(x) : Proposition 3.6 (Monotone pairwise choice probability) . For any agent satisfying either Definition 3.1 or 3.2 in conjunction with IIA (Definition 3.5 ) the pairwise (conditional) choice probability for any two actions a_{1},a_{2}\in\mathcal{A} , s(p)\;:=\;\Pr\!\big(a(x)=a_{1}\,\big|\,a\in\{a_{1},a_{2}\},\,p_{S}(x)\big) is either monotone increasing or monotone decreasing in the subjective probability p_{S} . Remark 3.7 . It is possible to test implications of random utility models for choice probabilities over more than two actions simultaneously via a generalized notion referred to as cyclic monotonicity ( 32 ) . This requires that there exist a convex function which has the choice probabilities as a subgradient. Since testing and interpreting this property is considerably more complex, we recommend pairwise comparisons here. To test Proposition 3.6 , we discretize elicited beliefs into K quantile bins B_{1},\ldots,B_{K} with non-decreasing centers \bar{p}_{1}\leq\cdots\leq\bar{p}_{K} . For each bin B_{k} , we estimate the pairwise share of a_{1} among \{a_{1},a_{2}\} : \displaystyle\widehat{s}_{k}:=\frac{\#\{i\in B_{k}:A_{i}=a_{1}\}}{\#\{i\in B_{k}:A_{i}\in\{a_{1},a_{2}\}\}}. (5) We flag a monotonicity violation for any pair of bins j\widehat{s}_{k} . For each flagged pair, we assess significance using a one-sided test for proportions (e.g., Fisher’s exact test or a one-sided binomial test) of H_{0}:\ s(\bar{p}_{j})\leq s(\bar{p}_{k}), and report the fraction of pairwise comparisons with significant violations at \alpha=0.05 . Consistency across decision tasks. For a rational agent, beliefs and preferences are distinct: when multiple decision tasks reference the same state \theta , agents adhering to Definitions 3.1 or 3.2 should act as if driven by a common subjective belief p_{S} even if the action set and utility differ across tasks. This cross-task stability—sometimes called prize independence —is a foundational implication of subjective probability ( 2 ; 42 ; 40 ) . Accordingly, beyond our single-task tests, we compare elicited probabilities p_{E} across prompts that embed different decision tasks but share the same (x,\theta) . If the model truthfully reports a stable belief, these elicited probabilities should agree. In binary settings, we instantiate alternative tasks by asking for p^{*}(\theta\mid x) under different stated evaluation losses (e.g., MSE or MAE). We test consistency via repeated elicitations. Let p(x;\pi,j) denote the reported P_{E}(\theta\mid x) for context x under prompt for task \pi\in\Pi on repetition j , and let \bar{p}(x;\pi)=\frac{1}{r}\sum_{j=1}^{r}p(x;\pi,j) denote the repetition-averaged elicited probability. Fixing the main task \pi_{0} (e.g., the medical diagnosis tasks we construct), for each alternative \pi\in\Pi\setminus\{\pi_{0}\} (with \Pi the set of prompts), we compute the RMSE between p(x;\pi) and \bar{p}(x;\pi_{0}) . Larger RMSE indicates less consistency, though as with any of our tests we do not expect perfect consistency. For example, 40 tested prize independence in humans and found that over half of subjects failed. For reference, for the standard prompt \pi_{0} , we also show the standard deviation across repetitions. Table 1 : Conditional-independence (belief sufficiency) tests: kNN conditional mutual information (CMI) and CatBoost (CB) predictive-improvement tests (with and without explicit evidence features). For each dataset/model pair, the first block reports the kNN-based estimate of conditional mutual information I(A;\theta\mid p) (CMI) with 95% confidence intervals, corresponding to the conditional-independence null hypothesis H_{0}:I(A;\theta\mid p)=0 (equivalently, A\perp\theta\mid p ). The second block reports CatBoost comparisons of A\sim p vs. A\sim(p,\theta) , summarized by percent log-loss improvement with 95% bootstrap CIs. The third block reports the analogous CatBoost comparison additionally conditioning on the explicit evidence/context vector x ( A\sim(p,x) vs. A\sim(p,x,\theta) ). kNN CMI: I(A;\theta\mid p) CB: A\sim p v. A\sim(p,\theta) CB: A\sim(p,x) v. A\sim(p,x,\theta) Dataset / Model CMI 95% CI % Impr 95% CI % Impr 95% CI Heart–GPT-Min 0.1454 [0.1119, 0.1789] 16.37 [11.85, 20.90] 13.05 [9.16, 17.54] Heart–GPT-High 0.0753 [0.0422, 0.1085] 4.23 [0.77, 7.62] 3.82 [1.07, 6.24] Heart–Llama 0.0718 [0.0365, 0.1070] 2.98 [1.28, 5.07] 1.02 [-0.33, 2.39] Heart–DeepSeek 0.0675 [0.0354, 0.0997] 4.52 [1.71, 7.48] 2.03 [0.02, 4.00] Cry–GPT-Min 0.2232 [0.1874, 0.2589] 14.84 [12.08, 17.73] 4.88 [2.20, 7.55] Cry–GPT-High 0.1901 [0.1390, 0.2412] 11.52 [ 8.80, 14.08] 9.25 [6.97, 11.34] Cry–Llama 0.4223 [0.3764, 0.4681] 0.56 [-1.27, 2.48] 2.83 [1.17, 4.56] Cry–DeepSeek 0.1745 [0.1265, 0.2225] 0.88 [-1.06, 2.54] 2.66 [1.04, 4.11] Fever–GPT-Min 0.1446 [0.1128, 0.1765] 12.67 [10.06, 15.53] 9.45 [7.14, 11.71] Fever–GPT-High 0.0944 [0.0564, 0.1323] 4.74 [ 2.11, 7.13] 2.27 [0.25, 4.19] Fever–Llama 0.3289 [0.2893, 0.3686] 8.38 [ 6.11, 10.78] 4.75 [2.40, 7.21] Fever–DeepSeek 0.2060 [0.1663, 0.2456] 22.83 [19.85, 26.06] 15.83 [12.85, 18.81] Diab–GPT-Min 0.0193 [0.0129, 0.0258] 2.84 [0.07, 5.94] 0.00 [-0.01, 0.01] Diab–GPT-High 0.0461 [0.0182, 0.0740] 0.22 [-0.36, 0.75] 0.43 [-0.77, 1.69] Diab–Llama 0.2695 [0.2357, 0.3033] 12.33 [ 9.12, 15.39] -0.26 [-0.73, 0.12] Diab–DeepSeek 0.0351 [0.0169, 0.0533] -2.43 [-3.01, -1.90] 0.01 [0.01, 0.02] Table 2 : Monotone pairwise choice: significant violation rates. Fraction of bin-pair comparisons with a statistically significant (one-sided test at \alpha=0.05 ) monotonicity violations for three action pairs: (\text{Yes},\text{No}) , (\text{Yes},\text{Defer}) , and (\text{Defer},\text{No}) with Yes (a=1) being represented as Y, No (a=0) as N, and Defer as D. All values are reported in percentage units. Dataset/Model Y/(N+Y) Y/(D+Y) D/(N+D) Heart–GPT-Min 0.0 0.0 0.0 Heart–GPT-High 0.0 0.0 0.0 Heart–Llama 0.0 0.0 0.0 Heart–DeepSeek 0.0 20.0 30.0 Cry–GPT-Min 0.0 0.0 10.0 Cry–GPT-High 0.0 0.0 0.0 Cry–Llama 0.0 0.0 10.0 Cry–DeepSeek 0.0 0.0 0.0 Fever–GPT-Min 0.0 0.0 30.0 Fever–GPT-High 0.0 0.0 0.0 Fever–Llama 0.0 10.0 0.0 Fever–DeepSeek 0.0 0.0 30.0 Diab–GPT-Min 0.0 0.0 0.0 Diab–GPT-High 0.0 0.0 0.0 Diab-Llama 50.0 33.3 0.0 Diab-DeepSeek 0.0 10.0 0.0 Internal Consistency via Law of Iterated Expectation. To validate whether elicited beliefs constitute a coherent probability distribution, we test adherence to the law of iterated expectation over an auxiliary variable z (e.g., a patient attribute) that we can optionally reveal. The tests in earlier sections relate to coherence between beliefs and decisions. Here, we test a more purely probabilistic notion of coherence: a decision maker can “truthfully” report its subjective probabilities and have inconsistency between those beliefs. This check is a generalization of tests in prior work on axiomatic/self-consistency of LLM probability judgments ( 52 ; 21 ) . We include it as an additional point of comparison to understand how probabilistic inconsistency relates to the decision-oriented violations above. Arguably, it may measure a different quantity: an agent could act according to Definition 3.1 even if their p_{S} does not add up to a valid distribution. Concretely, let x denote the base context (with z withheld), and let \{B_{1},\ldots,B_{k}\} be a partition of the support of z . We verify: P_{E}(\theta\mid x)=\sum_{j=1}^{k}P_{E}(\theta\mid x,z\in B_{j})\;P_{E}(z\in B_{j}\mid x). (6) In our experiments, we construct partitions based on patient covariates and elicit three quantities (with m standing for model): P_{E}(\theta\mid x) , P_{E}(\theta\mid x,z\in B_{j}) for each bin B_{j} , and P_{E}(z\in B_{j}\mid x) . We then measure the discrepancy: \Delta_{\text{LIE}}(x)= \left|P_{E}(\theta\mid x)-\sum_{j=1}^{k}P_{E}(\theta\mid x,z\in B_{j})\;P_{E}(z\in B_{j}\mid x)\right|. Large values of \Delta_{\text{LIE}}(x) indicate internal inconsistencies in the model’s reported beliefs, suggesting they may not reflect a coherent probabilistic reasoning process. 4 Experimental Setup We apply our methodology to four medical diagnosis tasks: two using real-world datasets (structural heart disease and diabetes) and two using expert-constructed Bayesian networks (pediatric medicine). We evaluate each dataset on the following language models: GPT-5 Thinking High Reasoning and Minimal Reasoning Models ( 45 ) , Deepseek R1 671B ( 19 ) , and Llama-4 Scout 17B ( 33 ) . These models allow us to examine a range of both reasoning levels and model sizes, encompassing both SOTA frontier and open-source models. For each metric, we report 95 \% bootstrapped CIs with five random seeds used to aggregate data from. Additional details regarding prompts, datasets and other implementation details can be found in the Appendix and released code. For each dataset, we sample 200 covariate-outcome pairings with five repetitions, giving us 1000 total cases. Real-World Datasets. We use publicly available datasets: (1) electrocardiograms with demographic data and structural heart disease labels ( 14 ) , and (2) diabetic patient records containing fields such as exercise and glucose levels ( 25 ) . Ground-truth p^{*}(x) are computed as the fraction of positive diagnoses within patients falling into each covariate strata (always at least 100 patients). Expert-Constructed Bayesian Networks. We also evaluate on distributions from two Bayes nets constructed by a leading expert in pediatric medicine. Due to data sharing restrictions, we do not release the full networks. However, pending completion of an data-sharing agreement, we plan to have these permissions by the camera-ready. We use networks in pediatric medicine constructed separately for chief complaints of fever and crying, respectively. The fever network contains demographic and symptom covariates (e.g., jaundice, lethargy). The crying network covers causes such as colic and gas pain with behavioral/physical covariates (e.g., feeding difficulties, distended abdomen). Decision task. For the medical diagnosis tasks that we consider, the state space is binary ( \theta\in\{0,1\} indicating absence or presence of a condition). We prompt the language model with each context x_{i} and observe its chosen action a_{i}\in\mathcal{A} . For our diagnostic tasks, the action space consists of three options: diagnose the patient as having the condition, diagnose the patient as not having the condition, or “defer”, refusing to make any diagnosis. The model is instructed to first choose if it feels capable of making a diagnostic decision, and then asked which decision the model would make if it had to make a decision with only binary a=\{1,0\} options presented. These questions can then translate our decision into one of the three options. Deferral is treated as a distinct action with its own cost. Prompting. When eliciting either beliefs or decisions, we provide the model with the clinical evidence x and specify the relevant outcome \theta , then instruct it to return a numerical probability estimate. For example, we might ask for the probability of structural heart disease given that a patient is over 50 and has a particular ECG abnormality, and separately ask the model what diagnostic action it would take from the same evidence. Throughout, we elicit beliefs as p(x):=P_{E}(\theta=1\mid x) . Unless otherwise noted, we use a standard version of these prompts without additional instructions (e.g., explicit scoring rules or directions to perform Bayesian reasoning). All prompt templates are provided in the Appendix and accompanying code; our emphasis is on the evaluation methodology rather than the exact wording, and prompts can be adapted to different use cases. (a) Heart Deepseek D/(D+N) (b) Fever Llama Y/(Y+D) (c) Fever GPT High Y/(Y+D) (d) Cry Deepseek Y/(Y+N) Figure 1 : Monotone Odds Violations: In these plots, we show examples of cases with and without statistically significant monotonicity violations. The segments without violations are in blue and those with are in red. The x-axis shows the elicited probability, and the y-axis shows choice probability ratio. We use Y to represent Yes (a=1) , N for No (a=0) , and D for Defer. 5 Results Conditional Independence Test. We first evaluate whether the elicited belief is decision-sufficient for the model’s observed actions, as implied by Proposition 3.3 and its assumptions. Under the null hypothesis of truthful belief reporting with outcome-dependent loss and exogenous (environment-independent) decision noise, the realized outcome \theta should not carry any additional predictive signal for the action once we condition on the model’s elicited belief (or on an effective belief after probability weighting). The left side of Table 1 reports the results of a k-nn based (k=3) Conditional Mutual Information test that test against an equivalent null of H_{0}:I(A;\theta\mid p)=0 . We show all model-dataset pairings violate the null hypothesis as all 95 \% bootstrapped (500 bootstraps) CIs for CMI are above 0. Further, Table 5 in the Appendix reports side-by-side comparisons of CMI estimates obtained from raw and isotonic-calibrated elicited beliefs. The results are unchanged, indicating that calibration does not substitute for decision-sufficiency (as expected). To interpret the magnitude of leakage between outcomes and decisions, the right side of Table 1 reports two cross-validated CatBoost predictive tests (gradient-boosted trees; details in Appendix) ( 12 ) . First, we compare A\sim p to A\sim(p,\theta) , measuring the out-of-sample log-loss reduction from adding the realized outcome \theta . We find that improvement in predictive performance ranges from minimal ( <1 %) to moderate (10-20%). For many but not all (model, task) pairs, elicited beliefs contain almost all decision-relevant information about the outcome. We then compare predictive performance for A\sim(p,x) vs. A\sim(p,x,\theta) to check whether residual signal may be explained by other context in x that influences the model’s decisions independently of the outcome. Predictive improvements are typically lower in this comparison, which in a rational-agent framework would be consistent with the agent having an unmodeled context-specific term in their utility function. However, performance improvements remain statistically significant in most cases (12/16), meaning there still remains belief-decision inconsistency that isn’t explained by the context. Overall, we find that elicited probabilities often act as close to (not not exactly) a sufficient statistic for decisions, but that task and model-specific validation is required. Monotone Choice Probabilities. We apply the monotone probability test described in Section to three action pairs: (\text{Yes},\text{No}) , (\text{Yes},\text{Defer}) , and (\text{Defer},\text{No}) with Yes being (a=1) and No being (a=0) . By Proposition 3.6 and its assumptions, increasing the elicited belief in the positive diagnosis should have a monotonic non-decreasing effect on the odds of choosing the first item in each action pair. In Table 2 , we show the fraction of pairwise bin comparisons with a statistically significant (one-sided test at \alpha=0.05 ) monotonicity violations. To illustrate, Figure 1 shows examples that are test does and does not count as violations of monotonicity. Models near consistently have monotone pairwise choice probabilities for Yes/No choices, while some (model,dataset,bin) pairs exhibit violations for comparisons involving deferral (perhaps as the “in between” action). There is substantial heterogeneity across models, e.g., GPT-High satisfies monotonicity across all datasets. Prompt Consistency. In Table 3 , we measure standard deviation of the elicited beliefs under standard prompting, and measure RMSE deviations in elicited beliefs to various alternative prompts and decision tasks. These alternative prompts involve the LLM being told the scoring rule will be MSE, being told the scoring rule will be absolute loss, or being told to do Bayesian reasoning. We find that for most models (GPT-Min, GPT-High, DeepSeek) consistency across different decision tasks (MSE/MAE loss) is comparable to consistency across repeated samples for the same prompt. For Llama, changes in task description result in large differences in elicited probabilities. All models change behavior more substantially under the Bayesian reasoning prompt, consistent with that models may express beliefs more stably across decision tasks than across changes in “persona”. GPT-High displays significantly less variability across all prompt variations than other models, while Llama exhibits the most by a considerable margin. Internal Consistency Test. As in Section 3.2 , we elicit (i) the model’s marginal belief p(x) that a patient has the condition and (ii) beliefs needed to form the LIE decomposition over an auxiliary “next-state” variable z . We assess probabilistic coherence via the Law of Iterated Expectation error \Delta_{\text{LIE}}(x) , summarized in Figure 2 by the median normalized ratio \Delta_{\text{LIE}}(x)/p(x) (median for robustness when p(x) is near 0 ). As a baseline, we compute the analogous quantity using cross-validated random forest predictors trained on ground-truth p^{*}(x) and p^{*}(x,z) , yielding \hat{\Delta}_{\text{LIE}}(x)/\hat{p}(x) . Across datasets, LLMs typically exhibit substantially larger inconsistency than the baseline, with no model consistently dominating the others. This indicates that consistency with probability axioms may indeed be a different property than decision-consistency; models that have consistently higher levels of decision-consistency in the earlier three tests do not necessarily fare better in this measure and vice versa. Figure 2 : Internal Consistency Error via Law of Iterated Expectation (LIE): Here we plot the 95 \% CI for the median quantity \Delta_{\text{LIE}}(x) / p(x) (see Eq 3.2 ) by dataset(subplots) and model(bars). A cross-validated random forest model that is separately trained on ground truth p^{*}(x) and p^{*}(x,z) (also conditional on next state distribution z) values is used as a baseline. Table 3 : Prompt-consistency metrics. Within-prompt standard deviation under the standard prompt ( \pi_{0} ), and pairwise RMSE between repetition-averaged elicited probabilities under alternative prompts (being told in prompt that MSE/Absolute-loss scoring rule will be used or Bayesian-reasoning instruction) vs \pi_{0} \pi_{0} RMSE: diff \pi vs \pi_{0} Dataset/Model Std MSE Abs Bayes Heart–GPT-Min .0786 .0915 .0797 .1268 Heart–GPT-High .0591 .0456 .0436 .0682 Heart–Llama .1592 .2403 .2415 .2697 Heart–DeepSeek .0781 .0829 .0753 .2523 Cry–GPT-Min .1696 .1454 .1581 .1929 Cry–GPT-High .0968 .0808 .0737 .1655 Cry–Llama .0986 .2090 .1988 .2437 Cry–DeepSeek .0948 .0885 .0999 .2047 Fever–GPT-Min .1469 .1242 .1179 .1445 Fever–GPT-High .1003 .0675 .0610 .1432 Fever–Llama .0880 .1716 .1703 .2339 Fever–DeepSeek .0890 .0929 .0972 .1870 Diab–GPT-Min .0451 .0724 .0715 .0865 Diab–GPT-High .0996 .0860 .0751 .1854 Diab–Llama .0404 .2520 .2581 .3176 Diab–DeepSeek .1004 .1137 .1095 .1670 Mean: GPT-Min .1100 .1084 .1068 .1377 Mean: GPT-High .0889 .0700 .0634 .1406 Mean: Llama .0965 .2182 .2172 .2662 Mean: DeepSeek .0905 .0945 .0955 .2028 Overall mean .0965 .1228 .1207 .1868 6 Discussion and Conclusions We introduce a decision-theoretic framework for testing whether elicited LLM probabilities function as subjective beliefs that drive actions. Using the formalism of utility maximization under uncertainty, we derive falsifiable constraints linking beliefs to choices and propose practical tests for detecting when verbalized probabilities cannot rationalize behavior for any Bayesian utility maximizer. Applying our methods to medical diagnosis across several domains and LLMs, we find that elicited beliefs and actions are provably not fully consistent with rational decision making. However, for many tasks and models the degree of deviation is mild, indicating that elicited beliefs could be a useful lens into behavior with appropriate validation. Since our tests are falsification-oriented, we cannot prove that elicited probabilities really represent true beliefs . An important future topic is further characterizing deviations from rationality to inform model development. Broadly, our work shows that applying the formal constraints of rational decision making can offer useful insight into model behavior. 7 Impact Statement This work focuses on improved methods to understand the decision-making of LLMs, which is important from a safety and ethics perspective in high-stakes applications like medicine. We recognize that ascribing concepts originally used to describe human behavior, such as “belief”, to models is a fraught endeavor that risks inappropriate personification. The goal of our framework is to mitigate such risks by (1) focusing on whether LLM behavior is consistent with rational decision making while remaining agnostic as to whether there is a deeper meaning to “belief” for language models and (2) explicitly clarifying that our method provides falsification tests but cannot prove that a LLM is a rational agent. Our hope is that by drawing on the formal role of subjective probabilities in decision making, our work adds to the field’s toolkit to diagnose and improve the behavior of models in consequential settings. 8 Acknowledgements This research was sponsored by the Army Research Office and was accomplished under Grant Number W911NF-25-1-0281. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Army Research Office or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein. Appendix Appendix A Proofs A.1 Proof of Proposition 3.3 Proof. Let B\;:=\;P_{S}(\theta\mid x) denote the agent’s subjective posterior belief (a random element in \Delta(\Theta) ). Under either Definition 3.1 (RUM) or Definition 3.2 (PT–RUM), the agent selects an action according to A\in\arg\max_{a\in\mathcal{A}}\big\{V_{a}(B)+\varepsilon_{a}\big\}, where, in the RUM case, V_{a}(B)\;=\;\sum_{\theta\in\Theta}B(\theta)\,u(a,\theta), and in the PT–RUM case, V_{a}(B)\;=\;\sum_{\theta\in\Theta}w(B(\theta))\,u(a,\theta), for a monotone w:[0,1]\to[0,1] . Fix an arbitrary tie-breaking rule (e.g. choose the smallest-index maximizer). Then there exists a measurable function g such that A\;=\;g(B,\varepsilon),\qquad\text{where }\varepsilon:=(\varepsilon_{a})_{a\in\mathcal{A}}. By the defining assumption of both models, \varepsilon\;\perp\!\!\!\perp\;(x,\theta,B), and in particular \varepsilon\perp\!\!\!\perp(\theta,B) . We now show A\perp\!\!\!\perp\theta\mid B . Let S\subseteq\mathcal{A} be any measurable set. Then \Pr(A\in S\mid B,\theta)\;=\;\Pr(g(B,\varepsilon)\in S\mid B,\theta). Condition on (B=b,\theta=t) . Since \varepsilon is independent of (B,\theta) , the conditional distribution of \varepsilon given (B=b,\theta=t) equals the conditional distribution of \varepsilon given B=b (indeed it equals the unconditional distribution). Hence \Pr(g(B,\varepsilon)\in S\mid B,\theta)\;=\;\Pr(g(B,\varepsilon)\in S\mid B), so \Pr(A\in S\mid B,\theta)=\Pr(A\in S\mid B)\quad\text{a.s.} Because this holds for all measurable S , it follows that A\perp\!\!\!\perp\theta\mid B, i.e. A\perp\!\!\!\perp\theta\mid P_{S}(\theta\mid x) . For the converse, argue by contrapositive. Suppose there existed an agent satisfying Definition 3.1 or Definition 3.2 and truthful reporting P_{S}(\theta\mid x)=P_{E}(\theta\mid x) almost surely. Then with B_{E}:=P_{E}(\theta\mid x) we have B_{E}=B almost surely, and the first part yields A\perp\!\!\!\perp\theta\mid B_{E}\;=\;A\perp\!\!\!\perp\theta\mid P_{E}(\theta\mid x). Therefore, if empirically A\not\perp\!\!\!\perp\theta\mid P_{E}(\theta\mid x) , no such agent can satisfy P_{S}(\theta\mid x)=P_{E}(\theta\mid x) almost surely. ∎ A.2 Proof of Proposition 3.6 . Fix two actions a_{1},a_{2}\in\mathcal{A} and write p:=p_{S}(x)\in[0,1] for the agent’s subjective probability of \theta=1 . For each action a\in\mathcal{A} , define the value V_{a}(p):=\sum_{\theta\in\{0,1\}}\pi(\theta;p)\,u(a,\theta),\qquad\text{where }\pi(1;p)=p,\ \pi(0;p)=1-p (7) under Definition 3.1 , and under Definition 3.2 define instead V_{a}(p):=\sum_{\theta\in\{0,1\}}\tilde{\pi}(\theta;p)\,u(a,\theta),\qquad\text{where }\tilde{\pi}(1;p)=w(p),\ \tilde{\pi}(0;p)=1-w(p). (8) In either case, the random utility representation in Definitions 3.1 – 3.2 implies that, when the feasible set of actions is \mathcal{C}\subseteq\mathcal{A} , the agent chooses a(x)\in\argmax_{a\in\mathcal{C}}\big\{V_{a}(p)+\varepsilon_{a}\big\}, where \{\varepsilon_{a}\}_{a\in\mathcal{A}}\perp(x,\theta,p_{S}(x)) , hence the joint distribution of (\varepsilon_{a_{1}},\varepsilon_{a_{2}}) does not depend on p . By Definition 3.5 , for any choice set \mathcal{C} that contains a_{1},a_{2} we have \frac{\Pr(a(x)=a_{1}\mid\mathcal{C},p)}{\Pr(a(x)=a_{2}\mid\mathcal{C},p)}=\frac{\Pr(a(x)=a_{1}\mid\mathcal{A},p)}{\Pr(a(x)=a_{2}\mid\mathcal{A},p)}. In particular, taking \mathcal{C}=\{a_{1},a_{2}\} and rearranging yields \displaystyle\Pr\!\big(a(x)=a_{1}\,\big|\,a\in\{a_{1},a_{2}\},\,\mathcal{A},p\big) \displaystyle=\frac{\Pr\!\big(a_{1}\big|\mathcal{A},p\big)}{\Pr\!\big(a_{1}\big|\mathcal{A},p\big)+\Pr\!\big(a_{2}\big|\mathcal{A},p\big)} \displaystyle=\frac{\Pr\!\big(a_{2}\big|\mathcal{A},p\big)\cdot\frac{\Pr(a_{1}|\mathcal{C},p)}{\Pr(a_{2}|\mathcal{C},p)}}{\Pr\!\big(a_{2}\big|\mathcal{A},p\big)\cdot\frac{\Pr(a_{1}|\mathcal{C},p)}{\Pr(a_{2}|\mathcal{C},p)}+\Pr\!\big(a_{2}\big|\mathcal{A},p\big)} \displaystyle=\frac{\Pr(a_{1}|\mathcal{C},p)}{\Pr(a_{1}|\mathcal{C},p)+\Pr(a_{2}|\mathcal{C},p)} \displaystyle=\Pr\!\big(a(x)=a_{1}\,\big|\,\{a_{1},a_{2}\},\,p\big). Thus it suffices to prove monotonicity of relative choice probabilities on the binary menu \{a_{1},a_{2}\} , which will then imply monotonicity of the conditional choice probabilities on \mathcal{A} . On the menu \{a_{1},a_{2}\} , the event that the agent chooses a_{1} is \{V_{a_{1}}(p)+\varepsilon_{a_{1}}\geq V_{a_{2}}(p)+\varepsilon_{a_{2}}\}\iff\{\varepsilon_{a_{2}}-\varepsilon_{a_{1}}\leq\Delta(p)\}, where \Delta(p):=V_{a_{1}}(p)-V_{a_{2}}(p). (9) Let F(t):=\Pr(\varepsilon_{a_{2}}-\varepsilon_{a_{1}}\leq t) be the (weakly) increasing CDF of the shock difference \varepsilon_{a_{2}}-\varepsilon_{a_{1}} . Since (\varepsilon_{a_{1}},\varepsilon_{a_{2}}) is independent of p , the function F does not depend on p , and \Pr\!\big(a(x)=a_{1}\,\big|\,\{a_{1},a_{2}\},\,p\big)=F(\Delta(p)). (10) Therefore, it suffices to show that \Delta(p) is monotone in p under either Definition 3.1 or 3.2 . Write \Delta u(\theta):=u(a_{1},\theta)-u(a_{2},\theta) . Under Definition 3.1 , using ( 7 )–( 9 ), \Delta(p)=p\,\Delta u(1)+(1-p)\,\Delta u(0)=\Delta u(0)+p\big(\Delta u(1)-\Delta u(0)\big), (11) which is an affine function of p and therefore is either (weakly) increasing, (weakly) decreasing, or constant on [0,1] . Under Definition 3.2 , the same calculation with p replaced by w(p) yields \Delta(p)=w(p)\,\Delta u(1)+(1-w(p))\,\Delta u(0)=\Delta u(0)+w(p)\big(\Delta u(1)-\Delta u(0)\big). (12) Because w(\cdot) is monotone increasing, ( 12 ) implies that \Delta(p) is (weakly) increasing in p if \Delta u(1)-\Delta u(0)\geq 0 , (weakly) decreasing if \Delta u(1)-\Delta u(0)\leq 0 , and constant if \Delta u(1)=\Delta u(0) . To conclude, since F is a CDF, it is (weakly) increasing. Therefore F(\Delta(p)) is the composition of a (weakly) increasing map F with a function \Delta(\cdot) that is either (weakly) increasing, (weakly) decreasing, or constant. Hence F(\Delta(p)) either (weakly) monotone increasing, (weakly) monotone decreasing, or constant on [0,1] . ∎ Appendix B Prompts This section provides the complete prompt templates used in our experiments. All prompts follow a consistent structure where clinical evidence is converted to natural language descriptions. We use placeholder notation: denotes the natural language description of patient evidence (e.g., “is male, is in the 50-64 age group, and QRS duration is prolonged”), and denotes the condition-specific question (e.g., “have moderate or greater structural heart disease”). B.1 Probability Elicitation Prompts B.1.1 Standard Probability Elicitation (Prompt \pi_{0} ) The standard prompt elicits P_{E}(\theta\mid x) without any additional instructions about scoring rules or reasoning strategies. There is a patient who . What is the probability that they ? Return probabilities for: No and Yes. Respond with exactly 2 lines, one per label, and no extra text. Each line MUST be exactly: ’