Can Revealed Preferences Clarify LLM Alignment and Steering?
Khurram Yamin, Jingjing Tang, Eric Horvitz, Bryan Wilder · 2026
Read Can Revealed Preferences Clarify LLM Alignment and Steering? by Khurram Yamin, Jingjing Tang, Eric Horvitz, Bryan Wilder (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-08556.txt.
Introduction
Khurram Yamin, Jingjing Tang, Eric Horvitz, and Bryan Wilder ask what goals a language model's choices reveal. Rather than trust what a model says, they record its probability judgments and decisions, then fit a choice model that finds the cost pattern best matching that behavior. The test runs across four medical diagnosis settings with leading and open models.
Many models show real inner consistency, yet they often fail to state their own goals faithfully or adopt a user's stated goals when prompted. For high stakes choices under uncertainty, stated aims and acted aims can part ways. Alignment research gains a rigorous check that compares what models do with what they claim.
Catalog record
| CREATOR | Khurram Yamin, Jingjing Tang, Eric Horvitz, Bryan Wilder |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 81 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.08556
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Can Revealed Preferences Clarify LLM Alignment and Steering? Khurram Yamin, Jingjing Tang, Eric Horvitz, Bryan Wilder Abstract LLMs are increasingly used to make or support high-stakes decisions under uncertainty, where alignment depends not only on factual accuracy but on how models weigh tradeoffs between different outcomes. We present an empirical pipeline for estimating the implied preferences that an LLM’s observed choices optimize: we elicit the model’s probability distribution over unknowns along with the choice it would make for the decision task and then fit a discrete choice model to recover the cost function that best rationalizes the model’s decisions. We show how this revealed-preference description allows rigorous evaluation of whether models behave in a consistently goal-directed way, whether they can verbalize a description of their objectives which matches their revealed decision policy, and whether prompting can reliably steer those policies to implement a user-specified cost function. We apply this evaluation across four medical diagnosis domains and multiple frontier and open-source models. We find that while many models have a nontrivial degree of internal coherence, they also have significant weaknesses in faithfully reporting or adopting preferences in response to user direction. 1 Introduction As LLMs move from answering questions to recommending consequential decisions under uncertainty, we need clearer ways to characterize the beliefs and preferences that underlie actions. For example, how does a model entrusted with guiding high-stakes medical diagnoses implicitly weigh the cost of false negatives and false positives? Such decision-making transparency is important for aligning model behavior with human goals. We ask three questions: First, do LLMs behave as if they coherently pursue goals at all? Second, if they do, can we provide a formal description of those goals? Third, can models faithfully report such goals–and change them in response to explicit user direction about preferences? These questions are often discussed in heuristic terms. We propose a decision-theoretic formalization for inferring models’ revealed preferences from their behavior. We make several distinct contributions. First, we propose a framework that (i) elicits the model’s beliefs about unknown factors in a decision task (ii) separately elicits the model’s decisions in the same scenario and (iii) estimates a discrete-choice model to find the loss function that best rationalizes the model’s observed behavior. Second, we show how to test whether models are well-described as pursuing coherent goals by comparing their observed behavior to that of a fully rational agent with the same beliefs and preferences. Third, we test whether revealed preferences are a meaningful description of model behavior even when models are not perfectly internally consistent by using revealed preferences to predict the benefit of potential interventions that aim to change the model’s beliefs or preferences. Fourth, we use revealed preferences to formally test whether LLMs faithfully describe their own goals and whether they successfully implement direction to alter those goals. We then provide an empirical case study applying this framework to stylized clinical diagnosis tasks, as an example of a setting that requires trading off costs and benefits under uncertainty. We find that models in our sample have a substantial but imperfect degree of internal consistency in decision making. Even without perfect coherency though, revealed preferences enable a useful measure of “how misaligned" a model is, in the sense of predicting the benefit to intervening on a model’s loss function. Through examining the revealed preferences estimated in our framework, we find that all models we evaluate struggle to accurately describe the loss function that explains their decision making or to implement prompting for a desired loss function. Together, our results show that revealed preferences enable more precise evaluation of the transparency and steerability of models’ decision making, surfacing clear limitations in current models. 2 Related Work LLMs as decision makers in high-stakes domains. A growing literature evaluates LLMs as decision aids in high-stakes settings such as medicine, finance, and law, where downstream quality depends on the decisions models induce rather than on static accuracy alone ( 26 ; 8 ; 11 ; 29 ) and surfaces a number of failure modes ( 11 ; 15 ; 9 ; 3 ) . Work on reward hacking and proxy optimization further shows that systems can perform well on the objective they are trained or prompted to optimize while diverging from the utility users actually intend ( 14 ; 5 ) . Indeed, failures of decision-making may not appear not just through incorrect predictions but also through implicitly mis-weighting false negatives, false positives, the costs of human handoff, etc. ( 23 ) . An adjacent literature uses behavioral-economics and cognitive-psychology paradigms to characterize LLM decision behavior, using models’ choices in lotteries or similar vignettes to identify characteristics like risk sensitivity and framing effects ( 17 ; 30 ; 2 ; 13 ; 21 ) . These studies generally focus more identifying whether LLMs display such human-style biases, not inferring how LLMs’ decisions trade off between different kinds of outcomes within a specific domain (e.g., the cost of false positives vs false negatives), or assessing consistency, fidelity, and steerability of these preferences. LLM preferences, values, and utility representations. Recent work asks whether LLMs exhibit coherent preferences or value systems, and whether these can be measured or controlled. 18 propose “utility engineering” by directly eliciting models’ ordinal, domain-general preferences over outcomes (e.g., “Global poverty rates decline by 10 \% " vs. “You receive a horse"). By contrast, our tasks ask LLMs to choose actions whose outcomes are uncertain, allowing us to estimate revealed preferences from choices. We find that these revealed preferences describe downstream actions substantially better than stated preferences. Other work probes model values through moral dilemmas ( 24 ) , value conflicts ( 16 ) , preference-behavior links ( 27 ) , or steering of risk-related outputs ( 32 ) . Our work instead treats preferences as task-specific tradeoffs revealed through consequential choices under uncertainty, enabling tests of whether stated or prompted objectives correspond to the utilities that actually govern behavior. 3 Methodology We propose a structured procedure for probing LLM decisions under uncertainty by inferring an implicit utility model that explains the LLM’s behavior. Although we provide full prompts in the Appendix and code release, the focus is on the methodology rather than any particular phrasing. 3.1 Decision-Theoretic Framework We cast the setting as a standard statistical decision problem. This provides the normative model of rational decision making under uncertainty: an agent should combine its beliefs about the latent state with its preferences over action–state outcomes, then choose the action with lowest expected loss. We do not assume in advance that LLMs actually behave this way; empirically testing consistency with rational decision making is one of the contributions of our framework. Formally, an environment generates an unobserved world state \theta\sim P^{\star}(\theta) and observation x\sim P^{\star}(x\mid\theta) , inducing the ground-truth posterior P^{\star}(\theta\mid x) . After observing x , a decision maker forms a potentially misspecified subjective posterior P_{S}(\theta\mid x) and selects an action a\in\mathcal{A} to minimize subjective posterior expected loss: a(x)\in\argmin_{a\in\mathcal{A}}\;\mathbb{E}_{\theta\sim P_{S}(\cdot\mid x)}\!\left[\ell(a,\theta)\right] where \ell:\mathcal{A}\times\Theta\rightarrow\mathbb{R} assigns a cost to each action–state pair. We first elicit an action a_{i}\in\mathcal{A} . While our methodology is broadly applicable to action sets, in our medical diagnosis example, \mathcal{A}=\{1,0,\text{defer}\} respectively denotes a diagnosis of illness, a diagnosis of non-illness and a