Optimal Use of Preferences in Artificial Intelligence Algorithms
Joshua S. Gans · 2026
Read Optimal Use of Preferences in Artificial Intelligence Algorithms by Joshua S. Gans (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-18732.txt.
Introduction
Joshua Gans asks when a smart tool should be taught with folk's likes built in, and when likes should be laid on late. A tool can learn likes in two spots. It can bake them into its training aim. Or it can learn the facts clean, then have likes laid on its calls at the point of use. Which path serves folk best is a live ask for firms and for law.
Gans uses a frame from the maths of how facts are shared by plan. He finds terms on which the split path — clean first, likes late — is best. The terms hold across all tasks of choice, not just one. That breadth is the force of the claim. It says teams need not re-ask the ask for each new use. A clean split could also make tools more easy to check and to bend to new likes.
Catalog record
| CREATOR | Joshua S. Gans |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 125 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-18732
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Optimal Use of Preferences in Artificial Intelligence Algorithms Joshua S. Gans Abstract Machine learning systems embed preferences either in training losses or through post-processing of calibrated predictions. Applying information design methods from Strack and Yang (2024), this paper provides decision-problem-agnostic conditions under which separation—training preference-free and applying preferences ex post is optimal. Unlike prior work that requires specifying downstream objectives, the welfare results here apply uniformly across decision problems. The key primitive is a diminishing-value-of-information condition: relative to a fixed (normalised) preference-free loss, preference embedding makes informativeness less valuable at the margin, inducing a mean-preserving contraction of learned posteriors. Because the value of information is convex in beliefs, preference-free training weakly dominates for any expected-utility decision problem. This provides theoretical foundations for modular AI pipelines that learn calibrated probabilities and implement asymmetric costs through downstream decision rules. However, separation requires users to implement optimal decision rules. When cognitive constraints bind—as documented in human-AI decision-making—preference embedding can dominate by automating threshold computation. These results provide design guidance: preserve optionality through post-processing when objectives may shift; embed preferences when decision-stage frictions dominate. Journal of Economic Literature Classification Numbers: C45, C53, D81, D82, D83. Keywords : loss functions, proper scoring rules, Bayesian decision theory, information design, convex order, cost-sensitive learning, post-processing. 1 Introduction Prediction is rarely an end in itself. Predictions are inputs into decisions: lending and insurance, admissions and hiring, medical triage, and the targeting of scarce public resources. This creates a fundamental architectural choice that appears in almost every applied pipeline, but is most salient in the deployment of artificial intelligence (AI). When building a prediction algorithm, should the training objective directly encode the eventual user’s specific preferences (e.g., asymmetric costs or safety priorities), or should the algorithm be trained to learn the underlying truth (calibrated probabilities), leaving preferences to be implemented afterwards? This is not merely a technical question of hyperparameter tuning; it is a choice between building a specialised tool tied to a single, static objective or a general capability that preserves the option value of the information for diverse and changing downstream tasks. A classical decision-theoretic view suggests that preferences belong in the loss function. In particular, 10 show how a decision problem induces an evaluation loss: the utility shortfall from acting on a forecast rather than on perfect information. When the forecasting method is taken as fixed, this logic is compelling: changing the evaluation loss changes how forecasts are ranked and how actions are chosen, but it does not change what information the forecast contains. 1 1 1 This approach has its roots in the work of 21 ; 22 ; see 8 for a discussion in the context of AI and machine learning. To illustrate the main findings of the paper, it is useful to put this in the context of a simple statistical decision problem. Suppose that there is an unknown state of the world, Y , that can take on values of 0 or 1 . There are two actions, a\in\{0,1\} , where the decision maker receives 0 if they match the action to the state but face costs c_{\mathrm{FP}} if (a,Y)=(1,0) and c_{\mathrm{FN}} if (a,Y)=(0,1) . Suppose the prior probability of Y=1 is \frac{1}{2} . There are two ways to approach making this decision. Each involves generating a signal from available data with posterior, q=\Pr[Y=1|\cdot] . First, you could generate a signal that embeds preferences in the signal itself. For instance, that signal could use class-weighted cross-entropy, L^{w}(p,1)=-w_{1}\log p and L^{w}(p,0)=-w_{0}\log(1-p) , that generates a Bayes-optimal report at belief q of: s(q)=\frac{w_{1}q}{w_{1}q+w_{0}(1-q)}. That is, the report is a cross-weighting of the posterior. In this case, the decision-maker will choose a=1 if and only if s(q)\geq\frac{1}{2} yielding an expected payoff of: \mathbb{E}[u_{\mathrm{emb}}]=-\mathbb{E}\!\left[\,q\,c_{\mathrm{FN}}\mathbf{1}\!\left\{s(q)<\tfrac{1}{2}\right\}+(1-q)\,c_{\mathrm{FP}}\mathbf{1}\!\left\{s(q)\geq\tfrac{1}{2}\right\}\right]. The second approach would be for the decision-maker to obtain a signal without any weights. The decision-maker then post-processes preferences . The agent forms posterior belief q and will optimally choose a=1 if and only if q\geq\tau , where \tau=\frac{c_{\mathrm{FP}}}{c_{\mathrm{FP}}+c_{\mathrm{FN}}}. This yields an expected payoff of: \mathbb{E}[u_{\mathrm{post}}]=-\mathbb{E}\!\left[\,q\,c_{\mathrm{FN}}\mathbf{1}\{q<\tfrac{c_{\mathrm{FP}}}{c_{\mathrm{FP}}+c_{\mathrm{FN}}}\}+(1-q)c_{\mathrm{FP}}\mathbf{1}\{q\geq\tfrac{c_{\mathrm{FP}}}{c_{\mathrm{FP}}+c_{\mathrm{FN}}}\}\right]. To compare this with preference-embedding note that, since s(q)\geq\tfrac{1}{2}\Longleftrightarrow q\geq\frac{w_{0}}{w_{0}+w_{1}} , \mathbb{E}[u_{\mathrm{emb}}] can be written as \mathbb{E}[u_{\mathrm{emb}}]=-\mathbb{E}\!\left[\,q\,c_{\mathrm{FN}}\mathbf{1}\!\left\{q<\tfrac{w_{0}}{w_{0}+w_{1}}\right\}+(1-q)\,c_{\mathrm{FP}}\mathbf{1}\!\left\{q\geq\tfrac{w_{0}}{w_{0}+w_{1}}\right\}\right]. Note that \mathbb{E}[u_{\mathrm{post}}]=\mathbb{E}[u_{\mathrm{emb}}] if, following the 10 recommendation, w_{0}:w_{1}=c_{\mathrm{FP}}:c_{\mathrm{FN}} . This equivalence highlights the challenge associated with preference-embedding. The problem is that, in practice, in machine learning, the loss is used not only to evaluate predictions but to train the algorithm that produces them. When learning is endogenous, changing the loss changes what the algorithm learns. This wedge is central in the cognitive-economic view of learning as endogenous information acquisition ( 6 ) . Recently, 4 (ACMM) have sharpened this critique by identifying a specific incentive failure: they show that utility-weighted objectives “flatten” the marginal incentives to learn in parts of the belief space, producing predictors that are, ironically, less useful even for the objective they were trained on. While they characterise this empirically as an incentive failure, operationally, training a preference-free probability model and then adjusting decisions ex post can outperform preference-embedded training even under the preference-weighted evaluation criterion. Suppose, for instance, that preference embedding is implemented by training with a weighted classification loss (e.g. class-weighted log loss) and then using a default cutoff on the model’s output (often 0.5 ), on the theory that the weights have already encoded the preference asymmetry. The crucial difference from the frictionless illustrative above is that modern ML training almost never minimises the weighted loss alone: it minimises a weighted loss plus standard controls on complexity such as weight decay, early stopping, or other regularisation. These controls penalise extreme decision boundaries and therefore shrink predicted probabilities toward the middle. Importantly, the regularisation term does not scale with the cost weights in the same way as the data-fit term, so the mapping from “desired cost ratio” to “implemented decision rule” becomes unstable and model-dependent: changing the weights changes not just the implied cutoff but also what distinctions the model bothers to represent. To see the consequence in the simplest possible way, suppose the underlying signal sometimes yields a low posterior q_{L} and sometimes a high posterior q_{H} , with q_{L}<\tau<q_{H} . Under post-processing or separation , a calibrated learner can preserve this spread and the decision-maker acts exactly in the high-posterior cases, as the threshold rule prescribes. Under weighted training with regularisation, the learned outputs are systematically pulled inward (the high cases become less extreme and the low cases become less low). In the extreme, the learner can collapse the two cases to nearly the same score, or at least eliminate the gap relative to the economically relevant threshold. Once that happens, no ex post thresholding can restore the lost option value: if the model output no longer separates the high- q states from the low- q states, then every downstream rule based only on that output must treat them (almost) the same. Separation avoids this failure mode because it targets the stable object (calibrated beliefs) and postpones the preference choice to a transparent, easily adjusted post-processing step. This paper generalises these insights into a robust separation principle, framing the core architectural question as one of learning versus choosing: should the algorithm learn calibrated beliefs about the world, or should it directly learn which actions to choose? While ACMM demonstrate the