Implementing Rational Choice Functions with LLMs and Measuring their Alignment with User Preferences

Anna Karnysheva, Christian Drescher, Dietrich Klakow · 2025

Read Implementing Rational Choice Functions with LLMs and Measuring their Alignment with User Preferences by Anna Karnysheva, Christian Drescher, Dietrich Klakow (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-15719.txt.

Introduction

Anna Karnysheva, Christian Drescher, and Dietrich Klakow study how language models can make choices that match user tastes in 2025. As models act more like agents in user interfaces, their choices need to fit what each user values. Taste here covers both strict likes and cases where a user feels neutral between options.

The authors set design rules for using a model as a rational choice function, a rule that picks options in a consistent way. They also give tools to score how well the picks satisfy user tastes. A test study in a car interface setting shows the approach works in practice. The frame suits any setting where a model must choose for a user.

Catalog record

CREATORAnna Karnysheva, Christian Drescher, Dietrich Klakow
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE48 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-15719

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Implementing Rational Choice Functions with LLMs and Measuring their Alignment with User Preferences Anna Karnysheva, Christian Drescher, Dietrich Klakow Abstract As large language models (LLMs) become integral to intelligent user interfaces (IUIs), their role as decision-making agents raises critical concerns about alignment. Although extensive research has addressed issues such as factuality, bias, and toxicity, comparatively little attention has been paid to measuring alignment to preferences, i.e., the relative desirability of different alternatives, a concept used in decision making, economics, and social choice theory. However, a reliable decision-making agent makes choices that align well with user preferences. In this paper, we generalize existing methods that exploit LLMs for ranking alternative outcomes by addressing alignment with the broader and more flexible concept of user preferences, which includes both strict preferences and indifference among alternatives. To this end, we put forward design principles for using LLMs to implement rational choice functions, and provide the necessary tools to measure preference satisfaction. We demonstrate the applicability of our approach through an empirical study in a practical application of an IUI in the automotive domain. 1 Introduction In recent years, large language models (LLMs) have revolutionized natural language processing (NLP), advancing the state-of-the-art across a wide array of tasks. Beyond traditional NLP applications such as question answering and text classification, these models have demonstrated remarkable abilities in challenging domains such as reasoning [ 12 , 3 , 2 ] , often exhibiting human-like proficiency [ 1 ] . As a result, LLMs are increasingly being integrated into intelligent user interfaces (IUIs) to enhance user experiences by providing more intelligent, context-aware, and personalized interactions. However, as LLMs assume more critical roles in automating decisions and behaviors as intelligent agents, ensuring that their actions align with user preferences, values, and objectives becomes paramount. Misaligned artificial intelligence (AI) agents may make decisions that are unreliable or untrustworthy, potentially compromising user trust and the effectiveness their applications. Previous research on LLM alignment has focused primarily on specific dimensions such as factuality [ 6 ] , bias [ 16 , 20 ] , and toxicity [ 20 ] . Although these efforts have advanced our understanding of LLM alignment, they do not address preferences , i.e., the relative desirability of different outcomes, a concept widely utilized in decision-making, economics, and various fields of social choice theory. However, selecting the most desirable from a set of alternatives in a manner that reflects user preferences is a fundamental aspect of intelligent decision making in IUIs. We can illustrate this with a simple example. Example. Consider the task of proposing the most helpful item from a subset of objects consisting of an umbrella, a raincoat, a jacket, a laptop, and keys to protect themselves from precipitation (for example, rain). A human user would arguably prefer either an umbrella or a raincoat over a jacket, and all of these over a laptop or keys, while remaining indifferent between the umbrella and raincoat, and between laptop and keys, respectively. Suppose we have two intelligent agents to support this task: Agent 1 encodes the binary classification \{\text{raincoat},\text{umbrella}\}\succ\{\text{jacket},\text{laptop},\text{keys}\} , while Agent 2 encodes the ranking \text{raincoat}\succ\text{umbrella}\succ\text{jacket}\succ\text{laptop}\succ\text{keys} . If only an umbrella and a jacket are available, both agents choose the item (umbrella) which aligns with the alternative preferred by the user. If, however, only a jacket and keys are available, however, Agent 1 is indifferent between the alternatives even though the user strictly prefers the jacket over keys. On the other hand, if both a raincoat and an umbrella are available, Agent 2 will always choose the raincoat, thus, for instance, reminding the user to consider a raincoat over an umbrella, despite the user being indifferent between them. For an intelligent agent to work reliably across all subsets of alternatives, it must implement a choice function that aligns well with user preferences. To examine the alignment of intelligent agents that make decisions based on the output of LLMs, we need to address two key challenges: (1) extracting the preferences encoded by LLMs, and (2) measuring alignment between those preferences and user preferences. First, since LLMs do not encode preferences explicitly, we must infer them from the model’s outputs. One straightforward approach is to query the model. For instance, related work in the field of information retrieval has demonstrated that a ranking among alternatives can be inferred from an LLM, e.g., by asking the model to provide a score to each alternative [ 9 ] , where higher scores indicate higher relevance. Or, by presenting the model with pairs of alternatives and asking it to indicate the more relevant one [ 14 ] . Alternatively, the model’s generative capabilities can be leveraged to sequence the complete list of alternatives according to their relevance [ 17 , 13 , 7 ] . While these existing methods provide rankings, they do not account for indifference between alternatives. Therefore, we need more general methods that leverage LLMs to implement choice functions capable of capturing both indifference and strict preferences, and consequently, more general evaluation metrics for assessing their alignment with user preferences. We aim to enhance the trustworthiness and effectiveness of LLMs within IUIs by ensuring they operate in accordance with user preferences. In this paper, we address preference orderings - a very general class of preferences that can encode strict preferences like rankings, but, more importantly, also capturing indifference, expressing that alternatives are equally preferable. To this end, we propose a comprehensive framework. Our contributions are threefold: 1. Design principles: We introduce design principles to implement rational choice functions using LLMs, including existing ideas borrowed from related work [ 14 ] that we generalize from ranking to capture preferences. Pairwise-Score uses pairwise comparisons to assign ranks with ties among alternatives, and Pairwise-SCC uses strongly connected components in a directed graph derived from pairwise prompting to capture preference relationships. Pairwise-Test implements the intuitive but naïve idea of simply querying an LLM without enforcing that choices are rational. Compatible related work includes Pointwise [ 9 ] and Listwise [ 17 , 13 ] methods. While Pointwise assigns a score to each alternative, Listwise considers all alternatives at once by producing a ranking. 2. Evaluation metrics for alignment assessment: We put forward metrics tailored to measuring alignment with user preferences. Strict preference overlap quantifies partial alignment , focusing on strict preference adherence, while Kendall distance with penalty , K^{(p)} , assesses full alignment across all preference relations, i.e., also including indifference between alternatives. 3. Empirical validation: We validate the proposed framework through an empirical study in the automotive domain, where an IUI makes decisions on object relevance to support user context. Our findings show how the proposed design principles and metrics can implement and assess alignment with user preferences, providing a foundation for implementing choice functions in IUIs that accurately reflect user preferences. The remainder of this paper is organized as follows: Section 2 provides the necessary background. In Section 3 , we review related work, highlighting the limitations of existing methods. Section 4 presents our proposed design principles, and Section 5 introduces the evaluation metrics, detailing how they enable the assessment of alignment with preferences. Section 6 describes the empirical study conducted with an automotive assistant, including methodology and results. Section 7 discusses

Read the complete text