The Burden of Interactive Alignment with Inconsistent Preferences
Ali Shirali · 2025
Read The Burden of Interactive Alignment with Inconsistent Preferences by Ali Shirali (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-16368.txt.
Introduction
Ali Shirali, writing alone, asks what it costs users to steer an algorithm toward what they truly want. Platforms learn from clicks and watch time, but users are inconsistent: an impulsive side lingers on content the reflective side values little. That noise can train the algorithm toward the wrong tastes.
Shirali models the user as two systems, a rational planner and an impulsive browser, in a Stackelberg game with the algorithm. The planner commits to a strategy, and the algorithm best-responds to what it sees. He defines the burden of alignment as the shortest span over which users must plan ahead to win. A sharp threshold appears: plan far enough and alignment succeeds, fall short and the algorithm aligns you instead. One small, costly signal, like an extra click, can shrink that burden a lot.
Catalog record
| CREATOR | Ali Shirali |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 74 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-16368
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
The Burden of Interactive Alignment with Inconsistent Preferences Ali Shirali Abstract From media platforms to chatbots, algorithms shape how people interact, learn, and discover information. Such interactions between users and an algorithm often unfold over multiple steps, during which strategic users can guide the algorithm to better align with their true interests by selectively engaging with content. However, users frequently exhibit inconsistent preferences: they may spend considerable time on content that offers little long-term value, inadvertently signaling that such content is desirable. Focusing on the user side, this raises a key question: what does it take for such users to align the algorithm with their true interests? To investigate these dynamics, we model the user’s decision process as split between a rational “system 2 2 ” that decides whether to engage and an impulsive “system 1 1 ” that determines how long engagement lasts. We then study a multi-leader, single-follower extensive Stackelberg game, where users, specifically system 2 2 , lead by committing to engagement strategies and the algorithm best-responds based on observed interactions. We define the burden of alignment as the minimum horizon over which users must optimize to effectively steer the algorithm. We show that a critical horizon exists: users who are sufficiently foresighted can achieve alignment, while those who are not are instead aligned to the algorithm’s objective. This critical horizon can be long, imposing a substantial burden. However, even a small, costly signal (e.g., an extra click) can significantly reduce it. Overall, our framework explains how users with inconsistent preferences can align an engagement-driven algorithm with their interests in a Stackelberg equilibrium, highlighting both the challenges and potential remedies for achieving alignment. 1 Introduction We study the interactions between human users and an algorithm over multiple steps. While these interactions may benefit both parties, the user and the algorithm can have misaligned interests. Our focus is on the user side, asking what it takes for a user to align the algorithm with her interests . We explore this alignment problem in an interactive environment where users may exhibit inconsistent preferences within an incentive-aware framework. The following explains our setting in detail. Our setting and model. When users have consistent preferences—i.e., the engagement length is in proportion to user’s true reward —the alignment problem reduces to engagement maximization. Recent advances in designing instruction-following language models rely on this assumption of consistent preferences, often using models such as Bradley-Terry [ 1 ] to directly or indirectly infer human rewards and optimize them to maximize human approval [ 2 , 3 , 4 ] . Similarly, recommender systems are typically designed to optimize recommendations that maximize user engagement [ 5 ] . Users can, however, have inconsistent preferences, where their actions may not reflect their true interests. This often occurs when a user’s decision results from a combination of impulsive system 1 and rational system 2 processes [ 6 , 7 , 8 ] , or when consumption choices are influenced by both long-term benefits (enrichment) and a desire for instant gratification (temptation) [ 9 ] . In such cases, revealed preferences do not necessarily align with the user’s true preferences. In general, due to inconsistent preferences or misaligned objectives [ 10 , 11 , 12 ] , the algorithm’s goal may diverge from the user’s. Can a strategic user then still succeed in aligning the algorithm with her interests? This question is central to our study, where we examine the conditions and strategies that enable users to steer the algorithm despite such misalignments. We model the alignment problem as a multi-leader, single-follower Stackelberg game, where users act as leaders by committing to strategies, and the algorithm best responds. We group users with similar interests under the same user type to capture preference heterogeneity. Our model reflects real-world settings in which users can initiate or terminate interactions, and excludes scenarios such as strategic classification [ 13 , 14 ] where the algorithm first commits to a strategy, such as a classifier. By assuming the algorithm best responds, we abstract away its learning process [ 15 , 16 ] . This is reasonable as modern systems interact frequently with users and learn from abundant behavioral data. Our Stackelberg game captures a dynamic setting where a user interacts with the algorithm over multiple steps within a session. After each interaction, the algorithm updates its posterior over the user’s type, gradually personalizing its responses. As a result, the strategies of other users affect how well an individual can steer the algorithm. The user’s goal is to guide the algorithm toward higher-reward recommendations over time. However, she must trade off long-term signaling with short-term rewards, creating a tension that places the burden of alignment on the user. Our results. To quantify the burden of alignment on the user, we ask: over what future horizon must the user optimize her strategy for the cost of alignment to be worthwhile? We propose this horizon as a measure of burden. When users exhibit inconsistent preferences, system 2 must be sufficiently foresighted to align the algorithm with user’s true interests. We show that each user type has a critical threshold: only if the user’s horizon exceeds this can she steer the algorithm; otherwise, similar to [ 17 ] , the algorithm effectively aligns her to its own objective, despite the user leading the game. The significant burden on users prompts the question: what design features can ease alignment? We explore a setting where users can exert observable effort during each interaction—such as clicking a small, non-beneficial button—as a way to signal system 2 ’s strong disinterest. Though the effort yields no direct reward, it enables users to convey preferences without disengaging, simplifying their strategies. As in the baseline setting, alignment still requires a critical optimization horizon. However, allowing users to “burn effort” as a signal [ 18 ] substantially reduces the burden of alignment. In summary, we present a framework to analyze the alignment problem between an optimal algorithm and users with inconsistent preferences. By fully characterizing the equilibria in a multi-leader Stackelberg game, where users lead, we quantify the burden of alignment in terms of the horizon over which users must optimize. Our framework also guides the design of improved alignment environments by quantifying how costly signaling through effort-burning can reduce this burden. As noted by Dean et al. 2024a , we believe that such formal mathematical models can provide valuable insights for the broader field of interaction design in future studies. Additional related work. The alignment problem has been studied across computer science, social science, and economics as a game between users and platforms/algorithms. The most closely related setting is a Stackelberg game in which the leader—typically the platform—commits to a strategy over multiple interactions, and the follower—a user—responds. Because recommendations unfold over multiple steps, users may act strategically, anticipating how their responses influence future outcomes. Haupt et al. 2023 term such users strategic , in contrast to myopic users who optimize locally. Similar to our work, they show that users tend to engage in behaviors that accentuate differences relative to users with other preference profiles. Haupt et al. 2023 and Cen et al. 2024 provide empirical support for this behavior, while Cen et al.