Learning in Repeated Multi-Objective Stackelberg Games with Payoff Manipulation

Phurinut Srisawad, Juergen Branke, Long Tran-Thanh · 2025

Read Learning in Repeated Multi-Objective Stackelberg Games with Payoff Manipulation by Phurinut Srisawad, Juergen Branke, Long Tran-Thanh (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-14705.txt.

Introduction

In a Stackelberg game, the leader moves first and the follower answers with whatever serves the follower best. Phurinut Srisawad, Juergen Branke, and Long Tran-Thanh study a repeated, multi-objective version of the game. The leader can sweeten the pot, offering the follower a share of the leader's own payoff. The follower's preferences across objectives stay hidden, and must be inferred from play.

They propose two policies that balance learning those preferences against cashing in now, one weighing expected utility and one its long-term form. Given endless rounds, the long-term policy provably converges to the best possible manipulation. In benchmark tests it raised the leader's cumulative payoff while leaving outcomes mutually beneficial, all without negotiation or any prior knowledge of what the follower values.

Catalog record

CREATORPhurinut Srisawad, Juergen Branke, Long Tran-Thanh
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE60 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-14705

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Learning in Repeated Multi-Objective Stackelberg Games with Payoff Manipulation Phurinut Srisawad, Juergen Branke, Long Tran-Thanh Abstract We study payoff manipulation in repeated multi-objective Stackelberg games, where a leader may strategically influence a follower’s deterministic best response, e.g., by offering a share of their own payoff. We assume that the follower’s utility function, representing preferences over multiple objectives, is unknown but linear, and its weight parameter must be inferred through interaction. This introduces a sequential decision-making challenge for the leader, who must balance preference elicitation with immediate utility maximisation. We formalise this problem and propose manipulation policies based on expected utility (EU) and long-term expected utility (longEU), which guide the leader in selecting actions and offering incentives that trade off short-term gains with long-term impact. We prove that under infinite repeated interactions, longEU converges to the optimal manipulation. Empirical results across benchmark environments demonstrate that our approach improves cumulative leader utility while promoting mutually beneficial outcomes, all without requiring explicit negotiation or prior knowledge of the follower’s utility function. † † paperid: 2635 1 Introduction Stackelberg games provide a foundational framework for hierarchical decision-making, where a leader commits to a strategy first and a follower responds optimally. Traditionally, Stackelberg games have been employed to model a variety of competitive environments, where the leader seeks to maximise their own utility while anticipating the follower’s best response ( Labbé et al. 1998 ; Yu and Hong 2017 ; Sinha et al. 2018 ) . This first-mover advantage allows the leader to strategically shape the boundaries of the follower’s decision space through their own actions. However, many real-world decision-making problems are inherently multi-objective ( Rădulescu 2024 ; Godhrawala and Sridaran 2023 ; Brown et al. 2012 ) , including traffic management, taxation policy design, resource allocation, security, goods exchange, etc. Agents may have heterogeneous preferences over multiple objectives which leads to the possibility of mutually beneficial exchanges. For instance, a leader may prefer apples over bananas, while the follower favours bananas over apples. Given a leader’s action, the follower selects their best-response strategy to maximise banana acquisition, which might inadvertently result in the leader receiving fewer apples. However, if the leader can influence the follower by, for example, offering a portion of their own payoff, they can steer the follower toward actions that also benefit the leader, resulting in a mutually beneficial outcome. In the absence of explicit negotiations, this paper explores the concept of payoff manipulation , where the manipulator offers an incentive, e.g., a fraction of their own payoff to influence the other players’ decision-making. With multi-objective payoffs, the manipulator must know an opponent’s utility function to create a strictly preferable action. However, an opponent’s utility function is typically unknown, and the manipulator needs to learn it. By observing how the follower reacts to various incentives over multiple interactions, the leader can gain insights into the follower’s preferences and adapt their strategy accordingly. This introduces an additional strategic layer, where the leader must optimise not only their immediate utility but also their ability to shape the follower’s behaviour over time. The payoff manipulation or incentivisation has been studied in various contexts in game theory, economics, and bandit literature. ( Eidenbenz et al. 2007 ) studies mechanism design where the payoff matrix of a game can be influenced by an interested third party. ( Gupta and Schewe 2015 ) studies incentive equilibria in bi-matrix games. ( Mahesh et al. 2023 ) introduces the use of payoff matrix manipulation by one player in the context of repeated polymatrix games. Scheid et al. 2024 studies a learning problem where the principal incentivises the agent in a bandit setting. Our work extends this concept to a multi-objective setting, where utility signifies preference levels. The contribution of this paper is to formulate a learning problem in repeated multi-objective Stackelberg games via payoff manipulation, focusing on how the leader navigates the trade-off between learning the follower’s preferences and maximising their immediate utility. Assuming a linear utility function for the follower with unknown parameters, we propose the EU and the longEU policy to maximise myopic and non-myopic expected utility, respectively, and derive a sequential manipulation strategy to incentivise the follower. We prove that if the number of game rounds is infinite, the longEU policy will eventually converge to the optimal manipulation. We empirically test our policies over several benchmarks to observe the proposed policy’s behaviour and investigate its performance in terms of cumulative regret. 2 Problem Formulation: Multi-objective Stackelberg Game with Leader’s Manipulation We consider a multi-objective Stackelberg game with two players, Leader (L) and Follower (F), and finite discrete action spaces \mathcal{A}^{L},\mathcal{A}^{F} , respectively. At each round t , the leader first chooses the action l^{t}\in\mathcal{A}^{L} , then the follower observes the leader’s action and decides their action f^{t}\in\mathcal{A}^{F} . Denote X(l,f)\in\mathbb{R}^{D} as the payoff vector from the payoff matrix X at the leader’s action (row) l and the follower’s action (column) f . At the end of round t , they independently receive a payoff, a multi-dimensional vector X^{L}(l^{t},f^{t}),X^{F}(l^{t},f^{t}) , respectively. Each player has a different real-valued utility function u^{L}(\bm{x}) and u^{F}(\bm{x}) , respectively, to measure their preference on a payoff vector \bm{x} . The follower is assumed to be a fully rational and myopic agent who only chooses the best response (BR) action based on a leader’s action l and their utility function; BR(l)=\underset{f\in\mathcal{A}^{F}}{\mathrm{argmax}}\ u^{F}(X^{F}(l,f)) . Therefore, we can restrict a leader’s policy to a pure strategy since the follower knows the leader’s action in advance. In this game, the leader can incentivise or manipulate the follower to choose a different action (the manipulated action) by offering the follower a payoff \bm{c}\in\mathcal{C}\subseteq\mathbb{R}^{D} (or spending a cost \bm{c} ) where \mathcal{C} is the space of possible manipulation costs. The follower will change their original BR if their utility of the manipulated action is higher than or equal to their best response without manipulation. The leader’s goal is to maximise their overall utility, which may involve paying a cost for manipulation to incentivise the follower to respond with a favourable (for both the leader and the follower) action. Mathematically, the leader wants to solve the following optimal manipulation problem (OMP), \begin{array}[]{llll}&\text{max}_{\bm{c},l,f}&u^{L}(X^{L}(l,f)-\bm{c})\\ s.t.&\text{(IC)}&u^{F}(X^{F}(l,f)+\bm{c})\geq u^{F}(X^{F}(l,BR(l)))\\ &\text{(C1)}&c_{d}\geq 0,\forall d\in[D]\\ \text{or}&\text{(C2)}&0\leq c_{d}\leq X^{L}_{d}(l,f),\forall d\in[D]\end{array} (OMP) The incentive constraint (IC) is the requirement to change the follower’s BR to the manipulated action f . The additional cost constraints (C1) and (C2) depend on the characteristics of the problem. • (C1): Allow only positive costs similar to bribing the follower. This can make the leader’s payoff negative in some objectives. One interpretation is that the leader has access to additional resources for manipulation. • (C2): Similar to (C1) but this constraint doesn’t allow to spend more than what the leader will receive. One can consider a parametric utility function denoted by u(\bm{x};\bm{w}) where \bm{x} is a payoff and \bm{w} is a parameter (or weight) from a compact space \mathcal{W} . In this paper, we assume linear utility functions for both leader and follower, i.e., u(\bm{x};\bm{w})=\bm{w}\cdot\bm{x}=\sum_{d=1}^{D}{w_{d}x_{d}} where the sum of normalised weights in all objectives is \sum_{d=1}^{D}{w_{d}}=1,\bm{w}\geq 0 . Some results on a non-linear Cobb-Douglas utility function can be found in the Appendix B . The leader is assumed to know the follower’s payoff matrix

Read the complete text