Reward Shaping for Inference-Time Alignment: A Stackelberg Game Perspective Haichuan Wang, Tao Lin, Lingkai Kong, Hezi Jiang, Milind Tambe Abstract Existing alignment methods directly use the reward model learned from user preference data to optimize an LLM policy, subject to KL regularization with respect to the base policy. This practice is suboptimal for maximizing user’s utility because the KL regularization may cause the LLM to inherit the bias in the base policy that conflicts with user preferences. While amplifying rewards for preferred outputs can mitigate this bias, it also increases the risk of reward hacking. This tradeoff motivates the problem of optimally designing reward models under KL regularization. We formalize this reward model optimization problem as a Stackelberg game, and show that a simple reward shaping scheme can effectively approximate the optimal reward model. We empirically evaluate our method in inference-time alignment settings and demonstrate that it integrates seamlessly into existing alignment methods with minimal overhead. Our method consistently improves average reward and achieves win–tie rates exceeding 66% against all baselines, averaged across evaluation settings. $\dagger$ $\dagger$ footnotetext: Equal contribution 1 Introduction Large language models (LLMs) have achieved remarkable capabilities and are now widely deployed across a broad range of language generation tasks. Despite these advances, model outputs often fail to align with users’ preferences, causing both safety-critical harms and benign personalization mismatches. The former include harmful biases – such as racial ( 1 ) , gender ( 25 ) , or cultural ( 26 ) bias – that motivate safety alignment. The latter stems from benign yet consequential deviations from user intent; for example, models may systematically favor particular notions of social equity ( 51 ) or linguistic styles ( 37 ) that conflict with individual user preference. These failure modes across safety and personalization, where models fail to optimize user utility, motivate a large literature on alignment. Existing alignment approaches can be broadly categorized into train-time alignment, which updates model parameters using preference feedback ( 32 ; 34 ) , and inference-time alignment, which steers a fixed model during decoding to adapt to diverse user preferences ( 23 ; 30 ) . Most alignment approaches, either train-time or inference-time, are reward-based : a reward model is learned from users’ preference data and then used to guide LLM policy optimization or inference-time decoding. A key assumption underlying the reward-based alignment pipeline is that directly optimizing for the learned reward is sufficient for alignment. However, this assumption is fragile in practice. In existing alignment practices, LLM policies are constrained to remain close to a base model via a KL-divergence regularizer ( 34 ) . When the base model exhibits strong bias, user preferences may be insufficient to induce desired behavior under the KL constraint. For example, given a strongly politically left-leaning LLM ( 46 ) , directly aligning it to a neutral reward model may still result in a left-leaning model after alignment. Appropriately shaping the reward, such as amplifying the reward assigned to neutral outputs, can more effectively shift the LLM’s output distribution toward user preferences. But excessive reward shaping, on the other hand, risks reward hacking or degenerate outputs ( 13 ) . This tension motivates us to study the following fundamental question: How should we optimally shape the reward model used for alignment? To address this question, we formalize alignment as a Stackelberg game. In our formulation, the reward model provider (the leader ) determines how user preferences are conveyed to the LLM policy by selecting a reward model, while the LLM (the follower ) best responds by optimizing an alignment objective under a KL-divergence constraint. Our theoretical framework explicitly accounts for maximizing user utility and mitigating reward hacking. We derive a closed-form characterization of the optimal reward model that the leader should select, and show that it can be efficiently approximated via simple reward shaping using Monte Carlo samples from the base LLM. We also empirically evaluate our reward shaping scheme, focusing on inference-time alignment, which is lightweight and enables adaptation to diverse user preferences without retraining the model. The main contributions of our work are: (1) We show that directly using a reward model learned from user preference data in the existing alignment pipeline can be suboptimal under KL-regularized objectives (Section 3 ). (2) We formulate a Stackelberg game to solve for the optimal reward model and show that the optimal reward model can be efficiently approximated via Monte Carlo sampling (Section 4 ), enabling easy integration into existing inference-time alignment methods with little additional inference-time overhead (Section 5 ). (3) We empirically evaluate our reward shaping approach on popular inference-time alignment methods and demonstrate consistent improvements in reward while achieving over 66% average win–tie rates against baselines as judged by GPT-4 (Section 6 ). 2 Related Work Train-time and Inference-time Alignment Most alignment methods operate at training time, either by learning a reward model and fine-tuning the policy via RLHF 32 ; 40 , or by directly optimizing on preference data as in DPO 34 and NLHF 31 . While effective, these approaches require modifying model weights and are therefore computationally expensive, and they commit the model to a fixed preference specification that is difficult to adapt post hoc. Inference-time alignment methods address these limitations by modifying the token distribution at decoding time using an external reward signal. Existing approaches differ in their mechanisms – e.g., directly shaping logits with next-token rewards 23 , casting decoding as a heuristic search problem 19 , learning prefix value functions 30 ; 6 , learning a token-level reward model 47 , or perturbing representations to increase predicted reward 24 – but share a common objective: steering a fixed base model using a fixed user-specified reward. In contrast, our work addresses an orthogonal question: how to construct an optimal reward model that improves alignment outcome for users in the first place. Reward Shaping for Alignment Many prior works study reward shaping for improving alignment. 27 argue that trajectory-level rewards are too sparse for effective RL training and introduce token-level rewards. 38 propose contrastive rewards that subtract the offline mean to penalize uncertainty. 45 apply a log-sigmoid transformation to centered rewards to boost low-quality outputs and mitigate reward hacking. 22 propose a minimum Bayes risk objective, which can be interpreted as a Wasserstein-distance regularization in contrast to KL-based methods. Finally, 13 show that mitigating reward hacking requires bounded rewards with rapid initial growth followed by gradual saturation, which they realize using a sigmoid function. These works study how specific reward transformations affect alignment behavior. In contrast, we take a game-theoretic perspective and formulate reward design itself as an optimization problem, explicitly characterizing the reward model that maximizes user preference under the alignment objective. Game-theoretic Alignment The alignment problem has been studied from a game-theoretical perspective. 31 ; 49 ; 35 ; 42 views alignment as two-player simultaneous move game between the policy and any competing policy and aims to approximate the corresponding Nash equilibrium. 7 leverages self-play to generate synthetic data to fine-tune the model. When reward model and policy are iteratively trained in RLHF, the training process could be viewed as a Stackelberg game, where one player moves first and another player follows. Different from us, 29 ; 8 ; 21 view policy as the leader and reward model as the follower. Similar to our work, 5 also views the reward model as the leader and policy as the follower, but they target general RLHF alignment instead of LLMs. Their algorithm requires the Hessian of policy which is impractical to obtain for LLMs, while we leverage the structure of LLM alignment to design practical algorithms. 4 show that labelers have incentives to strategically manipulate their reported preferences under existing RLHF mechanisms, whereas we study optimal design of reward model to improve user utility. Our model is conceptually related to contract design; however, we do not explicitly use tools from this literature and therefore defer a detailed discussion to Appendix A . 3 Limitation of Standard Alignment Pipeline Notation: Let \bm{x}\in X be a prompt, \bm{y}\in Y be a full response. We use \Delta(Y) to denote the set of probability distributions over the space of responses Y . Let y_{t} be the t -th token in the response, and \bm{y}_{m(\bm{x})\\ \end{cases}. Theorem 1 (Optimality of threshold reward) . The optimal solution r^{*} to problem ( 3 ) is a threshold reward model r_{m^{*}} . Moreover, the threshold function m^{*} of the optimal reward model should satisfy the following condition: m^{*}(\bm{x})\,=\,\mathbb{E}_{\bm{y}\sim\rho_{r_{m^{*}}}(\cdot|\bm{x})}\big[\,r_{U}(\bm{x},\bm{y})\,\big],\quad\forall\bm{x}\in X. (4) The proof of this theorem is deferred to Appendix D.1 . Theorem 1 says that, for each prompt \bm{x} , the leader should partition responses by whether their true reward r_{U}(\bm{x},\bm{y}) is above or below a prompt-dependent threshold m^{*}(\bm{x}) , assigning rewards B or 0 to them respectively. Moreover, Equation ( 4 ) requires that the user’s expected utility obtained from the LLM policy \rho_{r_{m^{*}}} guided by the threshold reward model must be equal to the threshold m^{*}(\bm{x}) itself. By providing such a “shaped” reward model to the follower (LLM), the leader maximizes user utility computed on the users’ true reward model. Intuitively, the leader boosts sufficiently preferred responses as much as possible and penalizes all other responses. Our threshold reward structure aligns with the motivating example in Section 3.2 : to counteract base policy bias, the leader exaggerates their preferences relative to the true reward r_{U} , rather than reporting r_{U} directly. 4.3 Computing the Optimal Threshold m^{*}(\bm{x}) Although Theorem 1 shows the optimality of a threshold reward model r_{m^{*}} , it did not discuss how to compute the optimal threshold function m^{*} . In particular, m^{*}(\bm{x}) is a solution to Equation ( 4 ). This subsection provides an efficient algorithm to find the optimal threshold m^{*}(\bm{x}) . Definition 2 . We first define a helper function. Given prompt \bm{x}\in X and threshold m\in\mathbb{R} , let F_{\bm{x}}(m)=\mathbb{E}_{\bm{y}\sim\rho_{\mathrm{base}}(\cdot|\bm{x})}\Big[w_{\bm{x},\bm{y}}(m)\cdot\big(r_{U}(\bm{x},\bm{y})-m\big)\Big] where w_{\bm{x},\bm{y}}(m):=\begin{cases}1&\text{if }r_{U}(\bm{x},\bm{y})0 be a hyperparameter that controls the sharpness of the transition around the threshold, which we refer to as the shaping strength. Given a prompt-dependent threshold \hat{m}^{*}(\bm{x}) (estimated using the Monte Carlo procedure in Section 4.2), define the soft threshold reward: r_{\hat{m}^{*},\alpha}(\bm{x},\bm{y}):=B\cdot\sigma\Big(\alpha\cdot\big(r_{U}(\bm{x},\bm{y})-\hat{m}^{*}(\bm{x})\big)\Big), (6) where \sigma(u)=\frac{1}{1+\exp(-u)} is the sigmoid function. We show that as the shaping strength \alpha varies, SRS (soft) smoothly interpolates between no alignment effect (when \alpha=0 ) and the analytical optimal solution r^{*} (when \alpha\to\infty) . We denote by U_{\mathrm{base}} the user’s expected utility under \rho_{\mathrm{base}} , and by U(r) the expected utility under \rho_{r} . Theorem 3 . Denote the optimal reward model as r^{*} . U(r_{m^{*},\alpha}) is continuous in \alpha , \lim_{\alpha\to 0}U(r_{m^{*},\alpha})=U_{\mathrm{base}} , and \lim_{\alpha\to\infty}U(r_{m^{*},\alpha})=U(r^{*}) . The proof of this theorem is deferred to Appendix D.3 . An immediate consequence of Theorem 3 is that the SRS -shaped reward improves user utility compared to directly using the user utility function r_{U} as the reward model. Corollary 1 . For any user utility function r_{U} bounded by B , there exists an \alpha_{0}>0 such that U(r_{m^{*},\alpha_{0}})>U(r_{U}) . To simplify terminology, we refer to SRS (soft) as SRS , unless otherwise specified. 5 Integrating SRS to Inference-Time Alignment This section integrates our Stackelberg Reward Shaping framework to inference-time alignment. While user preferences vary across individuals, it is generally infeasible to train a separate model to satisfy each user. Inference-time alignment instead enables per-request adaptation by shaping a deployed model’s output distribution, making it a natural setting for aligning model behavior with user-specific objectives. We therefore focus on inference-time alignment as a practical instantiation of the optimal reward shaping framework developed in this work. 5.1 Introduction to Inference-Time Alignment We first introduce the inference-time alignment problem, modeled as a Markov Decision Process (MDP). In a token-level MDP \mathcal{M}=\{\mathcal{S},\mathcal{A},P,R\} , a state \bm{s}_{t}\in\mathcal{S} is the concatenation of the prompt and the generated tokens before time t , i.e., \bm{s}_{t}=[\bm{x},\bm{y}_{t})] , serving as a proxy for Q^{*}(\bm{s}_{t},y_{t}) . Under SRS-CD, we perform reward shaping on \mathcal{D}_{\text{CD}} offline. Since the M responses per prompt are Monte Carlo samples from \rho_{\mathrm{base}} , we can construct the Estimator given by Equation ( 5 ) and apply the SRS shaping rule defined in Equation ( 6 ) for every \bm{x} in \mathcal{D}_{\text{CD}} . This yields the shaped dataset \mathcal{D}_{\text{SRS}}=\Bigl\{\bigl\{\bigl((\bm{s}_{t}^{h,i})_{t=1}^{T},\;\bm{y}^{h,i},\;{\color[rgb]{1,0,0}{r_{\hat{m}^{*},\alpha}}}^{h,i}\bigr)\bigr\}_{i=1}^{M}\Bigr\}_{h=1}^{H} . The same training objective as in CD is then used to learn the state-action function Q_{\phi}^{\text{SRS}} on \mathcal{D}_{\text{SRS}} . The offline shaping is summarized in Algorithm 1 , with additional loss definitions and training details provided in Appendix B.1 . Algorithm 1 Training the Q function for SRS-CD 1: base policy \rho_{\mathrm{base}} ; reward model r_{U} ; reward bound B ; reward strength 1/\beta ; sample size M ; shaping strength \alpha ; prompt dataset D 2: Q function Q_{\phi}^{\text{SRS}}(\bm{s}_{t},y_{t}) 3: Offline data generation and reward shaping: 4: for each prompt \bm{x}\sim\mathcal{D} do 5: Sample trajectories \{\bm{y}^{i}\}_{i=1}^{M}\sim\rho_{\mathrm{base}}(\cdot\,|\,\bm{x}) and extract the states along with the trajectories \{\,\bm{s}_{1:T}^{\,i}\,\}_{i=1}^{M} . 6: Compute rewards \{r_{U}(\bm{x},\bm{y}^{i})\}_{i=1}^{M} . 7: Construct the Monte Carlo estimator \widehat{F}_{\bm{x}}(m) by ( 5 ). 8: Solve \widehat{F}_{\bm{x}}(m)=0 via bisection to obtain threshold \hat{m}^{*}(\bm{x}) . 9: Apply reward shaping: r_{\hat{m}^{*},\alpha}(\bm{x},\bm{y}^{i})=B\,\sigma\Big(\alpha\big(r_{U}(\bm{x},\bm{y}^{i})-\hat{m}^{*}(\bm{x})\big)\Big). 10: end for 11: Q function training: 12: Train the Q function based on modified reward \mathcal{D}_{\text{SRS}}=\Bigl\{\bigl\{\bigl((\bm{s}_{t}^{h,i})_{t=1}^{T},\;\bm{y}^{h,i},\;r_{\hat{m}^{*},\alpha}^{h,i}\bigr)\bigr\}_{i=1}^{M}\Bigr\}_{h=1}^{H}. following standard CD training procedure. Details of training are deferred to Appendix B.1 . 13: return Q_{\phi}^{\text{SRS}}(\bm{s}_{t},y_{t}) 6 Experiment Results In this section, we evaluate the effectiveness of our SRS-integrated inference-time alignment methods in steering LLMs toward helpful behavior. 6.1 Experiment Setup We evaluate our methods on the HH-RLHF ( 2 ) and SHP ( 12 ) datasets, which are popular benchmarks for alignment. These datasets aim to align AI assistants to become more helpful to user and less harmful. We use Qwen3-8B ( 48 ) and Llama3-8B-Instruct ( 14 ) as the backbone for answering prompts. For reproducibility, we use publicly available reward models fine-tuned on preference dataset 2 2 2 Skywork-Qwen: https://huggingface.co/Skywork/Skywork-Reward-V2-Qwen3-8B 3 3 3 Skywork-Llama: https://huggingface.co/Skywork/Skywork-Reward-Llama-3.1-8B. ( 28 ) as a proxy for r_{U} . For controlled decoding, we train the Q function on the last layer of the hidden state, and we include the architecture details in Appendix E.5 . Table 1: Eval setups: datasets, backbones, and reward models. Eval Dataset Backbone Reward model Eval-1 HH-RLHF Qwen Skywork-Qwen Eval-2 SHP Qwen Skywork-Qwen Eval-3 HH-RLHF Llama Skywork-Llama Eval-4 SHP Llama Skywork-Llama Following ( 23 ) , we use the following evaluation metrics: (1) Diversity : This metric assesses a model’s ability to produce linguistically diverse text by measuring the frequency of repeated n -grams. (2) Coherence : This metric quantifies semantic consistency between each prompt and its response by computing the cosine similarity of their SimCSE embeddings ( 41 ) . (3) Average reward : Mean reward of the generation, scored by r_{U} . 6.2 Baseline We evaluate the proposed SRS-integrated CD and ARGS methods. While prior works on reward shaping focuses on train-time settings – where rewards for complete trajectories generated by both the base and trained policies are available ( 45 ; 13 ) – such information is not accessible in inference-time alignment. As a result, existing train-time reward shaping techniques are not directly transferable to inference-time setting. To enable a meaningful comparison under these constraints, we consider, in addition to the base policy (no alignment) and the vanilla CD and ARGS baselines, two heuristic reward-shaping schemes proposed by 13 that can be implemented using Monte Carlo samples from the base policy: • Minmax : Rewards are normalized using the minimum and maximum values among M Monte Carlo samples. To control the reward scale, we introduce a hyperparameter B and define: \tilde{r}=B\frac{r-r_{\text{min}}}{r-r_{\text{max}}} . • Meanstd : Rewards are normalized using the empirical mean and standard deviation computed from M running samples, given by: \tilde{r}=\frac{r-\mu}{\text{std}} , where \mu and std represent sample mean and standard deviation. We denote a method by \{ Reward Shaping Mechanism \} - \{ Decoding Policy \} , and compare SRS-CD and SRS-ARGS against their vanilla counterparts as well as these heuristic reward-shaping baselines. For each evaluation setting, we sweep the reward strength \frac{1}{\beta} for the vanilla decoding policy and select the value that achieves the best performance prior to the onset of reward hacking or oversteering (Definitions of both concepts are provided in Appendix C.1 ). This reward strength is then fixed and used consistently across all methods within the same evaluation. We use sample size M=10 and apply greedy-based decoding for all methods. We reserve a validation set for hyperparameter selection. All additional experiment details are provided in Appendix E . Table 2: Performance comparison between SRS and baseline methods. Our methods are highlighted. Within each evaluation setting, the best result is shown in bold; bolding is omitted when more than two methods are tied. Eval Method Div. Coh. Reward Eval-1 Base policy 0.80 0.61 2.76 ARGS 0.78 0.62 3.23 Minmax-ARGS 0.78 0.62 3.24 Meanstd-ARGS 0.79 0.62 3.02 SRS-ARGS 0.78 0.62 3.33 CD 0.79 0.62 3.09 Minmax-CD 0.79 0.62 3.17 Meanstd-CD 0.79 0.62 3.00 SRS-CD 0.79 0.62 3.23 Eval-2 Base policy 0.79 0.64 2.95 ARGS 0.82 0.66 3.26 Minmax-ARGS 0.83 0.67 3.14 Meanstd-ARGS 0.83 0.67 3.15 SRS-ARGS 0.81 0.66 3.40 CD 0.77 0.65 3.10 Minmax-CD 0.69 0.61 3.09 Meanstd-CD 0.80 0.65 2.65 SRS-CD 0.78 0.65 3.37 Eval-3 Base policy 0.81 0.60 -0.24 ARGS 0.80 0.61 1.87 Minmax-ARGS 0.80 0.61 1.84 Meanstd-ARGS 0.82 0.61 0.37 SRS-ARGS 0.81 0.61 2.04 Eval-4 Base policy 0.82 0.66 1.62 ARGS 0.85 0.65 2.97 Minmax-ARGS 0.85 0.65 3.20 Meanstd-ARGS 0.83 0.66 2.83 SRS-ARGS 0.85 0.66 3.29 6.3 Main result Table 2 reports the performance of all methods across four evaluation settings. 4 4 4 We omit results for Controlled Decoding (CD) on Eval-3 and Eval-4, as it shows no improvement over the base model even when trained with the unshaped reward. Since this behavior arises prior to applying reward shaping, it reflects a limitation of the vanilla CD procedure rather than our method. Further analysis is provided in Appendix C.3 . We report three main findings. First, SRS consistently attains the highest reward across all evaluated settings while maintaining comparable levels of diversity and coherence. This highlights the effectiveness of our Stackelberg game framework in maximizing user utility. Second, methods with an explicit reward bound is better at adapting to different reward strengths \frac{1}{\beta} . In contrast, Meanstd – an unbounded shaping scheme – fails to scale appropriately, sometimes leading to reward less than base policy (e.g., Eval-2 (CD)). Third, though Minmax employs a scale-adjusting bound and can approach the performance of SRS in certain cases (e.g., Eval-1 (CD)), its performance is less consistent across evaluations. This might be because Minmax is highly sensitive to extreme reward values; in the presence of outliers, the remaining rewards are compressed to a narrow range, preventing the distinction between favorable and unfavorable samples and leading to worse performance compared to SRS. 6.4 GPT-4 Evaluation To monitor reward hacking and evaluate aspects of language generation not captured by the metrics in Table 2 , we use GPT-4 as judge to provide complementary quality assessment. For each prompt, we provide explicit instruction that asks GPT-4 to score two responses on a scale from 1 to 10 along dimensions including helpfulness, harmlessness, relevance, accuracy, depth, creativity, and level of detail; the exact evaluation prompt is provided in Appendix E.8 . We randomly sample 300 test prompts and conduct head-to-head comparisons between our method and each baseline. Following 23 ; 6 , we report Win-Tie rate, the percentage at which our response is rated as better or equal to that of the baseline. To mitigate position bias, we randomize the order of the two responses as in 50 . As shown in Table 3 , our method achieves average Win-Tie rate of \{66.83\%,69.6\%,66.65\%\} against the Vanilla, Minmax, and Meanstd baselines, respectively, indicating that the observed improvements are not driven by reward hacking. We further provide qualitative examples comparing our method with vanilla inference-time methods in Appendix E.9 . Table 3: Each cell is the Win–Tie rate from head-to-head comparisons between our method and the baseline (column). Eval Method Vanilla Minmax Meanstd Eval-1 ARGS 66.7% 64.3% 63.0% CD 69.0% 76.0% 69.0% Eval-2 ARGS 75.3% 75.0% 72.3% CD 64.0% 76.0% 69.0% Eval-3 ARGS 59.3% 59.3% 59.3% Eval-4 ARGS 66.7% 67.0% 67.3% 7 Further Experimental Analysis 7.1 Ablation Study We already motivated the importance of bounded reward model in the Stackelberg game formulation, and here we focus on the importance of the threshold reward structure. Threshold Reward We evaluate the effectiveness of the proposed threshold reward structure in the SRS mechanism. We compare our method against two baselines: (i) direct reward capping, denoted by r_{U}^{\text{CAP}}=\min\{r_{U}(\bm{x},\bm{y}),\,B\} , and (ii) a threshold reward that uses the Monte Carlo mean reward as the threshold, defined as r^{\text{MEAN}}_{\alpha}(\bm{x},\bm{y})=B\,\sigma\Big(\alpha\big(r_{U}(\bm{x},\bm{y})-\tfrac{1}{M}\sum\nolimits_{i=1}^{M}r_{U}(\bm{x},\bm{y}^{i})\big)\Big). For fair comparison, we fix all shared hyperparameters to be identical across SRS and baselines with the same parameterization. As shown in Table 4 , while reward capping mitigates reward hacking, it collapses distinctions among high-utility responses, causing strictly preferred outputs to be treated the same as acceptable ones and degrading alignment performance – sometimes even relative to the vanilla benchmark. In contrast to heuristic transformations (e.g., thresholding at the empirical mean), SRS selects the threshold \hat{m}^{*}(\bm{x}) in a principled and self-consistent manner by directly optimizing the user utility objective, leading to improved alignment outcomes across both CD and ARGS experiments. Table 4: Performance comparison between SRS and baselines. Our methods are highlighted. Within each evaluation setting, the best result is shown in bold. Eval Method Div. Coh. Reward Eval-2 Base policy 0.79 0.64 2.95 ARGS 0.82 0.66 3.26 CAP-ARGS 0.81 0.67 3.29 MEAN-ARGS 0.74 0.63 3.30 SRS-ARGS 0.81 0.66 3.40 CD 0.77 0.65 3.10 CAP-CD 0.82 0.67 2.96 MEAN-CD 0.80 0.66 3.18 SRS-CD 0.78 0.65 3.37 7.2 Hyperparameter Study We defer a detailed hyperparameter study for reward bound B , shaping strength \alpha , and number of Monte Carlo samples M to Appendix C.2 . In particular, we empirically evaluate SRS (hard) as the limiting case of the shaping strength \alpha\to\infty . 7.3 Inference Time Overhead For SRS-CD, reward shaping is performed offline and therefore incurs no additional inference-time overhead. For SRS-ARGS, reward shaping involves solving the threshold equation (Eq. 5 ) during decoding via bisection, which incurs an average overhead of 2.69\times 10^{-4} seconds per token (approximately 0.03 seconds for a 128-token response). This cost is negligible relative to the average ARGS response time of 14.05 seconds per prompt. 8 Conclusion We identify that the common practice of directly using reward models trained from user preference data is suboptimal under KL-regularized alignment objectives. Motivated by this observation, we study reward shaping mechanisms that jointly maximize user utility while mitigating reward hacking. We formulate reward optimization as a Stackelberg game and show that the optimal reward model admits a threshold structure, which enables the design of efficient algorithms to approximate the optimal reward model. Empirically, we focus on inference-time alignment and demonstrate that our method can be seamlessly integrated into existing alignment approaches with minimal inference-time overhead. Across all evaluation settings, our reward shaping mechanism consistently improves average reward while maintaining diversity and coherence comparable to all baselines. Consistent with these gains, GPT-4 evaluations report an average win–tie rate of 66% over all baselines. Impact Statement This paper studies reward shaping mechanisms for alignment that jointly aim to maximize user utility while mitigating reward hacking. We formulate reward model optimization as a Stackelberg game, develop efficient algorithms to approximate the optimal reward model, and empirically integrate our approach into inference-time alignment methods and yield consistent performance improvements. Our framework assumes a benign user whose preferences are aligned with human values. If the reward model reflects malicious or misaligned preferences, the resulting policy may itself become misaligned. This limitation is not specific to our method and applies broadly to alignment approaches that rely on learned reward models. Consequently, safe application of our framework requires that the reward model be learned from preference data that is carefully curated to reflect human values. Appendix A Additional Related Work: Algorithmic Contract Design Algorithmic Contract Design Contract design studies how a principal can design incentive mechanisms (contracts) to align an agent’s actions with her objectives. This framework has been extensively studied in economics 18 , 15 , leading to conceptual breakthroughs that is recognized with the 2016 Nobel Prize in Economics. In recent years, their algorithmic aspects have attracted significant attention from the computer science community, leading to a growing theory and ML literature on their computational properties 11 , 44 , 10 and inspiring several approaches that apply contract-based ideas to align agents in multi-agent settings 17 , 20 . Three lines of work are most closely related to ours. The first is 16 , which interprets the reward model in alignment as a contract and explains reward hacking through the lens of incomplete contract, followed by empirical work demonstrating principal-agent conflicts among diverse end users and LLM 33 . However, 16 ’s work is mostly a conceptual framework, whereas we formulate a principal agent optimization problem and provide an implementable framework for improving test-time alignment. The second is 3 , which theoretically studies how a principal can subsidize rewards on selected states in an MDP to induce desired behavior from the agent. Our formulation differs in that the agent’s best response is subject to an entropy regularization, yielding different optimal solutions, and our work is grounded in alignment application. Lastly, 36 also studies principal–agent problem arising in LLM service providing, though their agent is the service provider, whereas in our setting the LLM itself is the agent. A.1 The Classical Contract Model In contract design, a principal commits to an incentive contract to induce an agent 5 5 5 The contract design problem is also a Stackelberg game, in which the principal moves first and the agent subsequently responds. Throughout this section, the term agent refers specifically to the follower in this game-theoretic setting and should be distinguished from agentic AI systems. to act in alignment with the principal’s objective [ 18 ] . A principal delegates a task to an agent who can choose an action \mu\in\mathcal{U} , which will induce a distribution q(\cdot|\mu) over outcome o\in\mathcal{O} . The principal first offers a contract c:\mathcal{O}\to\mathbb{R}_{\geq 0} , which maps the realized outcome to a payment. Both the principal’s utility U_{P} and the agent’s utility U_{A} depend on the realized outcome and contract, while the agent additionally incurs an action-dependent cost. The principal seeks a contract that maximizes her expected utility under a budget constraint B , anticipating the agent’s optimal response. \displaystyle\max_{c(\cdot)}~~ \displaystyle E_{o\sim q(\cdot|\mu^{*}(c))}[U_{P}(o,c(o))] s.t. \displaystyle\mu^{*}(c)\in\argmax_{\mu\in\mathcal{U}}E_{o\sim q(\cdot|\mu)}[U_{A}(\mu,c(o))] \displaystyle 0\leq c(o)\leq B\quad\forall o\in\mathcal{O} The first constraint ensures that the agent optimally responds to the contract in expectation. A.2 Mapping to our KL-regularized LLM alignment setting Our model can be seen as a variant of the contract design model. Fix a prompt \bm{x} . • Outcome: in our setting, the realized “outcome” is the full model response \bm{y} in the response space Y . • Principal: The reward model provider. The principal’s action is to choose a reward model r , and her utility is given by the user utility function r_{U}(\bm{x},\bm{y}) . • Agent: LLM. The LLM’s action is to select a response distribution \rho(\cdot|\bm{x}) over Y that maximizes its own utility as specified by the alignment objective (Equation 2 ) • Contract: Reward Model r(\bm{x},\bm{y}) . Given a prompt \bm{x} , the reward map maps the realized response \bm{y} to a numerical reward which incentivizes the LLM to generate outputs preferred by the user. • Limited liability / bounded incentives: we impose 0\leq r(\bm{x},\bm{y})\leq B , which is the same role as bounded payments in contracts For each prompt \bm{x} , our Stackelberg reward-design problem can be written as: \displaystyle\max_{r(\bm{x},\cdot)}~~ \displaystyle E_{\bm{y}\sim\rho_{r}(\cdot|\bm{x})}[r_{U}(\bm{x},\bm{y})] s.t. \displaystyle\rho_{r}(\cdot|\bm{x})\in\argmax_{\rho_{r}(\cdot|\bm{x})}\{E_{\bm{y}\sim\rho_{r}(\cdot|\bm{x})}[r(\bm{x},\bm{y})-\beta\cdot D_{\mathrm{KL}}(\rho(\cdot|\bm{x})||\rho_{base}(\cdot|\bm{x}))]\} \displaystyle 0\leq r(\bm{x},\bm{y})\leq B\quad\forall\bm{y}\in Y. Under this KL-regularized best response, the agent’s optimal response has a closed form: \rho_{r}(\bm{y}|\bm{x})\propto\rho_{base}(\bm{y}|\bm{x})\exp(r(\prompt, \response)/\beta) . So the contract tilts the base model’s distribution, but the KL term penalizes large deviations from \rho_{base} . Compared to the classical model, our setting differs in that the mapping q(|\cdot) from actions to outcomes is not exogenously specified, but is instead endogenously induced by the agent’s choice of the response distribution \rho\in\Delta(Y) . As a result, new techniques are required to address our setting. Appendix B Algorithms of SRS-ARGS and SRS-CD A key distinction between CD and ARGS lies in how expectations with respect to the base policy are realized. In controlled decoding, the value function is trained on offline trajectories sampled from \rho_{\mathrm{base}} ; consequently, expectations over \rho_{\mathrm{base}} can be estimated directly via Monte Carlo samples, without requiring any explicit probability reweighting. In contrast, ARGS deterministically enumerates the top- M next token candidates under \pi_{\mathrm{base}} and selects among them during decoding. Therefore, when constructing the helper function \hat{F}_{\bm{x}}(m) , each candidate token must be explicitly weighted by its probability under the base policy \pi_{\mathrm{base}} . B.1 Algorithms of SRS-CD Recall that \bm{s}_{t} denotes the context available up to time t . In our implementation, we represent \bm{s}_{t} using the LLM’s latent hidden representation, rather than the partially generated response itself. To integrate SRS mechanism into CD, we implement Alg 2 . Algorithm 2 Training the state-action function for SRS-CD 1: base model \rho_{\mathrm{base}} ; reward model r_{U} learned from user preference; reward upper bound B ; reward strength 1/\beta ; Monte Carlo sample size M ; personalization strength \alpha ; offline prompt dataset \mathcal{D} with |\mathcal{D}|=H 2: action-value function Q_{\phi}^{\text{SRS}}(\bm{s}_{t},y_{t}) 3: Offline data generation and reward shaping: 4: for each prompt \bm{x}\sim\mathcal{D} do 5: Sample trajectories \{\bm{y}^{i}\}_{i=1}^{M}\sim\rho_{\mathrm{base}}(\cdot\mid\bm{x}) and extract the states along with the trajectories \{\,\bm{s}_{1:T}^{\,i}\,\}_{i=1}^{M} . 6: Compute rewards \{r_{U}(\bm{x},\bm{y}^{i})\}_{i=1}^{M} . 7: Construct the Monte Carlo estimator \widehat{F}_{\bm{x}}(m) in Eq. ( 5 ). 8: Solve \widehat{F}_{\bm{x}}(m)=0 via bisection to obtain the threshold \hat{m}^{*}(\bm{x}) . 9: Apply reward shaping: r_{\hat{m}^{*},\alpha}(\bm{x},\bm{y}^{i})=B\,\sigma\Big(\alpha\big(r_{U}(\bm{x},\bm{y}^{i})-\hat{m}^{*}(\bm{x})\big)\Big). 10: end for 11: Value-function training: 12: We have \mathcal{D}_{\text{SRS}}=\Bigl\{\bigl\{\bigl((\bm{s}_{t}^{h,i})_{t=1}^{T},\;\bm{y}^{h,i},\;r_{\hat{m}^{*},\alpha}^{h,i}\bigr)\bigr\}_{i=1}^{M}\Bigr\}_{h=1}^{H}. from previous step. Our training objective is \mathcal{L}=\sum_{h=1}^{H}\sum_{i=1}^{M}\sum_{t}\left(Q_{\phi}^{\text{SRS}}(s_{t}^{h,i},y_{t}^{h,i})-\operatorname{stop\text{-}grad}\!\left(q_{t}^{h,i}\right)\right)^{2}. Here \text{stop-grad}(\cdot) indicates the gradient is not propagated through q_{t}^{h,i} , and we use \bm{s}_{t+1} to denote the concatenation of \bm{s}_{t} and y_{t} . The target state-action value q_{t}^{h,i} is computed as follows: q_{t}^{h,i}=\begin{cases}\mathbb{E}_{y_{t+1}\sim\pi_{\mathrm{base}}(\cdot|s_{t+1})}\left[Q^{\text{SRS}}_{\phi}(\bm{s}_{t+1},y_{t+1})\right],&y_{t}\neq\text{EOS},\\ r_{\alpha}(\bm{x},\bm{y}),&y_{t}=\text{EOS}.\end{cases} 13: return Q_{\phi}^{\text{SRS}}(\bm{s}_{t},y_{t}) After obtaining Q_{\phi}(\bm{s}_{t},y_{t}) , we decode in the manner specified in Algorithm 3 . Algorithm 3 SRS-CD 1: Previous context \bm{s}_{t} , number of candidates \tilde{M} , reward strength \frac{1}{\beta} , desired trajectory length T , base policy \pi_{\mathrm{base}} , state-action value Q^{\text{SRS}}_{\phi}(\bm{s}_{t},y_{t}) , decode rule 2: A generated sequence with T tokens 3: for l\leftarrow t to T-1 do 4: \{y_{t}\}_{i=1}^{\tilde{M}}\leftarrow top- \tilde{M} tokens with highest likelihood under \pi_{\mathrm{base}}(\cdot\mid\bm{s}_{t}) 5: for each y_{t}^{i}\in\{y_{t}\}_{i=1}^{\tilde{M}} do 6: \mathrm{score}(y_{t}^{i})\leftarrow\log\pi_{\mathrm{base}}(y_{t}^{i}\mid\bm{s}_{t})+\frac{1}{\beta}Q^{\text{SRS}}_{\phi}(\bm{s}_{t},y_{t}^{i}) 7: end for 8: if decode rule = greedy then 9: y_{t,\text{selected}}\leftarrow\operatorname*{arg\,max}_{y_{t}^{i}\in\{y_{t}\}_{i=1}^{\tilde{M}}}\mathrm{score}(y_{t}^{i}) 10: else if decode rule = sample then 11: y_{t,\text{selected}}\sim\mathrm{Softmax}\!\left(\{\mathrm{score}(y_{t})\}_{y_{t}^{i}\in\{y_{t}^{i}\}_{i=1}^{M}}\right) 12: end if 13: \bm{s}_{t+1}\leftarrow[\bm{s}_{t},y_{t,\text{selected}}] 14: end for 15: return \bm{s}_{T} B.2 Algorithms of SRS-ARGS For ARGS, given a state \bm{s}_{t} , the method selects the top- M tokens \{y_{t}^{i}\}_{i=1}^{M} under the base policy \pi_{\mathrm{base}} and evaluate each candidate using r_{U}([\bm{s}_{t},y_{t}^{i},\text{EOS}]) as a proxy for Q^{*}(\bm{s}_{t},y_{t}^{i}) . Since the candidate tokens are deterministically selected, we do not have \bm{y}\sim\rho_{\mathrm{base}}(\cdot|\bm{x}) , so we cannot directly use Eq. ( 5 ) to construct the Monte Carlo Estimator. To overcome this challenge, In SRS-ARGS, we approximate the expectation under \rho_{\mathrm{base}} by using a variant of Eq. ( 5 ) that is weighted by base policy probability, which is available during decoding. We propose the following weighted version of the helper function \displaystyle\hat{F}^{\mathsf{ARGS}}_{\bm{x}}(m)=\sum_{i=1}^{M}\pi_{\mathrm{base}}(y_{t}^{i}|\bm{s}_{t})w_{\bm{x},y_{\leq t}^{i}}(m)\cdot\Big(r_{U}([\bm{s}_{t},y_{t}^{i},\text{EOS}])-m\Big) (8) where w_{\bm{x},y_{\leq t}^{i}}(m):=\begin{cases}1&\text{if }r_{U}([\bm{s}_{t},y_{t}^{i},\text{EOS}])\mathbb{E}_{\bm{y}^{\prime}\sim\rho_{r}(\cdot|\bm{x})}\big[\,r_{U}(\bm{x},\bm{y}^{\prime})\,\big]\\ \in[0,B]&\text{ \ if \ }r_{U}(\bm{x},\bm{y})=\mathbb{E}_{\bm{y}^{\prime}\sim\rho_{r}(\cdot|\bm{x})}\big[\,r_{U}(\bm{x},\bm{y}^{\prime})\,\big]\\ 0&\text{ \ if \ }r_{U}(\bm{x},\bm{y})<\mathbb{E}_{\bm{y}^{\prime}\sim\rho_{r}(\cdot|\bm{x})}\big[\,r_{U}(\bm{x},\bm{y}^{\prime})\,\big].\end{cases} Proof. Let J(r)=\mathbb{E}_{\bm{y}\sim\rho_{r}(\cdot|\bm{x})}\big[\,r_{U}(\bm{x},\bm{y})\,\big] . Suppose there are N possible responses in total: Y=\{\bm{y}^{1},\ldots,\bm{y}^{N}\} . Denote r^{i}:=r(\bm{x},\bm{y}^{i}) . The reward model we are optimizing can then be expressed as a vector r=(r^{1},\ldots,r^{N}) . Define the Lagrangian: L(r,\mu,\nu)=J(r)+\sum_{i=1}^{N}\mu^{i}r^{i}+\sum_{i=1}^{N}\nu^{i}(B-r^{i}) . Stationarity gives \displaystyle\frac{\partial J}{\partial r^{i}}+\mu^{i}-\nu^{i}=0. (9) Using Lemma 1 , we have \displaystyle\frac{\partial J}{\partial r^{i}} \displaystyle=\sum_{\bm{y}^{\prime}\in Y}r_{U}(\bm{x},\bm{y}^{\prime})\cdot\frac{\partial\rho_{r}(\bm{y}^{\prime}\,|\,\bm{x})}{\partial r^{i}} \displaystyle=\sum_{\bm{y}^{\prime}\in Y}r_{U}(\bm{x},\bm{y}^{\prime})\cdot\frac{1}{\beta}\cdot\rho_{r}(\bm{y}^{\prime}\,|\,\bm{x})\cdot\Big(\delta_{\bm{y}^{\prime},\bm{y}^{i}}-\rho_{r}(\bm{y}^{i}\,|\,\bm{x})\Big) \displaystyle=\frac{1}{\beta}\Big[r_{U}(\bm{x},\bm{y}^{i})\rho_{r}(\bm{y}^{i}\,|\,\bm{x})-\rho_{r}(\bm{y}^{i}\,|\,\bm{x})\sum_{\bm{y}^{\prime}\in Y}r_{U}(\bm{x},\bm{y}^{\prime})\rho_{r}(\bm{y}^{\prime}\,|\,\bm{x})\Big] \displaystyle=\frac{1}{\beta}\rho_{r}(\bm{y}^{i}\,|\,\bm{x})\Big(r_{U}(\bm{x},\bm{y}^{i})-\mathbb{E}_{\bm{y}^{\prime}\sim\rho_{r}(\cdot|\bm{x})}\big[\,r_{U}(\bm{x},\bm{y}^{\prime})\,\big]\Big). Complementary slackness and feasibility are \begin{cases}\mu^{i}\cdot r^{i}=0,\quad&\nu^{i}(B-r^{i})=0,\\ \mu^{i}\geq 0,\quad&\nu^{i}\geq 0,\\ 0\leq r^{i}\leq B.\end{cases} Interior Solution: Let’s first consider the interior solution 00 , we obtain r_{U}(\bm{x},\bm{y}^{i})-\mathbb{E}_{\bm{y}^{\prime}\sim\rho_{r}(\cdot|\bm{x})}\big[\,r_{U}(\bm{x},\bm{y}^{\prime})\,\big]=0. This means that r^{i} cannot be an interior solution unless r_{U}(\bm{x},\bm{y}^{i})=\mathbb{E}_{\bm{y}^{\prime}\sim\rho_{r}(\cdot\,|\,\bm{x})}\big[\,r_{U}(\bm{x},\bm{y}^{\prime})\,\big] . Lower Bound r^{i}=0 : By complementary slackness, we have \mu^{i}\geq 0 and \nu^{i}=0 . The stationarity condition then becomes: \frac{1}{\beta}\rho_{r}(\bm{y}^{i}\,|\,\bm{x})\Big(r_{U}(\bm{x},\bm{y}^{i})-\mathbb{E}_{\bm{y}^{\prime}\sim\rho_{r}(\cdot|\bm{x})}\big[\,r_{U}(\bm{x},\bm{y}^{\prime})\,\big]\Big)+\mu^{i}=0, which implies \frac{1}{\beta}\rho_{r}(\bm{y}^{i}\,|\,\bm{x})\Big(r_{U}(\bm{x},\bm{y}^{i})-\mathbb{E}_{\bm{y}^{\prime}\sim\rho_{r}(\cdot|\bm{x})}\big[\,r_{U}(\bm{x},\bm{y}^{\prime})\,\big]\Big)=-\mu^{i}\leq 0. Hence r_{U}(\bm{x},\bm{y}^{i})\leq\mathbb{E}_{\bm{y}^{\prime}\sim\rho_{r}(\cdot|\bm{x})}\big[\,r_{U}(\bm{x},\bm{y}^{\prime})\,\big] . If r_{U}(\bm{x},\bm{y}^{i})<\mathbb{E}_{\bm{y}^{\prime}\sim\rho_{r}(\cdot|\bm{x})}\big[\,r_{U}(\bm{x},\bm{y}^{\prime})\,\big] , then it must be the case of r^{i}=0 . Upper Bound r^{i}=B : By complementary slackness, we have \mu^{i}=0 and \nu^{i}\geq 0 . The stationary condition then becomes: \frac{1}{\beta}\rho_{r}(\bm{y}^{i}\,|\,\bm{x})\Big(r_{U}(\bm{x},\bm{y}^{i})-\mathbb{E}_{\bm{y}^{\prime}\sim\rho_{r}(\cdot|\bm{x})}\big[\,r_{U}(\bm{x},\bm{y}^{\prime})\,\big]\Big)-\nu^{i}=0, which implies \frac{1}{\beta}\rho_{r}(\bm{y}^{i}\,|\,\bm{x})\Big(r_{U}(\bm{x},\bm{y}^{i})-\mathbb{E}_{\bm{y}^{\prime}\sim\rho_{r}(\cdot|\bm{x})}\big[\,r_{U}(\bm{x},\bm{y}^{\prime})\,\big]\Big)=\nu^{i}\geq 0. Hence r_{U}(\bm{x},\bm{y}^{i})\geq\mathbb{E}_{\bm{y}^{\prime}\sim\rho_{r}(\cdot|\bm{x})}\big[\,r_{U}(\bm{x},\bm{y}^{\prime})\,\big] . If r_{U}(\bm{x},\bm{y}^{i})>\mathbb{E}_{\bm{y}^{\prime}\sim\rho_{r}(\cdot|\bm{x})}\big[\,r_{U}(\bm{x},\bm{y}^{\prime})\,\big] , then it must be the case of r^{i}=B . In summary, we have r(\bm{x},\bm{y}^{i})=\begin{cases}B&\text{ \ if \ }r_{U}(\bm{x},\bm{y}^{i})>\mathbb{E}_{\bm{y}^{\prime}\sim\rho_{r}(\cdot|\bm{x})}\big[\,r_{U}(\bm{x},\bm{y}^{\prime})\,\big]\\ \in[0,B]&\text{ \ if \ }r_{U}(\bm{x},\bm{y}^{i})=\mathbb{E}_{\bm{y}^{\prime}\sim\rho_{r}(\cdot|\bm{x})}\big[\,r_{U}(\bm{x},\bm{y}^{\prime})\,\big]\\ 0&\text{ \ if \ }r_{U}(\bm{x},\bm{y}^{i})<\mathbb{E}_{\bm{y}^{\prime}\sim\rho_{r}(\cdot|\bm{x})}\big[\,r_{U}(\bm{x},\bm{y}^{\prime})\,\big],\end{cases} which proves the lemma. ∎ Proof of Theorem 1 : Lemma 2 shows that the optimal reward model r^{*} has a threshold structure: r^{*}(\bm{x},\bm{y})=\begin{cases}B&\text{ \ if \ }r_{U}(\bm{x},\bm{y})>m^{*}(\bm{x})\\ \in[0,B]&\text{ \ if \ }r_{U}(\bm{x},\bm{y})=m^{*}(\bm{x})\\ 0&\text{ \ if \ }r_{U}(\bm{x},\bm{y})0 , we have F_{\bm{x}}(m)=0\iff\mathbb{E}_{\bm{y}\sim\rho_{r_{m}}(\cdot|\bm{x})}\big[r_{U}(\bm{x},\bm{y})\big]-m=0 . Continuity and strict monotonicity of F_{\bm{x}}(m) . We then prove that F_{\bm{x}}(m) is continuous and strictly decreasing in m . We first note that \displaystyle F_{\bm{x}}(m) \displaystyle=\int_{Y}\Big(r_{U}(\bm{x},\bm{y})-m\Big)\cdot\begin{cases}\rho_{\mathrm{base}}(\bm{y}\,|\,\bm{x})&\text{ if }r_{U}(\bm{x},\bm{y})0 , take the difference \displaystyle A(m+\delta)-A(m) \displaystyle=\int_{\bm{y}:r_{U}(\bm{x},\bm{y})\geq m+\delta}\Big(r_{U}(\bm{x},\bm{y})-m-\delta\Big)\rho_{\mathrm{base}}(\bm{y}\,|\,\bm{x})\differential\bm{y} \displaystyle\quad-\int_{r_{U}(\bm{x},\bm{y})\geq m}\Big(r_{U}(\bm{x},\bm{y})-m\Big)\rho_{\mathrm{base}}(\bm{y}\,|\,\bm{x})\differential\bm{y} \displaystyle=\int_{r_{U}(\bm{x},\bm{y})\geq m+\delta}\big(-\delta\big)\rho_{\mathrm{base}}(\bm{y}\,|\,\bm{x})\differential\bm{y}~-~\int_{m\leq r_{U}(\bm{x},\bm{y})m^{*}(\bm{x})\end{cases}. which coincides with the optimal threshold reward r^{*} with \frac{B}{2} assigned as threshold value. Therefore, r_{m^{*},\alpha}\to r^{*} point-wise. By continuity of \mathsf{Tilt}_{\beta}(\cdot) , we have \rho_{r_{m^{*},\alpha}}\to\rho_{r^{*}} . Since the response set is finite, we exchange limits and summation to obtain \displaystyle\lim_{\alpha\to\infty}\sum_{j}\rho_{r_{m^{*},\alpha}}(\bm{x},\bm{y}^{j})r_{U}(\bm{x},\bm{y}^{j}) \displaystyle=\sum_{j}\lim_{\alpha\to\infty}\rho_{r_{m^{*},\alpha}}(\bm{x},\bm{y}^{j})r_{U}(\bm{x},\bm{y}^{j}) \displaystyle=\sum_{j}\rho_{r^{*}}(\bm{x},\bm{y}^{j})\cdot r_{U}(\bm{x},\bm{y}^{j}) \displaystyle=U(r^{*}). ∎ D.4 Bounded KL A decoding policy should remain close to the base policy in KL divergence while achieving high reward. Let \hat{Q}(s_{t},y_{t}) denote the ARGS or CD approximation of Q^{*}(s_{t},y_{t}) , and let \hat{\rho}_{r_{\hat{m}^{*},\alpha}} denote the decoding response distribution induced under the SRS-shaped reward, for either decoding method. The next proposition shows that \hat{\rho}_{r_{\hat{m}^{*},\alpha}} ’s deviation from the baseline policy is bounded, highlighting the importance of bounded reward model. Proposition 1 . Suppose the response length is bounded by T , then the divergence from \hat{\rho}_{r_{\alpha}} to base policy is given by {D}_{\mathrm{KL}}(\hat{\rho}_{r_{\hat{m}^{*},\alpha}}(\cdot|\bm{x}),\rho_{\mathrm{base}}(\cdot|\bm{x}))\leq\frac{1}{\beta}TB. Proof. Let \hat{\pi}_{r_{\hat{m}^{*},\alpha}} denote the token-level policy corresponding to \hat{\rho}_{r_{\hat{m}^{*},\alpha}} , and let \hat{Q}_{r_{\alpha}} denote the action-value function approximated by the inference-time decoding method. Our analysis relies only on the bound 0\leq\hat{Q}_{r_{\hat{m}^{*},\alpha}}(\bm{s}_{t},y_{t})\leq B for all state–action pairs. We therefore verify that this condition holds for both ARGS and Controlled Decoding, after which the proof proceeds independently of the specific decoding method. Use r_{\text{max}}=\max_{\bm{x},\bm{y}}r_{\alpha}(\bm{x},\bm{y}) . For ARGS, since r_{\hat{m}^{*},\alpha} is bounded between 0 and B by design, 0\leq\hat{Q}_{r_{\hat{m}^{*},\alpha}}(\bm{s}_{t},y_{t})=r_{\hat{m}^{*},\alpha}([\bm{s}_{t},y_{t},\text{EOS}])\leq B . For CD, 0\leq\hat{Q}_{r_{\hat{m}^{*},\alpha}}(\bm{s}_{t},y_{t})=\mathbb{E}_{\bm{y}\sim\rho_{\mathrm{base}}}[r([\bm{s}_{t},y_{t}],\bm{y}_{>t})]\leq r_{\text{max}}\leq B . By definition, we have: D_{\mathrm{KL}}\!\left(\hat{\rho}_{r_{\hat{m}^{*},\alpha}}(\bm{y}|\bm{x}),\,\rho_{\mathrm{base}}(\bm{y}|\bm{x})\right)=\mathbb{E}_{\bm{y}\sim\hat{\rho}_{r_{\hat{m}^{*},\alpha}}(\bm{y}|\bm{x})}\left[\log\frac{\hat{\rho}_{r_{\hat{m}^{*},\alpha}}(\bm{y}|\bm{x})}{\rho_{\mathrm{base}}(\bm{y}|\bm{x})}\right]. (12) We decompose the trajectory distribution to token level distribution and apply \pi_{\mathrm{dec}}^{*}(y_{t}\,|\,\bm{s}_{t})=\pi_{\mathrm{base}}(y_{t}\,|\,\bm{s}_{t})\cdot\tfrac{\exp(\frac{1}{\beta}Q^*(\state, y_t))}{C_{\beta}(\bm{s}_{t})} , where C_{\beta}(\bm{s}_{t}) is the partition function, and then we have: \displaystyle\hat{\rho}_{r_{\hat{m}^{*},\alpha}}(\bm{y}|\bm{x}) \displaystyle=\hat{\pi}_{r_{\hat{m}^{*},\alpha}}(y_{1}|\bm{x})\,\hat{\pi}_{r_{\hat{m}^{*},\alpha}}(y_{2}|y_{1},\bm{x})\cdots\hat{\pi}_{r_{\hat{m}^{*},\alpha}}(y_{T}|\bm{y}_{\leq T-1},\bm{x}) \displaystyle=\frac{1}{C_{\beta}(\bm{x})}\cdot\pi_{\mathrm{base}}(y_{1}|\bm{x})\cdot\exp(\frac{1}{\beta} \cdot\hat{Q}_{r_{\hat{m}^*,\alpha}}(\prompt,y_1)) \displaystyle\quad\cdots\cdot\frac{1}{C_{\beta}(\bm{x},\cdots,y_{T-1})}\cdot\pi_{\mathrm{base}}(y_{T}|\bm{x},y_{1},\cdots,y_{T-1})\cdot\exp(\frac{1}{\beta}\hat{Q}_{r_{\hat{m}^*,\alpha}}(\prompt, y_1, \cdots, y_{T-1})). By rearranging the terms we have, \displaystyle\hat{\rho}_{r_{\hat{m}^{*},\alpha}}(\bm{y}|\bm{x}) \displaystyle=\rho_{\mathrm{base}}(\bm{y}|\bm{x})\cdot\exp\!\Bigg(\frac{1}{\beta}\cdot\Big[\hat{Q}_{r_{\hat{m}^{*},\alpha}}(\bm{x},y_{1})+\cdots \displaystyle\qquad\qquad+\hat{Q}_{r_{\hat{m}^{*},\alpha}}(\bm{x},y_{1},\cdots,y_{T-1})\Big]\Bigg)\cdot\frac{1}{C_{\beta}(\bm{x})C_{\beta}(\bm{x},y_{1})\cdots C_{\beta}(\bm{x},y_{1},\cdots,y_{T})}. Then taking logarithm on both sides we have: \displaystyle\log\frac{\hat{\rho}_{r_{\hat{m}^{*},\alpha}}(\bm{y}|\bm{x})}{\rho_{\mathrm{base}}(\bm{y}|\bm{x})} \displaystyle=\frac{1}{\beta}\Big[\hat{Q}_{r_{\hat{m}^{*},\alpha}}(\bm{x},y_{1})+\cdots+\hat{Q}_{r_{\hat{m}^{*},\alpha}}(\bm{x},y_{1},\cdots,y_{T-1})\Big]-\log C_{\beta}(\bm{x})-\cdots-\log C_{\beta}(\bm{x},y_{1},\cdots,y_{T-1}). Note that C_{\beta}(\bm{s}_{t})=\sum_{y_{t}}\pi_{\mathrm{base}}(y_{t}|\bm{s}_{t})\exp(\frac{1}{\beta}\hat{Q}_{r_{\hat{m}^*,\alpha}}(\state,y_t)) . Recall we have \hat{Q}_{r_{\hat{m}^{*},\alpha}}(\bm{s}_{t},y_{t})\geq 0 . Therefore C_{\beta}(\bm{s}_{t})\geq\sum_{y_{t}}\pi_{\mathrm{base}}(y_{t}|\bm{s}_{t})=1 , which implies \log C_{\beta}(\bm{s}_{t})\geq 0\ \forall\bm{s}_{t} . Recall we also have \hat{Q}_{r_{\hat{m}^{*},\alpha}}(\bm{s}_{t},y_{t})\leq r_{\max}\leq B . Hence, \log\frac{\hat{\rho}_{r_{\hat{m}^{*},\alpha}}(\bm{y}|\bm{x})}{\rho_{\mathrm{base}}(\bm{y}|\bm{x})}\leq\frac{1}{\beta}TB. Plug in Eq. ( 12 ) we conclude that D_{\mathrm{KL}}\!\left(\hat{\rho}_{r_{\hat{m}^{*},\alpha}}(\bm{y}|\bm{x}),\,\rho_{\mathrm{base}}(\bm{y}|\bm{x})\right)\leq\frac{1}{\beta}TB. ∎ Appendix E Experiment Details E.1 Computing Infrastructure Our experiment is done on a cluster wth NVIDIA A100 (80GB VRAM) GPUs. Experiments are implemented using Python 3.9.23 and Pytorch framework version 2.6.0 compiled with CUDA toolkit version 12.4. E.2 HH-RLHF Following 23 , we conduct our experiments on the Dahoas/full-hh-rlhf dataset 8 8 8 https://huggingface.co/datasets/Dahoas/full-hh-rlhf , a cleaned and curated version of the HH-RLHF dataset [ 2 ] , which is one of the most widely used benchmarks for alignment research. The dataset is designed to train language assistants to be more helpful and less harmful. It consists of approximately 112k training samples and 12.5k test samples. Each sample contains a prompt paired with two responses, along with a human preference indicating which response is favored. E.3 Stanford SHP We also evaluate on the Stanford SHP dataset 9 9 9 https://huggingface.co/datasets/stanfordnlp/SHP [ 12 ] , which contains large-scale collective human preference annotations over responses to Reddit posts. The dataset spans a wide range of topics, including everyday life, relationships, and legal advice. The dataset consists of approximately 349k training samples, 18.4k validation samples, and 18.4k test samples. Each example contains a prompt in the form of an instruction or question, along with two responses, where one response is annotated as more helpful by Reddit users. E.4 Evaluation Details For reproducibility, we use publicly available reward models fine-tuned on preference dataset 10 10 10 Skywork-Qwen: https://huggingface.co/Skywork/Skywork-Reward-V2-Qwen3-8B 11 11 11 Skywork-Llama: https://huggingface.co/Skywork/Skywork-Reward-Llama-3.1-8B. [ 28 ] as a proxy for r_{U} . Note that Skywork-Llama shares the same tokenizer as Llama3-8B-SFT, and Skywork-Qwen shares the same tokenizer as Qwen3-8B. This tokenizer compatibility is required for implementing ARGS. For controlled decoding, we use the first 10k prompts from the training split as the prompt set. For each evaluation setting listed in Table 1 , we sample 10 responses per prompt. For Vanilla CD, we train the state–action value network directly on this generated dataset with rewards scored by r_{U} . For SRS-CD, we first apply reward shaping to the collected rewards and then train the state–action value network on the shaped dataset. We use a separate validation set of 1k prompts for hyperparameter tuning in controlled decoding. For ARGS, the validation set is reduced to 300 prompts due to its higher decoding cost. For both CD and ARGS, we evaluate on 1,000 randomly selected test prompts from the test split of the dataset. All methods generate outputs on the same set of prompts, and we report evaluation metrics as defined in Section 6.2 . Our CD implementation is adapted from 39 12 12 12 https://github.com/williambankes/robust-multi-objective-decoding . As their work focuses on multi-objective decoding, we re-implemented substantial portions of the codebase to fit our setting. The ARGS implementation is based on 23 13 13 13 https://github.com/deeplearning-wisc/args . E.5 Architecture Details For CD, we train a neural network to approximate the state–action value function. We adapt the codebase from 24 14 14 14 https://github.com/Lingkai-Kong/RE-Control to train this value function, using the hidden states of the LLM as input features. The hyperparameters are detailed in Table 6 . Table 6 : Summary of hyperparameters used to train the Q function for Qwen3-8B in Eval-1 (CD) and Eval-2 (CD). Base Model \rho_{\mathrm{base}} Parameter Value Qwen3-8B Number of epochs 30 Learning rate 1\times 10^{-4} Batch size 64 Optimizer Adam Floating point format fp32 Hidden dimension [8192, 4096] E.6 Implementation Details Note that Program 3 is defined for each prompt \bm{x}\in X , and therefore the optimal reward bound may vary across prompts. In practice, we implement a data-dependent, prompt-level reward bound. For a given prompt \bm{x} , we draw M Monte Carlo samples from the base policy \rho_{\mathrm{base}} and denote by r_{\max} and r_{\min} the maximum and minimum rewards among these samples. Given a global reward bound B , we define an effective prompt-level reward bound B_{\text{eff}} for \bm{x} as follows. For SRS-CD, we use B_{\text{eff}}=\min\bigl\{1.5\cdot(r_{\max}-r_{\min}),\;B\bigr\}. For SRS-ARGS, we use B_{\text{eff}}=\min\bigl\{(r_{\max}-r_{\min}),\;B\bigr\}. This design prevents SRS from artificially amplifying reward differences when candidate responses for a prompt exhibit little reward variation, thereby avoiding unnecessary over-shaping. For controlled decoding, rewards are obtained by applying r_{U} to complete responses in the offline dataset. Empirically, these rewards exhibit a relatively small dynamic range. We therefore scale (r_{\max}-r_{\min}) to allow a larger effective bound B_{\text{eff}} , which expands the feasible space available to the reward model provider when searching for an optimal reward model. In contrast, ARGS evaluates rewards using r_{U}([\bm{s}_{t},\bm{y}_{t},\text{EOS}]) at inference time. Since r_{U} is trained on complete responses, applying it to partial responses typically results in larger reward variation. As a result, we do not apply the same scaling factor in ARGS to account for this larger variation. In experiment, k in Eq 5 is clipped at 2 to add numerical stability. E.7 Hyperparameter For each (evaluation setting, inference time method) pair, we sweep the reward strength \frac{1}{\beta} for the vanilla decoding policy on the validation dataset and select the value of \frac{1}{\beta} that achieves the best performance prior to the onset of reward hacking or oversteering. We then fix the reward strength for all methods in that evaluation setting. In our experiments, we use \tfrac{1}{\beta}=0.5 for ARGS and \tfrac{1}{\beta}=1.5 for CD. We hypothesize that ARGS requires a smaller reward strength because the reward model r_{U} is trained on complete responses but is applied to scoring partial responses during ARGS’s decoding, which can be less accurate and exhibit higher variance, necessitating a tighter reward strength to avoid reward hacking (See Figure 2 for an example of ARGS reward hacking in Eval-1). We report the hyperparameters, the reward bound B and shaping strength \alpha , used for the results in Table 2 . The reward bound B reported is global and the per prompt reward bound is computed via procedures in Appendix E.6 . For all baseline methods and evaluation settings, we fix the number of Monte Carlo samples to M=10 . Hyperparameters are selected using a validation sweep over B\in\{5,8,10,12,15\} and \alpha\in\{1.0,1.5,2.0\} . Vanilla inference-time methods and MEANSTD do not involve tunable hyperparameters and are therefore omitted from the table. Table 7: Hyperparameter used in Table 2 . Eval Method B \alpha Eval-1 SRS-ARGS 15 2.0 Eval-1 Minmax-ARGS 10 N/A Eval-1 SRS-CD 12 1.5 Eval-1 Minmax-CD 10 N/A Eval-2 SRS-ARGS 10 2.0 Eval-2 Minmax-ARGS 5 N/A Eval-2 SRS-CD 10 2.0 Eval-2 Minmax-CD 15 N/A Eval-3 SRS-ARGS 15 1.0 Eval-3 Minmax-ARGS 5 N/A Eval-4 SRS-ARGS 8 1.5 Eval-4 Minmax-ARGS 10 N/A E.8 GPT-4 Evaluation Following 23 , 6 , we use a GPT-based judge to conduct head-to-head comparisons of responses generated by different methods. Figure 7 shows the full evaluation prompt used during GPT-based scoring, which is the same prompt as in 23 . Figure 7 : Prompt Template for the GPT-4 Evaluation. E.9 Qualitative Examples We provide qualitative examples comparing our method with vanilla inference-time methods. Figure 8 : Qualitative examples showing the alignment performance of SRS. Figure 9 : Qualitative examples showing the alignment performance of SRS. Figure 10 : Qualitative examples showing the alignment performance of SRS. Figure 11 : Qualitative examples showing the alignment performance of SRS.