PITA: Preference-Guided Inference-Time Alignment for LLM Post-Training
Sarat Chandra Bobbili, Ujwal Dinesha, Dheeraj Narasimha, Srinivas Shakkottai · 2025
Read PITA: Preference-Guided Inference-Time Alignment for LLM Post-Training by Sarat Chandra Bobbili, Ujwal Dinesha, Dheeraj Narasimha, Srinivas Shakkottai (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-20067.txt.
Introduction
Bobbili and co-authors introduce in 2025 PITA, a method that aligns language model output with user preferences at inference time, with no retraining. Guidance model methods steer token choices but lean on a separate reward model fit to human feedback, an unstable step. PITA removes that dependency.
A small guidance policy learns from preference feedback and shifts token probabilities as text is generated. The task is framed as finding a hidden preference distribution, solved by stochastic search and repeated refinement. Tests cover math reasoning and sentiment classification. Results show aligned output at lower compute cost than fine tuning routes, which is the practical appeal.
Catalog record
| CREATOR | Sarat Chandra Bobbili, Ujwal Dinesha, Dheeraj Narasimha, Srinivas Shakkottai |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 75 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-20067
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
PITA: Preference-Guided Inference-Time Alignment for LLM Post-Training Sarat Chandra Bobbili, Ujwal Dinesha, Dheeraj Narasimha, Srinivas Shakkottai Abstract Inference-time alignment enables large language models (LLMs) to generate outputs aligned with end-user preferences without further training. Recent post-training methods achieve this by using small guidance models to modify token generation during inference. These methods typically optimize a reward function KL-regularized by the original LLM taken as the reference policy. A critical limitation, however, is their dependence on a pre-trained reward model, which requires fitting to human preference feedback—a potentially unstable process. In contrast, we introduce PITA, a novel framework that integrates preference feedback directly into the LLM’s token generation, eliminating the need for a reward model. PITA learns a small preference-based guidance policy to modify token probabilities at inference time without LLM fine-tuning, reducing computational cost and bypassing the pre-trained reward model dependency. The problem is framed as identifying an underlying preference distribution, solved through stochastic search and iterative refinement of the preference-based guidance model. We evaluate PITA across diverse tasks, including mathematical reasoning and sentiment classification, demonstrating its effectiveness in aligning LLM outputs with user preferences. 1 Introduction Large language model (LLM) alignment traditionally relies on additional training, such as fine-tuning with human feedback, to ensure outputs conform to human desires. However, fine-tuning LLMs is computationally expensive and complex. This has motivated the study of inference-time alignment, where the pre-trained LLM’s weights remain frozen, and alignment is achieved by steering the model’s decoding process to meet end-user needs. Inference-time alignment aims to align generated outputs with user criteria (e.g., helpfulness, correctness, sentiment) without updating the LLM’s parameters. By intervening only during generation, this approach avoids retraining overhead and enables the alignment of opaque models whose internal weights are inaccessible. Recent post-training alignment methods employ small guidance models or value functions to influence token generation during decoding. These approaches frame decoding as an optimization problem, where the model maximizes a user-provided reward while maintaining proximity to its original style. They optimize a KL-regularized objective, balancing the base LLM’s log-likelihood with the reward from the guidance model. The reward signal originates from a pre-trained model that scores candidate tokens based on their anticipated impact on the alignment of subsequent tokens with the reward criteria. These methods integrate the guidance model into the decoding loop, adjusting the next-token distribution at each step. Although computationally intensive, these techniques enhance alignment without modifying the LLM’s weights. Figure 1 : Overview of PITA training and inference time implementation. During training, our algorithm learns a Action-Value function Q^{*} directly from the preference information. At Inference, the output distribution of the original LLM, \pi_{\mathrm{ref}} , is modified with exponentially weighted Q^{*} -values Reward-model-guided alignment methods face several challenges. First, they assume the availability of a pre-trained reward model, which is often not available. Reward models are typically learned from human-labeled preference data. However, preference data might be noisy or inconsistent, hindering the accurate learning of a reward model that reflects true user intent. Moreover, training a reward model can be unstable and prone to brittleness, where minor inaccuracies in the model can cause the LLM to generate misaligned outputs. These issues highlight the difficulty of learning a reliable reward model from preference data, motivating the need to eliminate an explicit reward predictor. In this work, we introduce PITA: Preference-Guided Inference-Time Alignment , a novel framework that eliminates the need for a reward model in LLM post-training. PITA directly uses preference feedback during inference to guide token generation. The key innovation is the use of a small, preference-based guidance policy that operates alongside a frozen LLM and adjusts token probabilities in real-time, without modifying the LLM’s weights. This guidance policy is trained using user preference signals, significantly reducing the requirement for extensive human-labeled data and avoiding the complexity of reward model training. PITA frames the alignment task as learning an underlying preference distribution over the model’s outputs. It uses stochastic search and iterative refinement to identify tokens that align better with user preferences. During each decoding round, the guidance policy explores multiple continuations and updates based on user feedback, progressively refining the model’s predictions. This process guides the model to converge towards outputs that better match user preferences, while keeping the base LLM unchanged. Figure 1 illustrates the differences between PITA and standard post-training. We first provide a performance analysis of PITA, using the notion of a linear function approximator for the preference model. As the learned preference model aligns with the true one, PITA’s guidance model aligns with the true reward-to-go, resulting in sub-linear regret. We then evaluate PITA on diverse tasks, including mathematical reasoning and sentiment alignment datasets. An example of the outputs generated by the reference LLM vs. PITA for a math reasoning problem from the GSM8K dataset is shown in Figure 2 . Here, PITA aligns the model’s steps with user-preferred solutions without fine-tuning, improving accuracy and consistency. In sentiment tasks, PITA ensures outputs meet user-defined sentiment or safety criteria, such as avoiding toxic language. Problem from GSM8K: Christina is planning a birthday party and needs .75 gift bags per invited guest, because 1/4 of attendees don’t show up. She invited 16 friends. Gift bags are $2 each. How much will she spend? Incorrect Solution from \pi^{\text{ref}} : Let’s break this down step by step: 1. Christina invited 16 friends. 2. Since 1/4 of attendees don’t show up, she’ll only need to give gift bags to 3/4 of the invited guests. To find this amount, multiply 16 by 3/4: 16 × (3/4) = 12. 3. Since she needs 0.75 gift bags per invited guest, she’ll need 0.75 × 12 = 9 gift bags. 4. Each gift bag costs $2, so the total cost will be 9 × $2 = $18. So, Christina will spend \boxed{\$18} on gift bags. Correct Solution from PITA: Since Christina invited 16 friends, and 1/4 of attendees don’t show up, the number of friends who will actually attend the party is 16 - (1/4)*16 = 16 - 4 = 12. Since Christina needs 0.75 gift bags per invited guest, and she invited 16 friends, she needs 0.75*16 = 12 gift bags. The cost of each gift bag is $2, so the total cost of the gift bags is 12*2 = $24. Final Answer: The final answer is [formula] . I hope it is correct. Figure 2 : Comparison of Incorrect Solution from the Reference model and the Correct Solution from PITA to a math reasoning problem. We compare PITA against two reward-based alignment schemes: Q# ( Zhou et al. 2025 ) uses a known reward, and Q#-HF where we learn reward from preference data via maximum likelihood estimation, followed by alignment using Q#. PITA performs comparably to Q#, while outperforming the learned-reward-based alignment method (as well as the base LLM), demonstrating its superior efficiency and effectiveness in utilizing preference data. PITA achieves this level of performance without requiring a pre-trained reward model, highlighting its advantage in preference-guided inference-time alignment. 2 Related Work Preference learning has a long history in RL and bandit literature ( Akrour et al. 2012 ; Auer et al. 2014 ; Sadigh et al. 2017 ) . Alongside the empirical success of training LLMs to generate outputs aligned with