Test-Time Personalization: A Diagnostic Framework and Probabilistic Fix for Scaling Failures
Linhai Zhang, Yulan He · 2026
Read Test-Time Personalization: A Diagnostic Framework and Probabilistic Fix for Scaling Failures by Linhai Zhang, Yulan He (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-10991.txt.
Introduction
Linhai Zhang and Yulan He study test-time personalization, scaling inference compute for personalized language models instead of rebuilding the model. Sampling many candidates from a personalized policy and picking the best with a personalized reward model should raise utility, but standard reward models fail to deliver the gain.
Theory shows oracle selection grows expected utility logarithmically with candidate count, setting a ceiling for test-time scaling. A unified scaling law decomposes any reward model's best-of-N curve into four measurable quantities, exposing two failure modes: user-level collapse and query-level reward hacking. A probabilistic reward model whose learned variance counters both failures restores consistent scaling across models and tasks. Personalization teams get both a diagnosis and a fix for test-time scaling.
Catalog record
| CREATOR | Linhai Zhang, Yulan He |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 94531 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.10991
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Test-Time Personalization: A Diagnostic Framework and Probabilistic Fix for Scaling Failures Linhai Zhang, Yulan He Abstract Existing approaches to LLM personalization focus on constructing better personalized models or inputs, while treating inference as a single-shot process. In this work, we study Test-Time Personalization (TTP) along an unexplored axis: scaling inference-time computation by sampling N N candidates from a personalized policy model and selecting the best with a personalized reward model. We prove that oracle selection yields expected utility growing logarithmically with the number of sampled candidates, establishing a theoretical ceiling for test-time scaling. However, standard reward models fail to realize this potential. To diagnose why, we derive a unified scaling law that decomposes any reward model’s Best-of- N N curve into four measurable quantities and reveals two failure modes, user-level collapse (near-constant prediction for some users) and query-level reward hacking (negative correlation with true quality for some queries). Guided by this law, we propose a probabilistic personalized reward model whose learned variance effectively mitigates both failure modes. Experiments confirm both elements of our framework: TTP delivers consistent scaling across multiple policy models and personalized text generation tasks, and our scaling law closely matches observed scaling curves across reward-model variants. 1 Introduction Large language models excel across diverse tasks, yet they predominantly produce one-size-fits-all responses that ignore individual user preferences [ 7 ] . This limitation has motivated growing research interest in LLM personalization. Existing approaches fall into three categories. Personalized Prompting augments input context with retrieved user history [ 15 , 17 ] . Personalized Adaptation directly fine-tunes model parameters on user data [ 21 , 25 ] . Personalized Alignment combines multi-objective reward models with user-specific weights [ 3 , 1 ] . Despite their differences, these approaches share a common paradigm: they focus on constructing better personalized models or inputs, while treating inference as a single-shot generation process. Meanwhile, scaling test-time computation has emerged as a powerful axis for improving LLM performance, particularly in reasoning tasks [ 22 , 19 , 18 ] . Recent work has begun to combine test-time scaling with personalization, but only along two axes: scaling user interactions to learn each user’s preferences online [ 11 ] , and scaling reward-model reasoning via generative reward models [ 26 ] . A third, equally natural axis, scaling the policy model itself , has not been studied for personalization. This raises a natural question: Can we improve personalization by scaling test-time computation for a weak personalized model? A straightforward instantiation of test-time personalization is parallel sampling [ 17 , 4 ] : a personalized policy generates N candidates per query, and a personalized reward model selects the best. We first establish the theoretical promise of this approach: when the number of candidates increases, the expected utility of the optimal sample grows logarithmically (Section 3 ). This provides a theoretical ceiling for what test-time personalization can achieve. Realizing this ceiling in practice, however, is far from trivial. As Figure 1 shows, both a global reward model trained on pooled data and a per-user reward model trained on individual histories perform near random selection across N , leaving the oracle gap wide open. What prevents standard reward models from scaling? Figure 1 : Test-time personalization on LaMP-4: News Headline Generation. Oracle shows logarithmic scaling that surpasses training-based baselines, while standard Reward Models (RMs) fail to scale, performing close to, or worse than, random selection. To diagnose this gap, we develop an analytical framework that connects a reward model’s correlation with the golden score to its Best-of- N scaling behavior (Section 4 ). The analysis surfaces two distinct failure patterns, user-level collapse where the reward model degenerates to near-constant predictions for some users and query-level reward hacking where the reward model’s predictions negatively correlate with quality for some queries. We then generalize the oracle scaling law into a general expression that forecasts the scaling curve of any reward model from four measurable quantities. Guided by this expression, we propose a probabilistic personalized reward model that mitigates both failure patterns via learned variance, enabling stable test-time scaling in practice (Section 5 ). Experiments on two benchmarks covering five personalized text generation tasks confirm both elements of the framework: our probabilistic reward model scales reliably and surpasses training-based baselines on most tasks, and the general expression closely matches observed scaling curves. In summary, our contributions are three-fold: • We introduce Test-Time Personalization (TTP), a new paradigm focused on scaling policy-model compute for personalized text generation. • We detect two failure patterns and a predictive scaling law, a general expression that forecasts any reward model’s scaling curve from four measurable quantities. • We propose a probabilistic personalized reward model that mitigates both failure patterns and scales reliably across multiple policy types and tasks. 2 Preliminaries We focus on personalized text generation tasks, where each user has demonstrated stylistic preferences through written examples and the system must mimic these preferences at inference time. 2.1 Problem Formulation Consider a set of users \mathcal{U} , where each user u\in\mathcal{U} has an underlying preference function r_{u}^{*}:\mathcal{Q}\times\mathcal{X}\to\mathbb{R} that maps a query-response pair (q,x) to a scalar reward reflecting how well x aligns with the user’s preferences for query q . Each user is associated with historical data \mathcal{D}_{u}=\{(q_{i},x_{i})\}_{i=1}^{n_{u}} . Given a query q , a personalized policy \pi_{u} conditioned on user history generates N candidate responses: \{x_{1},x_{2},\ldots,x_{N}\}\sim\pi_{u}(\cdot\mid q) . A personalized reward model \hat{r}_{u} trained on \mathcal{D}_{u} selects the best candidate: x^{*}=\arg\max_{x_{i}}\hat{r}_{u}(q,x_{i}) . The effectiveness of TTP is measured by the expected true reward of the selected response: U(N)=\mathbb{E}_{q,\,x_{1:N}\sim\pi_{u}}\left[r_{u}^{*}(q,x^{*})\right]. (1) The oracle strategy selects candidates using the true preference function r_{u}^{*} , yielding the optimal utility U_{\text{oracle}}(N) . The goal of TTP is to approach this oracle performance. 2.2 Experimental Setup We use five personalized text generation tasks from LaMP [ 17 ] and LongLaMP [ 4 ] , covering news headlines, scholarly titles, abstracts, product reviews, and topical posts. The personalized policy follows a retrieval-augmented generation (RAG) recipe [ 17 ] , conditioning generation on user-history examples retrieved per query. Since r_{u}^{*} is unobservable, we use ROUGE between generated responses and user-written references as a ground-truth-reward proxy. For each user, we sample candidate responses from the policy model and compute their ROUGE scores based on the golden response to construct training data. We consider two standard reward models: Global RM , trained on pooled data across users, and User RM , trained per user on \mathcal{D}_{u} . For evaluation, the policy model generates N candidates per query and different reward models select the best candidate. We report ROUGE scores averaged across users. 3 The Promise of Test-Time Personalization We begin by establishing the theoretical foundation of TTP: given access to an oracle reward function, the expected utility of selected responses grows logarithmically with the number of candidates. We then empirically validate this scaling law and show that oracle TTP can surpass training-based methods [ 21 ] . 3.1 Theoretical Foundation Our theoretical analysis establishes a scaling law for oracle selection: Theorem 3.1 ( Oracle Scaling Law ) . Assume the true reward of responses sampled from \pi_{u}(\cdot\mid q) is sub-Gaussian with mean \mu_{u} and variance proxy \sigma_{u}^{2} . Let x^{*}_{\mathrm{oracle}}=\arg\max_{i}r_{u}^{*}(q,x_{i}) denote the oracle selection from N