Personalizing Large Language Model Agents with Small Policy Models

Dian Jin, Zhi Zhang, Huichao Li, Yihe Pan, Rundong Huang, Doudou Zhou · 2026

Read Personalizing Large Language Model Agents with Small Policy Models by Dian Jin, Zhi Zhang, Huichao Li, Yihe Pan, Rundong Huang, Doudou Zhou (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2608-00215.txt.

Introduction

Jin and co-authors personalize LLM agents with a small policy model called FABLE. Agents can fetch memory, call tools, and vary style. Adapting those execution choices to one user stays hard. Fine-tuning a separate model is costly or barred for closed systems. Prompts expose user facts but do not teach execution choices from feedback.

FABLE sits outside the frozen host agent as a light policy layer. It factorizes memory, information, and response decisions for shared learning. An external feasible set filters actions before exploration. Bayesian contextual Thompson sampling learns each user's residual tastes. A calibrated variant carries an expected-regret bound. Tests show gains on preference-sensitive behaviours, they report. End-to-end task performance stayed competitive.

Catalog record

CREATORDian Jin, Zhi Zhang, Huichao Li, Yihe Pan, Rundong Huang, Doudou Zhou
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE120 KB
LICENSEhttp://creativecommons.org/publicdomain/zero/1.0/
RIGHTSCC0
SOURCEarXiv

Source: https://arxiv.org/abs/2608.00215

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Personalizing Large Language Model Agents with Small Policy Models Dian Jin, Zhi Zhang, Huichao Li, Yihe Pan, Rundong Huang, Doudou Zhou Abstract Large language model (LLM) agents can retrieve memory, call tools, ask clarifying questions, and vary response style, yet adapting these execution decisions to an individual user remains difficult. Fine-tuning a separate LLM is costly or impossible for proprietary systems, while prompts and memory primarily expose user information to the agent rather than adapt its execution decisions from feedback. We formulate personalization of a frozen agent as online learning of a per-user execution policy from scalar feedback observed only for the executed action. We propose FABLE (Factorized Adaptive Bandit Layer for Execution), a lightweight policy layer outside a potentially black-box host agent. FABLE factorizes memory, information-acquisition, and response decisions so feedback updates related choices; filters actions through an externally specified feasible set before exploration; and learns user-specific residual preferences relative to a fixed default-and-cost score via Bayesian contextual Thompson sampling. Under a linear residual-reward model, FABLE with a theoretically calibrated Gaussian sampling scale admits a O ~ ​ ( d 3 / 2 ​ n ) \widetilde{O}(d^{3/2}\sqrt{n}) regret guarantee against the best feasible action at each round. On a four-domain tau2-bench evaluation, FABLE attains the highest observed personalized reward and synthetic verbosity alignment while tying the highest task-success mean. Matched comparisons support factorization, onboarding, and online adaptation for preference-sensitive metrics, but not task-success improvement. An interactive demo is available at https://fable-agent.github.io/ . 1 Department of Statistics and Data Science, National University of Singapore 2 Department of Statistics and Data Science, University of California, Los Angeles 3 Department of Data Science, Fudan University 1 Introduction For agents based on large language models (LLMs), personalization concerns both the user information available to an agent and the execution policy governing how it acts on that information. This policy determines how the agent uses memory and tools, when it seeks clarification or confirmation, and how it presents the final answer. In response to the same travel-planning request, one user may expect the agent to retrieve relevant past trips, verify current options online, and confirm before booking, whereas another may prohibit memory access and prefer a direct answer without follow-up questions. Choosing between these behaviors for a particular user is a decision problem that the host model’s capabilities alone do not settle. Existing approaches leave this decision problem open in different ways. Per-user fine-tuning adapts model parameters, but it is costly and unavailable when the host is proprietary ( Tan et al. 2024 ) . Profiles, prompts, and memory systems supply user information to a fixed model, yet a rule stated in a prompt is never revised when the user’s later reactions contradict it, and these mechanisms do not decide when the agent should consult memory, invoke a tool, or ask a question ( Salemi et al. 2024 ; Chhikara et al. 2025 ) . Learned decision layers for frozen agents come closest: they select actions around a fixed host from offline rollouts or online feedback ( Yi and Song 2026 ; Yu et al. 2026 ) . However, they learn one shared policy for task execution rather than a policy per user, and the online variant conditions on the host’s hidden states, which a black-box API does not expose. Finally, recent personalized-agent methods adapt a single mechanism from interaction, such as decoding, memory use, or tool selection ( Qu et al. 2025 ; Liang et al. 2026 ; Yoon et al. 2026 ) . In an agent, however, the user reacts to the delivered interaction as a whole, and that reaction reflects the combined effect of the memory, information-acquisition, and response choices; a method that adapts one mechanism in isolation has no defined way to attribute this single signal across the choices that produced it. This feedback structure is the central difficulty. Each interaction executes one combination of execution decisions, and the user’s reaction or the task outcome is observed only for that combination; how the alternatives would have performed is never revealed. Observations may therefore be frequent yet individually uninformative about most of the action space. The combinations nevertheless share components. Evidence that a user dislikes unnecessary clarification should carry over to actions that ask for clarification under different memory or tool settings, whereas a flat model over complete combinations would relearn this preference once per combination. Two further properties of the setting shape our formulation. Onboarding information is useful but fallible, so it should enter as revisable prior evidence rather than as a fixed rule. Permissions, tool availability, and mandatory confirmations are not preferences at all: they determine which actions may be attempted, and they must be enforced before any exploration. We therefore formulate personalization of a frozen agent as online learning of a per-user execution policy over a factorized action space, from scalar feedback observed only for the selected action, subject to externally specified feasibility constraints. FABLE ( Factorized Adaptive Bandit Layer for Execution ) instantiates this formulation as a compact Bayesian policy layer that requires no access to host weights, gradients, or hidden states and no changes to tool implementations. Onboarding initializes a revisable residual state. At each interaction, a context adapter summarizes the request, interaction history, non-preference background, and hard state; a feasibility filter removes inadmissible actions; and the policy selects a factorized action over memory use, information acquisition, and response behavior, which is compiled into instructions for the host. Expected feedback is decomposed into a prespecified default score, an operational cost, and a user-specific residual, and Bayesian Thompson sampling is applied only to the residual, so a single scalar observation updates the feature directions shared by related actions. Our contributions are as follows. • We formalize per-user execution-policy learning for a frozen, potentially black-box agent as a feasibility-constrained contextual bandit over factorized execution actions with selected-action scalar feedback, and instantiate it as the FABLE algorithm (Section 3 ). • We establish a feasible-oracle regret guarantee for FABLE under a linear residual-feedback model. For a predictable known default–cost offset that may vary with context and action, a general Gaussian working initialization—including the onboarding-informed initialization used by FABLE—and the calibrated Gaussian sampling schedule \nu_{t}=\sqrt{9d\log(t/\delta)} , we adapt linear Thompson-sampling analysis to obtain \widetilde{O}(d^{3/2}\sqrt{n}) high-probability regret against the best action in each supplied feasible set, together with an expected-regret guarantee of the same order (Theorem 1 ). Supplementary analyses further characterize exactly which preference directions remain identifiable when feasibility constraints persistently exclude actions (Supplementary Proposition 2 and Corollary 1), and provide an anytime-valid confidence-sequence rule that controls false promotion of learned preferences (Appendix G). • We evaluate FABLE in a four-domain tau2-bench protocol (Section 5 ). The full policy has the highest observed personalized reward and alignment and ties the highest task-success mean. Relative to the host, its paired gains are +0.077 reward and +0.281 alignment, both with positive 95% CIs; task success remains unresolved. Matched controls further support cross-action sharing, revisable onboarding, and online adaptation for preference-sensitive metrics. Appendix C reports complementary PAHF and Math500 studies and negative cases. 2 Related Work Execution control for fixed agents. ReAct interleaves language reasoning with environment actions, while Toolformer trains a language model to invoke external tools ( Yao et al. 2023

Read the complete text