Latent Inter-User Difference Modeling for LLM Personalization
Yilun Qiu, Tianhao Shi, Xiaoyan Zhao, Fengbin Zhu, Yang Zhang, Fuli Feng · 2025
Read Latent Inter-User Difference Modeling for LLM Personalization by Yilun Qiu, Tianhao Shi, Xiaoyan Zhao, Fengbin Zhu, Yang Zhang, Fuli Feng (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-20849.txt.
Introduction
Yilun Qiu and co-authors personalize large language model output by comparing users with each other in a 2025 paper. Most personalization leans on a user's own history alone. Yet how a user differs from peers who liked similar things carries strong signals. Writing those differences into text prompts, though, often fails to capture them well.
Their framework works in the model's hidden space instead. It builds soft prompts from the gap between a user's embedding and peers' embeddings. A sparse filter then keeps only features relevant to the task before feeding them into a frozen model. On review-writing tests, the method beat baselines across several measures. The authors share their code for others to try.
Catalog record
| CREATOR | Yilun Qiu, Tianhao Shi, Xiaoyan Zhao, Fengbin Zhu, Yang Zhang, Fuli Feng |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 51 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-20849
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Latent Inter-User Difference Modeling for LLM Personalization Yilun Qiu, Tianhao Shi, Xiaoyan Zhao, Fengbin Zhu, Yang Zhang, Fuli Feng Abstract Large language models (LLMs) are increasingly integrated into users’ daily lives, leading to a growing demand for personalized outputs. Previous work focuses on leveraging a user’s own history, overlooking inter-user differences that are crucial for effective personalization. While recent work has attempted to model such differences, the reliance on language-based prompts often hampers the effective extraction of meaningful distinctions. To address these issues, we propose Difference-aware Embedding-based Personalization (DEP), a framework that models inter-user differences in the latent space instead of relying on language prompts. DEP constructs soft prompts by contrasting a user’s embedding with those of peers who engaged with similar content, highlighting relative behavioral signals. A sparse autoencoder then filters and compresses both user-specific and difference-aware embeddings, preserving only task-relevant features before injecting them into a frozen LLM. Experiments on personalized review generation show that DEP consistently outperforms baseline methods across multiple metrics. Our code is available at https://github.com/SnowCharmQ/DEP . 1 Introduction With continuous advancements in general-purpose intelligence, large language models (LLMs) Achiam et al. 2023 ; Grattafiori et al. 2024 ; Yang et al. 2025 ; Team et al. 2025 ; Guo et al. 2025 are increasingly integrated into everyday life, assisting users in making decisions Yao et al. 2023 ; Zhao et al. 2025c , retrieving information Zhao et al. 2024a ; Asai et al. 2024 , and task management Shen et al. 2024b . This growing presence has raised expectations for LLMs to go beyond generic, one-size-fits-all responses and instead produce responses that align with individual users’ unique preferences. To meet these heightened expectations, there has been a growing interest in LLM personalization Chen et al. 2024b ; Zhang et al. 2024b ; Xu et al. 2025 ; Liu et al. 2025b , which aims at tailoring model outputs based on user-specific information. Most existing methods adopt the memory-retrieval paradigm Salemi et al. 2024 ; Richardson et al. 2023a , where user history is stored in memory, and key information is then retrieved as a steering prompt to guide model generation. Earlier works Li et al. 2023 ; Mysore et al. 2024 focused solely on retrieving information about the user themselves for personalization. However, recent work such as DPL Qiu et al. 2025 argues that effective personalization should also capture how a user differs from others. This view is grounded in insights from psychology and behavioral science Snyder and Fromkin 1977 ; Snyder and Fromkin 2012 ; Irmak et al. 2010 , which highlight that inter-user variability determines individuality and shapes users’ distinct preferences. Accordingly, DPL incorporates inter-user comparison in the retrieval history, formulating the comparison as a natural language inference task performed by the LLM. Despite DPL’s demonstrated effectiveness, we argue that its language-based inter-user comparison paradigm using LLMs is structurally ill-suited for accurately extracting inter-user differences. On one hand, controlling the extraction of differences using an LLM is challenging; although providing extraction criteria can help, some aspects of distinction may be missed due to the difficulty of defining comprehensive standards. On the other hand, including other users’ raw data for comparison in LLMs can result in verbose prompts that strain the model’s context window, ultimately hindering the extraction of meaningful inter-user differences. To address these limitations, we propose shifting to latent-space difference modeling, where task-relevant differences between users are structurally represented and compared in the latent embedding space Doddapaneni et al. 2024 ; Liu et al. 2025c ; Zeldes et al. 2025 ; Ning et al. 2025 . Compared to natural language, latent embeddings offer two key advantages: (1) they encode fine-grained, context-dependent behavioral patterns in a compact form; and (2) they inherently support inter-user comparison through vector operations, enabling direct integration of comparison signals. Together, these properties make latent embeddings a more suitable medium for modeling inter-user differences within LLMs. Building on this idea, we propose a new method called Difference-aware Embedding-based Personalization (DEP), which models task-relevant inter-user differences in the latent space and injects them into LLMs as soft prompts for personalization. DEP extracts a difference-aware embedding as a soft prompt by subtracting and aggregating the user’s embedding against those of other users who engaged with similar items. At the same time, the original user-specific embedding is provided as a reference to supply contextual information. Both embeddings are essential: the user-specific embedding defines the behavioral context, while the difference-aware embedding captures deviations from that context. Together, they form a contextualized inter-user signal that reflects both individualized preferences and relative differences. Taking a step further, latent differences can be redundant, as not all aspects are task-relevant—some may simply introduce noise. To extract essential information while filtering out irrelevant signals, we process both user-specific and difference-aware embeddings using a sparse autoencoder (SAE) Huben et al. 2024 , which enforces sparsity to retain only key features. The resulting compressed representations are then injected into a frozen LLM as soft prompts. The SAE is fine-tuned to align these representations with the LLM’s internal understanding, allowing the model to effectively leverage inter-user differences for improved personalization. We conduct extensive experiments on a representative task, review generation Ni et al. 2019 , where DEP achieves state-of-the-art performance across multiple evaluation metrics. Our main contributions are as follows: • We propose modeling inter-user differences in the latent space to enable more comprehensive and flexible extraction of preference signals for LLM personalization. • We introduce a novel method, DEP, to achieve latent difference modeling, equipped with a sparse autoencoder to extract task-relevant differences while filtering out noise. • Extensive experiments show that our DEP consistently outperforms baseline methods with significant improvements. 2 Preliminary Problem Formulation. This work studies the task of LLM personalization, where the goal is to produce user-aligned output that reflects the individual preferences of a given user. We assume that each user has a set of historical texts. These historical texts are utilized to help the LLM infer the user’s interests and generate personalized content. Formally, let D denote the collection of historical records from all users. Each record in D is represented as (u,i,y_{u}^{i}) , where u is a user, i is an item (or object) the user has focused on, and y_{u}^{i} denotes the text written or preferred by user u for item i . When the target user u^{\prime} submits a request to generate text for a target item i^{\prime} , the LLM is expected to produce an output that aligns with the preference of u^{\prime} based on D . Without loss of generality, this work focuses on review generation, a representative personalization task. The goal is to generate reviews tailored to a user’s style and preferences, ensuring the output aligns with how the user typically expresses opinions on items such as movies or products. Figure 1: Overview of the proposed DEP method, which introduces user-specific and difference-aware embeddings to capture both individual preferences and inter-user differences. A sparse autoencoder (SAE) refines these representations, which are then injected into a frozen LLM as soft prompts to guide personalized text generation. Memory-retrieval framework. A common approach to enabling LLMs to perform personalized generation is to store users’ history and retrieve relevant signals at inference.