Aligning Language Models from User Interactions
Thomas Kleine Buening Jonas Hübotter Barna Pásztor, [1pt] Idan Shenfeld Giorgia Ramponi Andreas Krause · 2026
Read Aligning Language Models from User Interactions by Thomas Kleine Buening Jonas Hübotter Barna Pásztor, [1pt] Idan Shenfeld Giorgia Ramponi Andreas Krause (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-12273.txt.
Introduction
Kleine Buening and co-authors learn alignment directly from multi-turn user interactions. Follow-up messages often reveal that a reply missed the mark, yet training usually discards them. Models can already use such follow-ups in context to revise their behavior. Their method conditions the model on the follow-up and compares its new token distribution with the old policy. That hindsight distribution becomes a target, distilled back into the current policy. Training on real WildChat conversations improved alignment scores and enabled personalization without explicit feedback labels.
Catalog record
| CREATOR | Thomas Kleine Buening Jonas Hübotter Barna Pásztor, [1pt] Idan Shenfeld Giorgia Ramponi Andreas Krause |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 54725 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.12273
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Aligning Language Models from User Interactions Thomas Kleine Buening Jonas Hübotter Barna Pásztor, [1pt] Idan Shenfeld Giorgia Ramponi Andreas Krause Abstract Multi-turn user interactions are among the most abundant data produced by language models, yet we lack effective methods to learn from them. While typically discarded, these interactions often contain useful information: follow-up user messages may indicate that a response was incorrect, failed to follow an instruction, or did not align with the user’s preferences. Importantly, language models are already able to make use of this information in context. After observing a user’s follow-up, the same model is often able to revise its behavior. We leverage this ability to propose a principled and scalable method for learning directly from user interactions through self-distillation. By conditioning the model on the user’s follow-up message and comparing the resulting token distribution with the original policy, we obtain a target for updating the policy that captures how the model’s behavior changes in hindsight. We then distill this hindsight distribution back into the current policy. Remarkably, we show that training on real-world user conversations from WildChat improves language models across standard alignment and instruction-following benchmarks, without regressing other capabilities. The same mechanism enables personalization, allowing models to continually adapt to individual users through interaction without explicit feedback. Our results demonstrate that raw user interactions that arise naturally during deployment enable alignment, personalization, and continual adaptation. Introduction In modern language models, inference has overtaken training as the dominant consumer of compute, with models serving massive volumes of user queries every day. Yet the information revealed through these interactions is typically discarded and does not contribute to improving the model itself, representing a significant missed opportunity. At scale, users engage in extended conversations, refining prompts, requesting revisions, and responding directly to model outputs. These interactions are rich with implicit learning signals: follow-up messages may indicate that a response was incorrect, failed to follow an instruction, or did not align with the user’s preferences ( 11 ) . For example, a user may report an error after executing generated code, point out that a required format was not followed, or ask for a response to be rewritten in a different style or tone. Such signals arise organically during normal use and reflect how model outputs are received and acted upon during deployment. Finding ways to leverage this data source can open the door to continual learning from deployment at an unprecedented scale. Despite their scale and richness, we still lack effective methods to learn directly from user interactions. Unlike standard datasets ( 30 ; 9 ) , user interactions do not come with explicit labels, expert demonstrations, preference comparisons, or rewards. Instead, feedback is implicit and expressed through natural language responses whose meaning depends on the surrounding interaction context. As a result, it is unclear how to train directly on real-world conversations in a principled manner. At the same time, we observe that language models already demonstrate the ability to leverage this interactive information in context, a capability known as in-context learning ( 4 ; 50 ) . In multi-turn conversations, models often revise their behavior effectively after observing a user’s follow-up. When a user reports an error in generated code, the model can frequently infer which part of its previous response was incorrect and propose a fix. When a user points out that a required format was not followed, the model is able to correct the structure in a revised answer. When a user expresses dissatisfaction with tone or style, the model can adapt its response to better match the user’s preferences. In these cases, conditioning on the user’s follow-up message leads to responses that are more aligned with the task and the user’s intent. These observations suggest a simple but powerful perspective: having seen the user’s follow-up message, the model’s behavior is often better aligned than before. The user interaction reveals information that the model can already interpret and act upon, but only after the fact. In hindsight . Crucially, this improvement arises without additional supervision and reflects how the model’s behavior changes when it has access to the user’s follow-up. This suggests that a model’s in-context learning ability can be used as a lever for learning directly from user interactions in a principled way. Based on this idea, we introduce a simple and scalable method for learning directly from user interactions by comparing a model’s original behavior to what it would have done in hindsight. Concretely, after observing a user’s follow-up message, we reprompt the same model with this additional context and obtain a hindsight token distribution that reflects how the model would respond if it had access to the information revealed by the user. By comparing the original policy to this hindsight policy at every token of the original generation, we obtain a comparative learning signal that identifies how the model’s behavior should change. We then distill this signal back into the original policy using only the observed interaction. In other words, we distill the model into itself. Building on recent work on self-distillation ( 18 ) , we refer to this approach as Self-Distillation Policy Optimization (SDPO) from User Interactions. We show that this approach (illustrated in Figure 1 ) is simple and scalable, and enables language models to improve from raw, real-world user conversations without explicit supervision, reward models, or preference labels. Remarkably, when applied to real-world user interactions from WildChat ( 59 ) , SDPO from User Interactions improves alignment and instruction-following performance across standard benchmarks without degrading other capabilities. We also demonstrate that the same self-distillation mechanism naturally supports personalization and continual adaptation, allowing models to adapt to individual users purely through continued interaction. Figure 1: Direct Learning from User Interactions via Self-Distillation. From multi-turn user conversations, we obtain several interactions (x,y,o) that consist of the conversation history x , the model’s response y , and the subsequent user message o . By conditioning on the user’s follow-up, we form the hindsight policy and compare it to the original policy, producing token-level advantages that reinforce or penalize parts of the model’s original response. In this example, the user’s follow-up requests a more direct answer, leading to penalizing filler tokens and reinforcing the answer. Problem Formulation The interaction between a language model and a user consists of a sequence of alternating assistant messages y_{t} and user messages o_{t} . At the t -th turn, the language model observes the conversation history x_{t}=(o_{0},y_{1},o_{1},\dots,o_{t-1}) , and generates a response y_{t}\sim\pi_{\theta}(\cdot\mid x_{t}) . 1 1 1 In practice, the assistant may condition on a representation of x_{t} , such as a sliding context window, a learned embedding, or a summary. For simplicity, we treat x_{t} as the full interaction history here. In turn, the user responds with o_{t} , assuming the conversation does not terminate. In case of a fresh conversation initiated by the user, the first interaction reduces to the initial prompt o_{0} , the assistant’s answer y_{1} , and the user’s follow-up o_{1} . We define the triple (x_{t},y_{t},o_{t}) as a single interaction. Consequently, a user conversation (o_{0},y_{1},\dots,y_{t},o_{t}) yields t interactions (x_{1},y_{1},o_{1}),\dots,(x_{t},y_{t},o_{t}) , which overlap in their histories, since x_{t+1} contains all of x_{t} plus the most recent interaction. For convenience, we