KARMA: Karma-Aligned Reward Model Adaptation

Jared Scott, Jesse Roberts · 2026

Read KARMA: Karma-Aligned Reward Model Adaptation by Jared Scott, Jesse Roberts (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-26738.txt.

Introduction

Scott and Roberts build KARMA, a framework that teaches language models conversational pragmatics, the tone and norms that shape how messages land, from Reddit karma signals. A reward model trained on Reddit conversations guides reinforcement learning, and the tuned models behave more appropriately in pragmatics-heavy tasks. Surprisingly, the best downstream gains come from a context-only reward model, while factuality drops in every condition, a tension the authors trace to the reward signal itself.

Catalog record

CREATORJared Scott, Jesse Roberts
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34346
LICENSEhttp://creativecommons.org/licenses/by-sa/4.0/
RIGHTSCC BY SA
SOURCEarXiv

Source: https://arxiv.org/abs/2605.26738

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

KARMA: Karma-Aligned Reward Model Adaptation Jared Scott, Jesse Roberts Abstract Human communication depends on implicit social signals where effectiveness is shaped by tone, context, and conversational norms rather than semantic content alone. We introduce KARMA (Karma-Aligned Reward Model Adaptation), a framework for LLM learning of context-sensitive conversational behavior from large-scale social interaction data. KARMA trains a reward model on Reddit conversations to predict response valuation conditioned on context, and uses this signal to fine-tune language models via reinforcement learning to improve performance on pragmatics-mediated tasks. Critically, we find that the highest performing reward model does not lead to better downstream model alignment: a reward model relying exclusively on conversational context was a worse predictor of Reddit karma but yielded substantially better downstream performance. We evaluate the effects of KARMA applied to a downstream model with and without direct exposure to the social media data. The resulting models show improved pragmatics-mediated behaviors with largely mitigated undesirable side effects. Factuality is consistently diminished by KARMA across all conditions, including when the downstream model has no direct exposure to Reddit data, suggesting that this tension is embedded in the reward signal itself rather than introduced by noisy training data. 1 Introduction Human language is inherently situated. The success of a response depends not only on its semantic correctness, but also on its pragmatic fit with the audience, conversational context, and social expectations of the interaction. Put simply, communication at the highest level requires an ability to “read the room”; the pragmatic ability to recognize when humor, empathy, formality, restraint, directness, or other nuance is most appropriate. Like most target LLM behaviors (instruction-following, helpfulness, safety, etc), contextually well-situated language is not consistently represented in human data. Therefore, we frame this problem as one of implicit alignment. While recent advances in large language model (LLM) alignment have substantially improved some target behaviors, pragmatics have been less explored. As a result, models may have a pragmatic gap: an ability to generate responses that are factually correct and helpful yet socially misaligned with a specific conversational setting. Online discussion platforms like Reddit provide a naturally occurring source of human preference data through voting systems, which we expect to be sensitive to pragmatic dimensions such as relevance, tone alignment, humor, and situational appropriateness. We introduce (contribution 1) KARMA (Karma-Aligned Reward Model Adaptation), a framework for improving pragmatics-mediated behaviors using social feedback derived from Reddit discussions. KARMA learns a reward model over conversational context and candidate responses using Reddit interaction signals and subsequently applies reinforcement learning through Proximal Policy Optimization (PPO) to adapt a language model toward these preferences. Pragmatic understanding is multi-faceted and complex. Rather than attempting to be exhaustive, we adopt two tasks to evaluate KARMA’s potential to impact tasks that rely on the ability to read the room . KARMA significantly improves these pragmatics-mediated tasks. We hypothesize (contribution 2) that direct exposure to Reddit conversational data during fine-tuning may increase negative side-effects when compared against indirect exposure through the learned reward signal alone. Direct exposure to the Reddit data distribution amplifies undesirable behaviors, whereas reward-mediated adaptation mitigates negative side-effects while preserving core pragmatic benefits. A notable exception is a consistent decrease in factuality. Finally, we investigate the relationship between reward model capability and downstream model impact (contribution 3). Providing metadata in addition to conversational context improves reward model metrics but erodes downstream alignment outcomes. 2 Related Work Recent approaches to aligning large language models (LLMs) have largely relied on instruction tuning and reinforcement learning from human feedback (RLHF), where models are optimized using reward functions derived from curated human preference data Ouyang et al. 2022 ; Schulman et al. 2017 . While effective, these methods primarily target correctness, safety, and instruction adherence, and do not explicitly model a key aspect of conversational intelligence: the ability to “read the room,” i.e., adapt responses to implicit social cues such as tone, context, and situational appropriateness. Social media platforms provide a scalable alternative source of feedback, where engagement signals such as replies, upvotes, and comment karma act as implicit indicators of response quality. Prior work has explored engagement prediction as either a regression or classification task, demonstrating that successful prediction depends not only on response content but also on surrounding conversational context Fang et al. 2016 ; Zayats and Ostendorf 2017 . More recent models incorporate conversational structure and long-range dependencies, further reinforcing that engagement is highly context-dependent and shaped by interaction dynamics within a thread. At the same time, directly training on social media data introduces significant challenges. Large-scale conversational corpora often contain noise, bias, and toxic content, and models exposed to such data can inherit or amplify these undesirable behaviors Gehman et al. 2020 ; Luong et al. 2024 . Additionally, engagement signals themselves may reflect platform-specific or demographic biases, complicating their use as alignment objectives. In contrast to prior work, we frame “reading the room” as an implicit but under-addressed component of alignment, distinct from both explicit preference optimization and generic engagement modeling. Rather than treating it as a surface-level prediction task, we aim to capture the underlying ability to infer what is appropriate given conversational context. We learn a reward model that predicts engagement from context alone, encouraging the model to infer relevance, tone, and situational fit directly from interaction signals. This reward is then used within a reinforcement learning framework to guide alignment. Through this design, we leverage large-scale social feedback while avoiding direct optimization on raw social media data. Scott and Roberts Scott and Roberts 2026 describe a similar intent but provide no execution or empirical evaluation. 3 Reward Modeling from Reddit To model conversational effectiveness within the karma framework, we construct a dataset from Reddit discussions using the publicly available Pushshift dataset Baumgartner et al. 2020 . We filter out explicit content, non-text entries, and non-English comments, and retain only posts with multiple substantive responses. This yields approximately 80k posts and 400k comments spanning 14k communities, providing broad topical and stylistic diversity. Each comment is paired with its parent post and a set of sibling responses to capture conversational context. This representation captures both hierarchical structure (via parent–child relationships) and local competition between responses (via sibling comments). To investigate whether reward models should leverage platform-specific contextual signals, we intentionally construct two reward modeling settings: a generalized setting that excludes Reddit metadata, and a conditioned setting that incorporates such metadata. In the generalized setting, conversations are transformed into a format resembling standard chat interactions. Post–comment pairs are linearized into instruction-style inputs, where the original post is treated as a user prompt and the comment as a candidate response. Contextual information from parent comments is included as preceding turns in a dialogue, yielding a multi-turn chat representation. The karma signal captures correlated pragmatic dimensions such as humor, relevance, tone, and situational appropriateness, and this richness makes it a candidate signal for learning broad pragmatic competence. Importantly, subreddit identity, timestamps, and author information are removed in this setting. While these features may improve prediction of engagement within Reddit, we hypothesize that they may enable heuristics which may obscure or obviate the pragmatic utility of the model. By excluding these signals and standardizing inputs into a chat-like format, the reward model is encouraged to learn features

Read the complete text