Position: Good Embodied Reward Models Need Bad Behavior Data
Ran “Thomas” Tian, Yilin Wu, Andrea Bajcsy · 2026
Read Position: Good Embodied Reward Models Need Bad Behavior Data by Ran “Thomas” Tian, Yilin Wu, Andrea Bajcsy (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-01036.txt.
Introduction
Tian, Wu, and Bajcsy argue that robot reward models need bad robot data in this 2026 position paper. Reward models steer how embodied AI learns. Most are trained on success: clean, safe, finished tasks. The failures that teach what not to do are costly to gather. Datasets often filter them out or keep them locked away.
The trio audit three top embodied reward models. The models over reward acts that human graders would punish, they find. Unsafe contact, sloppy work, and shortcut tricks all score too well. The cause, they argue, is the missing diet of negative cases. Modest doses of real bad behavior data help. Alignment with human taste improves and costly false positives fall. Their call to the field is blunt. Curate and release your failed robot runs. Build engines that synthesize bad behavior, and grade reward models on fine splits of good and bad.
Catalog record
| CREATOR | Ran “Thomas” Tian, Yilin Wu, Andrea Bajcsy |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 43 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.01036
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Position: Good Embodied Reward Models Need Bad Behavior Data Ran “Thomas” Tian, Yilin Wu, Andrea Bajcsy Abstract This position paper argues that to obtain reliable embodied reward models, the community must invest in “bad” robot data: failed, suboptimal, error-prone, and even hazardous behaviors. While reward models are central to any foundation model’s lifecycle, today’s embodied reward models are trained primarily on successful behaviors. We analyze three state-of-the-art embodied reward models and find that they systematically over-reward behaviors that real human evaluators would penalize, including unsafe interactions, poor execution, and shortcut strategies that only superficially satisfy tasks. We attribute these failures to a key data gap: the scarcity of negative embodied data which is costly to collect and often filtered out or withheld in existing robotics datasets. Furthermore, we show that even modest exposure to real bad behavior data can improve alignment with human preferences and reduce costly false positives. We therefore call on the embodied AI community to curate and release their bad robot data, build synthetic bad data generation engines, develop more decentralized physical evaluation systems, and design benchmarks for fine-grained embodied reward model evaluations. 1 Introduction Reward models are central to the lifecycle of any foundation model, from reinforcement learning-based post-training ( 31 ) , to test-time compute ( 35 ) , to large-scale evaluation ( 40 ) . For example, in non-embodied domains such as large language models (LLMs) and vision-language models (VLMs), learned rewards ( 8 ; 49 ; 31 ) have been key for hard-to-verify tasks like open-ended generation, dialogue, summarization, and perceptual reasoning ( 44 ; 36 ; 23 ) . In the past two years, embodied domains such as autonomous driving and household robotics have started developing foundation models with architectures and training paradigms that mirror LLMs and VLMs; a prime example are vision–language–action models (VLAs) ( 7 ; 14 ; 41 ) . Similar to the non-embodied tasks described above, in robotics the quality of a behavior is often subjective, hard-to-specify, or defined by fine-grained physical consequences (e.g., tearing part of a napkin while folding it). Thus, there is a growing demand for general-purpose reward models that can evaluate a physical agent’s behavior directly from visual observations and a textual task specification ( 37 ; 21 ) , reducing the need for costly human evaluation ( 5 ) while providing signals for post-training ( 2 ) and test-time compute ( 42 ; 20 ) . However, today’s embodied reward models fall short of evaluating the nuances of physical behavior in the way humans do. Using real robot videos and human evaluator labels from the RoboArena benchmark ( 5 ) , we experimentally show that three state-of-the-art (SOTA) embodied reward models systematically over-reward behaviors that human evaluators would reject, including low-quality execution, unsafe interactions, and “shortcut” strategies that mimic completion while violating human preferences. The gap between the reward models and human evaluators also widens as the physical task complexity increases from simple pick-and-place to complex bi-manual tool use tasks. We argue that today’s embodied reward models are overly-optimistic because we lack “bad” embodied behavior data necessary for calibration. In non-embodied domains, LLMs and VLMs benefit from exceptionally diverse and scalable reward-training ecosystems in which “bad” data (from toxic language to logical fallacies) is abundant and cheap to generate, enabling a sufficiently large coverage of outcomes and their (human) evaluations. However, embodied data collection is bounded by wall-clock time and hardware safety, making failures, and dangerous behaviors undesirable to collect at scale. Even when failures do occur, they are rarely preserved in the public data ecosystem, as the dominant imitation-based learning paradigm prioritizes expert demonstrations and intentionally filters out negative data ( 32 ) . Thus, current embodied AI datasets used for reward model training are biased toward successful behaviors, resulting in models that are systematically over-optimistic. Our position is that if we want to have good embodied reward models, we need to invest in more “bad” robot data. We need a deliberate shift away from expert-only datasets and toward the intentional collection and release of dangerous, failed, noisy, and error-prone robot data to obtain reliable embodied reward models. We demonstrate that even prompting SOTA embodied reward models with real robot failure videos (rather than just textual descriptions of “bad” behavior) as in-context examples results in better preservation of human preferences and fewer high-impact false positives. We call on the robotics and embodied AI community to release curated, large-scale robot failure datasets, develop new methods for the synthetic generation of “bad” robot data, develop more decentralized, physical evaluation systems, and develop benchmarks for evaluating general-purpose embodied reward models. 2 Setup: Embodied Reward Models An embodied reward model assigns (a sequence of) scalar scores to robot behavior conditioned on some task context. Formally, let the task context c\in\mathcal{V} be a free-form language instruction. Let a T -length robot behavior be represented by a sequence of high-dimensional observations, \tau:=o_{1:T} , sensed by the robot’s intrinsic and extrinsic sensors (also referred to as a rollout ). For example, in tabletop robotic manipulation, each o_{t} typically consists of RGB images collected by a third-person and wrist-mounted camera. We define an embodied reward model as a parameterized map from observations and context to rewards: R_{\theta}(o_{1:T};c)\rightarrow r_{1:T}, where r_{t}\in\mathbb{R},\forall t\in[0,T] is a real value measure of how well the behavior aligns with the user’s original task instruction c . For example, this can include task progress, execution quality, and safety. The learnable parameters of the model are denoted by \theta . Let an embodied foundation model be defined as a parameterized map from an observation and task context to a distribution over actions, \pi_{\phi}(\mathbf{a}\mid o,c) , where \mathbf{a}:=a_{t:t+H} is an H -step future sequence of actions starting from any timestep t , and \phi are the parameters. After an embodied foundation model is pre-trained via imitation on a large dataset of observation-action labels ( 32 ; 17 ) , the reward function plays a role during post-training and during deployment-time. During post-training , the reward function enables reinforcement-learning (RL) to improve the precision of the foundation model or to learn new skills ( 38 ; 47 ; 11 ; 46 ; 37 ; 45 ; 2 ; 43 ) . Mathematically, given an initial observation o_{0} , this process entails solving the following problem: \phi^{*}=\arg\max_{\phi}\mathbb{E}_{o_{1:T}\sim\mathbb{P}(\cdot\mid o_{0},\pi_{\phi})}\Big[R_{\theta}(o_{1:T};c)\Big], where o_{1:T}\sim\mathbb{P}(\cdot\mid o_{0},\pi_{\phi}) defines the distribution over future observations (i.e., rollouts) induced by the embodied foundation model \pi_{\phi} under the current model parameters. The reward model scores how aligned the current rollouts are with the task context, and thus directly determines how the policy \pi_{\phi} is improved. During test-time compute , the embodied foundation model is frozen and samples \mathbf{a}\sim\pi_{\phi}(\cdot\mid o;c) are taken from the model; let a set of K action samples be denoted by \{\mathbf{a}^{i}\}^{K}_{i=1} . The reward model scores each action sample and executes only the actions that maximally align with the task ( 10 ; 42 ; 9 ) : \mathbf{a}^{*}=\arg\max_{\{\mathbf{a}^{i}\}^{K}_{i=1}}\mathbb{E}_{o_{t:t+H}\sim\mathbb{P}(\cdot\mid o_{t},\mathbf{a}^{i})}\Big[R_{\theta}(o_{t:t+H},c)\Big], In both regimes, the reward model’s calibration establishes an upper-bound for how much the foundation model can be improved, or how reliably it can be steered at deployment-time. As we show in the next section, despite current embodied reward models being able to evaluate temporal progress in