Coloring Between the Lines: Personalization in the Null Space of Planning Constraints
Tom Silver, Rajat Kumar Jenamani, Ziang Liu, Ben Dodson, Tapomayukh Bhattacharjee · 2025
Read Coloring Between the Lines: Personalization in the Null Space of Planning Constraints by Tom Silver, Rajat Kumar Jenamani, Ziang Liu, Ben Dodson, Tapomayukh Bhattacharjee (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-15503.txt.
Introduction
Tom Silver and four co-authors give robots a way to fit user tastes without breaking safety rules. Their method, named for coloring inside lines, starts with a plan solver that locks safe and able acts. It then learns taste based limits from live chats with the user.
By using slack left by the hard rules and tracking doubt, it learns fast with no resets. Tests in sim worlds, a web study, and a real food aid robot all favor it. It fit tastes better in fewer chats than rivals. The frame suits long term home robots.
Catalog record
| CREATOR | Tom Silver, Rajat Kumar Jenamani, Ziang Liu, Ben Dodson, Tapomayukh Bhattacharjee |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 32 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-15503
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Coloring Between the Lines: Personalization in the Null Space of Planning Constraints Tom Silver, Rajat Kumar Jenamani, Ziang Liu, Ben Dodson, Tapomayukh Bhattacharjee Abstract Generalist robots must personalize in-the-wild to meet the diverse needs and preferences of long-term users. How can we enable flexible personalization without sacrificing safety or competency? This paper proposes Coloring Between the Lines (CBTL), a method for personalization that exploits the null space of constraint satisfaction problems (CSPs) used in robot planning. CBTL begins with a CSP generator that ensures safe and competent behavior, then incrementally personalizes behavior by learning parameterized constraints from online interaction. By quantifying uncertainty and leveraging the compositionality of planning constraints, CBTL achieves sample-efficient adaptation without environment resets. We evaluate CBTL in (1) three diverse simulation environments; (2) a web-based user study; and (3) a real-robot assisted feeding system, finding that CBTL consistently achieves more effective personalization with fewer interactions than baselines. Our results demonstrate that CBTL provides a unified and practical approach for continual, flexible, active, and safe robot personalization. Website: https://emprise.cs.cornell.edu/cbtl/ Figure 1: Real robot demonstration of Coloring Between the Lines (CBTL) . The robot personalizes to the user over five meals (four in sim). During the fifth meal, CBTL plans to reposition the plate to prevent violating the user’s learned occlusion preferences. It then chooses ketchup based on learned dipping preferences. Finally, CBTL generalizes to avoid occlusions with a new drink. Keywords: Personalization, planning, online learning 1 Introduction When a generalist robot first arrives at a new home, restaurant, or hospital, it should already know a great deal—how to move, how to manipulate objects, and how to interpret human-given commands, among many other general competencies. But the robot will not yet know about the aspects of its environment that make it unique. It will not yet know that a user loves tacos but hates cilantro; that they have limited range of motion [ 1 ] ; or that they sometimes like to watch TV while eating dinner. The robot should learn these nuances over time and personalize its behavior accordingly [ 2 , 3 , 4 , 5 ] . Continual personalization of a generalist robot has been a longstanding goal in robotics, but it has remained elusive because of the inherent tension between safety and flexibility . An over-constrained robot cannot adapt; an under-constrained robot can be dangerous. Previous work on personalization has largely deferred this challenge by focusing on task-specific forms of personalization [ 6 , 7 , 8 , 9 , 10 , 11 , 12 , 13 ] or by working in settings without critical safety requirements [ 14 , 15 , 16 ] . In this work, we propose a general method for resolving these competing concerns. We start by leveraging planning with compositional constraints [ 17 , 18 , 19 , 20 , 21 ] as a framework to enable a broad range of behavior while adhering to explicit programmer-specified restrictions [ 22 ] . For example, consider the mealtime assistance system shown in Figure 1 . Given a request for a bite of food, the robot uses its given non-personalized capabilities to generate a constraint satisfaction problem (CSP) with variables \mathrm{platePose}\in\text{SE(2)} , \mathrm{utensilAbovePlatePose}\in\text{SE(3)} , \mathrm{robotAbovePlateConf}\in\mathbb{R}^{7} , \mathrm{dip}\in\{\text{ketchup},\text{BBQ sauce},\text{none}\} , and \mathrm{readySignal}\in\{\text{mouth open},\text{button},\text{auto}\} , among others, which are mutually constrained, e.g., to enforce kinematic feasibility and safe collision-free motion. CSPs generally have multiple solutions, especially when continuous variables are involved. We refer to the set of solutions to a CSP as a null space by analogy to the concept in kinematics [ 23 , 24 , 25 ] which refers to the fact that a redundant-DoF manipulator can move without affecting the end-effector’s task, e.g., repositioning the elbow while keeping the hand fixed. Our key insight is that this CSP null space provides an opportunity for personalization. Rather than picking arbitrary solutions, the robot should “color between the lines” of the constraints to select solutions in the null space that are personalized to the user. In particular, we propose that robot engineers should not only implement one-size-fits-all constraints to ensure safety and general competency—they should also implement constraints with learnable parameters that can be rapidly personalized in-the-wild. For example, previous work in assisted feeding has shown that users vary in their bite preferences (e.g., dipping sauce) [ 26 ] , their interaction modes (e.g., ready signals) [ 13 ] , and their sensitivities to visual occlusions (e.g., by the robot arm) [ 27 ] . Our approach unifies these and other aspects of personalization that previous work has treated task-dependently. For instance, we generate a \mathrm{dipPreferred}(\mathrm{dip},\theta_{1}) constraint where \theta_{1}\in\mathrm{Text} is a natural language summary of the user’s dip preferences, and the constraint itself is implemented with a large language model (LLM) [ 28 ] ; a similar \mathrm{signalPreferred}(\mathrm{readySignal},\theta_{2}) constraint where \theta_{2}\in\mathrm{Text} ; and an \mathrm{occlusionFree}(\mathrm{conf},\theta_{3}) constraint, where \theta_{3}\in[0,1] encodes the user’s occlusion sensitivity and where the constraint itself uses a model from prior work [ 27 ] . As the robot gathers experience, it adapts the parameters \theta_{1},\theta_{2},\theta_{3} and continually personalizes its behavior. To maintain flexibility, it is essential that the robot learns full personalized constraints and does not simply memorize CSP solutions. For example, plate and drink positions that satisfied the user in the past may become impossible if the table changes or if obstacles are introduced. We therefore propose to actively learn personalized constraint parameters using an entropy-based method [ 29 , 30 ] . This enables the robot to rapidly learn personalized constraints that generalize across scenarios. We call this overall approach—planning with compositional constraints, learning personalized constraint parameters, and actively gathering data— Coloring Between the Lines (CBTL) . In experiments, we evaluate the extent to which CBTL achieves efficient and effective personalization. We first compare CBTL to four baselines in three simulation environments and find that CBTL consistently personalizes better and faster. We then conduct a web-based user study with 60 participants and find that they significantly prefer CBTL’s choices over a non-personalized baseline. Finally, we validate the method with the real robot in Figure 1 and show that CBTL generalizes: by reusing the personalized occlusion constraint learned from feeding, CBTL anticipates new occlusions and repositions the drink accordingly. Furthermore, to avoid collisions and maintain reachability, CBTL repositions the plate to make room for the drink. Together, these results demonstrate that CBTL provides a unified and practical approach for continual, flexible, active, and safe robot personalization. 2 Related Work Personalization in Robot Learning. In this work, we develop a method for robot personalization that is continual , flexible , active , and safe . Previous work considers one-time personalization in the context of single tasks [ 6 , 7 , 8 , 9 , 10 , 11 , 27 , 31 ] . For example, given natural-language user preferences such as “feed me spaghetti before meatballs” [ 26 ] or “sort the laundry by lights and darks” [ 12 ] , robot behavior can be adapted on-the-fly to take these preferences into account. Some of these works explicitly consider safety [ 20 , 32 , 33 ] . The works of Thumm et al. [20] and Wang et al. [34] are especially relevant; like us, they consider personalization within