IMPASTO: Integrating Model-Based Planning with Learned Dynamics Models for Robotic Oil Painting Reproduction
Yingke Wang 1 , Hao Li 1 , Yifeng Zhu 2 , Hong-Xing Yu 1 , Ken Goldberg 3 , Li Fei-Fei 1 , Jiajun Wu 1 , Yunzhu Li 4 , and Ruohan Zhang 1 · 2026
Read IMPASTO: Integrating Model-Based Planning with Learned Dynamics Models for Robotic Oil Painting Reproduction by Yingke Wang 1 , Hao Li 1 , Yifeng Zhu 2 , Hong-Xing Yu 1 , Ken Goldberg 3 , Li Fei-Fei 1 , Jiajun Wu 1 , Yunzhu Li 4 , and Ruohan Zhang 1 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-29315.txt.
Introduction
Wang and co-authors present IMPASTO, a robot system that reproduces oil paintings. Reproduction needs force control of soft brushes and multi-step stroke planning. No step-by-step human demos or faithful simulators are assumed. Learned models predict how each stroke changes the canvas. A receding-horizon planner then picks trajectories and forces. A force-sensitive controller runs the strokes on a seven-axis arm. The system learns only from robot self-play. It beat baselines on single strokes and full artworks.
Catalog record
| CREATOR | Yingke Wang 1 , Hao Li 1 , Yifeng Zhu 2 , Hong-Xing Yu 1 , Ken Goldberg 3 , Li Fei-Fei 1 , Jiajun Wu 1 , Yunzhu Li 4 , and Ruohan Zhang 1 |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 41027 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.29315
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
IMPASTO: Integrating Model-Based Planning with Learned Dynamics Models for Robotic Oil Painting Reproduction Yingke Wang 1 , Hao Li 1 , Yifeng Zhu 2 , Hong-Xing Yu 1 , Ken Goldberg 3 , Li Fei-Fei 1 , Jiajun Wu 1 , Yunzhu Li 4 , and Ruohan Zhang 1 Abstract Robotic reproduction of oil paintings using soft brushes and pigments requires force-sensitive control of deformable tools, prediction of brushstroke effects, and multi-step stroke planning, often without human step-by-step demonstrations or faithful simulators. Given only a sequence of target oil painting images, can a robot infer and execute the stroke trajectories, forces, and colors needed to reproduce it? We present IMPASTO , a robotic oil-painting system that integrates learned pixel dynamics models with model-based planning. The dynamics models predict canvas updates from image observations and parameterized stroke actions; a receding-horizon model predictive control optimizer then plans trajectories and forces, while a force-sensitive controller executes strokes on a 7-DoF robot arm. IMPASTO integrates low-level force control, learned dynamics models, and high-level closed-loop planning, learns solely from robot self-play, and approximates human artists’ single-stroke datasets and multi-stroke artworks, outperforming baselines in reproduction accuracy. Project website: https://impasto-robopainting.github.io/ . I Introduction Robot painting reproduction—enabling machines to accurately replicate physical artworks with brushes and pigments—has a rich history and presents unique technical challenges [ 35 , 20 ] . In modern times, these robots often paint images on canvas by executing sequences of brushstrokes and adjusting based on sensory feedback [ 35 , 27 , 36 ] . Unlike digital image generation, a painting robot must master complex low-level control of deformable tools and fluids, force-sensitive manipulation of a brush against a surface, visual perception of the evolving artwork, and high-level planning of stroke sequences. These requirements make robot painting reproduction a significant challenge: the robot must achieve the same nuanced, variable brushstrokes as a human artist based on real-world physics. We ask: Given only a sequence of static images of an oil painting by an expert artist, can a robot infer corresponding control actions, such as trajectory, orientation, and applied force, to approximately reproduce the painting? This setting is similar to art training exercises—can our robot reach the level of beginning art students who are able to replicate an oil painting? This is a challenging setting for robotics, as it simultaneously 1) requires precise low-level control to replicate strokes with soft brushes and wet paints; 2) requires high-level planning to compose multiple strokes; 3) assumes no access to any form of action demonstration data, either through teleoperation or tracking the brushes. One approach to tackling this challenge is learning a 2D dynamics model for the oil painting domain. Given the current visual state of the canvas, and a parameterized action, the robot should predict the consequence of its brush action, i.e., the resulting state of the canvas. We conjecture that such a model can be trained through self-play. Once trained, the robot can integrate this model with model predictive control (MPC) applied within local stroke windows to infer actions to replicate each brushstroke, plan multiple strokes to approximate a complex painting. Although learning neural dynamics models and combining them with MPC have become increasingly popular in different robotic tasks [ 11 , 41 , 46 , 49 , 18 ] , we believe this is the first attempt to leverage them for robot painting. We present IMPASTO ( I ntegrating M odel-Based P lanning with Le A rned Dynamic S Models for Robo T ic O il Painting Reproduction), a robotic system that combines learned dynamics models with force-sensitive control and model-based planning to approximately reproduce human oil paintings. Our contributions include: • Learned pixel dynamics model: We develop a neural network dynamics model of brushstrokes, trained on a dataset of robot self-play data. This model learns the complex relationship between the robot’s action parameters (end-effector trajectory, applied force) and the resulting stroke outcomes to predict stroke shape and appearance. • Model predictive trajectory and force planning: We integrate the learned model into a model-based planning framework for single or consecutive stroke planning. Given a desired appearance of stroke(s), the MPC optimizer computes precise trajectories and forces to reproduce each brushstroke. • An integrated robot painting system: We implement a complete system on a 7-DoF robotic arm with a force-torque sensor. The system autonomously handles dipping the brush in paint, cleaning the brush between colors, and painting the strokes on a canvas. We evaluate IMPASTO on both individual strokes and full paintings drawn by expert human artists. Our results suggest that the learned model plus model-based planning approach yields strokes that closely match the appearances of the human brushstrokes. We demonstrate that the learned dynamics model can better replicate individual and overlaid strokes than the baseline methods. We also show closed-loop painting experiments in which the robot plans and paints a multi-stroke artwork. II Related Work II-A Modern Robot Painting The idea of automating art dates back centuries [ 35 ] . Harold Cohen’s AARON in 1986 is widely considered the first modern robot painter with an open-loop programmed approach [ 5 ] . Recently, painting robots have employed closed-loop control. e-David is an industrial robotic arm equipped with a camera that paints with real brushes and continuously adjusts its strategy based on visual feedback [ 27 , 7 ] . Another state-of-the-art system is the FRIDA series [ 36 , 38 , 4 ] , a collaborative painting robot that introduces a differentiable simulation for brushstrokes and a real2sim2real planning loop. FRIDA simulates how each stroke will appear and, during execution, periodically captures an image of the canvas to re-plan the remaining strokes, though FRIDA’s primary goal was not exact reproduction of existing artworks. These modern robotic painting systems emphasize sophisticated stroke planning algorithms, from high-level image segmentation and stroke sequencing [ 27 ] to optimization of stroke parameters in simulation [ 36 ] , all aimed at achieving human-like painting results under robotic control. Recent advances in robotic painting leverage machine learning methods. Researchers have applied reinforcement learning (RL) to optimize paint coverage [ 3 , 44 ] and stroke planning [ 37 , 22 , 19 ] , used genetic algorithms for stroke planning [ 2 ] , used imitation learning (IL) to leverage human demonstrations [ 31 , 13 ] , and employed deep generative models to synthesize and stylize brushstrokes [ 50 , 28 , 12 , 14 ] . Together, these approaches push robotic art beyond hand-designed programming toward more autonomous and creative behaviors. It is worth noticing that our problem setting is among the most challenging ones: We aim to closely approximate human experts’ brushstrokes with soft brushes and wet pigments. Unlike in the RL settings, we do not rely on a painting simulator; unlike in the IL setting, we do not have access to human demonstrations. II-B Planning with Learned Neural Dynamics Models Deep neural networks trained on interaction data have become a standard way to learn dynamics models for manipulation, with broad empirical success [ 41 , 46 ] . Such models can be trained directly in pixel space [ 10 , 9 , 8 , 51 , 43 ] or in compact latent spaces that