AnimatePainter: A Self-Supervised Rendering Framework for Reconstructing Painting Process

Junjie Hu, Shuyong Gao, Qianyu Guo, Yan Wang, Qishan Wang, Yu’ang Feng, Wenqiang Zhang · 2025

Read AnimatePainter: A Self-Supervised Rendering Framework for Reconstructing Painting Process by Junjie Hu, Shuyong Gao, Qianyu Guo, Yan Wang, Qishan Wang, Yu’ang Feng, Wenqiang Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-17029.txt.

Introduction

Junjie Hu and six co-authors teach a video model to repaint a picture the way a human painter would, in a 2025 paper. People can look at a finished image and imagine the stroke order behind it. Earlier process generators worked only for narrow image types and needed costly hand-labeled data. The new framework, AnimatePainter, is self-supervised: it builds its own training data using depth estimates and stroke rendering.

The model treats painting as refinement plus layering, working backward by removing strokes from a reference image. Depth fusion layers help the video model copy that human-like order. A system that recovers process from any image could teach art students, verify authenticity, and drive more natural drawing robots.

Catalog record

CREATORJunjie Hu, Shuyong Gao, Qianyu Guo, Yan Wang, Qishan Wang, Yu’ang Feng, Wenqiang Zhang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE32 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-17029

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

AnimatePainter: A Self-Supervised Rendering Framework for Reconstructing Painting Process Junjie Hu, Shuyong Gao, Qianyu Guo, Yan Wang, Qishan Wang, Yu’ang Feng, Wenqiang Zhang Abstract Humans can intuitively decompose an image into a sequence of strokes to create a painting, yet existing methods for generating drawing processes are limited to specific data types and often rely on expensive human-annotated datasets. We propose a novel self-supervised framework for generating drawing processes from any type of image, treating the task as a video generation problem. Our approach reverses the drawing process by progressively removing strokes from a reference image, simulating a human-like creation sequence. Crucially, our method does not require costly datasets of real human drawing processes; instead, we leverage depth estimation and stroke rendering to construct a self-supervised dataset. We model human drawings as ”refinement” and ”layering” processes and introduce depth fusion layers to enable video generation models to learn and replicate human drawing behavior. Extensive experiments validate the effectiveness of our approach, demonstrating its ability to generate realistic drawings without the need for real drawing process data. 1 Introduction Painting is a manifestation of human artistry, which can also be an AI ability. We already have Diffusion [ 28 ] and GAN [ 10 ] to generate impressive images that rival or even surpass humans. However, a complete image is only enough for us to appreciate, and cannot provide us with painting guidance. Painters need to deconstruct a painting into process fragments to learn and improve their painting skills, while painting robots [ 31 , 32 , 38 ] need intuitive fragments to draw vivid paintings. While limited to certain types of painting, some methods [ 37 , 3 , 44 ] can accurately reconstruct the painting process of a painting. Most of them train on real drawing data which is expensive and difficult to obtain. Specific and scarce training data limits the types of images they can generate. To extend the task of image process generation to any type of image, we present AnimatePainter , a self-supervised rendering framework for reconstructing painting process. AnimatePainter consists of an self-supervised data generation method based on cognitive priors and a end-to-end process painting generator. It only requires easily accessible text-image pairs data to operate efficiently, without requiring actual painting data. Building upon a stroke-based renderer, AnimatePainter first reconstructs the painting process through cognitive painting principles that emulate human artistic decision-making, as shown on Fig. 1 . To address the domain gap between synthesized and authentic painting sequences, we subsequently introduce the DF-Encoder module, which injects geometric awareness through depth-map guided painting scheduling. A more detailed explanation of these two parts will be systematically elaborated in subsequent sections. Figure 1 : Our self-supervised method does not require expensive real painting process data, can quickly synthesize a large amount of painting data, and supports any SBR backbone. In particular, we define this task as a self-supervised image erasure problem that is not limited to a specific topic. Given an arbitrary image, we reconstruct the image by reverse erasing its content from complete image to blank canvas. Considering that human artists first conceptualize the overall framework and distribution of the target image when painting, and then confirm the priority order of each semantic part. Generally speaking, the distant background and the subjects of foreground are independent of each other, and the farther parts should be painted first. Inspired by previous manual rule-based works [ 5 , 9 , 21 , 18 ] and the painting skills of human artists, we provide some cognitive priors such as deep-layered-painting, contours-to-details, attention-area-limitation. Based on the above observations, we propose an end-to-end drawing process generation method that does not require test-time fine-tuning. Our method consists of two parts. In the first part, we use a self-supervised data generation method to generate a unified video dataset that includes the drawing process of each image. Only text-image pairs are required in this process, rather than actual drawing process videos. Our method is compatible with any existing strokes-based render methods, such as [ 21 , 31 ] . However, the real-simulate gap is inevitable. The discrepancy between the generated data distribution and the real data distribution prevents us from simply using the generated data directly for training, which can lead to overfitting of the model on ”virtual data”. To alleviate this issue, we introduce DF-Encoder to realize hierarchical painting control using a depth map generated by depth estimation. Like human artists typically divide the scene into foreground and background, we also utilizing the depth map to segment the entire scene into multiple painting sections. In the second part, a renderer based on a video generation diffusion model is proposed, which utilizes the depth hierarchical map and reference images to generate the canvas content for each frame. Different from previous approaches like Inverse Painting [ 3 ] , we do not explicitly determine the painting area for each step, but instead use depth maps to guide the painting content in layers, as shown in Fig. 3 . We summarize our main contributions as follows: • We frame the process image generation task as a video generation problem, where intermediate step frames are generated between the first and last frames. To the best of our knowledge, AnimatePainter is the first to incorporate a video generation model into process image generation. • To overcome the challenge of obtaining real data, we propose a novel self-supervised data generation method that can efficiently produce large volumes of drawing process videos without the need for additional real drawing data. Our method is compatible with any stroke-based rendering technique and generates training data with a similar style. • We introduce DF-Encoder to minimize the distribution gap between generated and real data, facilitating process image generation with a hierarchical structure. Figure 2 : Samples of dataset . Some key frames of the generated data. The first two rows are based on Frida [ 31 ] (ink painting and colorful painting); the third row is based on PaintTransformer [ 21 ] . 2 Relation 2.1 Strokes-based Render Strokes-based Rendering, which employs line strokes to render realistic images, is widely used in robotic drawing [ 31 , 32 ] , artistic creation [ 21 , 3 ] , image stylization, and other fields. The painting process of rendering an image can be achieved by placing brush strokes. However, most current brush stroke-rendering methods [ 31 , 32 , 24 , 22 , 11 , 8 ] focus on the modeling and image stylization of individual brush strokes, with only a small amount of work [ 22 , 21 ] focusing on the generation of the drawing process. In recent work, PaintTransformer [ 21 ] predicts strokes sequences as tokens and employs a self-supervised random stroke generation method, enabling the generation of stunning images in the oil painting genre. To generate content step by step, some methods utilize reinforcement learning (RL) to constrain the content of strokes. Timecraft [ 44 ] proposes a probabilistic model to predict changes between two adjacent frames, achieving the synthesis of long videos; Inverse Painting [ 22 ] employs a large language model to infer the next painting area, thereby generating a mask to guide the content generation at

Read the complete text