Loomis Painter: Reconstructing the Painting Process

Markus Pobitzer, Chang Liu, Chenyi Zhuang, Teng Long, Bin Ren, Nicu Sebe, [email protected], Corresponding author · 2025

Read Loomis Painter: Reconstructing the Painting Process by Markus Pobitzer, Chang Liu, Chenyi Zhuang, Teng Long, Bin Ren, Nicu Sebe, [email protected], Corresponding author (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-17344.txt.

Introduction

Markus Pobitzer and five co-authors present Loomis Painter, a system that reconstructs the process of making a painting, step by step, from a finished image. Painting tutorials teach technique, but static videos lack interactivity. Generative video models could fill the gap, yet training an image-to-video model on forward painting sequences fights its pretrained attention patterns, because the model leans too hard on the finished first frame. The authors' key move is to train on reversed sequences, which preserves the reference behavior the model already has.

A media-transfer strategy further separates the input's appearance from the target process, so a process learned in one medium can be rendered in another. Trained on curated real painting processes, Loomis Painter beat baselines on standard similarity metrics and was preferred by both expert and non-expert viewers in user studies.

Catalog record

CREATORMarkus Pobitzer, Chang Liu, Chenyi Zhuang, Teng Long, Bin Ren, Nicu Sebe, [email protected], Corresponding author
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-17344

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Loomis Painter: Reconstructing the Painting Process Markus Pobitzer, Chang Liu, Chenyi Zhuang, Teng Long, Bin Ren, Nicu Sebe, [email protected], Corresponding author Abstract Step-by-step painting tutorials are vital for learning artistic techniques, but existing video resources ( e.g. , YouTube) lack interactivity and personalization. While recent generative models have advanced artistic image synthesis, they struggle to generalize across media and often show temporal or structural inconsistencies, hindering faithful reproduction of human creative workflows. To address this, we propose a unified framework for multi-media painting process generation with a semantics-driven style control mechanism that embeds multiple media into a diffusion model’s conditional space and uses cross-medium style augmentation. This enables consistent texture evolution and process transfer across styles. A reverse-painting training strategy further ensures smooth, human-aligned generation. We also build a large-scale dataset of real painting processes and evaluate cross-media consistency, temporal coherence, and final-image fidelity, achieving strong results on LPIPS, DINO, and CLIP metrics. Finally, our Perceptual Distance Profile (PDP) curve quantitatively models the creative sequence, i.e., composition, color blocking, and detail refinement, mirroring human artistic progression. Code available at https://github.com/Markus-Pobitzer/wlp . Figure 1 : Loomis Painter: Our method reconstructs the painting process of any input image, either faithfully, as shown in (b), or in different art media, as in (a) and (c). The title of our work is inspired by the Loomis portrait method, which we also enable. Images with green borders are input reference images; all others are generated by our method. 1 Introduction Sketching and painting are creative processes that are hard to master. Many tutorials exist as step-by-step guides in the form of books and videos. However, books cannot show the complete dynamic process, and existing instructional videos are inherently passive, lacking interactivity and personalized guidance. While recent generative models have demonstrated impressive capabilities in artistic image synthesis 29 ; 25 ; 2 , they remain limited in process-level modeling: generated painting sequences often exhibit temporal discontinuities, structural jumps, and poor generalization across artistic media, making it difficult to faithfully reconstruct the coherent, sequential nature of human painting processes. For artists, it is not only important to replicate a scene but also to bring what they perceive to the canvas with their chosen tools. This includes media-specific steps such as layering color with oil 26 and faithfully reconstructing proportions. A prominent example is the Loomis Method 17 , developed by Andrew Loomis, which demonstrates a structural approach to drawing a head with correct proportions. This method can be applied to an existing reference photo, bridging the gap between a static representation and an artistic process. To address these challenges, we propose a unified framework for multi-media painting process generation, capable of modeling the evolution of artistic content across diverse traditional painting media. We introduce a control mechanism that embeds multiple artistic media into the conditional space of a diffusion model, enabling the model to capture medium-dependent textural evolution and procedural patterns within a shared latent space. Combined with cross-media style augmentation, this framework supports controllable transfer of painting procedures across artistic media while maintaining consistency and coherence in the generated sequences. Another contribution of our work is a reverse-painting training strategy, which learns to regress from the completed artwork back to a blank canvas. This formulation allows the model to learn how to uncover the painting from one generated frame to the next, while the first frame is grounded by the input image. By reversing the direction of the training trajectory and building on top of a video generation model, structural discontinuities and temporal jumps commonly observed in conventional forward-sequence prediction models do not show up. To support high-fidelity learning of real artistic workflows, we construct a large-scale dataset of real drawing and painting processes spanning multiple artistic media. Importantly, we introduce an automatic occlusion-removal procedure that eliminates hand occlusions and other visual clutter, allowing the model to learn accurate stroke-level transformations throughout the creative process. We conduct extensive experiments evaluating cross-media consistency, temporal coherence, and final-image fidelity. Our method achieves strong performance on LPIPS 37 , DINO 18 , and CLIP 22 metrics. Beyond conventional evaluations, we propose a new measure, Perceptual Distance Profile (PDP), that quantitatively models structural progression over time. PDP provides a principled way to characterize the trajectory of perceptual changes across frames and reveals that our model closely follows the human creative pattern of composition, color blocking, and detail refinement. To sum up, our contributions are: • We introduce Loomis Painter, a painting video generation model empowered by our dataset. We enhance the model to enable inference-time translation of diverse scenes into a variety of artistic media, thereby bridging the gap between input and artistic expression. • We show that the proposed reverse-painting strategy is crucial to accurately reconstructing the input painting. • We collect and curate a high-quality painting process video dataset that addresses the occlusion problem and covers diverse artistic media and styles beyond existing datasets. • We extensively evaluate our method with state-of-the-art methods, introducing a novel video-level evaluation for the realism and quality of the generated painting process. 2 Related Work Neural Painting. Neural painting aims to reconstruct visual imagery through a sequence of brushstrokes inferred by a neural network. Early efforts, such as Paint Transformer 16 , employed feed-forward architectures based on Transformers 30 ; 4 to progressively generate stroke parameters. Subsequent works such as 21 also include human interaction in the generation process, similar to 3 which lets the user compose an artwork with the help of the neural painter and leverages diversity with a Diffusion Process 24 ; 8 . Despite their progress, these approaches largely model painting as a parametric rendering problem rather than an authentic artistic process. Their synthesized stroke sequences often diverge from how human artists construct compositions in real tutorials. In contrast, our method directly learns from real-world painting sessions, capturing genuine artistic decision-making and temporal workflows. By operating in pixel space through a video diffusion framework, it overcomes the limitations of parameterized strokes and produces temporally coherent, photorealistic painting progressions akin to those created by skilled artists. Figure 2 : Overview of our painting process generation method. The curated video is first reversed to better align with the underlying video generation model. We LoRA-tune WAN 2.1 31 , a video generation model conditioned on an input image and a prompt. In our case, the input image corresponds to a painting, and the model learns to reconstruct the steps to paint it, starting from a finished painting to a blank canvas. In (b), the media transfer model is shown, which enables the video generation model to render any input image as an acrylic, oil, or pencil painting based on the text input. To achieve this, we generate variations of the reference image using image editing models and train the video generation model to reconstruct the original painting process. Pixel-Based Generation. Pixel-based methods synthesize painting sequences directly at the pixel level, conditioned on a reference image. Early convolutional methods 38 attempted to reconstruct painting workflows, while a more recent work Inverse Painting 2 advanced this painting workflow reconstruction with an autoregressive three-stage pipeline that compares intermediate frames with the reference, masks the next operation area, and updates pixels through diffusion. More recent

Read the complete text