Birth of a Painting: Differentiable Brushstroke Reconstruction

Ying Jiang 1∗, Jiayin Lu 1∗, Yunuo Chen 1∗, Yumeng He 1,2, Kui Wu 3, Yin Yang 4, Chenfanfu Jiang 1 · 2025

Read Birth of a Painting: Differentiable Brushstroke Reconstruction by Ying Jiang 1∗, Jiayin Lu 1∗, Yunuo Chen 1∗, Yumeng He 1,2, Kui Wu 3, Yin Yang 4, Chenfanfu Jiang 1 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-13191.txt.

Introduction

Ying Jiang and six co-authors reconstruct not just a painting but the strokes that made it. Generative models can synthesize a finished image or imitate process in patches, but they rarely recover explicit stroke structure with smooth, lifelike shading. The authors' system optimizes single- and dual-color Bézier strokes through a parallel differentiable paint renderer. A style module synthesizes geometry-conditioned textures across painting styles, and a smudge operator blends colors the way a thumb or brush would.

A coarse-to-fine strategy jointly optimizes stroke geometry, color, and texture under geometric and semantic guidance. On oil, watercolor, ink, and digital paintings the reconstructions look realistic and expressive, with smooth tonal transitions and rich stylization. A faithful stroke-level replay also lets scholars study an artist's technique.

Catalog record

CREATORYing Jiang 1∗, Jiayin Lu 1∗, Yunuo Chen 1∗, Yumeng He 1,2, Kui Wu 3, Yin Yang 4, Chenfanfu Jiang 1
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE61 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-13191

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Birth of a Painting: Differentiable Brushstroke Reconstruction Ying Jiang 1∗, Jiayin Lu 1∗, Yunuo Chen 1∗, Yumeng He 1,2, Kui Wu 3, Yin Yang 4, Chenfanfu Jiang 1 Abstract . † † * indicates equal contributions. Affiliations: 1 UCLA, 2 USC, 3 LIGHTSPEED, 4 Utah Painting embodies a unique form of visual storytelling, where the creation process is as significant as the final artwork. Although recent advances in generative models have enabled visually compelling painting synthesis, most existing methods focus solely on final image generation or patch-based process simulation, lacking explicit stroke structure and failing to produce smooth, realistic shading. In this work, we present a differentiable stroke reconstruction framework that unifies painting, stylized texturing, and smudging to faithfully reproduce the human painting–smudging loop. Given an input image, our framework first optimizes single- and dual-color Bézier strokes through a parallel differentiable paint renderer, followed by a style generation module that synthesizes geometry-conditioned textures across diverse painting styles. We further introduce a differentiable smudge operator to enable natural color blending and shading. Coupled with a coarse-to-fine optimization strategy, our method jointly optimizes stroke geometry, color, and texture under geometric and semantic guidance. Extensive experiments on oil, watercolor, ink, and digital paintings demonstrate that our approach produces realistic and expressive stroke reconstructions, smooth tonal transitions, and richly stylized appearances, offering a unified model for expressive digital painting creation. Figure 1 . Painting Gallery. We visualize the reconstructed paintings produced by our method, along with one example of an intermediate painting stage. Our method reconstructs the painting process by progressively generating realistic brushstrokes, resulting in visually coherent and complete paintings. 1. Introduction As Picasso once said, “Painting is just another way of keeping a diary.” Painting serves as a powerful medium for expressing human emotions and ideas in visual form. Most prior work on painting generation focuses on the finished artwork ( 10 ; 32 ; 42 ) , while the creative process itself remains concealed. Yet, observing how painters construct their work is equally meaningful. It not only allows audiences to appreciate and study the historical context and expressive techniques of art but also serves as a rich resource for learning how to paint for both humans and robots in reality ( 5 ; 35 ) . Since mastering painting requires substantial time and effort, and much instruction still depends on in-person teaching, generating painting processes offers new and more accessible pathways for learning and engaging with art. Previous works on painting process generation have taken primarily two directions: painting video generation ( 38 ; 51 ; 4 ) and painting stroke synthesis ( 27 ; 53 ; 15 ; 22 ; 43 ; 34 ) . For painting video generation, the task is formulated as a video synthesis problem, where each frame represents a stage of the painting process. ( 38 ) and ( 4 ) explore diffusion models to predict time-lapse videos of artistic creation, while ( 51 ) utilizes diffusion transformers to generate both past and future drawing processes. Although these works produce visually compelling videos thanks to powerful generative models, they do not provide explicit stroke information. Moreover, the transitions between frames are typically patch-based, making the generated painting workflows difficult to reproduce in real-world painting or existing digital painting software. In contrast, stroke-based painting generation methods explicitly represent brush strokes with parametric primitives, such as Bézier curves, to model individual stroke geometries and use differentiable renderers to synthesize vectorized images. Recent work has explored stroke parameter searching ( 53 ) , reinforcement learning ( 15 ) , and transformer-based models ( 22 ) to generate brush stroke sequences. Nevertheless, neural painting approaches ( 53 ; 22 ) often deviate from a realistic painting appearance. These methods render paint strokes defined by Bézier curves with single or dual colors on the canvas, lacking proper color blending between strokes. This limitation prevents them from producing complex textures, smooth tonal transitions, and natural shading effects characteristic of real paintings, resulting in hard stroke boundaries (Fig. 2 (a)) and an overall unrealistic visual appearance. Furthermore, when generating distinct painting styles such as oil, watercolor, or ink effects, these methods typically require training separate style-specific neural painters. This limitation restricts the generated outputs to a narrow set of predefined styles and hinders generalization beyond the training distribution. Inspired by the progressive human painting process, which involves applying flat color fills followed by blending or smudging to create natural shading effects ( 16 ) , we propose a differentiable stroke reconstruction framework that unifies stroke-based paint and smudge rendering with stylized texture generation, reproducing the paint-smudge loop to generate paintings with realistic shading and expressive textured strokes. Our pipeline is organized into three stages: (1) Paint Stroke Reconstruction , a parallel differentiable color-filling module; (2) Stroke Texture Stylization , a stylized texture synthesis module; and (3) Smudge Stroke Reconstruction , a differentiable smudging operation for color blending. Unlike traditional painting software, where strokes are rasterized sequentially to match dynamically created user input, we introduce a parallel differentiable painting algorithm that renders single- or dual-color strokes directly from open Bézier curves in one step. With a differentiable paint renderer, we optimize Bézier-curve geometry parameters and per-stroke colors under geometric and semantic guidance. After extracting dual-color strokes and their geometric structures, we employ a StyleGAN-based generator to optimize latent textures conditioned on stroke geometry, producing diverse stylized appearances (e.g., watercolor, oil, and ink). The textured strokes are then composited onto the canvas and further refined through a differentiable smudge renderer, which generates smudged strokes and smooth color transitions to achieve natural shading. Emulating the iterative human painting process, we adopt a coarse-to-fine optimization strategy that progressively refines paint and smudge parameters by looping through painting, styling, and smudging, while spatially subdividing the canvas from coarse to fine patches. We evaluate our method on analog oil, watercolor, and ink paintings, as well as digital paintings, demonstrating visually compelling results with accurate reconstruction of stylized painting and smudge strokes. In summary, our main contributions are as follows: • We propose a differentiable stroke reconstruction framework that unifies paint rendering, stylized texture generation, and smudge rendering, faithfully reproducing the paint–smudge loop with realistic and expressive strokes. • We introduce a parallel differentiable stamp-based paint renderer and a differentiable smudge renderer that captures both geometry and texture, enabling joint optimization of stroke geometry, color, and texture under novel semantic and gradient guidance. • We present a unified representation for stylized painting across diverse painting styles, and validate our approach through extensive experiments on various categories (oil, watercolor, ink, and digital paintings), demonstrating its effectiveness and expressiveness. 2. Related Work Figure 2 . From left to right: (a) hard shading effects, (b) smooth shading effects, and (c) textured painterly shading effects. Figure 3 . From left to right: (a) input pixel-based image, (b) vectorized painting with closed-region representation, and (c) vectorized painting with open-curve representation. While the closed-region vectorization effectively captures fine details, its zigzag boundaries composed of multiple Bézier curves are less suitable for stroke-based painting and can only produce hard shading effects. In contrast, the open-curve representation models each stroke as a single cubic Bézier curve, blending colors through smudging. This approach more closely resembles human-created painting strokes

Read the complete text