Can We Change the Stroke Size for Easier Diffusion?
Yunwei Bai, Ying Kiat Tan, Yao Shu, Tsuhan Chen · 2026
Read Can We Change the Stroke Size for Easier Diffusion? by Yunwei Bai, Ying Kiat Tan, Yao Shu, Tsuhan Chen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-26783.txt.
Introduction
Bai, Tan, and Chen ask whether varying stroke size can simplify flow matching for image generation. Painters lay rough strokes first and fine ones later, which eases their work. MultiStroke copies that rhythm by coarsening the clean-signal part of the state and target early, then refining it over time. A light corrector aligns steps between roughness levels during sampling. The Gaussian noise part stays untouched throughout. On ImageNet-1K, the method cuts FID by 22.5% against a matched flow-matching baseline.
Catalog record
| CREATOR | Yunwei Bai, Ying Kiat Tan, Yao Shu, Tsuhan Chen |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 82996 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.26783
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Can We Change the Stroke Size for Easier Diffusion? Yunwei Bai, Ying Kiat Tan, Yao Shu, Tsuhan Chen Abstract Diffusion models can be challenged in the low signal-to-noise regime, where they have to make pixel-level predictions despite the presence of high noise. The geometric intuition is akin to using the finest stroke for oil painting throughout, which may be ineffective. We therefore study stroke-size control as a controlled intervention that changes the effective roughness of the supervised target, predictions and perturbations across timesteps, in an attempt to ease the low signal-to-noise challenge. We analyze the advantages and trade-offs of the intervention both theoretically and empirically. Code will be released. 1 Introduction Oil painting artists usually change brush size: coarse strokes first to lay out global structure, then finer strokes to add detail. Diffusion image generation, in contrast, is trained as a per-pixel regression problem with noise, where the input has extremely low signal-to-noise ratio at high time steps. This creates a hard family of high-frequency regression targets with large stochastic-gradient variance, slows optimization, and causes reverse-chain error accumulation that often appears as high-frequency artifacts ( 6 ) . We study a mechanistic control knob (named “MultiStroke” for easy reference) for reverse-chain roughness, focusing on class-conditional DDPM-style image diffusion, and we do not claim universality across architectures or modalities. Experiments are compute-matched and diagnostic-driven . Intuitively speaking, we ask a simple yet under-explored question: can we change the effective stroke size across timesteps to simplify diffusion? In this work, we formalize a stroke-size operator family that attenuates within-block detail in both the supervised regression targets, predictions and perturbation used during training and sampling. Our theory is operator-theoretic and is anchored to the practical pooling-plus-upsampling stroke operator: it is an orthogonal projection onto a block-coarse subspace and induces an exact coarse/detail decomposition. Under this decomposition, MultiStroke preserves coarse components while shrinking detail components, yielding explicit reductions in multiresolution target complexity, and contracted propagated/injected detail energy along the reverse chain under an affine surrogate dynamics. Figure 1 : Samples under matched training and sampling budgets. DDPM ( 9 ) uses a fixed pixel-scale target at all timesteps, while stroke control uses coarser targets early. We also implement the intervention that instantiates the operator which coarsens the supervision targets, predictions and sampling perturbations via nearest-neighbor downsampling and upsampling. Our contributions include: 1. Mechanism identification. We formalize stroke-size control as a single operator S_{k} applied consistently to (i) the training prediction and objective and (ii) the sampling-time state and injected noise, yielding an explicit knob over reverse-chain roughness. 2. Operator-theoretic framework for the practical stroke map. For the implemented pooling-plus-upsampling operator, we prove S_{k} is an orthogonal projector, derive its Fourier attenuation envelope, and obtain an exact coarse/detail decomposition with closed-form projector algebra. Under an affine surrogate reverse dynamics, we derive a detail-energy recursion showing contraction of propagated detail and reduced injected detail noise. 3. Mechanism validation. Under compute-matched DDPM training and limited-step sampling on real datasets, we observe improved compute-limited quality and fewer high-frequency artifacts, together with a bias-stability trade-off with relatively aggressive stroke size control. Not just blurring. A natural concern is that MultiStroke simply “blurs” the learning problem. Proposition 5.1 rules out this interpretation at the population level: under our standing condition w_{\max}<1 (so A_{t} has an identity component), the population minimizer of the stroke-space objective coincides with the standard DDPM ( 9 ) conditional mean. The intervention therefore changes the optimization path and reverse-chain dynamics, not the population solution. 2 Related Work Diffusion models ( 9 ; 34 ; 23 ) have been improved via schedules/parameterizations ( 12 ; 16 ) , faster samplers ( 32 ; 30 ) , and guidance ( 11 ) ; these advances largely retain pixel-scale prediction at every timestep. Curriculum and loss reweighting. Several works reshape diffusion training via per-timestep curricula or objective reweighting, including Min-SNR- \gamma weighting ( 7 ) and Soft Diffusion ( 3 ) . Min-SNR is a scalar reweighting technique that balances time steps losses according to their corresponding SNR values, whereas Soft Diffusion encourages diffusion models to produce higher-quality samples that match the training target after corruption. In contrast, MultiStroke applies a stroke operator to the model input, predictions, supervised targets and injected noise, yielding a controllable high-frequency contraction effect (Proposition 5.4 ). To the best of our knowledge, using such a stroke operator as simplification for the high-noise training and sampling is under-explored , if not unprecedented. Frequency structure and U-Net pathways. Another complementary line of work studies how diffusion U-Nets route low- vs. high-frequency information through down/up-sampling and skip connections, and how modifying these pathways can affect artifacts and quality. For example, FreeU ( 31 ) adjusts U-Net feature and skip contributions at inference time to improve sample quality without retraining. These methods act at the architecture/feature level, whereas MultiStroke acts at the objective, prediction and perturbation levels: we apply a stroke operator to the supervised residual and to the injected noise, and analyze the induced coarse/detail contractions (Sections 5.3 - 5.4 ). The two directions are compatible: our analysis provides a principled view of suppressing detail-subspace stochasticity early in the reverse chain, which is closely related to heuristically damping high-frequency pathways. Additional works are provided in Appendix D . 3 Spectral Difficulty at Low SNR Diffusion training couples a family of regression problems across noise levels. For DDPM-style objectives, a network predicts Gaussian noise from a corrupted input \displaystyle x_{t} \displaystyle=\sqrt{\bar{\alpha}_{t}}\,x_{0}+\sqrt{1-\bar{\alpha}_{t}}\,\varepsilon, (1) \displaystyle\varepsilon\sim\mathcal{N}(0,I),\quad t\in\{1,\dots,T\}. The signal-to-noise ratio decreases with t as \mathrm{SNR}(t)=\frac{\bar{\alpha}_{t}}{1-\bar{\alpha}_{t}}, (2) so at high timesteps the input is dominated by noise. Nevertheless, the supervision remains a pixel-scale noise field. This mismatch is the source of two coupled difficulties. A high-frequency regression family. Viewed in a Fourier basis, the noise target allocates substantial energy to high spatial frequencies. At low-SNR, the conditional prediction function x_{t}\mapsto\mathbb{E}[\varepsilon\mid x_{t},t,y] must infer these fine-scale components from an input that contains little information about x_{0} . In practice, this turns the high-timestep portion of training into a difficult, high-frequency regression family. Large gradient noise and reverse-chain fragility. For squared-error regression, stochastic-gradient variance is driven by the second moment of the residual target. When the target retains substantial high-frequency energy, the variance of gradients along high-frequency directions can dominate, slowing optimization and making the learned score more sensitive to noise. During sampling, per-step errors produced at high timesteps frequently manifest as high-frequency artifacts. These observations motivate our stroke-size control design, a linear operator that attenuates high spatial frequencies early (when SNR is low), together with a schedule that returns to the standard objective in the final steps. The remainder of the paper analyzes its effect on training target complexity, and the reverse-chain inference. 4 Stroke-Size Control as a Controlled Intervention 4.1 Preliminaries We work with a conditional DDPM ( 9 ) over images x_{0}\in\mathbb{R}^{C\times H\times W} and labels y\in\{1,\dots,K\} . Given a variance schedule \{\beta_{t}\}_{t=1}^{T} with \alpha_{t}=1-\beta_{t} and \bar{\alpha}_{t}=\prod_{s=1}^{t}\alpha_{s} (with the convention \bar{\alpha}_{0}:=1 ), the forward process is x_{t}=\sqrt{\bar{\alpha}_{t}}\,x_{0}+\sqrt{1-\bar{\alpha}_{t}}\,\varepsilon,\;\varepsilon\sim\mathcal{N}(0,I),\;t=1,\dots,T. (3) A network \varepsilon_{\theta}(x_{t},t,y) is trained to predict \varepsilon by minimizing \mathcal{L}_{\mathrm{DDPM}}(\theta)=\mathbb{E}\bigl[\|\varepsilon_{\theta}(x_{t},t,y)-\varepsilon\|_{2}^{2}\bigr]. (4) A standard score estimator induced by the noise prediction parameterization is \hat{s}_{t}(x_{t},t,y):=-\frac{\varepsilon_{\theta}(x_{t},t,y)}{\sqrt{1-\bar{\alpha}_{t}}}. (5) DDPM reverse mean and variance. For completeness, the standard DDPM ancestral sampler uses \mu_{t}(x,\hat{\varepsilon}_{t})=\frac{1}{\sqrt{\alpha_{t}}}\Bigl(x-\frac{1-\alpha_{t}}{\sqrt{1-\bar{\alpha}_{t}}}\,\hat{\varepsilon}_{t}\Bigr),\;\sigma_{t}^{2}=1-\alpha_{t}. (6) Variance convention.