Dual Orthogonal Guidance for Robust Diffusion-based Handwritten Text Generation
Konstantina Nikolaidou 1 George Retsinas 2 Giorgos Sfikas 3 Silvia Cascianelli 4 Rita Cucchiara 4 Marcus Liwicki 1 1 Luleå University of Technology 2 National Technical University of Athens 3 University of West Attica 4 University of Modena and Reggio Emilia · 2025
Read Dual Orthogonal Guidance for Robust Diffusion-based Handwritten Text Generation by Konstantina Nikolaidou 1 George Retsinas 2 Giorgos Sfikas 3 Silvia Cascianelli 4 Rita Cucchiara 4 Marcus Liwicki 1 1 Luleå University of Technology 2 National Technical University of Athens 3 University of West Attica 4 University of Modena and Reggio Emilia (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-17017.txt.
Introduction
Konstantina Nikolaidou and co-authors present a steadier steering method for AI handwriting in a 2025 paper. Diffusion models can fake handwriting well for common words in neat styles, but they tend to memorize training samples and smudge hard styles into unreadable artifacts.
Their method, Dual Orthogonal Guidance, steers each generation step with a cleaned-up direction built from the prompt itself, rather than leaning on blank unconditional guesses. The trick pushes the pen away from distortions while keeping the intended words and adds healthy variety. A schedule adjusts the push across the denoising steps. The result is handwriting that stays legible across rare words and stubborn styles, a step toward generators that can imitate any hand on demand.
Catalog record
| CREATOR | Konstantina Nikolaidou 1 George Retsinas 2 Giorgos Sfikas 3 Silvia Cascianelli 4 Rita Cucchiara 4 Marcus Liwicki 1 1 Luleå University of Technology 2 National Technical University of Athens 3 University of West Attica 4 University of Modena and Reggio Emilia |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 32 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-17017
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Dual Orthogonal Guidance for Robust Diffusion-based Handwritten Text Generation Konstantina Nikolaidou 1 George Retsinas 2 Giorgos Sfikas 3 Silvia Cascianelli 4 Rita Cucchiara 4 Marcus Liwicki 1 1 Luleå University of Technology 2 National Technical University of Athens 3 University of West Attica 4 University of Modena and Reggio Emilia Abstract Diffusion-based Handwritten Text Generation (HTG) approaches achieve impressive results on frequent, in-vocabulary words observed at training time and on regular styles. However, they are prone to memorizing training samples and often struggle with style variability and generation clarity. In particular, standard diffusion models tend to produce artifacts or distortions that negatively affect the readability of the generated text, especially when the style is hard to produce. To tackle these issues, we propose a novel sampling guidance strategy, Dual Orthogonal Guidance (DOG), that leverages an orthogonal projection of a negatively perturbed prompt onto the original positive prompt. This approach helps steer the generation away from artifacts while maintaining the intended content, and encourages more diverse, yet plausible, outputs. Unlike standard Classifier-Free Guidance (CFG), which relies on unconditional predictions and produces noise at high guidance scales, DOG introduces a more stable, disentangled direction in the latent space. To control the strength of the guidance across the denoising process, we apply a triangular schedule: weak at the start and end of denoising, when the process is most sensitive, and strongest in the middle steps. Experimental results on the state-of-the-art DiffusionPen and One-DM demonstrate that DOG improves both content clarity and style variability, even for out-of-vocabulary words and challenging writing styles. 1 Introduction Handwritten Text Generation (HTG), or Styled HTG, is a task that has only relatively recently gained traction compared to the more “traditional” Document Imaging tasks, like Handwritten Text Recognition (HTR) [ 22 , 6 , 40 , 39 , 41 ] or Keyword Spotting (KWS) [ 37 , 38 , 19 ] , in terms of effective models and methods. One motivation for HTG systems is user personalization in digital applications, where it can be useful in aiding individuals with physical impairments to produce handwritten notes in a personalized style. Another crucial goal for HTG is to play the role of an efficient tool for data augmentation, acting in an auxiliary way for other main, downstream Document Analysis tasks [ 10 ] . This is especially useful in low-resource contexts [ 31 ] , where inadequately documented scripts or languages with few writers or with few digitized, annotated training samples constitute a serious impediment to creating automated document-imaging tools. Figure 1 : Qualitative examples of generation without (left) and with (right) our proposed DOG guidance strategy. Diffusion-based HTG models have recently shown promising results in producing readable handwritten images replicating an existing style. However, two major issues persist: 1. They frequently generate artifacts that degrade content clarity, even for common, in-distribution examples. 2. They tend to memorize style-content pairs seen during training, making them brittle in low-data regimes or for unseen combinations. To partially address the first issue, Classifier-Free Guidance (CFG) [ 17 ] has been proposed as a sampling method that interpolates between unconditional and conditional predictions, typically improving alignment with the target condition. However, CFG introduces a trade-off: high guidance scales may help with clarity but often result in over-saturation or degraded detail. In this work, we introduce Dual Orthogonal Guidance (DOG), a guidance strategy that pushes generation along an orthogonal direction derived from a negatively perturbed version of the conditioning prompt. The idea is to encourage clearer generation by suppressing entangled distortions from a perturbed condition (see Fig. 1 ), while still enabling structured variation. Unlike CFG, which relies on unconditional sampling, DOG uses a negative prompt derived from the actual condition ( e.g. , noised style or content), which is then orthogonalized with respect to the original. This produces a more targeted and stable modification of the sampling trajectory. To control the influence of this guidance throughout the diffusion process, we use a triangular schedule, which limits the effect at early and late steps, where the process is most sensitive, and maximizes it in the middle steps, where it can influence the structure without causing instability. In particular, our main contributions are the following: • Dual Orthogonal Guidance (DOG): A test-time sampling strategy that introduces an orthogonal direction derived from a negative prompt, enabling clearer generation and controlled variability. • Stability through triangular scheduling: We modulate the guidance scale across timesteps, peaking mid-process to avoid distorting global structure or introducing artifacts. • Plug-and-play integration: DOG is model-agnostic, requires no retraining, and can be applied directly to pre-trained diffusion-based HTG models. • Compelling Performance: We beat the state-of-the-art in terms of posterior sampling variety, with tangible benefits in terms of qualitative and quantitative results. The remainder of this paper is structured as follows. In Section 2 , we review the related work. In Section 3 , we describe the proposed Dual Orthogonal Guidance. Section 4 presents numerical experiments and showcases illustrative qualitative results. Finally, we conclude the paper with Section 5 , where we summarize our contributions and discuss future work. 2 Related work Styled HTG & Classifier-free Guidance . Styled HTG is the task of generating handwritten images given a style and a content condition. Early HTG methods relied on extensive hand-crafted feature extraction and text resynthesizing using rule-based methods [ 48 , 24 , 46 ] . Later, recurrent architectures demonstrated the ability to generate handwriting [ 15 ] . Building on early methods, GAN-based techniques [ 1 , 43 , 9 , 20 , 21 , 28 , 14 ] , including several that integrate transformer architectures [ 4 , 35 , 47 ] , have enhanced handwriting generation. However, these approaches still face challenges such as training instability and limited diversity [ 13 , 2 , 30 ] . More recently, Diffusion Models [ 49 , 32 , 8 , 33 , 5 , 16 ] have been put forward as an alternative to GAN-based models. While the paradigm of diffusion forms a cohesive foundation for styled HTG, several problems have proved to be non-straightforward to solve in a satisfactory manner, including style variability, rare character combinations, and training data memorization. Classifier-free guidance (CFG) [ 17 ] has shown improvement in the quality of the generation by performing a linear interpolation between the conditional and unconditional estimations, \lambda\epsilon(x|c)+(1-\lambda)\epsilon(x|\emptyset) . A few works have showcased the effect of CFG in HTG by exploring the standard approach [ 8 , 11 , 29 , 5 ] . However, only [ 5 ] explores the effect of CFG on the generation. Negative Prompting in Diffusion Models . Diffusion Models are state-of-the-art latent variable generative models that are consistently setting new benchmarks in numerous and diverse tasks [ 44 ] , including those that pertain to one or another form of prompting, in the sense of a textual condition to the model [ 18 , 7 ] . Negative Prompting seeks to steer the guidance away from unwanted attributes. In [ 25 ] , an objective that may include multiple conditions is broken down to a sum of composing directions in the latent space. Unwanted conditions are then assigned a negative weight, in principle allowing the end-user