CraftGraffiti : Exploring Human Identity with Custom Graffiti Art via Facial-Preserving Diffusion Models

Ayan Banerjee, Fernando Vilariño, Josep Lladós · 2025

Read CraftGraffiti : Exploring Human Identity with Custom Graffiti Art via Facial-Preserving Diffusion Models by Ayan Banerjee, Fernando Vilariño, Josep Lladós (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-20640.txt.

Introduction

Ayan Banerjee, Fernando Vilariño, and Josep Lladós turn portrait photos into graffiti while keeping the person recognizable. Graffiti's bold, abstract style easily warps eyes, noses, and mouths until identity vanishes. Their system, CraftGraffiti, applies the graffiti look first and locks in the face afterward.

Special attention layers carry identity details through the style transfer, and text prompts steer new poses without pose markers. Doing style before identity cut the drift seen in the reverse order. The system scored well on face consistency and human taste, and it ran live at the Cruilla Festival for real audiences.

Catalog record

CREATORAyan Banerjee, Fernando Vilariño, Josep Lladós
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-20640

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CraftGraffiti : Exploring Human Identity with Custom Graffiti Art via Facial-Preserving Diffusion Models Ayan Banerjee, Fernando Vilariño, Josep Lladós Abstract Preserving facial identity under extreme stylistic transformation remains a major challenge in generative art. In graffiti, a high-contrast, abstract medium—subtle distortions to eyes, nose, or mouth can erase the subject’s recognizability, undermining both personal and cultural authenticity. We present CraftGraffiti , an end-to-end text-guided graffiti generation framework designed with facial feature preservation as a primary objective. Given an input image and a style and pose descriptive prompt, CraftGraffiti first applies graffiti style transfer via LoRA-fine-tuned pretrained diffusion transformer, then enforces identity fidelity through a face-consistent self-attention mechanism that augments attention layers with explicit identity embeddings. Pose customization is achieved without keypoints, using CLIP-guided prompt extension to enable dynamic re-posing while retaining facial coherence. We formally justify and empirically validate the “style-first, identity-after” paradigm, showing it reduces attribute drift compared to the reverse order. Quantitative results demonstrate competitive facial feature consistency and state-of-the-art aesthetic and human preference scores, while qualitative analyses and a live deployment at the Cruïlla Festival highlight the system’s real-world creative impact. CraftGraffiti advances the goal of identity-respectful AI-assisted artistry, offering a principled approach for blending stylistic freedom with recognizability in creative AI applications. The code, demo, and details of the actual installation at the music Festival Cruilla 2025 in Barcelona are available at: github.com/ayanban011/CraftGraffiti. Figure 1: Customization with CraftGraffiti : It can accurately customize the input character pose in three different musical scenarios, preserving the facial attributes in graffiti style, as the only the pose of the face mesh changes with style while preserving the mesh structure of the input image. 1 Introduction Human identity preservation in the GenAI era: bias, aesthetics, and open experimentation: Generative systems such as GANs and diffusion models replicate and amplify cultural biases from their training data, entering into the risk of distorting human identity across multiple dimensions ( ghosh2024don ) . Since usually trained on not particularly curated datasets, it is known that current generative systems usually tend to disproportionately produce light-skinned, youthful, conventionally attractive faces, marginalizing other demographics, and concentrating on a narrow band of a specific concept of aesthetics ( munoz2023uncovering ) . On the other hand, both children and older adults are usually either underrepresented or not properly identified, even for age-progression models ( munoz2023uncovering ; li2018glca ) , presenting results that tend to transform the image of a young person into an adult face, or conversely, elderly people as unrealistically younger figures. Errors in the gender identification are usually more prone to affect women than men, and reinforce gender norms and heteronormativity, as seen in occupational and presentational stereotypes ( sun2023smiling ; zhou2024bias ) . Rooted in imbalanced datasets, these distortions risk homogenizing outputs unless addressed through curated data, fairness-aware training, and interdisciplinary critique ( ferrara2023fairness ) . For this reason, and in light of the relevant risks identified, it is essential to openly address the tensions to which our human identity is exposed in the new digital era; an era of interconnected individuals endowed with generative capacities. Such a topic is profound and complex from the philosophical, ethical, political, and technical dimensions ( binns2017fairness ; alkfairy2024ethical ) . We state that in order to address the discussion and public debate of these topics, we can make use of two main instruments: 1) An artistic installation providing a representation of the customized human pose, technically sound and preserving facial attributes as much as possible, and 2) an ecological environment that allows for the natural interaction with people, providing the explicit context of a living lab or open experimentation space. CraftGraffiti - Integrating Cultural Expression and Facial Consistency in Generative Street Art: For the first aim, we need a cultural bind that serves as an entry point for people to interact with this new era of GenAI, which has to be inclusive and accessible for all; from this perspective, graffiti appears as an excellent candidate. Graffiti is a culturally significant art form, merging creative expression with the diversity of human identity ( das2023power ; forster2012evaluating ) and often serving as a voice for marginalized communities ( bates2014bombing ) . While recent advances in GANs ( xu2021drb ; yang2019controllable ) and diffusion models ( zhang2023inversion ; wang2023stylediffusion ) offer automated generation, existing methods struggle to preserve facial attributes across varying styles and poses. Text-based image editing ( kawar2023imagic ; huang2025diffusion ) and style transfer models ( huang2024diffstyler ; wang2023stylediffusion ; guo2024focus ) fail to simultaneously adjust poses, while pose editing approaches ( cheong2023upgpt ; okuyama2024diffbody ) lack unified style transfer. Unified models ( pascual2024enhancing ; mokady2023null ; dong2023prompt ) attempt both but often lose facial details due to self-attention limitations. We introduce CraftGraffiti , an end-to-end generative framework that produces custom graffiti-style portraits of human characters while explicitly preserving the subject’s facial identity (see Fig. 1 ). This system is text-guided, allowing a user to specify high-level context (e.g., prompting a portrait of the person as a DJ, guitarist, or singer) and to customize the character’s pose, all within a graffiti art style. To accomplish this, CraftGraffiti integrates several components: a CLIP-based text encoder for semantic guidance, a graffiti-style fusion module implemented via a pre-trained diffusion transformer 1 1 1 https://civitai.com/models/1058970/graffiti-style-fluxFaces fine-tuned on graffiti aesthetics, a LoRA-based adapter ( hu2022lora ) that efficiently injects pose information without requiring full model retraining, and a novel face-consistent self-attention mechanism that ensures key facial features of the input are maintained throughout the diffusion process. The final output is rendered by a variational autoencoder (VAE) decoder, yielding a high-quality graffiti portrait that retains the individual’s identity. This architecture enables simultaneous pose adjustment and style transfer while keeping the person’s face recognizable. Our contributions include: (1) a unified framework for pose-guided, graffiti-themed portrait generation that preserves individual identity; (2) a novel self-attention module specifically designed for facial consistency in diffusion-based generation; and (3) state-of-the-art performance in both qualitative visual fidelity and quantitative identity-preservation metrics compared to existing methods. A public artistic installation in an ecological environment as a living lab: For our second aim, we propose to explore ecological environments to enhance user-experience experimentation by enabling real-world engagement in authentic contexts ( kieffer2017ecoval ) . Living labs provide user-centred, open innovation ecosystems with real-world experimentation, co-creation, and multi-stakeholder collaboration ( compagnucci2021livinglabs ; livinglab_wikipedia ) . Outdoor labs with pervasive sensing and crowdsensing further increase scalability and context-awareness ( pournaras2021crowdsensing ) , fostering inclusivity, diversity, and richer insights than artificial environments. From this perspective, music festivals appear as excellent ecological experimentation spaces for this type of trans- and multi-disciplinary investigation. In our case, we set our installation during the four days of the Festival Cruïlla, taking place 9-12 July 2025 (Please refer to the appendix for more details of the installation setup, outcomes, and human feedback). 2 Preliminaries Diffusion Models: Text-to-image diffusion models, such as Stable Diffusion rombach2022high ; podell2023sdxl ; everaert2023diffusion , generate images by learning a denoising process that maps latent noise to realistic images ( \mathbf{I} ), conditioned on textual input. Let \mathbf{x}_{0}\in\mathbb{R}^{H\times W\times C} denote a clean image and \mathbf{x}_{t} its noisy version at timestep t\in\{1,\dots,T\} . The forward diffusion process ( q(\cdot) ) adds Gaussian noise ( \mathcal{N}(\cdot) ) in

Read the complete text