ObjMST: Object-focused Multimodal Style Transfer

Chanda Grover Kamra, Indra Deep Mastan, Debayan Gupta · 2025

Read ObjMST: Object-focused Multimodal Style Transfer by Chanda Grover Kamra, Indra Deep Mastan, Debayan Gupta (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-04353.txt.

Introduction

Chanda Grover Kamra, Indra Deep Mastan, and Debayan Gupta propose ObjMST in a 2025 paper, a framework for object-focused multimodal style transfer. Style transfer paints the look of one image or text cue onto a new image, but past tools apply one style in a flat way across the whole scene.

ObjMST gives split style guidance to the main objects and to the parts around them. A new loss, built on CLIP, keeps image and text style views in line for both zones. A mapping step styles the key objects first, and a final blend makes the whole image hang together.

Catalog record

CREATORChanda Grover Kamra, Indra Deep Mastan, Debayan Gupta
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-04353

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ObjMST: Object-focused Multimodal Style Transfer Chanda Grover Kamra, Indra Deep Mastan, Debayan Gupta Abstract We propose ObjMST, an object-focused multimodal style transfer framework that provides separate style supervision for salient objects and surrounding elements while addressing alignment issues in multimodal representation learning. Existing image-text multimodal style transfer methods face the following challenges: (1) generating non-aligned and inconsistent multimodal style representations; and (2) content mismatch, where identical style patterns are applied to both salient objects and their surrounding elements. Our approach mitigates these issues by: (1) introducing a Style-Specific Masked Directional CLIP Loss, which ensures consistent and aligned style representations for both salient objects and their surroundings; and (2) incorporating a salient-to-key mapping mechanism for stylizing salient objects, followed by image harmonization to seamlessly blend the stylized objects with their environment. We validate the effectiveness of ObjMST through experiments, using both quantitative metrics and qualitative visual evaluations of the stylized outputs. Our code is available at: https://github.com/chandagrover/ObjMST. † † highlights: We propose a multimodality-guided ObjMST framework for image style transfer. We generate style encodings using cross-modal StyleGAN inversion via CLIP embeddings. Issue: Alignment issues; sol: masked directional CLIP loss. Issue: Content mismatch; sol: Salient-To-Key attention mechanism. Quantitative and qualitative results show that ObjMST outperforms relevant baselines. keywords Multimodal ,Style Transfer ,CLIP ,Segmentation ,Harmonization † † credit: Conceptualization, Methodology, Draft writing, Reviewing and Editing † † credit: Reviewing, Investigation, Supervision, Methodology † † credit: Supervision, Investigation, Reviewing † † corresponding: Corresponding author 1 Introduction Classical Image-guided Image Style Transfer (IIST) methods primarily rely on a style image to supervise the stylization process [ 1 , 11 , 10 , 6 ] . In contrast, multimodal learning [ 12 ] integrates additional modalities, such as textual descriptions [ 16 , 15 ] or audio cues [ 7 ] to guide or augment the stylization [ 10 , 6 ] . There also exist uni-modal (image or text) [ 11 , 10 , 6 ] and multimodal (image and text) [ 19 ] guided methods for IST. IIST methods that rely solely on image data are limited in practical use when style images are unavailable [ 10 ] or when unique styles are lacking [ 19 ] . Consequently, there has been a shift towards Text-Guided Image Style Transfer (TIST) methods [ 10 , 6 , 15 ] . However, TIST approaches often introduce textual artifacts [ 10 ] due to challenges in achieving effective style alignment and content preservation [ 10 , 6 ] . In a recent study, Wang et al. [ 19 ] proposed a novel cross-modal GAN inversion-based style generation mechanism. This mechanism is guided by both style text and style image, ensuring consistency across modalities. Table 1: The table compares Image Style Transfer (IST) methods. ’Single’ in columns one and three indicates a single text condition, while ’double’ in column two refers to double text conditions for foreground ( fg ) and background ( bg ) objects. Text-Based IST (Single) Text-Based IST (Double) Multimodal IST (Single) CS [ 10 ] \surd X X SemCS [ 6 ] \surd \surd X LDAST [ 2 ] \surd X X MMIST [ 19 ] \surd X \surd ObjMST (Ours) \surd \surd \surd We observed that MMIST [ 19 ] encounters difficulties in preserving semantics in the style transfer. For instance, Fig. 1 , top row (a-d) shows that MMIST [ 19 ] fails to capture color and texture tones corresponding to the style-text description ”Copper plate engraving”. This limitation is primarily due to the difficulty in aligning text and image [ 21 ] features within the shared CLIP (Contrastive Language-Image Pre-training) [ 16 ] embedding space. Another issue in MMIST [ 19 ] is content mismatch, where a similar style is applied to semantically different objects, resulting in unintended spill-over. For example, Fig. 1 (e-f) shows that the texture or patterns from the style features spill over into regions such as the background or other objects, producing a visually incoherent result in MMIST [ 19 ] . SemCS [ 6 ] and CLVA [ 2 ] were also observed to be suffering from content mismatch (Fig. 1 ). (a) Content Image (b) Multimodal Input (c) MMIST [ 19 ] (d) Ours (e) Content Image (f) Multimodal Input (g) SemCS [ 6 ] (h) Ours Copper Plate Engraving. F: Ice B: Starry Night by Vincent Van Gogh. (i) Style Text (j) Content Image (k) CS [ 10 ] (l) SemCS [ 6 ] (m) CLVA [ 2 ] (n) MMIST [ 19 ] (o) Ours A cubism style painting. Figure 1: The comparative stylized outputs are presented as follows: (i) Top Row, Left Side: MMIST (Single); (ii) Top Row, Right Side: TIST (Double); and (iii) Bottom Row: TIST (Single). Columns (a, e, j) represent the content images, while columns (b, f, i) show the multimodal style inputs. In MMIST [ 19 ] (column c), misalignment is evident as the texture and color of the copper plate features are inconsistent compared to Ours-ObjMST (column d). In TIST (Double), SemCS [ 6 ] (column g) introduces undesired distortions, whereas ObjMST (column h) correctly applies the ”Starry Night” style features to the background (sky) and ice features to the foreground. In TIST (Single), ObjMST (column o) effectively preserves the content features while accurately applying the desired style. We address the issue of misalignment in style transfer by introducing ObjMST, the first object-focused multimodal style transfer framework. The main challenge lies in performing StyleGAN inversion, guided by directional CLIP loss, to generate intermediate style representations. These representations are then used in image-based style transfer to produce the final output. Our analysis of the intermediate style representations (Table 2 and Fig. 7 ) demonstrates improved visual quality when both salient objects and their surrounding elements are considered. To further enhance the representations of salient objects, we mask out arbitrary content features from the multimodal style input during the StyleGAN inversion process. This ensures the preservation of essential style elements in the intermediate stages. This entire process leads to the development of the ”masked” directional CLIP loss, which ensures consistent and aligned generation of intermediate style representations. ObjMST resolves the content mismatch between salient objects and surrounding elements by employing salient-to-key feature matching [ 23 ] . The alignment of each salient object’s content features with stable key positions of style features (S2K) facilitates semantically preserved stylization, focusing exclusively on the salient object within the image. For example, in Fig. 1 -(h), ice features are applied solely to the building. ObjMST generates distinct style representations for salient objects and surrounding elements using cross-model GAN inversion, assisted by masked directional CLIP loss for the foreground ( fg ) and background ( bg ) objects. Separating the style representation generation for fg and bg is essential to minimizing content mismatches. To ensure consistency in the final output, self-supervised image harmonization is employed [ 5 ] . Our major contributions are as follows: • We propose the ObjMST framework, which provides separate style supervision for salient objects and surrounding elements, while addressing alignment issues in multimodal representation learning (Fig. 6 and Fig. 5 ). • The ObjMST framework maps salient content features to stable style key features (S2K) to preserve the semantic

Read the complete text