Scale-Interaction Transformer: A Hybrid CNN-Transformer Model for Facial Beauty Prediction

Djamel Eddine Boukhari · 2025

Read Scale-Interaction Transformer: A Hybrid CNN-Transformer Model for Facial Beauty Prediction by Djamel Eddine Boukhari (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-05078.txt.

Introduction

Judging facial beauty from a photograph depends on features at several scales at once. Djamel Eddine Boukhari proposes a hybrid model for that prediction task in a solo-authored 2025 computer vision paper. Convolutional networks extract features well but usually at a fixed scale, missing links between fine and global traits.

The Scale-Interaction Transformer first applies parallel convolutions at varied receptive fields to capture multi-scale features. A transformer encoder then models interactions among those representations through self-attention. On a widely used benchmark, the model sets a new leading correlation score in the author's experiments, beating previous methods.

Catalog record

CREATORDjamel Eddine Boukhari
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-05078

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Scale-Interaction Transformer: A Hybrid CNN-Transformer Model for Facial Beauty Prediction Djamel Eddine Boukhari Abstract Automated Facial Beauty Prediction (FBP) is a challenging computer vision task due to the complex interplay of local and global facial features that influence human perception. While Convolutional Neural Networks (CNNs) excel at feature extraction, they often process information at a fixed scale, potentially overlooking the critical inter-dependencies between features at different levels of granularity. To address this limitation, we introduce the Scale-Interaction Transformer (SIT), a novel hybrid deep learning architecture that synergizes the feature extraction power of CNNs with the relational modeling capabilities of Transformers. The SIT first employs a multi-scale module with parallel convolutions to capture facial characteristics at varying receptive fields. These multi-scale representations are then framed as a sequence and processed by a Transformer encoder, which explicitly models their interactions and contextual relationships via a self-attention mechanism. We conduct extensive experiments on the widely-used SCUT-FBP5500 benchmark dataset, where the proposed SIT model establishes a new state-of-the-art. It achieves a Pearson Correlation of 0.9187, outperforming previous methods. Our findings demonstrate that explicitly modeling the interplay between multi-scale visual cues is crucial for high-performance FBP. The success of the SIT architecture highlights the potential of hybrid CNN-Transformer models for complex image regression tasks that demand a holistic, context-aware understanding. K eywords Facial Beauty Prediction, Vision Transformer, Hybrid CNN-Transformer, Multi-Scale Feature Extraction, Self-Attention, Computational Aesthetics, Computer Vision. 1 Introduction Automated Facial Beauty Prediction (FBP) is a challenging and long-standing task in computer vision, residing at the intersection of machine learning, signal processing, and computational aesthetics [ 1 ] . The objective is to develop algorithms that can assign a beauty score to a facial image that correlates highly with human perception. Beyond its academic interest, FBP has potential applications in areas such as photo editing and recommendation, cosmetic surgery simulation, and entertainment [ 2 ] . However, the task is inherently difficult due to the subjective and multi-faceted nature of beauty, which is influenced by a complex interplay of cultural, social, and personal preferences [ 3 ] . Early machine learning approaches to FBP relied on handcrafted features based on geometric ratios, facial symmetry, and anthropometric measurements [ 4 ] . While these methods provided valuable initial insights, they were often limited in their expressive power and failed to capture the subtle, texture-based, and holistic cues that humans perceive [ 5 ] [ 6 ] . The advent of deep learning, particularly Convolutional Neural Networks (CNNs) [ 7 ] , marked a paradigm shift. Architectures such as AlexNet [ 8 ] , and ResNet [ 9 ] became the de facto standard, achieving state-of-the-art performance by automatically learning a rich hierarchy of features directly from image data. Despite their success, a potential limitation of conventional CNN architectures lies in their feature aggregation strategy. Typically, features are extracted through a sequence of convolutional layers, resulting in a final feature map with a fixed receptive field [ 10 ] . While effective for object classification, this approach may not be optimal for FBP, where perception is guided by a combination of global features (e.g., facial structure, harmony, symmetry) and local details (e.g., skin texture, eye clarity, nose shape). Capturing features at multiple scales is crucial, but explicitly modeling the relationships between these scales remains an open challenge. Concurrently, the Transformer architecture, originally developed for natural language processing [ 11 ] , has been successfully adapted for computer vision tasks in the form of the Vision Transformer (ViT) [ 12 ] . The core strength of the Transformer lies in its self-attention mechanism, which excels at modeling long-range dependencies and contextual relationships within a sequence of input tokens [ 13 ] . This provides a powerful alternative to the localized receptive fields inherent in CNNs. In this work, we bridge the gap between these two successful paradigms by proposing the Scale-Interaction Transformer (SIT) , a novel hybrid architecture designed specifically for facial beauty prediction. We hypothesize that explicitly modeling the interactions between features extracted at different spatial scales will lead to a more robust and human-correlated prediction of facial beauty. Our approach first employs a multi-scale CNN module to extract parallel feature representations at varying receptive fields. These multi-scale features are then framed as a sequence and fed into a Transformer encoder, which uses self-attention to weigh and fuse these features, effectively learning the complex interplay between local details and global facial aesthetics. The main contributions of this paper are as follows: 1. We propose a novel hybrid CNN-Transformer architecture, the Scale-Interaction Transformer (SIT), which synergizes the powerful feature extraction of CNNs with the relational modeling capabilities of Transformers. 2. We introduce a multi-scale feature extraction module that captures facial characteristics at fine, medium, and coarse levels simultaneously. 3. We demonstrate the efficacy of treating these multi-scale features as a sequence and applying a Transformer to explicitly model their inter-dependencies for the FBP task. 4. Our proposed SIT model achieves new state-of-the-art results on the widely-used SCUT-FBP5500 benchmark dataset, outperforming previous methods in both correlation and error metrics. The remainder of this paper is structured as follows: Section 2 reviews related work. Section 3 details our proposed methodology. Section 4 presents the experimental results, including comparisons with state-of-the-art methods and ablation studies. Finally, Section 5 concludes the paper and discusses future work. 2 Related Work Our research is positioned at the confluence of several key domains in computer vision: Facial Beauty Prediction (FBP), deep learning with Convolutional Neural Networks (CNNs), multi-scale feature representation, and the emerging paradigm of Vision Transformers. This section reviews the literature in these areas to contextualize the contribution of our proposed method. 2.1 Facial Beauty Prediction The quest to computationally model facial attractiveness has a long history. Early research predominantly focused on handcrafted features derived from domain knowledge in psychology and anthropometry. These methods typically extracted geometric features, such as facial ratios (e.g., the golden ratio), distances between facial landmarks, and measures of symmetry [ 4 ] . While these approaches provided a foundational understanding, their reliance on predefined, low-level features limited their expressive power, as they often failed to capture the holistic and texture-based cues that are integral to human perception [ 5 ] . The advent of deep learning instigated a paradigm shift in the field. By learning features directly from data, deep learning models obviated the need for manual feature engineering and consistently outperformed their traditional counterparts, marking the beginning of the modern era of FBP research. 2.2 Deep Learning Approaches for FBP Convolutional Neural Networks (CNNs) have become the de facto standard for FBP. Initial applications saw the successful transfer of standard classification architectures, such as AlexNet [ 8 ] , and later, much deeper models like ResNet [ 9 ] and ResNeXt , to the FBP regression task [ 14 ] . These models were typically fine-tuned on FBP datasets, treating the problem as a regression task by replacing the final classification layer with a dense layer predicting a single beauty score. To further improve performance, subsequent research has focused on augmenting these baseline CNN architectures [ 15 ] . For instance, Cao et al. [ 16 ] introduced a Squeeze-and-Compaction Attention (SCA) module to help the network focus

Read the complete text