Lightweight Vision Transformer with Window and Spatial Attention for Food Image Classification

Xinle Gao, Linghui Ye, Zhiyong Xiao* · 2025

Read Lightweight Vision Transformer with Window and Spatial Attention for Food Image Classification by Xinle Gao, Linghui Ye, Zhiyong Xiao* (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-18692.txt.

Introduction

Sorting food photos automatically helps factories check quality and watch safety on production lines. Xinle Gao, Linghui Ye, and Zhiyong Xiao present a lightweight vision transformer for that task in this 2025 paper. Full transformer models are accurate but heavy, with many parameters and high compute cost. That weight blocks use on modest hardware.

Their design adds two attention tricks. A window mechanism splits the image into patches and reads local and global context at lower cost. A spatial mechanism then highlights the image regions that matter most for the call. On the Food-101 data set the model scored about 95 percent accuracy. On a second food set it scored about 94 percent. Both came with far fewer parameters and operations than baselines. The authors aim the model at resource-limited settings.

Catalog record

CREATORXinle Gao, Linghui Ye, Zhiyong Xiao*
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-18692

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Lightweight Vision Transformer with Window and Spatial Attention for Food Image Classification Xinle Gao, Linghui Ye, Zhiyong Xiao* Abstract With the rapid development of society and continuous advances in science and technology, the food industry increasingly demands higher production quality and efficiency. Food image classification plays a vital role in enabling automated quality control on production lines, supporting food safety supervision, and promoting intelligent agricultural production. However, this task faces challenges due to the large number of parameters and high computational complexity of Vision Transformer models. To address these issues, we propose a lightweight food image classification algorithm that integrates a Window Multi-Head Attention Mechanism (WMHAM) and a Spatial Attention Mechanism (SAM). The WMHAM reduces computational cost by capturing local and global contextual features through efficient window partitioning, while the SAM adaptively emphasizes key spatial regions to improve discriminative feature representation. Experiments conducted on the Food-101 and Vireo Food-172 datasets demonstrate that our model achieves accuracies of 95.24% and 94.33%, respectively, while significantly reducing parameters and FLOPs compared with baseline methods. These results confirm that the proposed approach achieves an effective balance between computational efficiency and classification performance, making it well-suited for deployment in resource-constrained environments. K eywords Lightweight food image classification \cdot Vision Transformer \cdot Window multi-head attention mechanism \cdot Spatial attention mechanism \cdot Deep learning 1 Introduction With the rapid development of society and continuous advances in science and technology, the food industry faces increasingly stringent demands for high-quality and efficient production [ 1 ] . In this context, food image classification has become a critical technology, enabling automated quality inspection on production lines, strengthening food safety supervision, and accelerating the intelligent transformation of agricultural production. Despite its importance, food image classification remains a challenging task due to the diversity and complexity of food categories [ 2 ] . Foods within the same category often share highly similar attributes, such as color and texture, while exhibiting variations in shape, perspective, and fine-grained details. These similarities hinder precise subcategory classification, while differences can lead to confusion in feature learning, increasing classification complexity. In addition to advances in food image classification, research on hyperspectral image (HSI) processing has provided valuable insights into efficient feature extraction and lightweight model design [ 3 , 4 , 5 ] . In recent years, deep learning has significantly advanced food image classification. Convolutional Neural Networks (CNNs) have played a central role by enabling robust feature extraction and effectively addressing the challenges of high intra-class similarity and inter-class variability [ 6 ] . Several CNN-based approaches have been proposed, ranging from region-based feature extraction to lightweight architectures tailored for resource efficiency. Wei and Wang [ 7 ] proposed a model employing the Faster R-CNN architecture for food region detection in dietary images, followed by CNN-based feature extraction from detected food regions. Yonis et al. [ 8 ] implemented a modified version of MobileNet V2, termed TL-MobileNet V2, for fruit classification. CNN architectures also achieved remarkable advancements in medical image analysis [ 9 , 10 , 11 , 12 , 13 ] . However, CNNs are inherently limited in their ability to capture long-range dependencies, which constrains their performance in complex classification tasks. The introduction of Vision Transformers (ViTs) has provided a new perspective for food image classification [ 14 ] . Using self-attention mechanisms, ViTs model global semantic dependencies across image patches, leading to improved representation learning . The Swin Transformer further improves efficiency by employing shifted window attention, reducing parameter counts and computational complexity while maintaining competitive performance in vision tasks [ 15 ] . Recently, several studies have further advanced food image classification by addressing fine-grained challenges and improving model generalization. For instance, Gao et al. successfully tackled the intricate task of classifying food images with highly similar morphologies by employing sophisticated data augmentation and feature enhancement strategies within the Vision Transformer framework (AlsmViT) [ 16 ] . Similarly, Xiao et al. introduced a deep convolutional module that, when integrated with the global representations of a Swin Transformer, significantly enhanced both local and global feature representations [ 17 ] . Xiao et al. proposed a high-accuracy food image recognition model (FoodCSWin) that enhances dietary assessment by combining diffusion-based data augmentation and local feature enhancement to handle visually diverse but nutritionally similar foods [ 18 ] . Beyond transformer-based approaches, researchers have also explored hybrid optimization strategies. By translating the hierarchical parallelism and evolutionary optimization philosophy of hybrid parallel genetic algorithms into food classification, notable improvements have been achieved, particularly in handling class-imbalanced and fine-grained recognition scenarios [ 19 ] . However, the structural characteristics of ViTs present two key limitations: (1) the large number of parameters and computational overhead significantly increase training and inference costs, and (2) their resource-intensive nature restricts deployment in real-world applications, particularly on mobile or embedded devices with limited computing power. These challenges underscore the need for lightweight transformer-based models that balance classification performance with computational efficiency. In related domains, recent studies have also highlighted the effectiveness of combining CNNs and Transformers for medical image analysis tasks. Xiao et al. proposed a semi-supervised optimization strategy based on the teacher–student paradigm that integrates the strengths of CNNs and Transformers, significantly improving the utilization of large volumes of unlabeled medical images and enhancing segmentation accuracy [ 20 ] . Ji et al. introduced an MLP-based model that employs matrix decomposition and rolling tensor operations for skin lesion segmentation, effectively replacing the self-attention mechanism in Transformers. This approach not only achieved superior performance with fewer parameters but also demonstrated that revisiting traditional methods can yield innovative breakthroughs in deep learning [ 21 ] . Furthermore, the adaptability of hybrid architectures has been illustrated through the Light3DHS model, which combines CNN and ViT components for 3D hippocampal structure segmentation. Light3DHS improved segmentation accuracy while reducing computational cost, offering valuable contributions to brain disease research and clinical applications [ 22 ] . Hybrid CNN-Transformer architectures have also been widely applied in other fields [ 23 , 24 , 25 , 26 ] . More recently, lightweight architectures have emerged as a promising direction. For example, a fine-grained food recognition model integrating an FIP-Attention module for modeling complex ingredient–dish relationships, together with an FCR-Classifier for refining texture–color features, has achieved state-of-the-art performance across multiple benchmarks and demonstrated practical applicability in mobile dietary monitoring applications [ 27 ] . Although hybrid CNN-Transformer architectures have shown promising results in food classification, medical imaging, and hyperspectral analysis, they are not without limitations. The integration of convolutional modules and self-attention layers often increases architectural complexity, leading to larger parameter counts and higher FLOPs [ 28 ] . To address these issues, we propose a lightweight food image classification algorithm that integrates a Window Multi-Head Attention Mechanism (WMHAM) with a Spatial Attention Mechanism (SAM). The WMHAM partitions input features into nonoverlapping local windows, reducing computational cost while retaining global contextual information through interwindow interactions. The SAM adaptively emphasizes discriminative spatial regions, enhancing the focus of the model on key features without increasing the number of parameters. Together, these mechanisms effectively reduce the complexity of the model while improving the classification performance. We evaluate our approach on two widely used benchmark datasets, Food-101 and Vireo Food-172, achieving classification accuracies of 95.24%

Read the complete text