End-to-end human parsing and detection optimized for resource-constrained devices Authors: Md Imran Hosen, Tarkan Aydin, Md Baharul Islam Published: 2025 ABSTRACT Human parsing, a vital task in human-centric analysis, involves segmenting clothing and body parts for individual association. Existing methods often rely on auxiliary inputs like detection and edge prediction, limiting their suitability for resource-constrained devices. To address this, we propose an end-to-end framework that integrates a transformer based self-attention module to enhance contextual understanding while being optimized for low-resource environments. We also introduce bounding-polygon annotations to facilitate simultaneous detection and parsing. Our method achieves fine-grained results in a single pass, significantly improving inference speed without sacrificing accuracy. Real-world validation on Raspberry Pi demonstrates its effectiveness and efficiency in resource-constrained scenarios. Introduction Instance-level human parsing is an essential and demanding computer vision task that involves segmenting different body parts of a human and linking them with the appropriate instances. This task has extensive implications for human-centric analysis in various scenarios, including security monitoring, human-centric video analysis and autonomous driving1, person re-identification2, human-object interaction3, artistic creation4, dense pose estimation5. Since the introduction of convolutional networks, researchers have explored various approaches to improve the human parsing task. For instance, two-stage detection-based techniques, such as Faster R-CNN6, demonstrated significant advancements in multi-human parsing. However, these methods suffer from misalignment issues between network inputs and outputs at the pixel level. Mask R-CNN7 partially addressed this by preserving spatial locations but remained constrained by multi-stage processing and hyperparameter sensitivity. Subsequent detection-based methods8–10 sought to enhance parsing efficiency through end-to-end approaches. For example,8 introduced a component-aware convolutional structure, while10 applied global semantic representation to improve parsing maps. However, these methods heavily depend on human detection, leading to sub-optimal representation learning and error accumulation11. Graphonomy12 and Grapy-ml13 addressed granularity and generalizability challenges through graph-based structures, but their computational complexity limited real-time deployment and applicability in resource-constrained scenarios. The integration of part detection in human parsing represents a significant advancement, offering a more detailed understanding of the human body. This fine-grained analysis captures intricate spatial distributions and relationships among different body parts. In contrast, existing works, like5,10,11,14,15, lack individual part detection, limiting their utility, especially in critical domains like healthcare and safety monitoring16. The existing methods suffer from several drawbacks, including: (a) slow processing, which necessitates running auxiliary stages, resulting in slow processing times; (b) a lack of global context embedding, making it difficult to differentiate between similar regions, such as left and right hands or legs; and (c) limited applicability. Motivated by these challenges, we propose a low-cost, real-time single-shot human parsing network inspired by YOLO based architecture17. Our approach incorporates a transformer based modified Self-Attention Module (mSAM) to enhance contextual information. The mSAM combines learnable layers with standalone positional embeddings to better capture spatial relationships in human parsing, in contrast to its original implementation in18, which used a learnable linear layer optimized for detection tasks. Unlike prior implementations, our mSAM omits LayerNorm layers to improve memory efficiency, making it ideal for resource-constrained environments. The key contributions of our method are summarized as follows:Our end-to-end framework eliminates the inefficiencies of multi-stage pipelines and achieves real-time performance suitable for deployment on devices like Raspberry Pi.To address the lack of global context embedding in existing methods, our framework incorporates mSAM, which combines learnable layers with standalone positional embeddings for enhanced spatial relationship modeling. The inclusion of Lightweight ResNet Bloc (LRB) in the prediction head improves sensitivity to informative features, ensuring accurate parsing of symmetric and visually similar regions.Extending the Look Into Person (LIP) and Crowd Instance-level Human Parsing (CIHP) datasets by providing polygon annotations. We validate our framework through extensive experiments on real-world data and demonstrate its efficiency and effectiveness on resource-constrained devices, addressing practical deployment challenges.The organization of this paper is as follows: In “Related works”, we briefly discuss the related literature. In “Methodology”, we present the proposed method in detail. We provide experimental setup, dataset, and evaluation criteria in “Data generation and experiment”. Experimental results, analysis, and discussion are carried out in “Results and discussion”. Then, we conclude our work with future research directions. Our end-to-end framework eliminates the inefficiencies of multi-stage pipelines and achieves real-time performance suitable for deployment on devices like Raspberry Pi. Our end-to-end framework eliminates the inefficiencies of multi-stage pipelines and achieves real-time performance suitable for deployment on devices like Raspberry Pi. To address the lack of global context embedding in existing methods, our framework incorporates mSAM, which combines learnable layers with standalone positional embeddings for enhanced spatial relationship modeling. The inclusion of Lightweight ResNet Bloc (LRB) in the prediction head improves sensitivity to informative features, ensuring accurate parsing of symmetric and visually similar regions. To address the lack of global context embedding in existing methods, our framework incorporates mSAM, which combines learnable layers with standalone positional embeddings for enhanced spatial relationship modeling. The inclusion of Lightweight ResNet Bloc (LRB) in the prediction head improves sensitivity to informative features, ensuring accurate parsing of symmetric and visually similar regions. Extending the Look Into Person (LIP) and Crowd Instance-level Human Parsing (CIHP) datasets by providing polygon annotations. We validate our framework through extensive experiments on real-world data and demonstrate its efficiency and effectiveness on resource-constrained devices, addressing practical deployment challenges. Extending the Look Into Person (LIP) and Crowd Instance-level Human Parsing (CIHP) datasets by providing polygon annotations. We validate our framework through extensive experiments on real-world data and demonstrate its efficiency and effectiveness on resource-constrained devices, addressing practical deployment challenges. Related works Human parsing Numerous studies on human parsing have been reported in the literature. It can be categorized into two main groups: single and multi-human parsing. Single human parsing (SHP) SHP forms by assuming one human instance is present in the scene. As a result, only pixel-level supervision is required for the semantic category. Hierarchical Human Parsing19 leverages the representational capacity of Graph Convolutional Networks to understand hierarchical human layouts. In20, Liu et al. proposed a dual-task cascaded framework (DTCF) to enhance edge-aware features in human parsing. To improve accuracy, it integrates parsing and edge features in a progressive refinement process. HTCorrM21 extends CE2P22 and increases its capabilities by incorporating part edges to better model the relationship between different parts. HIPN23 introduced a semi-supervised learning approach combined with denoising techniques, enabling both positive and negative learning to create a noise-tolerant hybrid framework capable of handling noisy pseudo labels effectively. Liu et al.24 proposed CDGNet, which uses human parsing labels gathered horizontally and vertically to guide learning, incorporating attention to balance local and global features. Zhou et al.25 introduced PAGRnet, which learns a global human posture representation through multi-level feature fusion and attention mechanisms. Recently, Chen et al.26 introduced a self-supervised learning method for human-centric tasks, using unlabeled human images to deepen semantic understanding in learned representations. Since real-world problems often involve more than one human instance, the applicability of these models may be limited to single human parsing. Multi-human parsing Multiple human parsing (MHP) handles parsing multiple human instances simultaneously, with key challenges being efficiency and parsing without auxiliary modules. MHP typically follows either a “segmentation first” approach, as in27, where segments are merged using boundary detection, or a “detection first” approach, as in5, which parses components of detected human instances. MHP methods use two main pipelines for instance-level parsing: bottom-up and top-down. Bottom-up approaches: the bottom-up method views multiple human parsing as a fine-grained semantic segmentation task, where each pixel is assigned a category and grouped into the corresponding human instance. Grapy-ml13 investigates the semantic label relationships underlying the hierarchical graph structure. In28, a heterogeneous non-local block is proposed, which utilizes crucial points locations and human semantics to fully exploit the connection among parser, posture, and edge. To facilitate the transmission of human structural information across various components, the Part-Aware Context Network (PCNet)29, a dispersion module, a part-class module, and a relational module are integrated. A similar approach as PCNet was followed in30. The main challenge of bottom-up human parsing is accurately distinguishing between human instances, particularly when people are close or overlapping. Fine-grained semantic segmentation often struggles to differentiate pixels belonging to different individuals, leading to errors in instance-level human parsing. Top-down approaches: the top-down method uses a bounding box to detect humans and extract regions of interest (ROI) for segmentation. It can be divided into two-stage and one-stage approaches. In the two-stage method, an object detector selects the ROI to produce complete human parsing. Ruan et al.22 used two human parsing models. A braiding network was proposed in31 that integrates local structural information with semantic knowledge through interconnected sub-networks. Ji et al.32 introduced a semantic tree for human components, and Li et al.33 developed a self-correcting technique to handle label noise. Yang et al.34 presented QANet to assess output quality and parsing. Despite their high performance, two-stage methods face limitations in flexibility and inference time for real-world applications35. The key difference between one-stage and two-stage approaches is in their operation. Huang et al.36 introduced Mask Scoring R-CNN, which uses a network block to assess the accuracy of predicted instance masks by computing mask IoU. Cheng et al.37 proposed BMask R-CNN, which incorporates edge clues to improve boundary accuracy. Nevertheless, these approaches face challenges in accurately parsing small regions because of insufficient contextual information. To address this, Yang et al.10 developed Renovating Parsing R-CNN, using global semantic representation to enhance multi-scale features. Zhang et al.11 further introduced an instance-level network with edge-guided parsing for better differentiation of adjacent parts. One-stage top-down methods still face challenges in parsing small regions due to insufficient context.Fig. 1Overview of the proposed end-to-end framework for human parsing. The architecture comprises three main modules: Backbone, Neck, and Head, each labeled and color-coded to illustrate their roles in feature extraction, aggregation, and prediction.Table 1Hierarchical structure of the proposed lightweight network (input \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$640\times 640$$\end{document}).StageLayer (type)Output size\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C_{out}$$\end{document}ParamsBackbone0ConvBlock\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$320\times 320$$\end{document}323,5201ConvBlock\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$160\times 160$$\end{document}6418,5602ResBloc\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$160\times 160$$\end{document}6418,8163ConvBlock\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$80\times 80$$\end{document}12873,9844ResBloc (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\times 2$$\end{document})\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$80\times 80$$\end{document}128115,7125ConvBlock\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$40\times 40$$\end{document}256295,4246ResBloc (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\times 3$$\end{document})\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$40\times 40$$\end{document}256625,1527ConvBlock\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$20\times 20$$\end{document}5121,180,6728mSAM\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$20\times 20$$\end{document}5121,321,9849SPPF\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$20\times 20$$\end{document}512656,89610ConvBlock\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$20\times 20$$\end{document}256131,584Neck (PANet)11Upsample\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$40\times 40$$\end{document}256012Concat\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$40\times 40$$\end{document}512013ResBloc\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$40\times 40$$\end{document}256361,98414ConvBlock\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$20\times 20$$\end{document}12833,02415Upsample\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$40\times 40$$\end{document}128016Concat\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$40\times 40$$\end{document}256017ResBloc\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$40\times 40$$\end{document}12890,88018ConvBlock\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$20\times 20$$\end{document}128147,71219Concat\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$20\times 20$$\end{document}256020LRB\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$20\times 20$$\end{document}256216,44821ConvBlock\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$20\times 20$$\end{document}256590,33622Concat\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$20\times 20$$\end{document}512023ResBloc\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$20\times 20$$\end{document}5121,182,720Head24Segment Headmulti-scale–615,133Total Parameters7,680,541GFLOPs26.1 Overview of the proposed end-to-end framework for human parsing. The architecture comprises three main modules: Backbone, Neck, and Head, each labeled and color-coded to illustrate their roles in feature extraction, aggregation, and prediction. Table 1Hierarchical structure of the proposed lightweight network (input \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$640\times 640$$\end{document}).StageLayer (type)Output size\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C_{out}$$\end{document}ParamsBackbone0ConvBlock\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$320\times 320$$\end{document}323,5201ConvBlock\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$160\times 160$$\end{document}6418,5602ResBloc\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$160\times 160$$\end{document}6418,8163ConvBlock\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$80\times 80$$\end{document}12873,9844ResBloc (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\times 2$$\end{document})\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$80\times 80$$\end{document}128115,7125ConvBlock\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$40\times 40$$\end{document}256295,4246ResBloc (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\times 3$$\end{document})\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$40\times 40$$\end{document}256625,1527ConvBlock\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$20\times 20$$\end{document}5121,180,6728mSAM\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$20\times 20$$\end{document}5121,321,9849SPPF\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$20\times 20$$\end{document}512656,89610ConvBlock\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$20\times 20$$\end{document}256131,584Neck (PANet)11Upsample\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$40\times 40$$\end{document}256012Concat\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$40\times 40$$\end{document}512013ResBloc\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$40\times 40$$\end{document}256361,98414ConvBlock\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$20\times 20$$\end{document}12833,02415Upsample\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$40\times 40$$\end{document}128016Concat\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$40\times 40$$\end{document}256017ResBloc\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$40\times 40$$\end{document}12890,88018ConvBlock\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$20\times 20$$\end{document}128147,71219Concat\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$20\times 20$$\end{document}256020LRB\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$20\times 20$$\end{document}256216,44821ConvBlock\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$20\times 20$$\end{document}256590,33622Concat\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$20\times 20$$\end{document}512023ResBloc\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$20\times 20$$\end{document}5121,182,720Head24Segment Headmulti-scale–615,133Total Parameters7,680,541GFLOPs26.1 Hierarchical structure of the proposed lightweight network (input \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$640\times 640$$\end{document}). Transformer based human parsing The Transformer architecture, initially designed for NLP tasks, has seen significant success in computer vision, outperforming traditional neural networks. ViT38 showed strong performance in image classification, while DETR39 applied Transformer models to target detection using CNN-extracted image features. Self-attention mechanisms have emerged as a crucial component in human parsing. Song et al.40 introduced the Global Transformer Module (GTM) with a self-attention mechanism to capture long-range dependencies, focusing on single-human parsing. Guan et al.41 used a graph transformer for multi-human parsing, generating graph features to infer semantic correlations, though with some limitations. Yang et al.16 introduced Mask2Former Parsing that outperforms prior techniques but is computationally demanding for real-time use. Inspired by the success of self-attention, we adopt it in our human parsing approach, focusing on the encoder to capture global information. Object detection Two types of learning-based object detection methods are available: two-stage7,42 and single-stage17,43,44 network. Two-stage networks, such as R-CNN45, use complete connection layers to obtain proposals through a region proposal network (RPN), then classify and regress them in two steps. However, these approaches are computationally expensive and inefficient for real-time applications46. One-stage networks, such as YOLO17,47,48, can directly predict bounding boxes without a proposal stage, resulting in faster inference speed. While earlier one-stage detectors compromised accuracy, recent breakthroughs in the YOLO series have achieved competitive accuracy and efficiency. For example, YOLOv347 uses a single neural network to detect classes with high accuracy. Although numerous studies have reported on human parsing, existing methods either have high computational costs due to costly processing or rely on parallel branches, including edge prediction and human detection and need more context attention. Therefore, there is still need a method to learn local and global features, capture global dependencies, and perform end-to-end without requiring parallel models such as detection and edge prediction. Therefore, the method should be efficient and easy to deploy on real-time, edge, and resource-constrained devices. Inspired from the recent detection networks47,48, we propose a resource-efficient framework for real-time multi-human parsing with detection to address these issues. The proposed method directly predicts the bounding box and segment human instances, making it much simpler and more flexible. Our model can be categorized as a unified approach since the framework comprises multiple bottom-up and top-down paths. Methodology Overview of the architecture Our model comprises three principal components: a lightweight backbone for multi-scale feature extraction, a neck module for hierarchical feature fusion, and a unified Head responsible for both parsing and detection. Given an input image tensor \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$I \in \mathbb {R}^{H \times W \times 3}$$\end{document}, the backbone processes I through a series of convolutional transformations to yield intermediate feature maps at multiple spatial resolutions. This stage includes conventional convolutional blocks as well as our proposed modified Self-Attention Module (mSAM), which enriches the spatial context by capturing long-range dependencies. The resulting feature hierarchy is denoted by \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\{F_s\}$$\end{document}, where each \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$F_s$$\end{document} represents a feature map at scale s. These are subsequently passed to the neck module, which aggregates them via a combination of top-down and bottom-up pathways, producing a fused representation \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{F}$$\end{document}. Finally, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{F}$$\end{document} is fed into the Head, which jointly predicts semantic masks, bounding box coordinates, class labels, and confidence scores. As depicted in Figure 1, the overall architecture consists of three well-defined stages, Backbone, Neck, and Head—each of which is visually distinguished using color-coded blocks for clarity. Table 1 provides the detailed hierarchical structure of the proposed model, including the number of layers, output resolution, channel dimension \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C_{out}$$\end{document} and corresponding parameter counts (Params). Backbone network The backbone serves as the initial stage of the architecture and is responsible for extracting rich visual features from the input image at multiple spatial resolutions. It transforms the raw input \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$I \in \mathbb {R}^{H \times W \times 3}$$\end{document} into a set of feature maps \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\{F_s\}$$\end{document} using a series of convolutional and residual operations. This stage is designed to balance representational capacity with computational efficiency, making it well-suited for deployment on edge devices. Notably, we integrate the proposed mSAM module into this stage to enhance the model’s ability to capture global contextual dependencies, which is critical for accurate human parsing in complex scenes. The backbone of our architecture is composed of several essential modules: ConvBlock, Bottleneck, ResBlock, SPPF, and the mSAM attention module, each playing a crucial role in feature extraction and representation learning. The first four modules are derived from YOLOv5. The ConvBlock serves as the foundational building block, combining convolution, batch normalization (BN), and the SiLU activation function49. The Bottleneck module is a residual block that leverages two skip connections, facilitating enhanced gradient backpropagation and addressing the vanishing gradient problem. By adjusting the channel width through an expansion factor, the Bottleneck module improves the flow of information across the network, making it more effective at learning complex features. The ResBlock is a modified version of the CSP bottleneck50, which introduces multiple bottleneck modules and convolutions, connected via a shortcut structure. The SPPF (Spatial Pyramid Pooling Fast) module builds upon the original spatial pyramid pooling approach51, applying max-pooling with multiple kernel sizes. By pooling features at different resolutions, the SPPF module aggregates spatial information from varying scales, resulting in richer and more robust feature representations that are crucial for handling instances with diverse sizes and poses.Fig. 2The structure of modified self-attention module. The structure of modified self-attention module. Modified self-attention module (mSAM) The modified self-attention module (mSAM) is designed to enhance the global contextual understanding of feature maps, which is particularly beneficial for parsing spatially distributed body parts. Our design draws inspiration from DETR 39, but introduces two key architectural changes: (1) the use of a fixed, standalone 2D positional embedding rather than a learnable one, and (2) the removal of LayerNorm to improve memory and runtime efficiency. Figure 2 shows structure of Modified Self-Attention Module. Unlike DETR and other transformer-based models that apply a learnable linear layer for positional encoding, mSAM incorporates a fixed 2D sine-cosine positional encoding, denoted as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$P \in \mathbb {R}^{H \times W \times C}$$\end{document}. For each spatial location (x, y) in the feature map, a unique vector is assigned using sinusoidal functions at varying frequencies:1\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \begin{aligned} P_{(x,y),2i}&= \sin \left( \frac{x}{10000^{2i/C}}\right) , \\ P_{(x,y),2i+1}&= \cos \left( \frac{x}{10000^{2i/C}}\right) \end{aligned} \end{aligned}$$\end{document}2\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \begin{aligned} P_{(x,y),2j}&= \sin \left( \frac{y}{10000^{2j/C}}\right) , \\ P_{(x,y),2j+1}&= \cos \left( \frac{y}{10000^{2j/C}}\right) \end{aligned} \end{aligned}$$\end{document}This positional encoding P is directly added to the input feature map \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X \in \mathbb {R}^{H \times W \times C}$$\end{document} before the attention operation. Compared to learnable positional encodings, this formulation avoids additional parameterization, is resolution-invariant, and retains the spatial interpretability essential for human parsing. Additionally, in contrast to conventional attention modules, we omit LayerNorm layers in mSAM. This design decision is motivated by deployment efficiency. LayerNorm introduces additional memory cost for storing per-channel statistics. Also, it can become a bottleneck on resource-constrained devices like Raspberry Pi, which lack dedicated support for such normalization layers. In our experiments, the absence of LayerNorm resulted in a reduction in average memory usage (approximately 0.5%) and improved inference latency (by 5–7%) without affecting training stability or convergence. Neck module The neck is responsible for aggregating multi-scale features from the backbone, which is essential for handling instances at varying sizes. The neck component in the target segmentation framework comprises multiple bottom-up and top-down paths. The features obtained from the backbone network were also passed through the neck network for convolution and up-sampling to increase their dimensions. These features were concatenated and sent to the ResBlock network for segmentation and detection. This process was repeated to produce small, medium, and significant levels of feature maps.Fig. 3The structure of prediction head. The structure of prediction head. Head The Head module is designed to produce the final outputs for both segmentation and detection tasks, leveraging features aggregated by the neck network at multiple scales. This multi-scale approach ensures robust performance across small, medium, and large instances. The architecture is inspired by YOLOv5 and employs a dual-branch structure to handle segmentation and detection in a unified manner. Figure 3 shows the diagram of Head.Fig. 4The structure of light ResBlock (LRB). The structure of light ResBlock (LRB). The segmentation branch focuses on generating coarse, instance-independent segmentation masks. It employs a three-layer FCN with SiLU activation functions to produce a set of prototype masks that span the entire image. These prototype masks provide a foundational representation of potential object regions. The Detection branch extends traditional object detection by predicting instance-specific coefficients for each anchor. These coefficients encode the spatial and semantic information necessary to refine the prototype masks generated by Segmentation branch. Detection branch also outputs class labels, bounding box coordinates, and confidence scores for detected instances. The final masks are produced by linearly combining the instance-independent prototype masks from Segmentation branch with the instance-specific coefficients predicted by Detection branch. The combined outputs are further processed through Non-Maximum Suppression (NMS) to eliminate redundant detections and ensure high-quality results. The Detection branch outputs 24 channels per anchor, including 19 for class labels, 4 for bounding box coordinates, and 1 for confidence scores. With three anchors, this totals 72 channels for detection outputs. The Segmentation branch branch contributes an additional 32 channels for prototype masks. Combined, this results in 168 total output channels. To ensure that segmentation masks remain confined within their respective bounding boxes, a clipping operation is applied to prevent mask overflow. To improve accuracy and efficiency, we have replaced the standard ResBlock module with a Lightweight ResNet Block (LRB) in the medium-scale detection head. Light ResBlock (LRB): the LRB module replaces the traditional ResBlock to improve efficiency without sacrificing accuracy. The LRB design uses adaptive pooling and squeeze-and-excitation52 mechanisms to reduce computational overhead while ensuring that important features are emphasized. This design choice enhances performance and speeds up inference, making it suitable for real-time applications in human parsing. The LRB architecture comprises two consecutive convolutional blocks (Conv+BN+SiLU), followed by adaptive pooling and two fully connected (FC) layers. The FC layers consist of ReLU activation and sigmoid activation, respectively. A detailed view of the LRB architecture is illustrated in Fig. 4. The attention can be constructed based on a transformation function \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$F_{tr}$$\end{document}, which maps an input \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X\in \mathbb {R}^{W'\times H' \times C'}$$\end{document} to a feature map \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$U\in \mathbb {R}^{W\times H\times C}$$\end{document}, where W, H, and C denote the width, height, and number of channels, respectively. The output feature map can be represented as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$U=[U_1,U_2,\dots ,U_C]$$\end{document}, where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$U_c$$\end{document} is defined in Eq. (3).3\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} U_c = V_c * X = \sum _{s=1}^{C'}v_c^s*x^s \end{aligned}$$\end{document}Where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$V=[v_1,v_2,\dots ,v_C]$$\end{document}, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$*$$\end{document} represents the convolution operation. The output feature map U has dimensions \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$W\times H$$\end{document}, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$v_c^s$$\end{document} is a 2D spatial kernel that acts on the corresponding channel of X. The convolutional filters collect local spatial correlations, while the channel dependencies are implicitly encoded in \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$v_c$$\end{document} since the output is a sum of all channels. These dependencies are intertwined with the local spatial correlations, and by explicitly modeling them, we improve the network’s ability to learn more informative and discriminative features. This explicit modeling of channel dependencies enhances the overall feature extraction process, making the network more sensitive to key features that are critical for parsing and detection tasks. We have integrated LRB at the middle scale head of our network. Activation and loss functions The SiLU activation function was chosen for its smooth gradient, which aids in stable training and enables the model to capture both linear and non-linear relationships. For the loss function, we utilize cross-entropy loss with logits and Dice loss to accurately predict pixel-wise segmentation and ensure strong overlap with ground truth masks. The detection loss uses CIoU to measure the accuracy of bounding box predictions. Activation function: we use the Sigmoid Linear Unit (SiLU)49, also known as the Swish activation function. It was proposed as an alternative to the traditional activation functions like the sigmoid or ReLU53 functions. The SiLU and sigmoid functions can be defined as Eq. 4 and Eq. 5.4\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \text {SiLU}(x) = x \cdot \text {sigmoid}(x) \end{aligned}$$\end{document}5\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \text {sigmoid}(x) = \frac{1}{1 + e^{-x}} \end{aligned}$$\end{document}The SiLU function combines the input value’s linearity with the sigmoid function’s non-linearity. When the input value is positive, the sigmoid function approaches 1. When the input value is negative, the sigmoid function approaches 0, which suppresses the input value. This behavior allows the SiLU function to capture linear and non-linear relationships in the data. Besides, the SiLU function has a smooth gradient, facilitating more stable and efficient training of neural networks using backpropagation. Loss function: the loss function consists of a detection loss and a segmentation loss represented as Eq. 6 below.6\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \text {Loss}_T = \alpha \text {Loss}_{dt} + \beta \text {Loss}_{sg} \end{aligned}$$\end{document}where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\beta$$\end{document} are tuning parameters (we set \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha$$\end{document} = \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\beta$$\end{document} =1) that balance the total loss (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\text {Loss}_T$$\end{document}), \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\text {Loss}_{dt}$$\end{document} represents the detection loss, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\text {Loss}_{sg}$$\end{document} indicates the segmentation loss. The detection loss (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\text {Loss}_{dt}$$\end{document}) can be further elaborated as Eq. 7.7\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \text {Loss}_{dt} = \alpha _1 \text {loss}_{cl} + \alpha _2 \text {loss}_{ob} + \alpha _3 \text {loss}_{bx} \end{aligned}$$\end{document}where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\text {loss}_{cl}$$\end{document} represents the loss for classifying the class, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\text {loss}_{ob}$$\end{document} represents the loss for the object confidence. Both of these losses are binary cross-entropy losses, which reduce the slope of the loss function and focus on misclassified examples. Besides, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\text {loss}_{bx}$$\end{document} measures the overlap between the ground truth box and the predicted box by computing the Complete Intersection over Union (CIoU). The segmentation loss (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\text {Loss}_{sg}$$\end{document}) is calculated using a combination of cross-entropy loss with logits and dice loss. Cross-Entropy Loss with Logits loss function quantifies the dissimilarity between predicted pixel-wise logits and ground truth labels. It guides the model to produce probability distributions that closely resemble the actual object masks. Dice loss measures the overlap between predicted and ground truth masks. It significantly contributes to the accuracy of the segmentation output. Model optimization During the process of optimizing our model for human parsing, we faced the typical challenge of balancing high accuracy with fast processing speeds in deep learning applications. While recent advancements have significantly improved accuracy, the need for faster model execution has driven the adoption of techniques such as quantization. Quantization reduces the precision of a model’s weights and activations, lowering memory requirements and computational overhead. This approach enables faster processing while incurring only a minor trade-off in accuracy. To overcome these challenges, we applied post-training static quantization similar as17. Specifically, we converted our parsing model from 32-bit floating-point precision (float32) to 16-bit floating-point precision (float16), achieving reduced precision and improved efficiency. Following this optimization step, we converted the model to the ONNX format for deployment on Raspberry Pi (specifically, Raspberry Pi 4). ONNX, known as Open Neural Network Exchange, provides a framework-agnostic representation of neural networks, facilitating efficient deployment across diverse hardware platforms54. Notably, all operators utilized in our model conform to standard deep learning libraries and are ONNX-compatible, ensuring seamless exportability to ONNX and enabling deployment across platforms. This standalone ONNX model is executable using ONNX Runtime, further streamlining deployment processes. Data generation and experiment Data generation: to assess the effectiveness of our method, we use two publicly available datasets, e.g., CIHP27 and Look Into Person (LIP)55. The CIHP dataset consists of 38280 diverse image data: training (28280), validation(5000), and test (5000). It has 20 pixel-wise human semantic parts and instance-level identification. On the other hand, the LIP dataset focused on a person’s semantic level annotation. It has 50,000 image data with detailed pixel-by-pixel annotations for 16 critical places in 2D human postures and 19 semantic human component labels. The segmentation ground truth is usually provided through dense pixel-wise annotations. However, for simultaneously parsing and detection, we provided the model bounding polygon annotation. Thus, a conversion tool is implemented to convert the segmentation masks into bounding polygon annotation. First, we separate each annotated class and convert them to binary masks based on their pixels, as shown in Fig. 5. To convert a binary mask to bounding polygon, we identify the mask boundaries using the canny edge detector and produce a binary edge image. Next, we find the boundary of the binary edge image. To simplify the contours, we apply the Douglas-Peucker56 to reduce the number of points representing the contour while preserving the object’s shape. The simplified contour is then converted to a polygon by removing holes present using the convex hull function. We repeat this process for all contours. Each resulting polygon is represented as a list of vertices. Figure 6 shows the quantity of each instance for both datasets.Fig. 5A visualization of each separated part of instances and their corresponding polygon coordinates.Fig. 6The details of instances of CIHP (left) and LIP (right) dataset. A visualization of each separated part of instances and their corresponding polygon coordinates. The details of instances of CIHP (left) and LIP (right) dataset. Experiment details: our method was implemented using the PyTorch (1.10) library with an image size of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$640\times 640$$\end{document}. We employed data augmentation techniques such as geometric and lighting distortion to enhance the training data. The Adam optimizer was used in all experiments. Training was performed for 100 epochs with a batch size of 4. Most experiments were conducted on a workstation running Windows OS equipped with an NVIDIA GeForce RTX 2070 GPU (8 GB VRAM) and 32 GB RAM. For the final set of experiments evaluating the effect of different activation functions and loss functions, we used a workstation with an NVIDIA GeForce RTX 3050 GPU (4 GB VRAM) under the same experimental setup. All experiments were conducted in 32-bit floating-point precision (float32) unless otherwise stated. Evaluation matrices: we report our performance based on two popular matrices, the mean Intersection over Union (mIoU) and mean average precision (mAP). The mIoU is used for measuring segmentation performance due to its simplicity and effectiveness57. The intersection over Union (IoU) quantifies the degree of overlap between the predicted output and the target mask. This metric is calculated using Eq. 8.8\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} IoU = \frac{X \cap Y}{X \cup Y} \end{aligned}$$\end{document}where X and Y refer to the ground truth and predicted masks, respectively. The union operation in \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X \cup Y$$\end{document} refers to all mask pixels, while the intersection in \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X \cap Y$$\end{document} refers to pixels in both masks. To obtain the average IoU score across all categories for our semantic prediction, the IoU score is computed independently for each class. The mAP is an extension of Average Precision (AP) commonly used in image segmentation tasks to evaluate the model performance across multiple object categories. It is calculated by independently computing the AP value for each category before the mean of these values across all categories. Thus, the mAP value represents the average AP across all object categories, as shown in Eq. 9.9\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} mAP = \frac{1}{C}\sum _{i=1}^{C}AP_i \end{aligned}$$\end{document}where C is the total object categories and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$AP_i$$\end{document} is the AP value for i category.Table 2Computational performance and efficiency comparison. Bold and italics text indicate the best and second-best performances, respectively.MethodsYearBackbonesParam. (M)mIoUFPSPGN272018ResNet101629.254.44.2Graphonomy122019Xception156.955.525.0Graphy-ml132020Xception176.056.216.6Li et al.582024ResNet5046.763.217.2Parsing R-CNN52019ResNet5054.356.37.4RP RCNN102020ResNet5058.458.35.0AIParsing112022ResNet5050.259.78.9Our’s2024Darknet53-TR46.664.140.3Our’s2024ResNet5050.563.236.4Our’s2024ResNet10169.563.925.4 Table 2Computational performance and efficiency comparison. Bold and italics text indicate the best and second-best performances, respectively.MethodsYearBackbonesParam. (M)mIoUFPSPGN272018ResNet101629.254.44.2Graphonomy122019Xception156.955.525.0Graphy-ml132020Xception176.056.216.6Li et al.582024ResNet5046.763.217.2Parsing R-CNN52019ResNet5054.356.37.4RP RCNN102020ResNet5058.458.35.0AIParsing112022ResNet5050.259.78.9Our’s2024Darknet53-TR46.664.140.3Our’s2024ResNet5050.563.236.4Our’s2024ResNet10169.563.925.4 Computational performance and efficiency comparison. Bold and italics text indicate the best and second-best performances, respectively. Results and discussion Computational performance analysis To evaluate the instance-level human parsing efficiency, we conducted experiments on the CIHP validation set and compared our method with state-of-the-art methods including four bottom-up methods (PGN27, Graphy-ml13, Li et al.58 and Graphonomy12) and three one-stage top-down methods (Parsing R-CNN5, RP R-CNN10, and AIParsing11). Table 2 summarizes the performance metrics, including mean Intersection over Union (mIoU), inference speed (frames per second), and model size (no. of parameters) for each method. Our model received a higher mean Intersection over Union (mIoU) score compared to both bottom-up and one-stage top-down approaches. Moreover, our method is about \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$2 \times$$\end{document} faster than the closest Graphonomy12 method without compromising the model size (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$3 \times$$\end{document} smaller). To evaluate the flexibility and robustness of our model, we conducted tests using both ResNet50 and ResNet101 as backbones. The results show that our model performs similarly well with both backbones, highlighting its robust performance across different architectures. However, the computational benefits of ResNet101 come at the expense of increased model complexity and resource requirements. Overall, this reflects a trade-off between model complexity, size, speed, and parsing accuracy. Our model strikes a better efficiency balance compared to other reported models.Table 3Quantitative comparison on the CIHP validation set. Bold and italics text indicate the best and second-best performances, respectively.MethodsYearApproachesBackboneEpochmIOUPGN+272018Bottom-upResNet-1018057.3DeepLab v3+592018Bottom-upXception10058.9Graphonomy+122019Bottom-upXception10058.6CorrPM282020Bottom-upResNet-10115060.2GPM +132020Bottom-upXception10060.3Grapy ML +132020Bottom-upXception20060.6PCNet292020Bottom-upResNet-10112061.1Guan et al.,412022Bottom-upViT15060.2PRM302022Bottom-upResNet-10112062.6Li et al.582024Bottom-upResNet5015063.2BraidNet312019Two-stage top-downResNet-10115060.6SemaTree322020Two-stage top-downResNet-10120060.9M-CE2P222019Two-stage top-downResNet-10115063.6SCHP +332020Two-stage top-downResNet-10115067.5M2FP162023Two-stage top-downViT15069.1Unified92019One-stage top-downResNet-1013755.2Parsing R-CNN52019One-stage top-downResNet-507556.3Mask Scoring362019One-stage top-downResNet-507556.0BMask R-CNN372020One-stage top-downResNet-507556.4RP RCNN102020One-stage top-downResNet-507558.2AIParsing112022One-stage top-downResNet-1017560.7ReSParser602023One stage top-downResNet1017558.9Our’s2024UnifiedDarknet53-TR10064.1Our’s2024UnifiedResNet5010063.2Our’s2024UnifiedResNet10110063.9Table 4Quantitative comparison on the LIP validation set. Bold and italics text indicate the best and second-best performances, respectively.MethodYearBackboneEpochsmIoUMMAN612018ResNet1013046.93MuLA622018Hourglass25049.30JPPNet632018ResNet1016051.37CE2P222019ResNet10115053.10CNIF642019ResNet10115057.74BraidNet312019ResNet10115054.42SLRS332020ResNet10115056.34HHP192020ResNet10115059.25DTCF202020ResNet10120057.82SemaTree322020ResNet10120054.73SCHP332020ResNet10115059.33HRNet652020HRNetW4815055.90HTCorrM212021HRNetW4818056.85HIPN232021ResNet10115059.61PRM302022ResNet10112058.86QANet342022HRNetW4814059.61PAGRnet252022HRNetW4815059.85Song et al.402022ResNet-10115054.50CDGNet242022ResNet10115060.30M2FP162023ViT15059.86Chen et al.262023Swin-S10060.21Zhang et al.662023Swin15061.26Our’s2024Darknet53-TR10064.44Our’s2024ResNet5010063.95 Table 3Quantitative comparison on the CIHP validation set. Bold and italics text indicate the best and second-best performances, respectively.MethodsYearApproachesBackboneEpochmIOUPGN+272018Bottom-upResNet-1018057.3DeepLab v3+592018Bottom-upXception10058.9Graphonomy+122019Bottom-upXception10058.6CorrPM282020Bottom-upResNet-10115060.2GPM +132020Bottom-upXception10060.3Grapy ML +132020Bottom-upXception20060.6PCNet292020Bottom-upResNet-10112061.1Guan et al.,412022Bottom-upViT15060.2PRM302022Bottom-upResNet-10112062.6Li et al.582024Bottom-upResNet5015063.2BraidNet312019Two-stage top-downResNet-10115060.6SemaTree322020Two-stage top-downResNet-10120060.9M-CE2P222019Two-stage top-downResNet-10115063.6SCHP +332020Two-stage top-downResNet-10115067.5M2FP162023Two-stage top-downViT15069.1Unified92019One-stage top-downResNet-1013755.2Parsing R-CNN52019One-stage top-downResNet-507556.3Mask Scoring362019One-stage top-downResNet-507556.0BMask R-CNN372020One-stage top-downResNet-507556.4RP RCNN102020One-stage top-downResNet-507558.2AIParsing112022One-stage top-downResNet-1017560.7ReSParser602023One stage top-downResNet1017558.9Our’s2024UnifiedDarknet53-TR10064.1Our’s2024UnifiedResNet5010063.2Our’s2024UnifiedResNet10110063.9 Quantitative comparison on the CIHP validation set. Bold and italics text indicate the best and second-best performances, respectively. Table 4Quantitative comparison on the LIP validation set. Bold and italics text indicate the best and second-best performances, respectively.MethodYearBackboneEpochsmIoUMMAN612018ResNet1013046.93MuLA622018Hourglass25049.30JPPNet632018ResNet1016051.37CE2P222019ResNet10115053.10CNIF642019ResNet10115057.74BraidNet312019ResNet10115054.42SLRS332020ResNet10115056.34HHP192020ResNet10115059.25DTCF202020ResNet10120057.82SemaTree322020ResNet10120054.73SCHP332020ResNet10115059.33HRNet652020HRNetW4815055.90HTCorrM212021HRNetW4818056.85HIPN232021ResNet10115059.61PRM302022ResNet10112058.86QANet342022HRNetW4814059.61PAGRnet252022HRNetW4815059.85Song et al.402022ResNet-10115054.50CDGNet242022ResNet10115060.30M2FP162023ViT15059.86Chen et al.262023Swin-S10060.21Zhang et al.662023Swin15061.26Our’s2024Darknet53-TR10064.44Our’s2024ResNet5010063.95 Quantitative comparison on the LIP validation set. Bold and italics text indicate the best and second-best performances, respectively. Quantitative performance We compare our results with state-of-the-art bottom-up, one-stage, and two-stage top-down parsing methods. Table 3 presents the quantitative evaluation on the CIHP validation set. The proposed method performs better than all one-stage top-down and bottom-up approaches. It also shows a comparable performance compared to two-stage top-down approaches. For example, the PRM30 achieves the best mIoU (62.6) among the bottom-up methods, and Graphy-ml13 is the second best at 60.6. Our method received around 3% and 6% better than the PRM and Graphy-ml methods, respectively. Besides, we receive around 4% better mIoU than recent AIParsing11 that shows the best result among all one-stage top-down methods. Although two-stage top-down methods SCHP+33 and M2FP16 a performs better than the proposed method, it significantly suffers from the flexibility and inference cost applied to real-world applications16.Table 5Performance comparison of our lightweight model on the LIP and CIHP datasets (first 1000 images).ModelDatasetInference (ms)ONNX (.onnx)LIP1538PyTorch (.pt)2444ONNX (.onnx)CIHP1514PyTorch (.pt)2875 Table 5Performance comparison of our lightweight model on the LIP and CIHP datasets (first 1000 images).ModelDatasetInference (ms)ONNX (.onnx)LIP1538PyTorch (.pt)2444ONNX (.onnx)CIHP1514PyTorch (.pt)2875 Performance comparison of our lightweight model on the LIP and CIHP datasets (first 1000 images). Fig. 7Real-time human parsing visualization on Raspberry Pi 4 generated by our approach. Real-time human parsing visualization on Raspberry Pi 4 generated by our approach. Table 4 showcases the quantitative evaluation of our method on the LIP validation set, where it significantly outperforms state-of-the-art approaches. For example, we achieve a mIoU of 64.44, while the closest methods, CDGNet24 and HIPN23, obtain 60.30 and 59.61, respectively. One of the reasons for this performance is to use mSAM that effectively captures the global context and aids to accurately parsing. We did not utilize test time augmentation, which is more time-consuming than regular testing and makes the model inefficient for resource-constrained devices. This choice was intentional, as we aimed to balance performance and efficiency. We also implemented our model using a ResNet50 backbone, achieving an mIoU of 63.95. This demonstrates the robust performance of our approach and validates the effectiveness of our method, regardless of the backbone used. As shown in the Tables, our method consistently outperforms existing approaches on both the LIP and CIHP datasets. In particular, the proposed model achieved higher segmentation accuracy while maintaining competitive inference speed, confirming its effectiveness for real-world applications. These improvements highlight the advantage of our lightweight design, which balances accuracy and efficiency more effectively than prior works.Fig. 8Qualitative comparison of our method with state-of-the-art methods on CIHP dataset. Row 1, 2: compared with top-down approaches, row 3, 4: bottom-up approaches. Qualitative comparison of our method with state-of-the-art methods on CIHP dataset. Row 1, 2: compared with top-down approaches, row 3, 4: bottom-up approaches. Performance on resource-constrained devices (Raspberry Pi 4) A resource-constrained device refers to an embedded system characterized by limited computing, storage, energy, and networking capabilities. To evaluate the suitability of our model for deployment on such devices, we conducted tests on a Raspberry Pi 4 (equipped with 4GB RAM but lacking a GPU, with a power consumption of approximately 2.5-5 watts, depending on the workload and attached peripherals). Our lightweight model demonstrated successful execution and compatibility on this platform. We evaluated the model’s performance using 1000 images (first 1000) from the CIHP and LIP datasets, each with a resolution of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$640\times 640$$\end{document}. Both PyTorch and ONNX versions of the model were evaluated, and the results are summarized in Table 5. Notably, the ONNX model exhibited superior inference speeds compared to the PyTorch model, with performance improvements of approximately 37% for the LIP dataset and 47% for the CIHP dataset. It is noteworthy that when tested on our primary environment (GPU-accelerated), both the ONNX and PyTorch models maintained their performance without degradation, albeit with a slight increase in inference time for the ONNX model (approximately 4 ms longer per image) in both datasets. For instance, while the PyTorch model required 9.3 ms per image inference on the CIHP dataset, the ONNX model took 13.6 ms. Furthermore, we deployed our ONNX models (trained on the CIHP dataset) on real-world data captured by the Raspberry Pi 4 (with a resolution of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$640\times 480$$\end{document}). The experimental results, depicted in Fig. 7, illustrate the model’s capability to accurately parse single and multiple humans, generating corresponding masks. However, it is important to note that a degradation in inference speed was observed when processing live video streams. For instance, while inference on CIHP data took approximately 2875 ms per image, it required around 3500 ms per frame for video processing. Despite the reduced inference speed on resource-constrained devices, our model maintained consistent accuracy and quality. This compatibility extends the applicability of our approach to a wide range of low-computing IoT devices and smartphones. Qualitative performance In Figure 8, we conduct a qualitative comparison between our method and state-of-the-art one-stage top-down (e.g., Parsing R-CNN5, RP R-CNN10, AIParsing11) and bottom-up (e.g., DeeplabV359, Gong et al.27, and Graphy-ml13) human parsing methods on the CIHP validation set. Our proposed model exhibits notable advantages, providing relatively more complete regions than other methods, particularly in challenging scenes such as occlusion or appearance approximation. For instance, Parsing R-CNN5, RP R-CNN10, and AIParsing11 fail to parse the biker’s right leg due to occlusion (as seen in the white elliptical region in the first row). In contrast, our method accurately and comprehensively parses it. Similarly, our approach demonstrates superior performance in predicting high-quality parsing results for symmetric regions, such as the left and right legs shown in the second row. Compared to bottom-up approaches, such as DeepLab v3+59, Gong et al.27, and Graphy-ml13, our method exhibits better results. For instance, these methods struggle to parse the player’s lower part in the third row due to the complex background, and they miss the second person’s left leg in the fourth row. Additionally, we present further qualitative results in Fig. 9, obtained from the validation sets of CIHP and LIP. These results underscore the effectiveness of our model across a spectrum of challenging scenarios.Fig. 9Enhanced human parsing results with detection by our model insights from LIP (first row) and CIHP (last row) datasets.Fig. 10Qualitative visualization of human parsing generated by our model on randomly selected YouTube video (first two rows) and real-time data (last row).Fig. 11Enhancing human body parsing with mSAM: impact analysis and performance comparison.Table 6Effect of different modules on the LIP dataset (trained for 5 epochs).ModuleParam.FLOPsPmAP50LatencyCBAM37M136.262.334.325.0 msmSAM (base)46M141.366.647.327.6 msmSAM46M141.068.648.727.4 msTable 7Effect of network size and model depth on performance: LIP and CIHP datasets.Model#Layers#ParametersBox(P)Box(R)Mask (P)Mask(R)mIoUFPSDatasetM. sizeModel49646.6M66.758.367.858.164.445.7LIP88MBModel(light)2907.7M64.553.364.953.059.7140.8LIP14MBModel49646.6M73.560.773.559.664.140.3CIHP88MBModel(light)2907.7M66.851.966.950.658.6106.4CIHP14MBTable 8Effect of different backbones on performance: LIP and CIHP datasets (trained for 20 epochs).BackboneMask (P)Mask(R)mIoUFPSDatasetMViT58.740.654.260LIPEfficientNet61.146.156.555LIPOur’s63.850.659.345LIPMViT58.730.548.854CIHPEfficientNet64.143.855.350CIHPOur’s66.348.858.540CIHP Enhanced human parsing results with detection by our model insights from LIP (first row) and CIHP (last row) datasets. Qualitative visualization of human parsing generated by our model on randomly selected YouTube video (first two rows) and real-time data (last row). Enhancing human body parsing with mSAM: impact analysis and performance comparison. Table 6Effect of different modules on the LIP dataset (trained for 5 epochs).ModuleParam.FLOPsPmAP50LatencyCBAM37M136.262.334.325.0 msmSAM (base)46M141.366.647.327.6 msmSAM46M141.068.648.727.4 ms Effect of different modules on the LIP dataset (trained for 5 epochs). Table 7Effect of network size and model depth on performance: LIP and CIHP datasets.Model#Layers#ParametersBox(P)Box(R)Mask (P)Mask(R)mIoUFPSDatasetM. sizeModel49646.6M66.758.367.858.164.445.7LIP88MBModel(light)2907.7M64.553.364.953.059.7140.8LIP14MBModel49646.6M73.560.773.559.664.140.3CIHP88MBModel(light)2907.7M66.851.966.950.658.6106.4CIHP14MB Effect of network size and model depth on performance: LIP and CIHP datasets. Table 8Effect of different backbones on performance: LIP and CIHP datasets (trained for 20 epochs).BackboneMask (P)Mask(R)mIoUFPSDatasetMViT58.740.654.260LIPEfficientNet61.146.156.555LIPOur’s63.850.659.345LIPMViT58.730.548.854CIHPEfficientNet64.143.855.350CIHPOur’s66.348.858.540CIHP Effect of different backbones on performance: LIP and CIHP datasets (trained for 20 epochs). Performance on real-time live data To demonstrate the practicality of our work in real-time applications, we evaluated it using real-world test data. Specifically, we tested our method with randomly selected YouTube videos and live webcam data to analyze its performance in diverse settings. The results from YouTube videos, depicted in Fig. 10 (first two rows), were highly promising. Our model demonstrated precise predictions for each frame while achieving an impressive inference speed of 63.6 frames per second (FPS) on a standard desktop GPU (details in the Experiment Details). Furthermore, we tested the model on live webcam-based data (resolution: 2 MP, frame size: \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$640\times 480$$\end{document} pixels, frame rate: 30 FPS). The outcomes, shown in Fig. 10 (last row), demonstrate that our approach maintained robust predictions despite varying lighting conditions and being tested on data not included in the training and validation sets. These outcomes showcase the effectiveness and efficiency of our approach in real-world scenarios, especially in real-time applications. Ablation study In this study, we extensively discuss the effectiveness of different modules in our method. Effect of mSAM Our model encountered difficulty accurately recognizing overlapped, crowded, and ambiguous human body regions, such as dresses and various leg parts (Fig. 11, 1st row, 2nd image), due to its reliance solely on convolutional blocks, which lacked effective global context. To overcome this limitation, we introduced a modified self-attention module (mSAM) into our architecture. This enhancement enabled the model to capture long-range dependencies and intricate relationships between different body parts, resulting in a significant improvement in parsing human bodies (Fig. 11, 1st row, 3rd image). Additionally, We analyzed the network’s features (taking after SPPF layer) with and without mSAM, revealing mSAM’s superior pattern capture (Fig. 11, 2nd row, 2nd & 3rd image). To further support our claim that the proposed mSAM module is computationally efficient, we compared its performance and complexity against widely used attention mechanisms, namely CBAM 67, as summarized in Table 6. The comparison includes key metrics such as the number of parameters (in millions), floating-point operations (FLOPs, in GFLOPs), latency (in milliseconds), precision, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$mAP_{50}$$\end{document}, all evaluated on the LIP validation set after 5 epochs of training. While mSAM introduces slightly more parameters and FLOPs compared to CBAM (46M vs. 37M parameters and 141.0 vs. 136.2 GFLOPs), it achieves substantially better accuracy—improving precision by over 6 percentage points and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$mAP_{50}$$\end{document} by more than 14 points. Notably, the latency difference between mSAM and CBAM is modest (27.4 ms vs. 25.0 ms), demonstrating that the proposed module maintains real-time capability despite increased representational capacity. Furthermore, we include a baseline variant, mSAM(base), which omits certain optimizations such as the simplified positional embedding and LayerNorm removal. Comparing mSAM to mSAM(base), we observe that our final design yields both higher accuracy and slightly reduced latency and FLOPs, highlighting the efficiency of our modifications. These results indicate that mSAM offers a favorable trade-off between computational cost and predictive performance, making it well-suited for deployment on resource-constrained devices. Effect of LRB We have replaced the conventional ResBlock with our Lightweight ResBlock (LRB), resulting in a notable 1-2% improvement in model performance while reducing the total number of parameters by approximately 1 million. For instance, when we substituted the traditional ResBlock with LRB in our larger model, the number of parameters decreased from 47 million to 46 million, and the GFLOPs reduced from 143 to 140. This not only streamlined our model but also boosted performance to an impressive 73.1 (precision) from the previous 72.1. Network depth and design In our study, we explored various network designs and depths for our model and observed a strong correlation between network depth and model performance, as depicted in Table 7. Specifically, we experimented with reducing the model depth by one-third and decreasing the width multiplier by half. Reducing the model depth by one-third entailed reducing the number of repetitions of ResBlocks. In our standard model, we consecutively repeated each ResBlock three, six, or nine times in the backbone, and three times each in the neck and head. Similarly, the mSAM and LRB were repeated three times. However, in the light version, we reduced these repetitions by a factor of three. Decreasing the width multiplier by half involved halving the number of output channels compared to the standard model. For example, in the standard model, the output channel of mSAM was 1024, whereas in the light version, it was reduced to 512. However, while reducing the model size significantly improved inference speed, it also led to a sacrifice in performance, as illustrated in Table 7. Our investigation underscores the importance of carefully considering the trade-offs associated with selecting the appropriate network depth.Table 9Effect of different activation and loss functions on performance (LIP valid. set, trained light model for 10 epochs).Loss functionBox (P)Box (R)Box (mAP50)Mask (P)Mask (R)Mask (mAP50)Memory usageTrain time /epochCE60.947.347.361.346.546.72.5G24 min.Focal65.340.438.964.640.138.42.4G23 min.Vari-focal65.440.538.965.540.338.72.4G32 min. Activ. functionBox (P)Box (R)Box (mAP50)Mask (P)Mask (R)Mask (mAP50)Memory usageInf. /imageSiLU60.947.347.361.346.546.72.5G17.2 msReLU67.643.243.967.872.843.32.3G15.7 msFReLU59.740.740.859.040.340.23.2G33.5 ms Table 9Effect of different activation and loss functions on performance (LIP valid. set, trained light model for 10 epochs).Loss functionBox (P)Box (R)Box (mAP50)Mask (P)Mask (R)Mask (mAP50)Memory usageTrain time /epochCE60.947.347.361.346.546.72.5G24 min.Focal65.340.438.964.640.138.42.4G23 min.Vari-focal65.440.538.965.540.338.72.4G32 min. Activ. functionBox (P)Box (R)Box (mAP50)Mask (P)Mask (R)Mask (mAP50)Memory usageInf. /imageSiLU60.947.347.361.346.546.72.5G17.2 msReLU67.643.243.967.872.843.32.3G15.7 msFReLU59.740.740.859.040.340.23.2G33.5 ms Effect of different activation and loss functions on performance (LIP valid. set, trained light model for 10 epochs). Effect of backbone We evaluated the impact of different backbones on the LIP and CIHP datasets over 20 epochs. The backbones considered were MobileViT (MViT)68 and EfficientNet69. The rationale for selecting EfficientNet and MobileViT as comparison backbones lies in their efficiency. The results, summarized in Table 8, highlight the variations in performance metrics across these backbones. Comparing the metrics across the backbones, our hybrid backbone (get merit from CNN and ViT) consistently outperforms both EfficientNet69 and MobileViT68 in terms of precision, recall, and mIoU across both datasets. For instance, our method achieves approximately 9% and 20% higher performance than MViT on the LIP and CIHP datasets, respectively. Similarly, it outperforms EfficientNet by 5% and 6% on the LIP and CIHP datasets, respectively. Although our method exhibits slightly slower processing times compared to MViT and EfficientNet, the performance gains are substantial and thus justifiable.Table 10Inference latency (in milliseconds) comparison between FP32 and FP16 model precisions on CIHP and LIP datasets.Model prec.CIHP datasetLIP datasetFull prec. (FP32)7.6 ms6.7 msHalf prec. (FP16)4.8 ms4.8 ms Table 10Inference latency (in milliseconds) comparison between FP32 and FP16 model precisions on CIHP and LIP datasets.Model prec.CIHP datasetLIP datasetFull prec. (FP32)7.6 ms6.7 msHalf prec. (FP16)4.8 ms4.8 ms Inference latency (in milliseconds) comparison between FP32 and FP16 model precisions on CIHP and LIP datasets. Activation and loss function The choice of both loss and activation functions plays a decisive role in determining model accuracy and efficiency. As summarized in Table 9, three loss functions were evaluated. Cross-Entropy (CE) loss achieved balanced results, with box precision (60.9), recall (47.3), and mean average precision (mAP50) of 47.3, while mask-level performance showed a similar balance (precision 61.3, recall 46.5, mAP50 46.7). In contrast, Focal and Vari-focal losses improved box precision to 65.3 and 65.4, respectively, but this came at the cost of substantially reduced recall (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\approx$$\end{document}40) and lower mAP50 (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\approx$$\end{document}39) at both the box and mask levels. Moreover, Vari-focal loss increased the training time per epoch from 24 minutes (CE) to 32 minutes, without providing meaningful accuracy gains. These findings suggest that CE offers the most effective trade-off between precision and recall while maintaining a reasonable computational cost, and it was therefore adopted as the default loss function. Regarding activation functions, SiLU provided stable results across precision, recall, and mAP50 (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\approx$$\end{document}47 for both box and mask) with moderate memory usage (2.5G) and inference time (17.2 ms per image). ReLU achieved the highest precision values (Box P = 67.6, Mask P = 67.8), but its recall was inconsistent (Box R = 43.2, Mask R = 72.8), resulting in lower mAP50 values (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\approx$$\end{document}43–44). FReLU required the highest memory (3.2G) and significantly degraded accuracy, with both box and mask mAP50 dropping to \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\approx$$\end{document}40. Considering these results, SiLU was retained as the default activation function, since it ensured balanced predictive performance without excessive computational overhead. Effect of model precision We conducted experiments to evaluate the effect of numerical precision on inference speed while maintaining model accuracy. Specifically, we compared full precision (FP32) and half precision (FP16) inference using the same architecture (our light model). As summarized in Table 10, we observed no degradation in segmentation accuracy when switching from FP32 to FP16. However, inference latency decreased significantly, highlighting the advantage of half-precision execution for real-time applications on edge devices. For instance, the inference time on the CIHP dataset decreased from 7.6 ms to 4.8 ms when using FP16, and a similar improvement was observed on the LIP dataset.Fig. 12Examples of failure cases by our approach. Examples of failure cases by our approach. Overall, the experimental analysis confirms that the proposed framework achieves a strong balance between segmentation accuracy, computational efficiency, and deployment flexibility. The ablation studies validated our design choices for loss and activation functions, while the precision analysis and deployment results demonstrate the practicality of the model for real-time applications on both high-performance GPUs and low-power edge devices. Failure cases Despite the overall success of our method, there are instances where the model fails to produce accurate parsing results, as illustrated in Fig. 12. These failures occur under several conditions: when the human region closely resembles other objects, when there is extensive overlap between instances, and when clothing items have a similar appearance. In particular, the top row of Fig. 12 demonstrates cases where the model struggles due to the background color being too similar to the human instance, leading to incorrect parsing. The second row showcases failures resulting from significant overlap between two human instances, causing the model to misidentify the boundaries of each individual. The addition of similar samples into the dataset may address these issues and provide richer annotations. Conclusion In this paper, we introduces a novel human parsing method designed to accurately detect and parse human body and clothing. Our model achieves end-to-end performance with a significantly faster speed than current state-of-the-art methods, eliminating the need for parallel modules such as detection and edge prediction. By incorporating the mSAM module, which utilizes self-attention to capture long-range dependencies, our model effectively generates superior global representations and distinguishes between symmetric regions. We implemented our model on resource-constrained devices and evaluated its performance qualitatively and quantitatively on multiple datasets, with experimental results validating its effectiveness. However, our method faces challenges in scenarios with highly overlapping regions and color ambiguity between instances and backgrounds. Potential solutions to address these challenges include enriching annotations, improving the model’s capacity to handle overlapping instances and similar colors, and considering the interplay between various human instances.