Identifying Ethical Biases in Action Recognition Models

Ana Băltărețu, Pascal Benschop, Jan van Gemert · 2026

Read Identifying Ethical Biases in Action Recognition Models by Ana Băltărețu, Pascal Benschop, Jan van Gemert (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-17971.txt.

Introduction

Baltaretu and two colleagues audit fairness in human action recognition models, which increasingly serve in high-stakes settings yet have rarely been checked across appearances. Using synthetic video with full control over visual identity traits such as skin color, they vary one trait at a time while keeping motion identical and preserving temporal consistency. Through the BEDLAM simulation platform, they test whether popular models show statistically significant skin-color bias. Results reveal systematic errors across groups, showing models can encode unwanted visual associations.

Catalog record

CREATORAna Băltărețu, Pascal Benschop, Jan van Gemert
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE42407
LICENSEhttp://creativecommons.org/licenses/by-sa/4.0/
RIGHTSCC BY SA
SOURCEarXiv

Source: https://arxiv.org/abs/2604.17971

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Identifying Ethical Biases in Action Recognition Models Ana Băltărețu, Pascal Benschop, Jan van Gemert Abstract Human Action Recognition (HAR) models are increasingly deployed in high-stakes environments, yet their fairness across different human appearances has not been analyzed. We introduce a framework for auditing bias in HAR models using synthetic video data, generated with full control over visual identity attributes such as skin color. Unlike prior work that focuses on static images or pose estimation, our approach preserves temporal consistency, allowing us to isolate and test how changes to a single attribute affect model predictions. Through controlled interventions using the BEDLAM simulation platform, we show whether some popular HAR models exhibit statistically significant biases on the skin color even when the motion remains identical. Our results highlight how models may encode unwanted visual associations, and we provide evidence of systematic errors across groups. This work contributes a framework for auditing HAR models and supports the development of more transparent, accountable systems in light of upcoming regulatory standards. Keywords: Human Action Recognition, Synthetic dataset, Bias analysis. / Video 1 Left: lighter skin tone, right: darker skin tone, prediction change from “cartwheeling” to “capoeira”. Video 2 Left: lighter skin tone, right: darker skin tone, prediction change from “jumpstyle dancing” to “juggling soccer ball”. Figure 1 : Qualitative analysis showcasing potential racial bias in action recognition models. Predicted labels per video at the bottom right of the frame. In Video 1 (top row), all sequences depict the same motion performed in the same environment, with only the actor’s skin tone changed. The model correctly labels the lighter-skinned person as “cartwheeling”, while misclassifying the darker-skinned person as performing “capoeira” an Afro-Brazilian martial art that blends dance and acrobatics. In Video 2 (bottom row), similar inconsistencies occur for the action “jumpstyle dancing” depending on the perceived race of the actor. See: https://youtu.be/amygAq-Sqc4 . This suggests that some action predictions rely on visual attributes rather than on the movement. 1 Introduction Can an action recognition model mistake a cartwheel for capoeira simply because the actor has a darker skin tone? Such questions are central to evaluating fairness in Human Action Recognition (HAR), where predictions may reflect underlying bias, see Figure 1 . We present a framework for auditing fairness in Human Action Recognition (HAR) models using synthetic data. By systematically varying visual attributes our approach exposes potential biases in model predictions that may otherwise remain hidden in real-world datasets. This is increasingly relevant as HAR models are integrated into high-stake domains such as security, autonomous driving, healthcare 60 , and regulatory frameworks like the EU AI Act 14 ; 16 demand greater transparency and accountability from developers of AI systems. With this paper we are trying to assist policy makers, law enforcers and AI engineers in developing more robust and fair systems, making it easier to verify their compliance with emerging legal and ethical standards. To this end, we developed a framework for generating synthetic video datasets with complete control over a subset of visual attributes. Modifying a single attribute allows us to isolate their effects and measure bias without introducing confounding factors. HAR models exhibit bias because they do not always learn to recognize actions based purely on human movement. Instead, they may rely on visual attributes that are not directly related to the action itself, such as clothing, background, or body appearance 29 ; 53 ; 27 ; 11 ; 12 . For instance, a model might associate a Hawaiian shirt with drinking a cocktail, even though the shirt has nothing to do with the action being performed. This suggests that the model could be picking up on superficial visual patterns in the training data, rather than truly understanding the motion, which raises concerns about fairness and reliability. In this project, we identify how sensitive models are to changing ethical attributes. Regulatory frameworks like the EU AI Act 14 ; 16 highlight risks such as opacity and data dependency, which can lead to discriminatory outcomes in AI systems. Motivated by these concerns, we focus on potential biases linked to race or skin color, which may undermine fundamental rights like non-discrimination 51 . Our analysis does not include general action recognition tasks like industrial processes, or animal monitoring, instead, due to the inherently human nature of our selected attributes, we limit the analysis to the HAR domain. Some approaches for evaluating biases have been to manually annotate 8 , or derive information from existing labels 54 , and then assess model performance for each group. An emerging trend has been to generate controlled synthetic images using neural networks, for facial recognition 36 , or human pose and shape estimation 33 . Similarly, for the task of Human Action Recognition we require video data in which specific attributes can be controlled. While prior work such as STAGE 33 has demonstrated bias analysis in individual frames, HAR introduces an additional challenge: motion across frames. This requirement makes extending frame-level approaches like STAGE to the video domain particularly challenging and prone to generation defects, resulting in less control. BEDLAM 6 allows us to generate physically plausible and temporally consistent videos while programmatically controlling visual attributes. To avoid introducing confounding factors because of potential artifacts, we adopt the fully simulated approach through BEDLAM. The simulated approach enables isolated interventions such as changing skin tone, so we can systematically evaluate how these attributes influence model predictions. This paper aims to investigate biases from human action recognition (HAR) models in a simulated environment. Specifically, it addresses the main research question: How sensitive are Human Action Recognition (HAR) models to single visual attribute changes in synthetically generated RGB videos? To explore this question, the paper focuses on two sub-research questions: SubQ1. How well do HAR models trained on real data generalize to synthetic videos? SubQ2. How can we measure significant differences in HAR predictions when changing personal characteristics? To address the research questions, we present the following contributions: we evaluated publicly available action recognition models directly on fully synthetic data without fine-tuning, observing that while some models under-perform, others yield promising results. Finally, we focused on skin color as a key parameter to investigate model bias, identifying cases where predictions shift significantly with changes in appearance. 2 Related work Past research has been exploring biases across various domains of deep learning. This includes studies on gender bias in text-to-image generative models 21 , and visual bias in vision-language models 28 . Within the field of action recognition, prior research has investigated model behavior under occlusion 24 , as well as their robustness to a wide range of noise types 47 . Our paper focuses on making a framework for auditing action recognition models and the datasets they are trained on. We want to identify if they have any inherent biases towards ethical attributes such as skin color. Previously there have been other papers who made similar auditing, testing and diagnostics frameworks 33 ; 15 ; 55 . STAGE 33 focuses on the task of Human Pose and Shape estimation (HPS) models, while 47 focuses on robustness analysis on 90 perturbations (e.g. noise, blur, camera movement). At their core these tasks are similar to action recognition, but operate on individual frames, while action recognition

Read the complete text