Metamorphic Testing for Pose Estimation Systems
Matías Duran 1, Thomas Laurent 1, Ellen Rushe 2, Anthony Ventresque 1 · 2025
Read Metamorphic Testing for Pose Estimation Systems by Matías Duran 1, Thomas Laurent 1, Ellen Rushe 2, Anthony Ventresque 1 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-09460.txt.
Introduction
Matías Duran and co-authors propose, in a 2025 paper, a way to test pose estimation systems without hand-labeled data. These systems track body joints for sports, health, and animal care, and their faults can matter. Testing is hard because the right answer must usually be marked by hand, at great cost, and marks made for one use rarely fit another. Their framework, MET-POSE, uses metamorphic testing instead.
The trick checks rules that should hold when an input changes in a known way, so no ground truth is needed. The authors list rules for common vision challenges and a way to score them. They try the framework on a top human pose system with two photo data sets. The outputs expose faults at a rate matching or beating labeled tests, giving users a way to probe a system on their own ground.
Catalog record
| CREATOR | Matías Duran 1, Thomas Laurent 1, Ellen Rushe 2, Anthony Ventresque 1 |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 68 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-09460
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Metamorphic Testing for Pose Estimation Systems Thanks: This work was supported in part with the financial support of grant 13/RC/2094_2 to Lero - the Research Ireland Research Centre for Software. Matías Duran 1, Thomas Laurent 1, Ellen Rushe 2, Anthony Ventresque 1 Abstract Pose estimation systems are used in a variety of fields, from sports analytics to livestock care. Given their potential impact, it is paramount to systematically test their behaviour and potential for failure. This is a complex task due to the oracle problem and the high cost of manual labelling necessary to build ground truth keypoints. This problem is exacerbated by the fact that different applications require systems to focus on different subjects (e.g., human versus animal) or landmarks (e.g., only extremities versus whole body and face), which makes labelled test data rarely reusable. To combat these problems we propose MeT-Pose , a metamorphic testing framework for pose estimation systems that bypasses the need for manual annotation while assessing the performance of these systems under different circumstances. MeT-Pose thus allows users of pose estimation systems to assess the systems in conditions that more closely relate to their application without having to label an ad-hoc test dataset or rely only on available datasets, which may not be adapted to their application domain. While we define MeT-Pose in general terms, we also present a non-exhaustive list of metamorphic rules that represent common challenges in computer vision applications, as well as a specific way to evaluate these rules. We then experimentally show the effectiveness of MeT-Pose by applying it to Mediapipe Holistic, a state of the art human pose estimation system, with the FLIC and PHOENIX datasets. With these experiments, we outline numerous ways in which the outputs of MeT-Pose can uncover faults in pose estimation systems at a similar or higher rate than classic testing using hand labelled data, and show that users can tailor the rule set they use to the faults and level of accuracy relevant to their application. I Introduction Pose Estimation Systems has applications in medicine [ 1 ] , sign language recognition [ 2 ] and high stakes sports events [ 3 ] , requiring them to be well-tested in order to provide a substantiated assessment of the correct behaviour when applied to sensitive domains. Such systems need to perform correctly and as expected under a variety of conditions. This work aims to provide practitioners with a means of assessing this. In this work we propose MeT-Pose , a metamorphic testing framework to test pose estimation systems without the significant cost of manual data labelling. Additionally, we propose a non exhaustive set of metamorphic rules for MeT-Pose , including flexible metrics to assess violations. These rules allow practitioners to apply various commonly encountered image changes and can easily be extended by users, should they want to explore different aspects of the system. We apply MeT-Pose to Mediapipe Hollistic [ 4 ] , a widely used state-of-the-art pose estimation system, on datasets from the literature used to train and assess human pose estimation systems in different domains. We show that our proposed framework can find numerous faulty outputs from the system. Results show that MeT-Pose provides results on par with classic, human annotated ground truth-based testing on the FLIC dataset. Furthermore, we illustrate how analysis of the results can highlight elements of the input that impact the system’s performance. Such analysis can then help practitioners better understand the settings under which the system functions properly, and can thus be used with confidence. The remainder of this paper is organised as follows. First, Section II provides an overview of the context and concepts that underpin this work. Next, Section III describes our proposed metamorphic testing framework for pose estimation systems, and Section IV describes a non-exhaustive set of possible metamorphic relations for this system. Section V describes the experiments performed to evaluate the framework, and Section VI describes and analyses the results of these experiments. Section VII gives an overview of related work, while Section VIII describes threats to the validity of this work, and steps taken to address them. Finally, Section IX concludes the paper and proposes avenues for future work. II Background This section provides an overview and definition of the concepts used in this work. First, Section II-A defines human pose estimation systems, which the proposed framework tests, then Section II-B defines challenges specific to testing Machine Learning (ML) systems, and Section II-C explains the idea of metamorphic testing, the basis of the proposed framework. II-A Pose Estimation Pose estimation is the task of estimating the locations of different landmarks on a subject from images or video frames, with the overall aim of providing an overview of their pose. This task is important to a wide variety of fields including human activity recognition [ 5 ] , sign language recognition [ 2 ] , and sports analytics [ 3 ] . It is typically solved using DL-based regression algorithms which estimate the coordinates of each body part. One of the most common open-source pose estimation frameworks available is MediaPipe’s Pose Landmark Detection system [ 6 ] , which is often incorporated directly into larger ML frameworks for a variety of tasks. This is typically done without fine-tuning, as this model is trained on a large and diverse set of people – often a far larger number of individuals than would be available for the lower-resource tasks on which this model is typically applied. It is thus important that these pose estimation systems be tested under different conditions when integrated into different systems that target different use cases and thus will provide images of different natures (e.g., dynamic applications will feed the pose estimation system images that are more blurry). II-B Challenges in Testing ML-based Systems Deep Learning (DL) research has gained enormous momentum over the last decade, with a majority of the development in this area being centred around computer vision. However, despite their success, these deep architectures have also demonstrated harmful decision-making [ 7 , 8 , 9 ] , bias [ 10 ] and a lack of “understanding” of common-sense concepts such as basic spatial relations [ 11 , 12 ] . These issues are made all the more challenging to identify given that DL methods lack interpretability, with the steps that lead to a particular decision often being unclear. This lack of transparency means that it is crucial that these systems are systematically tested and their results compared to their expected behaviour in order to understand their sensitivities, along with the conditions under which they can be expected to behave correctly. Another aspect that complicates the testing of complex systems is that they often present what is known as the Oracle Problem [ 13 ] , which describes a scenario where the correct output of the system under test is not known. Sometimes there are no automated ways to compute this oracle, or the act of querying this oracle can be prohibitively expensive. In the case of pose estimation, labelling keypoints in videos is extremely time consuming, making it expensive to collect this form of ground truth data for many applications. This complicates the testing of these systems and requires techniques