Accuracy and Fairness of Facial Recognition Technology in Low-Quality Police Images: An Experiment With Synthetic Faces
Maria Cuellar, Hon Kiu (James) To, Arush Mehrotra · 2025
Read Accuracy and Fairness of Facial Recognition Technology in Low-Quality Police Images: An Experiment With Synthetic Faces by Maria Cuellar, Hon Kiu (James) To, Arush Mehrotra (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-14320.txt.
Introduction
Cuellar, To, and Mehrotra test how image quality affects facial recognition systems of the kind police use. Most accuracy studies feed these systems clean portraits, but real investigative photos are often blurred, dark, or low in resolution. The authors ask whether errors and racial or gender gaps grow as images degrade.
They generated synthetic faces, applied five kinds of damage, and ran identification tests. False negatives rose as quality fell, especially with blur and low resolution, while error rates stayed higher for women and for Black individuals, with Black women most affected. Even so, the system beat many traditional forensic methods, so the authors argue for validation and oversight rather than abandonment.
Catalog record
| CREATOR | Maria Cuellar, Hon Kiu (James) To, Arush Mehrotra |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-14320
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Accuracy and Fairness of Facial Recognition Technology in Low-Quality Police Images: An Experiment With Synthetic Faces Maria Cuellar, Hon Kiu (James) To, Arush Mehrotra Abstract Facial recognition technology (FRT) is increasingly used in criminal investigations, yet most evaluations of its accuracy rely on high-quality images, unlike those often encountered by law enforcement. This study examines how five common forms of image degradation–contrast, brightness, motion blur, pose shift, and resolution–affect FRT accuracy and fairness across demographic groups. Using synthetic faces generated by StyleGAN3 and labeled with FairFace, we simulate degraded images and evaluate performance using Deepface with ArcFace loss in 1:n identification tasks. We perform an experiment and find that false positive rates peak near baseline image quality, while false negatives increase as degradation intensifies–especially with blur and low resolution. Error rates are consistently higher for women and Black individuals, with Black females most affected. These disparities raise concerns about fairness and reliability when FRT is used in real-world investigative contexts. Nevertheless, even under the most challenging conditions and for the most affected subgroups, FRT accuracy remains substantially higher than that of many traditional forensic methods. This suggests that, if appropriately validated and regulated, FRT should be considered a valuable investigative tool. However, algorithmic accuracy alone is not sufficient: we must also evaluate how FRT is used in practice, including user-driven data manipulation. Such cases underscore the need for transparency and oversight in FRT deployment to ensure both fairness and forensic validity. Keywords: synthetic faces, forensic science, accuracy, scientific validity, error rates. 1 Introduction Law enforcement agencies increasingly rely on facial recognition technology (FRT), a method that uses artificial intelligence (AI) to identify suspects from still images. Tools like Clearview AI, Amazon Rekognition, and Oosto, are used by police and ICE, providing access to over 50 billion identified images scraped from public websites, DMV records, border crossings, and more. FRT has become commonplace in criminal investigations, with at least 3,750 state and local law enforcement agencies and 20 federal agencies currently reporting using FRT, and many agencies have plans to expand their use of FRT in recent years ( [ 1 ] ). Figure 1: Images released by the Philadelphia Police Department in public appeals for help identifying individuals suspected of crimes often exhibit low contrast and brightness, significant blur, non-frontal poses, and low resolution. Images from the official YouTube channel of the Philadelphia Police Department, https://www.youtube.com/user/PhiladelphiaPolice , obtained 05/05/2025. Although FRT is increasingly used, and often promoted as highly accurate ( [ 2 ] ), the accuracy of FRTs has been tested primarily in settings with high-quality images ( [ 3 ] ). It is possible that the algorithms work less well in low-quality images such as those often used by law enforcement (see some examples of these in Figure 1 ). But, the promotion of FRTs being so accurate can lead to law enforcement being misled. For example, a media investigation revealed that some law enforcement officers treat FRT outputs as definitive rather than investigative leads, for example, referring to an unverified match as ‘100% match’ ( [ 4 ] ). However, the Innocence Project reports that there are “at least seven confirmed cases of misidentification due to facial recognition technology, six of which involve Black people who have been wrongfully accused.” ( [ 5 ] ). But how often are FRTs making errors? And is this happening equally throughout demographic groups? Without proper validation, using FRT in practice risks leading to a wrongful conviction or wrongful acquittal, and this could be worse for individuals of certain demographic characteristics. Law enforcement officers and other stakeholders might trust the FRT results more or less than they should. There is some research on how image quality influences FRT accuracy ( [ 6 ] , [ 7 ] , [ 8 ] , [ 9 ] , [ 10 ] ), but it is unclear how this varies across various levels and types of image quality. A notable resource is the vendor test performed by the National Institute on Standards and Technology, which tested over 100 commonly used FRT algorithms and reported their performance in 1:1 and 1:n tasks, across various factors, including race [ 3 ] . Several factors can affect image quality, including but not limited to, object motion ( [ 6 ] , [ 7 ] , [ 8 ] ), camera specifications and placement ( [ 9 ] , [ 6 ] , [ 7 ] ), illumination ( [ 9 ] , [ 8 ] ), pose ( [ 8 ] ), and facial expression ( [ 8 ] ). Differential accuracy of FRTs by gender and race (i.e., what we call a lack of fairness) is another critical concern. Studies show that false positive rates are higher for Black individuals, women, and certain age groups ( [ 11 ] , [ 12 ] , [ 13 ] , [ 14 ] ), a fact that risks the constitutionally protected equal protection under the law. Different methods have been proposed to measure fairness ( [ 15 ] , [ 16 ] , [ 17 ] , [ 18 ] ). However, a comprehensive evaluation of FRT accuracy by race and gender in low-quality images is lacking. Courts have begun to require disclosure when a lead was generated using FRT, and it is likely that FRT results will increasingly be introduced in evidentiary contexts. As of early 2025, 15 states had enacted laws regulating FRT use in policing ( [ 19 ] ), and courts are beginning to require disclosure when FRT is used. As such, courts’ admissibility standards for scientific evidence offer a valuable framework for evaluating the validity of these tools. Legal scholars have raised concerns about transparency, proprietary algorithms, and contextual bias ( [ 20 ] , [ 21 ] , [ 22 ] , [ 23 ] , [ 24 ] ). As FRT moves closer to evidentiary use, it must meet the same scientific standards courts apply to other forensic methods ( [ 25 ] ). Our approach draws on lessons learned from problems in traditional forensic science disciplines, specifically, their lack of demonstrated reliability (accuracy, repeatability, and reproducibility) ( [ 25 ] ). It applies the corresponding solutions proposed in the forensic science reform literature ( [ 26 ] ) to the study of facial recognition technologies (FRTs). While FRT is not typically categorized as a forensic science and is not yet widely introduced as evidence in court, its use in criminal investigations plays a similar role: identifying suspects. And, so far, it seems to be more successful than other forensic disciplines, as shown in Table 1 . Forensic Discipline False Positive Rate False Negative Rate Facial recognition (FRT) 0.0001% 0.06% Fingerprint comparison 0.1% 7.5% Firearm comparison 1.01% 0.37% Table 1: Example error rates for FRT ( [ 27 ] ) compared to fingerprint ( [ 28 ] ) and firearm ( [ 29 ] ) comparisons, as they are currently practiced. When applied to high-quality images, facial recognition technology (FRT) outperforms many traditional forensic disciplines in terms of accuracy. However, it is important to note that the reported error rates come from studies that have faced methodological criticism ( [ 30