Foundation versus Domain-specific Models: Performance Comparison, Fusion, and Explainability in Face Recognition

Redwan Sony, Parisa Farmanifard, Arun Ross, Anil K. Jain · 2025

Read Foundation versus Domain-specific Models: Performance Comparison, Fusion, and Explainability in Face Recognition by Redwan Sony, Parisa Farmanifard, Arun Ross, Anil K. Jain (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-03541.txt.

Introduction

Redwan Sony, Parisa Farmanifard, Arun Ross, and Anil Jain ask a pointed question: can big general-purpose AI models beat a specialist at face recognition? They pitted famous foundation models, including CLIP, BLIP, GPT-4o, and Grok-4, against a purpose-built face system across several benchmark sets, using the general models with no task training.

The specialist won everywhere. The general models did better on loosely segmented faces than on tight crops, a hint they lean on context clues. Blending a foundation model's score with the specialist's score improved accuracy where false matches must be rare. The big models could also explain decisions in words, and at times settled cases the specialist found doubtful. The authors favor a judicious mix: specialist muscle with foundation-model judgment on top.

Catalog record

CREATORRedwan Sony, Parisa Farmanifard, Arun Ross, Anil K. Jain
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-03541

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Foundation versus Domain-specific Models: Performance Comparison, Fusion, and Explainability in Face Recognition Redwan Sony, Parisa Farmanifard, Arun Ross, Anil K. Jain Abstract In this paper, we address the following question: How do generic foundation models (e.g., CLIP, BLIP, LLaVa, DINO) compare against a domain-specific face recognition model (viz., AdaFace or ArcFace) on the face recognition task? Through a series of experiments involving several foundation models and benchmark datasets, we are able to report the following findings: (a) In all datasets considered, domain-specific models outperformed zero-shot foundation models. (b) The performance of zero-shot generic foundation models improves on over-segmented face images than tightly cropped faces thereby suggesting the importance of contextual clues. For example, at a False Match Rate (FMR) of 0.01%, the True Match Rate (TMR) of OpenCLIP improved from 64.97% to 81.73% on the LFW dataset as the face crop increased from 112 × \times 112 to 250 × \times 250 while the TMR of domain-specific AdaFace dropped from 99.09% to 77.31%. (c) A simple score-level fusion of a foundation model with a domain-specific FR model improved the accuracy at low FMRs. For example, the TMR of AdaFace when fused with BLIP improved from 72.64% to 83.31% at an FMR of 0.0001% on the IJB-B dataset and from 73.17% to 85.81% on the IJB-C dataset. (d) Foundation models, such as ChatGPT, can be used to impart explainability to the FR pipeline (e.g., “Despite minor lighting and head tilt differences, the two left-profile images show high consistency in forehead slope, nose shape, chin contour…”). In some instances, foundation models are even able to resolve low-confidence decisions made by AdaFace (e.g., “Although AdaFace assigns a low similarity score of 0.21, both images exhibit high visual similarity…and the pair is likely of the same person”), thereby reiterating the importance of combining domain-specific FR models with generic foundation models in a judicious manner. 1 Introduction Automated Face Recognition (FR) is an important biometric technology that uses face images or videos to recognize an individual [ 19 ] . Face recognition systems typically operate in the verification or identification mode. In verification (1:1 matching), the goal is to determine whether two face images are of the same individual. In identification (1:N matching), the input face image (probe) is compared against a database of enrolled face images (gallery) in order to determine the best match. State-of-the-art (SoTA) face recognition models have been observed to achieve high accuracy on challenging benchmark datasets [ 23 ] . In the most recent NIST face recognition evaluations [ 1 ] , the leading FR model had a False Non-Match Rate (FNMR) of 0.0007 at a False Match Rate (FMR) of 10^{-6} in the constrained cooperative scenario (“VISA”). In the unconstrained non-cooperative scenario (“BORDER-KIOSK”), the top-performing FR model achieved an FNMR of 0.0338 at an FMR of 10^{-5} . This remarkable improvement from some of the early NIST evaluations has been possible due to the unprecedented rise of Deep Neural Networks (DNNs) that has upended the computer vision and natural language processing (NLP) fields. Several well-established and publicly available face recognition models have significantly advanced research in this domain. We refer to these models as domain-specific FR models since they have been exclusively designed for the FR domain. Notable among them and available as open source are AdaFace [ 22 ] , ArcFace [ 11 ] , CosFace [ 47 ] , MagFace [ 33 ] , FaceNet [ 42 ] , and SphereFace [ 30 ] . While FaceNet popularized the use of triplet loss for learning discriminative face embeddings, the other models’ loss functions primarily leverage variants of angular margin-based losses, each introducing measures to improve intra-class compactness and inter-class separability in the learned representation space. Table 1 : Reverse chronological summary of recent studies applying or analyzing foundation models for various face biometric tasks. Authors (Year) Summary of Contribution Sony et al. (2025) [ 44 ] Conducted benchmarking of 41 vision-language models on face recognition and soft-biometric prediction, highlighting strong zero-shot performance but no interpretability or identity-level analysis was done. Narayan et al. (2025) [ 34 ] Introduced FaceXBench, a large-scale VQA-style benchmark for face understanding with 5,000 multiple-choice questions across 14 tasks, and evaluated 28 LLMs to reveal key limitations in recognition performance and reasoning. Otroshi et al. (2025) [ 43 ] Presented the first comprehensive survey on foundation models in biometrics, reviewing their use in recognition, spoof detection, and generative attacks, while highlighting privacy and ethical issues. Chettaoui et al. (2024) [ 7 ] Proposed adapting foundation models for face recognition; showed that fine-tuning pre-trained FMs, even with limited data, outperforms training from scratch. Hassanpour et al. (2024) [ 16 ] Investigated ChatGPT’s ability to perform face recognition, gender detection, and age estimation through crafted prompts; revealed the risks of privacy leakage. DeAndres-Tame et al. (2024) [ 10 ] Used GPT-4 to perform face verification and face attribute extraction, emphasizing its ability to generate explanations for match decisions. Bhat et al. (2023) [ 2 ] Evaluated CLIP and OpenCLIP as zero-shot face recognizers. At the same time, and relatively recently, there has been huge strides made in the development of generic foundation models. A foundation model (FM) is a large-scale model trained on huge quantities of diverse and broad domain data, often spanning multiple modalities (image, video, text, audio,..), with the goal of enabling adaptability across a wide range of downstream tasks with minimal fine-tuning [ 3 ] . These models are typically pre-trained using self-supervised or unsupervised learning objectives that involve aligning visual features with corresponding textual embeddings. Such text-guided learning strategies empower the vision encoders within foundation models to capture semantically rich and transferable visual representations, making them highly effective feature extractors across multiple domains for a variety of tasks [ 39 , 6 , 26 , 20 ] . Some popular examples of foundation models include CLIP [ 39 ] , OpenCLIP [ 6 ] , BLIP [ 26 , 27 ] , DINO [ 4 ] , SAM [ 32 , 40 ] , LLaVA [ 28 , 29 ] , DeepSeek [ 48 , 50 , 9 ] , ALIGN [ 20 ] , and Kosmos [ 38 ] . Originally designed for general vision-language tasks, FMs are being explored for different biometric tasks such as face recognition [ 44 , 7 ] , attribute prediction [ 44 ] , iris segmentation [ 15 ] , iris recognition [ 14 ] , presentation attack detection [ 45 ] , and morph detection [ 37 , 25 ] due to their strong zero-shot transfer capabilities and scalability across multiple domains. In this paper, we raise the following questions. 1. On the task of face recognition - where domain-specific models have been effective so far - what are the benefits of using generic foundation models? 2. Can foundation models be used alongside domain-specific models to further improve recognition accuracy? 3. Can foundation models be used to impart explainability to the face recognition task? In order to investigate these questions, we conduct an extensive study involving 35 foundation models (corresponding to 14 different families) and 5 domain-specific FR models (corresponding to 2 different matchers) on 5 benchmark datasets. 1 1 1 For brevity

Read the complete text