Generalist Multimodal LLMs Gain Biometric Expertise via Human Salience

Jacob Piland, Byron Dowling, Christopher Sweet, Adam Czajka · 2026

Read Generalist Multimodal LLMs Gain Biometric Expertise via Human Salience by Jacob Piland, Byron Dowling, Christopher Sweet, Adam Czajka (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-17173.txt.

Introduction

Piland and co-authors test whether general multimodal models can detect iris presentation attacks with help from human experts. Building specialist models is hard: future attacks are unknown, data are costly, and sharing biometric data raises privacy issues. They find that vision encoders in these models already cluster many attack types without task training. Where clusters overlap, prompts carrying human salience, meaning verbal notes on attack signs, resolve the ambiguity. On 224 iris images, Gemini with expert prompts beat a specialist baseline and human examiners.

Catalog record

CREATORJacob Piland, Byron Dowling, Christopher Sweet, Adam Czajka
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE48514
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.17173

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Generalist Multimodal LLMs Gain Biometric Expertise via Human Salience Jacob Piland, Byron Dowling, Christopher Sweet, Adam Czajka Abstract Iris presentation attack detection (PAD) is critical for secure biometric deployments, yet developing specialized models faces significant practical barriers: collecting data representing future unknown attacks is impossible, and collecting diverse-enough data, yet still limited in terms of its predictive power, is expensive. Additionally, sharing biometric data raises privacy concerns. Due to rapid emergence of new attack vectors demanding adaptable solutions, we thus investigate in this paper whether general-purpose multimodal large language models (MLLMs) can perform iris PAD when augmented with human expert knowledge, operating under strict privacy constraints that prohibit sending biometric data to public cloud MLLM services. Through analysis of vision encoder embeddings applied to our dataset, we demonstrate that pre-trained vision transformers in MLLMs inherently cluster many iris attack types despite never being explicitly trained for this task. However, where clustering shows overlap between attack classes, we find that structured prompts incorporating human salience (verbal descriptions from subjects identifying attack indicators) enable these models to resolve ambiguities. Testing on an IRB-restricted dataset of 224 iris images spanning seven attack types, using only university-approved services (Gemini 2.5 Pro) or locally-hosted models (e.g., Llama 3.2-Vision), we show that Gemini with expert-informed prompts outperforms both a specialized convolutional neural networks (CNN)-based baseline and human examiners, while the locally-deployable Llama achieves near-human performance. Our results establish that MLLMs deployable within institutional privacy constraints offer a viable path for iris PAD. 1 Department of Computer Science and Engineering, University of Notre Dame, Notre Dame, IN 46556, USA [email protected], [email protected], [email protected] 2 Center for Research Computing, University of Notre Dame, Notre Dame, IN 46556, USA [email protected] Corresponding author: Adam Czajka — [email protected] Keywords: Biometric security, human salience, iris presentation attack detection, iris recognition, multimodal large language models, privacy-preserving biometrics, prompt engineering, UMAP clustering, vision transformers, zero-shot learning, few-shot learning. This material is based upon work supported by the U.S. National Science Foundation under grant No. 2237880 and work supported by the U.S. Department of Defense under contract No. W52P1J-20-9-3009. Any opinions, findings, conclusions, or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the U.S. National Science Foundation or U.S. Department of Defense. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes, notwithstanding any copyright notation hereon. Figure 1 : Experimental pipeline and paper contributions. We start with the generation of synthetic MESH salience from human annotations and corresponding image dataset. This salience, along with the control and human salience, is then combined with our novel prompts for testing. Results are obtained for both models to which we have ethical access and compared with a human and salience-guided CNN baseline. 1 Introduction 1.1 Background and Motivation Biometric presentation attack detection (PAD) stands at a critical intersection of security necessity and practical constraints. While deep learning approaches have shown promise for detecting various attack types ranging from artificial objects presented to biometric cameras to synthetic imagery, their deployment faces four fundamental challenges that motivate our investigation into alternative approaches, leveraging recent advances in foundational and multi-modal large models. First, the diversity of presentation attacks requires extensive data collection across multiple attack vectors, which is expensive, ethically complex when involving human subjects, and will never exhaust all possible attack types potentially to be seen in the future. Second, the rapid evolution of attack methodologies, particularly with advances in generative AI, demands systems that can adapt quickly, without complete retraining [ 66 ] . Thus, an effective PAD solution should ideally be based on models with a general “understanding” of visual tasks, and that are only directed by limited data (multi-modal, if available) towards the PAD task. Large multi-modal (vision-language) models may offer such capability [ 40 ] . Third, while biometric PAD algorithms in general surpass human-sourced classification, there is an important application of forensic expert-based judgment in court cases, where the machine serves an important supporting role, but the final decision is left – by law – for the expert. In such cases there is a need for seamless human-machine pairing approaches. Fourth, the sensitive nature of biometric data creates significant barriers to model development and deployment. Biometric images are personally identifiable information subject to stringent privacy regulations and Institutional Review Board (IRB) restrictions that explicitly prohibit transmission to public cloud services such as ChatGPT or Claude. These privacy-related constraints eliminate many state-of-the-art/commercial solutions from consideration and there’s a need for privacy-safe, usually locally-run approaches. 1.2 Proposed Solution Due to the impractically of attaining ever larger biometric datasets, which is the standard approach to training specialized CNNs, we propose a fundamentally different approach: leveraging the latent visual understanding of general-purpose MLLMs that can be deployed within institutional privacy boundaries. Our investigation is constrained to models that either have special institutional agreements for secure processing (Gemini 2.5 Pro through our university’s bilateral agreement with Google) or can be hosted entirely on local infrastructure (e.g., Llama 3.2-Vision). This constraint, while limiting our model choices, reflects real-world deployment scenarios where biometric data cannot leave institutional control. In this work, we demonstrate the solution for iris PAD as the example PAD domain, although the proposed methodology can be applied to design a PAD method for other biometric techniques, for which both images and language descriptions of these images are available. (a) (b) Figure 2 : Uniform Manifold Approximation and Projection (UMAP) visualization of iris samples encoded by (left) SigLIP vision-only embeddings and (right) SigLIP + Gemma multimodal embeddings using a simple binary prompt asking whether the iris is “real and healthy” or “synthetic/unhealthy.” Despite never being trained specifically for iris PAD, SigLIP alone achieves partial separation of attack types. However, adding even minimal semantic guidance through Gemma shows much improved visual separation between live vs. spoof discrimination, with clearer cluster boundaries and reduced overlap between classes. This visual separation motivates our investigation into whether general-purpose MLLMs can address specialized biometric security tasks through appropriate prompting. Our key insight is that privacy-compliant MLLMs, despite never being explicitly trained on iris PAD tasks, have learned visual features that partially separate different attack types. We illustrate this through exploratory Uniform Manifold Approximation and Projection (UMAP) visualization [ 36 ] using SigLIP [ 69 ] , a widely-used vision-language model, for image encoding and Gemma [ 53 ] , a large language model, for text embedding. To leverage both SigLIP’s strong visual representations and Gemma’s text processing capabilities (as SigLIP is limited to short prompts), we combine their embeddings through a shallow Multi-Layer Perceptron (MLP) fusion layer, a standard approach in multimodal learning [ 3 ] . The shallow MLP is a two-layer network with normalization and GELU activation, an input shape of 2048 and a hidden dimension size of 512. This combined representation, shown in Fig. 2 , uses only a simple binary prompt asking whether the iris is ’real and healthy’ or ’synthetic/unhealthy’ 1 1 1 Full prompt: ’Is this a real and healthy iris or synthetic/unhealthy iris? Return a single float number from 0 to 1 with 0 being real/healthy and 1 being synthetic/unhealthy and no other output.’ The (un)healthy specification is in reference to the diseased iris attack

Read the complete text