RealBirdID: Benchmarking Bird Species Identification in the Era of MLLMs

Logan Lawrence, Mustafa Chasmai, Rangel Daroya, Wuao Liu, Seoyun Jeong, Max Hamilton, Fabien Delattre, Oindrila Saha, Subhransu Maji, Grant Van Horn · 2026

Read RealBirdID: Benchmarking Bird Species Identification in the Era of MLLMs by Logan Lawrence, Mustafa Chasmai, Rangel Daroya, Wuao Liu, Seoyun Jeong, Max Hamilton, Fabien Delattre, Oindrila Saha, Subhransu Maji, Grant Van Horn (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-27033.txt.

Introduction

Lawrence and co-authors present RealBirdID, a benchmark for bird identification that rewards honest abstention. In the wild, a single photo often cannot settle a fine-grained species call. Cues may be hidden, or the answer may need the bird's song. RealBirdID pairs answerable images with curated unanswerable ones carrying labeled reasons. Models must name the species or abstain with a concrete rationale. Top multimodal models scored under 13 percent on the answerable set. Stronger classifiers were not better calibrated to abstain, either.

Catalog record

CREATORLogan Lawrence, Mustafa Chasmai, Rangel Daroya, Wuao Liu, Seoyun Jeong, Max Hamilton, Fabien Delattre, Oindrila Saha, Subhransu Maji, Grant Van Horn
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47349
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.27033

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

RealBirdID: Benchmarking Bird Species Identification in the Era of MLLMs Logan Lawrence, Mustafa Chasmai, Rangel Daroya, Wuao Liu, Seoyun Jeong, Max Hamilton, Fabien Delattre, Oindrila Saha, Subhransu Maji, Grant Van Horn Abstract Fine-grained bird species identification in the wild is frequently unanswerable from a single image: key cues may be non-visual (e.g. vocalization), or obscured due to occlusion, camera angle, or low resolution. Yet today’s multimodal systems are typically judged on answerable, in-schema cases, encouraging confident guesses rather than principled abstention. We propose the RealBirdID benchmark: given an image of a bird, a system should either answer with a species or abstain with a concrete, evidence-based rationale: “requires vocalization,” “low quality image,” or “view obstructed”. For each genus, the dataset includes a validation split composed of curated unanswerable examples with labeled rationales, paired with a companion set of clearly answerable instances. We find that (1) the species identification on the answerable set is challenging for a variety of open-source and proprietary models ( ≤ \leq 13 % 13\% accuracy for MLLMs including GPT-5 and Gemini-2.5 Pro), (2) models with greater classification ability are not necessarily more calibrated to abstain from unanswerable examples, and (3) that MLLMs generally fail at providing correct reasons even when they do abstain. RealBirdID establishes a focused target for abstention-aware fine-grained recognition and a recipe for measuring progress. Figure 1 : Preview of RealBirdID. In contrast to previous species identification datasets, in each of the genus associated with RealBirdID there is a corresponding set of unanswerable (UA) examples . The summary metric proposed gauges both (1) the ability for the classifier to disambiguate between confusing classes and (2) abstain from predicting on unanswerable examples. Incorrect abstention reasoning is shown in red whereas correct reasoning is shown in green . 1 Introduction Figure 2 : Peek into the dataset . Few examples for the answerable and unanswerable images in RealBirdID. Unanswerable examples are grouped by possible unanswerable reasons. Images from “Sound needed” may be harder to abstain than “Quality” for example, as more detailed knowledge about the particular genus would be needed to understand these reasons. Bird species identification has been crucial to measuring progress in fine-grained visual recognition (FGVR), with benchmarking capabilities advancing via datasets like CUB-200-2011 Wah et al. 2011 , NABirds Van Horn et al. 2015 , NeWT Van Horn et al. 2021 , INQUIRE Vendrow et al. 2024 , and the iNaturalist challenges Van Horn et al. 2018 which emphasize parts, attributes, and expert taxonomic structure. In particular, zero-shot visual classification has been revitalized by the introduction of Multimodal Large Language Models (MLLMs) gpt 2023 ; Dai et al. 2023 ; Liu et al. 2023 ; Liu et al. 2024 . These developments, along with open-vocabulary prompting and instruction-tuned multimodal supervision have pushed FGVR performance upward across fully supervised, few-shot, and zero-shot settings Hong et al. 2025 ; He et al. 2025 ; Saha et al. 2024 ; Saha et al. 2025 ; Lawrence et al. 2026 , enabling systems to handle increasingly subtle intra-class distinctions. However, real world deployment scenarios are often out-of-distribution or, even worse, out-of-schema from existing benchmarks Bendale and Boult 2015 ; Koh et al. 2021 ; Bendale and Boult 2016 . In particular, MLLMs are often trained with datasets comprised exclusively of answerable examples, whereas it is common to see examples in deployment that cannot be properly classified Goyal et al. 2017 ; Zhang et al. 2023 . Being forced to choose an answer can cause hallucinations Kalai et al. 2025 ; Huang et al. 2025 which can be misleading and potentially dangerous, especially in settings like medicine or law Thirunavukarasu et al. 2023 ; Guha et al. 2023 ; Wu et al. 2023 ; Tomani et al. 2024 . In such cases, one would like the model to give a reason for abstaining from picking one of the predefined classes, which could give the opportunity for human experts to intervene and refine the system. The idea of abstention has recently gained traction within text-only tasks Kirichenko et al. 2025 ; Tomani et al. 2024 ; Madhusudhan et al. 2025 . However, the study of abstention of MLLMs in visual settings is still in its infancy Zhang et al. 2023 . To the best of our knowledge, there has been no work focused on measuring abstention when keeping text fixed . Namely, the model can only visually reason in order to abstain. To address this gap, we introduce the RealBirdID dataset, a benchmark for evaluating the ability of MLLMs to exhaustively predict fine-grained species for a given genus, as well as the ability to abstain under realistic, unanswerable examples for that genus ( Fig. 1 ). We come to the following conclusions: 1. Exhaustive species identification is an unsolved problem in vision-language systems. When thresholding on model uncertainty, models generally do not exceed 0.3 Area Under the Curve (AUC) on species-level prediction on the answerable set, corresponding to roughly 17\% accuracy ( Section 5 ). Using range map information to restrict the list of species under consideration significantly increases classification performance across all models ( Fig. 6 ). 2. Current methods have difficulty forming tradeoffs between classification and abstention. Furthermore, they are barely better than random at separating unanswerable and answerable examples ( Table 2 ). Using range maps only has mild effects on abstention tradeoff ( Fig. 6 ). In fact, using range info for MLLMs actually degrades abstention performance. 3. Even when models abstain, their reasons are often incorrect. Across all models, we observed a strong bias in abstention reasons ( Section 6.2 ), in some cases foregoing vocalization completely ( Fig. 8 ). MLLMs abstention rates are very brittle in the face of wording variation on instructions ( Fig. 7 ), sometimes ranging from 3% to 23% abstention rate under semantically equivalent prompt wordings. 2 Related Work Benchmarking Abstention Capabilities In text-only tasks, the idea of unanswerability has gained traction (e.g. SQuAD2.0 Rajpurkar et al. 2018 , NewsQA Trischler et al. 2017 , BigBench (KnownUnknowns) Srivastava et al. 2023 , KUQ Amayuelas et al. 2024 , VQAv2 Goyal et al. 2017 ). However, these methods do not consider the reason for being unable to answer when given an example, which we refer to as abstention with reasoning, and will be focusing on in this work. AbstentionBench Kirichenko et al. 2025 introduces a 35k-example benchmark spanning 20 datasets to evaluate whether LLMs appropriately abstain on unanswerable, underspecified, subjective, or outdated questions, and shows that scaling models offers little benefit. SelfAware Yin et al. 2023 benchmarks LLM self-knowledge by contrasting unanswerable and answerable questions with an automated uncertainty-detection protocol, finding models can flag “unknowns” above chance yet remain overconfident with category and model-dependent variance. In contrast to the text-only datasets AbstentionBench and SelfAware, we focus on inputs with images as containing the evidence for abstention. RGQA Zhang et al. 2023 establishes a realistic VQA (RVQA) benchmark by pairing standard answerable questions with 29k human-annotated unanswerable questions (fine- and coarse-grained) to test rejection and answering jointly. In contrast to RGQA, we specifically focus on needing visual information to abstain rather than detecting unanswerable texts . Namely, the prompts in our data are identical or semantically equivalent (some meaning

Read the complete text