Specificity-aware reinforcement learning for fine-grained open-world classification
Samuele Angheben, Davide Berasi, Alessandro Conti, Elisa Ricci, Yiming Wang · 2026
Read Specificity-aware reinforcement learning for fine-grained open-world classification by Samuele Angheben, Davide Berasi, Alessandro Conti, Elisa Ricci, Yiming Wang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-03197.txt.
Introduction
Samuele Angheben, Davide Berasi, Alessandro Conti, and colleagues push image classifiers to name the exact species, not just the family. Open-world fine-grained classification has no fixed label list, and reasoning multimodal models often answer too generically despite knowing better. Their 2026 framework, SpeciaRL, trains for correctness and specificity together.
A dynamic reward, anchored to the best answers the model itself produces during online rollouts, rewards finer labels only when they stay within the model's verified reach. That design discourages wild guesses while rewarding precision. On out-of-domain fine-grained benchmarks, the method delivered the best balance of correctness and specificity among rivals. The code and model are public. The work shows latent knowledge can be surfaced by the right training signal.
Catalog record
| CREATOR | Samuele Angheben, Davide Berasi, Alessandro Conti, Elisa Ricci, Yiming Wang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 80887 |
| LICENSE | http://creativecommons.org/licenses/by-sa/4.0/ |
| RIGHTS | CC BY SA |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.03197
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Specificity-aware reinforcement learning for fine-grained open-world classification Samuele Angheben, Davide Berasi, Alessandro Conti, Elisa Ricci, Yiming Wang Abstract Classifying fine-grained visual concepts under open-world settings, i.e., without a predefined label set, demands models to be both accurate and specific. Recent reasoning Large Multimodal Models (LMMs) exhibit strong visual understanding capability but tend to produce overly generic predictions when performing fine-grained image classification. Our preliminary analysis reveals that models do possess the intrinsic fine-grained domain knowledge. However, promoting more specific predictions (specificity) without compromising correct ones (correctness) remains a non-trivial and understudied challenge. In this work, we investigate how to steer reasoning LMMs toward predictions that are both correct and specific. We propose a novel specificity-aware reinforcement learning framework, SpeciaRL, to fine-tune reasoning LMMs on fine-grained image classification under the open-world setting. SpeciaRL introduces a dynamic, verifier-based reward signal anchored to the best predictions within online rollouts, promoting specificity while respecting the model’s capabilities to prevent incorrect predictions. Our out-of-domain experiments show that SpeciaRL delivers the best trade-off between correctness and specificity across extensive fine-grained benchmarks, surpassing existing methods and advancing open-world fine-grained image classification. Code and model are publicly available at https://github.com/s-angheben/SpeciaRL . † † Corresponding author: [email protected] . 1 Introduction Figure 1 : In open-world image classification, improving prediction specificity without compromising correctness remains challenging. Existing techniques, such as prompting to be specific, supervised fine-tuning ( sft ) or reinforcement fine-tuning ( rft ), promote specificity but reduce correctness. Instead, our proposed method ( SpeciaRL ) significantly improves the specificity of the base Qwen2.5VL-7B model without compromising correctness. Gray arrows indicate that training is performed on a single-domain (bird) dataset, which is disjoint from the domains in the test set, therefore illustrating cross-domain generalization. Image classification has long been a cornerstone problem in computer vision, aiming to assign a semantic concept to the main object featured in an image 12 . Traditional image classification models typically operate under a closed-world setting, where all possible semantic categories are predefined within a fixed vocabulary 40 . However, in real-world environments models often need to handle emerging categories or novel concepts, highlighting the importance of open-world classification 5 , which removes the fixed vocabulary assumption. This more challenging and practically relevant setting can now be studied more effectively thanks to the emergence of large pre-trained vision–language models 39 ; 60 . Candidate concepts can be derived from large textual corpora 9 or directly generated by recent Large Multimodal Models (LMMs) 28 ; 25 ; 65 ; 3 ; 8 in response to open-ended prompts such as “What is the object in the image?” . Such advances have, in turn, motivated novel approaches as well as new evaluation protocols to assess the correctness of predicted concepts, addressing the unconstrained nature of LMM-generated outputs. Recent benchmarking works 64 ; 31 ; 10 extensively evaluated the classification performance of LMMs in both closed-world and open-world settings. Focusing on the latter, Conti et al. 10 introduced performance metrics based on large language models and textual embedding similarity, in an effort to comprehensively describe the behavior of LMMs. The study showed that the best-performing models are recent reasoning LMMs , such as Qwen2.5VL 4 , which are trained with reasoning-enriched multimodal datasets to connect visual evidence with linguistic inference. The study also revealed that LMMs mostly struggle in classifying fine-grained concepts , with the tendency of being overly generic ( e.g . , flower vs . daisy). However, naïvely encouraging more specific predictions ( i.e . , high specificity) may increase the number of wrong outputs ( i.e . , reduced correctness). For example, Conti et al . observed in 10 that simple prompting benefits LMMs in producing more fine-grained predictions, but at the cost of inferior correctness. Our own experimentation also confirms this compromised correctness when promoting specificity, either by directly querying the model to “be specific” or by fine-tuning the model with supervised fine-tuning ( sft ) or reinforcement fine-tuning ( rft ), as shown in Fig. 1 . Promoting more specific predictions requires indeed a delicate balance between specificity and correctness , a non-trivial challenge that remains greatly underexplored. This work addresses the limitation of LMMs being overly generic on fine-grained open-world classification, aiming to improve prediction specificity without compromising correctness . Before designing the method, we conduct an in-depth inspection of the models’ behavior to understand their capabilities and limitations. We analyzed the prediction distribution over several specificity levels, e.g . , more specific , specific , less specific , and generic , confirming the tendency of the model being overly generic. We further verify whether this limitation stems from a lack of domain-specific knowledge. Interestingly, our preliminary analysis on Qwen2.5VL 4 , the best-performing LMM in 10 , suggests that the model does possess substantial prior knowledge , as evidenced by its strong ability to correctly identify fine-grained categories when queried multiple times, despite a few samples remaining generic or less specific. Given these observations, we propose SpeciaRL, an effective reinforcement learning method with a novel specificity-aware dynamic reward design to elicit specificity within the model’s maximal capabilities. Intuitively, if a model’s best prediction for a given sample is inherently generic, penalizing it for lacking specificity may push it toward producing more incorrect outputs. Our sample-wise reward is therefore dynamically set based on the highest specificity level the LMM can achieve for that sample during multiple rollouts. This paradigm naturally blends into the GRPO 41 algorithm without compromising computational efficiency. SpeciaRL encourages the model’s genuine reasoning capability in fine-grained visual understanding, enabling strong out-of-domain generalization even when trained on a limited dataset from a specific domain. Empirically, SpeciaRL strikes the best balance between specificity and correctness across both fine-grained and very fine-grained datasets, outperforming zero-shot reasoning LMMs and fine-tuned baselines. Our main contributions are summarized below: • We tackle the non-trivial, underexplored challenge of promoting specificity without compromising correctness in fine-grained open-world image classification. • Our analysis confirms that LMMs are overly generic and provides insights on their potential and limitation. • We introduce SpeciaRL, an online reinforcement learning method with a novel specificity-aware dynamic reward. • SpeciaRL achieves the best trade-off between specificity and correctness compared to existing methods. 2 Related Works Large Multimodal Models and reasoning. Early vision-language models primarily focused on learning a joint embedding space that aligned textual and visual representations 39 ; 60 ; 19 . This paradigm later evolved into generative Large Multimodal Models 2 ; 28 ; 65 ; 43 ; 52 ; 8 ; 26 , which connect visual features from a pretrained encoder to the input space of a Large Language Model, enabling open-ended visual question answering and visual reasoning. Recent studies on Chain-of-Thought (CoT) prompting 53 ; 20 have demonstrated that eliciting multi-step reasoning in LMMs significantly improves their performance on several tasks. This insight has led to the development of reasoning LMMs such as OpenAI o1 18 and DeepSeek-R1 16 , which are specifically fine-tuned to perform complex multi-step reasoning before providing a final answer. In this context, Reinforcement Learning has emerged as an efficient and effective post-training strategy for improving the reasoning capabilities of LMMs 1 ; 37 ; 45 ; 66 . In this paper, we aim to investigate