Seeing as Experts Do: A Knowledge-Augmented Agent for Open-Set Fine-Grained Visual Understanding
Junhan Chen, Zilu Zhou, Yujun Tong, Dongliang Chang*, Yitao Luo, Zhanyu Ma School of Artificial Intelligence, Beijing University of Posts and Telecommunications, China {chenjunhan, zhouzilu2019, tongyujun, changdongliang, lyt2022, mazhanyu}@bupt.edu.cn · 2026
Read Seeing as Experts Do: A Knowledge-Augmented Agent for Open-Set Fine-Grained Visual Understanding by Junhan Chen, Zilu Zhou, Yujun Tong, Dongliang Chang*, Yitao Luo, Zhanyu Ma School of Artificial Intelligence, Beijing University of Posts and Telecommunications, China {chenjunhan, zhouzilu2019, tongyujun, changdongliang, lyt2022, mazhanyu}@bupt.edu.cn (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-03762.txt.
Introduction
Junhan Chen and five colleagues build an agent that identifies fine visual categories the way an expert would: by gathering evidence, not guessing labels. Fine-grained understanding is leaving closed lists behind, and models must now justify their calls in open settings where categories were never predefined. Their 2026 agent, KFRA, runs a three-stage loop that mimics expert analysis.
Open-vocabulary detection plus web-scale retrieval first proposes candidate categories. The agent then retrieves textual knowledge for each candidate and aligns it with the image, hunting the discriminative regions, a color patch, a structure, a behavior, that separate look-alikes. A large multimodal model finally weighs the grounded evidence into an interpretable verdict. A new benchmark, FGExpertBench, grades reasoning depth across six knowledge dimensions. KFRA beats standalone models and rival agents, gaining up to about a fifth in reasoning accuracy.
Catalog record
| CREATOR | Junhan Chen, Zilu Zhou, Yujun Tong, Dongliang Chang*, Yitao Luo, Zhanyu Ma School of Artificial Intelligence, Beijing University of Posts and Telecommunications, China {chenjunhan, zhouzilu2019, tongyujun, changdongliang, lyt2022, mazhanyu}@bupt.edu.cn |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 33767 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.03762
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Seeing as Experts Do: A Knowledge-Augmented Agent for Open-Set Fine-Grained Visual Understanding Junhan Chen, Zilu Zhou, Yujun Tong, Dongliang Chang*, Yitao Luo, Zhanyu Ma School of Artificial Intelligence, Beijing University of Posts and Telecommunications, China {chenjunhan, zhouzilu2019, tongyujun, changdongliang, lyt2022, mazhanyu}@bupt.edu.cn Abstract Fine-grained visual understanding is shifting from static classification to knowledge-augmented reasoning, where models must justify as well as recognise. Existing approaches remain limited by closed-set taxonomies and single-label prediction, leading to significant degradation under open-set or context-dependent conditions. We present the Knowledge-Augmented Fine-Grained Reasoning Agent (KFRA), a unified framework that transforms fine-grained perception into evidence-driven reasoning. KFRA operates through a three-stage closed reasoning loop that emulates expert analysis. It first performs open-vocabulary detection and web-scale retrieval to generate category hypotheses. It then conducts discriminative regions localisation by aligning textual knowledge with visual evidence through a global-to-local focusing mechanism. Finally, it integrates all multimodal evidence within a large multimodal model to perform interpretable reasoning. Unlike existing agents that treat retrieval and reasoning as independent processes, KFRA establishes a retrieval–grounding coupling that converts retrieved knowledge into spatially grounded evidence for verification. This design enables factual, interpretable, and task-agnostic reasoning across diverse fine-grained scenarios. To evaluate this capability, we construct FGExpertBench, a benchmark designed to assess reasoning depth and cross-task generalisation across six knowledge dimensions. Extensive experiments demonstrate that KFRA consistently surpasses both standalone large multimodal models and current agent frameworks, achieving up to 19 percent improvement in reasoning accuracy and delivering evidence-grounded interpretability in open-set fine-grained visual understanding. Figure 1 : Overview of the Knowledge-Augmented Fine-Grained Reasoning Agent (KFRA). The agent operates through a three-stage closed reasoning loop that emulates expert analysis. (1) Candidate List Generation : KFRA performs open-vocabulary detection and web-scale retrieval to propose category hypotheses. (2) Discriminative Regions Localisation : For each hypothesis, it retrieves relevant textual knowledge and aligns distinctive cues such as colour, structure, or behaviour with corresponding visual regions through a global-to-local focusing mechanism. (3) Knowledge and Region Guided Inference : The agent integrates all multimodal evidence to perform factual and interpretable reasoning across objects and tasks. † † footnotetext: † Equal contributions. * Corresponding author. 1 Introduction Fine-grained visual understanding is no longer about naming what an object is, but about explaining why it is what it is. The challenge is not recognition but reasoning, connecting perceptual evidence to the structured knowledge that defines expert understanding 47 . Despite years of progress, most fine-grained systems still behave like refined classifiers 11 ; 52 ; 44 , accurate within a closed taxonomy but incapable of reasoning beyond it. Recent advances in transformer architectures 17 ; 3 , large multimodal models post-training 25 ; 15 ; 46 , and retrieval-augmented generation 39 ; 14 have extended perception to open-vocabulary settings. Yet, the optimisation goal remains unchanged: predict a single label from a fixed taxonomy (as shown in Figure 2 ). External cues such as attributes, hierarchies, or prompts enrich semantics but do not alter this objective 5 ; 4 . Compressing a complex instance into one categorical token collapses expert knowledge into a flat decision boundary and leaves no capacity for reasoning about unseen subtypes, abnormal states, or contextual variations. In practice, existing fine-grained models lose up to 30–40% accuracy 6 when confronted with unseen species or domains, exposing the fragility of closed-set formulations. This motivates a paradigm shift from label prediction to evidence-driven reasoning. Fine-grained understanding is evolving from static classification to knowledge-grounded reasoning, where models must justify as well as recognise. Human experts, when faced with ambiguous or novel instances, do not rely on memory alone—they hypothesise candidate categories, retrieve references, focus on discriminative traits, and verify these hypotheses against factual knowledge before concluding. This iterative process of observing, hypothesising, and verifying forms a chain of evidence that connects perception with external knowledge. Reproducing this process computationally requires a system that can perceive, retrieve, localise, and reason across multiple tasks rather than a single classification objective. Existing approaches fall short of constructing such reasoning chains. To bridge this gap, we introduce the Knowledge-Augmented Fine-Grained Reasoning Agent (KFRA), which reformulates fine-grained understanding as an open-set reasoning process grounded in open-world knowledge (as shown in Figure 1 ). Figure 2 : Comparison of Fine-Grained Paradigms. KFRA advances fine-grained understanding from static classification to evidence-driven reasoning, leveraging dynamic knowledge and retrieval–grounding coupling to achieve factual, interpretable, and task-agnostic understanding. KFRA generalises beyond recognition to a spectrum of fine-grained reasoning tasks, including object identification, attribute comparison, action understanding, counting, causal analysis, and knowledge inference. It operates through a three-stage closed reasoning loop that emulates expert analysis: (1) Candidate List Generation: KFRA performs open-vocabulary detection to isolate visual entities and conducts web-scale image retrieval to gather visually similar examples, forming candidate hypotheses. (2) Discriminative Regions Localisation: For each hypothesis, it retrieves relevant textual knowledge describing distinctive cues such as colour, structure, or behaviour, and aligns these descriptions with image regions through a global-to-local focusing mechanism. When critical details are missing, an automatic detail-enhancement tool refines local regions for higher discriminability. (3) Knowledge and Region Guided Inference: KFRA integrates all available evidence, including hypotheses, textual knowledge, and grounded regions, within a large multimodal model to reason across objects and modalities and to produce interpretable conclusions. This closed-loop design ensures coherence between retrieval, localisation, and inference, allowing KFRA to adapt across fine-grained tasks without retraining. Unlike general-purpose agents 48 ; 40 ; 22 that treat retrieval and inference as loosely connected steps, KFRA establishes a retrieval–grounding coupling that transforms large multimodal models from passive label predictors into active evidence builders. Retrieved knowledge is not auxiliary context but an actionable signal that guides spatial grounding and hypothesis verification. This coupling enables factual, spatially-grounded, and task-agnostic reasoning across diverse fine-grained scenarios. Figure 3 : Overview of FGExpertBench . The benchmark introduces a structured taxonomy of fine-grained visual understanding tasks that span multiple perception and reasoning dimensions. Each category includes representative examples illustrating its task type, while the overall design enables systematic evaluation of large multimodal models in terms of reasoning depth, cross-task generalisation, and evidence grounding. To quantify these capabilities, we construct FGExpertBench , a benchmark that measures reasoning depth and cross-task generalisation rather than recognition accuracy. It comprises 300 images and 1,500 question–answer pairs across six knowledge dimensions: object recognition, attribute extraction, action understanding, counting, reasoning analysis, and knowledge inference (Figure 3 ). Each task requires synthesising perceptual cues with external knowledge to produce a justified answer. Experiments show that KFRA consistently surpasses both standalone large multimodal models and existing agentic frameworks. When integrated with strong backbones such as GLM-4.5V 42 or Qwen3-VL 51 , KFRA improves reasoning accuracy by up to 19% and delivers evidence-grounded interpretability across all dimensions. In summary, this work makes three key contributions: (1) It introduces KFRA, a knowledge-augmented reasoning framework that unifies diverse fine-grained tasks under a single evidence-driven paradigm. (2) It develops a closed-loop pipeline that couples retrieval, grounded localisation, and multimodal inference, enabling zero-shot reasoning across tasks and domains. (3) It releases FGExpertBench , a benchmark for evaluating reasoning depth and cross-task generalisation in fine-grained understanding. Together, these advances move fine-grained vision beyond static taxonomy recognition toward dynamic, knowledge-grounded reasoning—bringing machine perception one step closer to expert-level cognition. 2 Related Work Understanding how