KIRA: Knowledge-Intensive Image Retrieval and Reasoning Architecture for Specialized Visual Domains
Parthaw Goswami, Jaynto Goswami Deep, SAP Prague, [email protected] · 2026
Read KIRA: Knowledge-Intensive Image Retrieval and Reasoning Architecture for Specialized Visual Domains by Parthaw Goswami, Jaynto Goswami Deep, SAP Prague, [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-16915.txt.
Introduction
Goswami and Deep present KIRA, a five-stage framework that extends retrieval-augmented question answering to images, where text-heavy knowledge bases and visual queries often fail to meet. It chunks images into regions, adapts encoders to rare visual concepts, retrieves across modalities, reasons across several hops, and checks answers against visual evidence, scoring 0.97 retrieval precision in tests across four specialized domains.
Catalog record
| CREATOR | Parthaw Goswami, Jaynto Goswami Deep, SAP Prague, [email protected] |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37089 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.16915
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
KIRA: Knowledge-Intensive Image Retrieval and Reasoning Architecture for Specialized Visual Domains Parthaw Goswami, Jaynto Goswami Deep, SAP Prague, [email protected] Abstract Retrieval-augmented generation (RAG) has transformed text-based question answering, yet its extension to visual domains remains hindered by fundamental challenges: bridging the modality gap between image queries and text-heavy knowledge bases, constructing semantically meaningful visual knowledge bases, performing multi-hop reasoning over retrieved images, and verifying that generated answers are faithfully grounded in visual evidence. We present Kira ( K nowledge- I ntensive Image R etrieval and Reasoning A rchitecture), a unified five-stage framework that addresses ten core problems in visual RAG for specialized domains. Kira introduces: (1) hierarchical semantic chunking with DINO-based region detection for multi-granularity knowledge base construction, (2) domain-adaptive contrastive encoders with few-shot adaptation for rare visual concepts, (3) dual-path cross-modal retrieval with chain-of-thought query expansion, (4) chain-of-retrieval for multi-hop visual reasoning with temporal and multiview support, and (5) evidence-conditioned grounded generation with post-hoc hallucination verification. We also propose DomainVQA-R , a benchmark suite that evaluates visual RAG along three axes (retrieval precision, reasoning faithfulness, and domain correctness) going beyond standard recall metrics. Experiments across four specialized domains (medical X-ray, circuit diagrams, satellite imagery, and histopathology) with a progressive six-variant ablation demonstrate that Kira achieves 0.97 retrieval precision, 1.0 grounding scores, and 0.707 domain correctness averaged across domains, while the ablation reveals actionable insights about when each component helps and when components introduce precision-diversity tradeoffs that must be managed. Code will be released upon acceptance. 1 Introduction Retrieval-augmented generation (RAG) has emerged as a powerful paradigm for knowledge-intensive tasks in natural language processing Guu et al. 2020 ; Lewis et al. 2020 , enabling language models to access external knowledge bases rather than relying solely on parametric memory. While text-based RAG is now well-established, extending this paradigm to visual domains introduces a qualitatively different set of challenges that current methods fail to adequately address. Consider a radiologist querying a knowledge base with a chest X-ray to find similar cases of early-stage pneumonia, or an engineer searching a circuit diagram repository to identify a specific topology. These scenarios require the system to: (1) understand what an image means in a specialized domain, not merely what it looks like; (2) bridge the gap between visual queries and potentially text-heavy knowledge entries; (3) chain multiple retrieved images to reach a conclusion; and (4) verify that the generated answer is actually grounded in the retrieved visual evidence. No existing system addresses all of these challenges in a unified framework. We identify ten core technical problems that must be solved to make visual RAG practical for specialized domains, organized into four categories: • Core Technical: retrieval quality for visual content ( P1 ), cross-modal alignment ( P2 ), and knowledge base construction ( P3 ). • Reasoning & Integration: retrieval-augmented reasoning beyond simple retrieval ( P4 ), and multi-hop visual reasoning ( P5 ). • Domain-Specific: handling rare and fine-grained visual concepts ( P6 ), and temporal/multiview reasoning ( P7 ). • Evaluation & Trust: lack of suitable benchmarks ( P8 ), explainability of retrieved evidence ( P9 ), and hallucination from visual retrieval ( P10 ). To address these problems jointly, we propose Kira ( K nowledge- I ntensive Image R etrieval and Reasoning A rchitecture), a unified five-stage framework illustrated in Fig. 1 . Each stage is designed to solve a specific subset of the identified problems: Stage 1: Knowledge Base Ingestion (P3, P6) uses a hierarchical semantic chunker with DINO Caron et al. 2021 self-attention-based region detection and domain-adaptive contrastive encoders with few-shot adaptation. Stage 2: Cross-Modal Query Processing (P1, P2) introduces dual-path retrieval combining visual embedding and text description indices, with chain-of-thought query expansion using BLIP-2 Li et al. 2023 . Stage 3: Multi-hop Retrieval Engine (P5, P7) implements chain-of-retrieval with residual query construction, augmented by temporal sequence and multiview handlers. Stage 4: Grounded Reasoning Module (P4, P10) performs evidence-conditioned generation with structured evidence packs and post-hoc grounding verification. Stage 5: Evaluation & Benchmark Layer (P8, P9) proposes the DomainVQA-R benchmark with five metrics and produces retrieval rationale cards for explainability. We evaluate Kira across four specialized domains (medical X-ray, circuit diagrams, satellite imagery, and histopathology) using a progressive six-variant ablation study. Our contributions are: 1. A unified architecture that systematically addresses ten identified problems in visual RAG for specialized domains. 2. Novel components including DINO-based hierarchical chunking, few-shot domain-adaptive encoders, chain-of-thought query expansion, chain-of-retrieval, cross-encoder re-ranking, and grounding verification. 3. The DomainVQA-R benchmark evaluating visual RAG along three axes (retrieval precision, reasoning faithfulness, and domain correctness). 4. Comprehensive experiments revealing when each component helps and an honest analysis of current limitations. 2 Related Work Text-Based RAG. Retrieval-augmented generation was popularized by RAG Lewis et al. 2020 and REALM Guu et al. 2020 , which augment language models with a non-parametric retrieval component. Subsequent work has explored dense passage retrieval Karpukhin et al. 2020 , multi-hop retrieval Xiong et al. 2021 , and self-reflective RAG Asai et al. 2024 . While these methods are highly effective for text, they do not address the unique challenges of visual modalities. Vision-Language Models. Recent foundation models such as CLIP Radford et al. 2021 , BLIP-2 Li et al. 2023 , and LLaVA Liu et al. 2023 enable joint vision-language understanding. CLIP provides a shared embedding space for images and text, while BLIP-2 introduces a Querying Transformer (Q-Former) that bridges frozen image encoders with language models. However, these models are trained on web-scale data and often fail to capture fine-grained distinctions in specialized domains such as medical imaging or circuit design Wang et al. 2022 . Visual Question Answering. VQA benchmarks Antol et al. 2015 ; Hudson and Manning 2019 typically operate on single images without retrieval. Knowledge-based VQA Marino et al. 2019 ; Schwenk et al. 2022 requires external knowledge but focuses on general-domain facts. Medical VQA He et al. 2020 ; Lau et al. 2018 targets domain-specific reasoning but lacks the retrieval component that is central to our work. Image Retrieval for Specialized Domains. Content-based image retrieval (CBIR) Smeulders et al. 2000 has a long history, but modern approaches increasingly rely on learned representations. Domain-adapted retrieval has been explored for medical images Chen et al. 2022 and remote sensing Cheng et al. 2018 . These approaches focus purely on retrieval and do not address downstream reasoning, grounding, or explainability. Self-Supervised Region Detection. DINO Caron et al. 2021 demonstrated that self-supervised Vision Transformers learn to attend to semantically meaningful image regions, producing attention maps that can serve as unsupervised object detectors. We leverage this property for region-level chunking during knowledge base construction. Position of Kira . Unlike prior work that addresses individual components in isolation, Kira provides an end-to-end framework spanning knowledge base construction, cross-modal retrieval, multi-hop reasoning, grounded generation, and evaluation. To our knowledge, this is the first system to jointly address all ten identified problems in visual RAG. 3 Method Kira is a five-stage pipeline, illustrated in Fig. 1 . We describe each stage and the specific problems it addresses. Figure 1 : Kira Five-Stage Architecture Overview. 3.1 Stage 1: Knowledge Base Ingestion Hierarchical Semantic Chunking (P3). Most image RAG systems index images as monolithic entities. Kira instead chunks each image