CPJ: Explainable Agricultural Pest Diagnosis via Caption-Prompt-Judge with LLM-Judged Refinement

Wentao Zhang 1 , Tao Fang 2 , Lina Lu 1 , Lifei Wang 2 , Weihe Zhong 2 · 2025

Read CPJ: Explainable Agricultural Pest Diagnosis via Caption-Prompt-Judge with LLM-Judged Refinement by Wentao Zhang 1 , Tao Fang 2 , Lina Lu 1 , Lifei Wang 2 , Weihe Zhong 2 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-24947.txt.

Introduction

Wentao Zhang, Tao Fang, Lina Lu, and colleagues present CPJ, an explainable method for diagnosing agricultural pests and diseases from images without training. Existing methods need costly supervised fine-tuning and degrade when conditions shift. Their Caption-Prompt-Judge pipeline uses large vision-language models to write multi-angle captions of an image, then refines those captions with a language model acting as judge.

Refined captions feed a dual-answer question process covering both recognition and management advice. On the CDDMBench benchmark, performance improves significantly over baselines. Farming technology readers get diagnosis that shows its reasoning in words. Interpretability matters when advice guides real treatment decisions in the field.

Catalog record

CREATORWentao Zhang 1 , Tao Fang 2 , Lina Lu 1 , Lifei Wang 2 , Weihe Zhong 2
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE25 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-24947

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CPJ: Explainable Agricultural Pest Diagnosis via Caption-Prompt-Judge with LLM-Judged Refinement Wentao Zhang 1 , Tao Fang 2 , Lina Lu 1 , Lifei Wang 2 , Weihe Zhong 2 Abstract Accurate and interpretable crop disease diagnosis is essential for agricultural decision-making, yet existing methods often rely on costly supervised fine-tuning and perform poorly under domain shifts. We propose Caption–Prompt–Judge (CPJ), a training-free few-shot framework that enhances Agri-Pest VQA through structured, interpretable image captions. CPJ employs large vision-language models to generate multi-angle captions, refined iteratively via an LLM-as-Judge module, which then inform a dual-answer VQA process for both recognition and management responses. Evaluated on CDDMBench, CPJ significantly improves performance: using GPT-5-mini captions, GPT-5-Nano achieves +22.7 pp in disease classification and +19.5 points in QA score over no-caption baselines. The framework provides transparent, evidence-based reasoning, advancing robust and explainable agricultural diagnosis without fine-tuning. Our code and data are publicly available. 1 1 1 https://github.com/CPJ-Agricultural/CPJ-Agricultural-Diagnosis I Introduction Accurate and interpretable crop disease diagnosis is critical for agricultural decision-making. Conventional vision models (e.g., YOLO) only provide categorical labels without integrating essential context such as environment or crop variety, hindering practical utility as real-world diagnosis demands deeper reasoning combining visual evidence and expert knowledge. The opacity of such black-box systems undermines trust, emphasizing the need for explainable AI (XAI) [ 1 ] . Advanced large vision-language models (LVLMs), such as Qwen-VL and GPT-series, demonstrate strong performance in open-domain visual question answering (VQA) [ 2 , 3 ] . Yet when applied to Agri-Pest VQA, these models often rely on raw image data alone, resulting in incomplete interpretations that miss critical diagnostic details distinguishing pathogen types. Domain-specific approaches like ITLMLP for cucumber disease recognition [ 4 ] and multimodal foundation models for agricultural applications [ 5 ] show promising directions but lack comprehensive explainability frameworks. Existing works, including multimodal datasets like AgMMU and CDDM [ 6 , 7 ] , focus on fine-tuning [ 8 , 9 ] for accuracy but neglect structured explainability, leading to unreliable outputs under domain shifts [ 10 ] . Without intermediate representations, models cannot connect visual cues to causal explanations, reducing interpretability and robustness. We propose enhancing Agri-Pest VQA with explicit, interpretable image captions as auxiliary inputs. We employ a larger LVLM to generate multi-angle descriptive captions detailing plant types, disease symptoms, severity, and uncertainties. These captions act as transparent intermediates, augmenting the model’s understanding like human experts verbalizing observations before diagnosis, thereby improving explainability through traceable, human-readable artifacts [ 11 ] . Fig. 1: Overview of the “CPJ” pipeline for explainable Agri-Pest VQA, featuring three cohesive stages: (i) Generative Explanational Captioning : A large vision-language model analyzes input crop disease images to generate multi-angle descriptive captions detailing plant morphology, disease symptoms, severity levels, and diagnostic uncertainties. These captions undergo iterative refinement via LLM-as-a-Judge evaluation to ensure accuracy, completeness, and neutrality. (ii) Task-Specific Prompt-Based VQA Generation : Using the refined captions alongside original images and few-shot exemplars, the VQA model generates dual complementary answers addressing both disease recognition (crop type, disease identification, visual features) and actionable management guidance (treatment methods, prevention strategies). (iii) LLM-as-a-Judge Answer Selection : A stronger LLM evaluates both candidate answers against reference answers using multi-dimensional criteria (factual accuracy, completeness, specificity, practical relevance), selecting the superior response to deliver reliable, interpretable diagnostic recommendations. We propose the Caption–Prompt–Judge (CPJ) framework, a training-free pipeline leveraging LLM-as-a-Judge self-refined captions for debiased, evidence-grounded Agri-Pest VQA. As illustrated in Fig. 1 , CPJ operates through: (i) generative interpretable captioning with iterative refinement via LLM-as-a-Judge, (ii) caption-informed VQA using task-specific prompts, and (iii) answer selection based on diagnostic validity and practical relevance. Evaluated on CDDMBench [ 7 ] using lightweight LVLMs (Qwen-VL-Chat, GPT-5-Nano), CPJ achieves significant improvements via strict keyword matching for classification and LLM-based scoring for QA. GPT-5-Nano with GPT-5-mini captions achieves +22.7 pp in disease classification and +19.5 points in QA score over baselines, demonstrating enhanced accuracy and transparency without supervised fine-tuning. Our contributions include: • Framework : A training-free CPJ framework prioritizing interpretability through iteratively refined captions, enabling transparent diagnostic reasoning and addressing the gap between black-box predictions and explainable AI needs. • Methodology : Integration of multi-angle descriptions with LLM-as-a-Judge refinement and dual-answer VQA generation, bridging visual-semantic gaps while maintaining objectivity through crop/disease name exclusion in caption prompts. • Empirical Validation : Comprehensive evaluation on CDDMBench showing significant gains: GPT-5-Nano achieves +22.7pp in disease classification and +19.5 points in QA score with GPT-5-mini captions, with human expert validation (94.2% agreement, Cohen’s k=0.88 ) confirming LLM-as-a-Judge reliability. II RELATED WORK II-A Agricultural Vision-Language Models Recent advancements in Vision-Language Models (VLMs) have enabled domain-specific adaptations for agricultural disease diagnosis. Traditional approaches to crop disease identification relied on handcrafted features and classical machine learning [ 12 ] , but the advent of deep learning, particularly Vision Transformers (ViT) [ 13 ] , has significantly improved classification accuracy. Pre-trained language models have also shown promise in agricultural text classification tasks [ 14 ] . Zhu et al. [ 15 ] comprehensively review VLM applications in agriculture, covering pest detection, weed identification, and seed quality assessment, while addressing challenges such as data scarcity, domain shift, and training efficiency. Specialized models like AgroGPT and AgriGPT-VL [ 16 ] demonstrate the efficacy of domain-specific training: Yang et al. train AgriGPT-VL on the large-scale Agri-3M-VL dataset via a progressive curriculum from text grounding to multimodal alignment, achieving state-of-the-art performance on agricultural benchmarks. Similarly, AgriVLM [ 17 ] integrates ViT and ChatGLM via Q-Former with LoRA fine-tuning, achieving over 90% accuracy in crop growth stage and disease recognition through parameter-efficient adaptation. Few-shot learning approaches [ 18 ] further enhance model adaptability in low-resource scenarios, while specialized agricultural tasks benefit from vision language model leveraging [ 19 ] . Benchmark evaluations reveal significant challenges in agricultural VQA [ 20 ] . Shinoda et al. [ 21 ] introduce AgroBench, an expert-annotated benchmark covering 203 crop and 682 disease categories across seven tasks, revealing that over 50% of errors stem from lack of domain-specific knowledge. Ranario and Earles [ 22 ] benchmark 27 agricultural classification tasks, finding that zero-shot VLMs underperform supervised baselines and that multiple-choice prompting significantly outperforms open-ended generation. While large-scale datasets (e.g., MIRAGE [ 23 ] , CDDMBench [ 7 ] ,AgMMU [ 6 ] ) enable supervised fine-tuning (SFT) methods to achieve strong in-domain performance [ 24 ] , these approaches tend to overfit narrow distributions and lack transparent reasoning. This limits their robustness to domain shifts (e.g., geography, season) and scalability in resource-limited agricultural settings [ 10 ] . II-B Explainable AI in Vision-Language Models The opacity of VLMs necessitates explainability frameworks to enhance transparency and trustworthiness across diverse domains. Dinga et al. employ Shapley Values to interpret VLM-generated map descriptions, quantifying region-specific contributions to textual outputs and revealing how map labels and scale influence caption accuracy [ 25 ] . In medical imaging, Kamal et al. integrate U-Net, Vision Transformer, and Medical Language Graphs with LRP and LIME to interpret image captioning models, improving transparency in diagnostic report generation [ 26 ] . Masala and Leordeanu introduce GEST, a spatio-temporal event graph framework for explainable video description, converting structured graphs into natural language to enhance interpretability in vision-to-language tasks [ 27 ] . Natarajan

Read the complete text