AgriDoctor: A Multimodal Intelligent Assistant for Agriculture
Mingqing Zhang, Zhuoning Xu, Peijie Wang, Rongji Li, Liang Wang, Qiang Liu, Jian Xu, Xuyao Zhang, Shu Wu, Liang Wang · 2025
Read AgriDoctor: A Multimodal Intelligent Assistant for Agriculture by Mingqing Zhang, Zhuoning Xu, Peijie Wang, Rongji Li, Liang Wang, Qiang Liu, Jian Xu, Xuyao Zhang, Shu Wu, Liang Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-17044.txt.
Introduction
Mingqing Zhang and a large team build a farming assistant in a 2025 paper. Crop disease diagnosis guides sustainable farming and food security. Image-only tools miss farm knowledge and cannot chat with growers. Their assistant, AgriDoctor, pairs vision with language and expert knowledge for interactive diagnosis.
Training rests on a large new benchmark with hundreds of thousands of disease images. It adds expert entries and bilingual prompts for tool choice. Tests show the assistant beats top vision-language models on fine farm tasks. A knowledgeable, chat-based tool could bring expert-grade diagnosis to small farms far from agronomists.
Catalog record
| CREATOR | Mingqing Zhang, Zhuoning Xu, Peijie Wang, Rongji Li, Liang Wang, Qiang Liu, Jian Xu, Xuyao Zhang, Shu Wu, Liang Wang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 39 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-17044
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
AgriDoctor: A Multimodal Intelligent Assistant for Agriculture Mingqing Zhang, Zhuoning Xu, Peijie Wang, Rongji Li, Liang Wang, Qiang Liu, Jian Xu, Xuyao Zhang, Shu Wu, Liang Wang Abstract Accurate crop disease diagnosis is essential for sustainable agriculture and global food security. Existing methods, which primarily rely on unimodal models such as image-based classifiers and object detectors, are limited in their ability to incorporate domain-specific agricultural knowledge and lack support for interactive, language-based understanding. Recent advances in large language models (LLMs) and large vision-language models (LVLMs) have opened new avenues for multimodal reasoning. However, their performance in agricultural contexts remains limited due to the absence of specialized datasets and insufficient domain adaptation. In this work, we propose AgriDoctor , a modular and extensible multimodal framework designed for intelligent crop disease diagnosis and agricultural knowledge interaction. As a pioneering effort to introduce agent-based multimodal reasoning into the agricultural domain, AgriDoctor offers a novel paradigm for building interactive and domain-adaptive crop health solutions. It integrates five core components: a router, classifier, detector, knowledge retriever and LLMs. To facilitate effective training and evaluation, we construct AgriMM , a comprehensive benchmark comprising 400,000 annotated disease images, 831 expert-curated knowledge entries, and 300,000 bilingual prompts for intent-driven tool selection. Extensive experiments demonstrate that AgriDoctor, trained on AgriMM, significantly outperforms state-of-the-art LVLMs on fine-grained agricultural tasks, establishing a new paradigm for intelligent and sustainable farming applications. 1 Introduction Accurate identification of crop diseases plays a pivotal role in promoting sustainable agriculture, ensuring crop productivity, and maintaining global food security. However, most existing crop disease recognition approaches are based on unimodal techniques, such as image-based classification models agarwal2020potato ; khamparia2020seasonal ; sakkarvarthi2022detection ; dai2024dfn and object detection frameworks saleem2019plant ; panchal2023image ; wang2024pd .While effective in visual recognition, these methods lack deep modeling of agricultural knowledge and cannot interactively understand natural language queries, limiting their real-world use in agriculture. In recent years, artificial intelligence has undergone rapid development, particularly in the field of natural language processing (NLP). Among these advancements, the emergence of LLMs has further accelerated progress. Trained on massive textual corpora, LLMs have demonstrated remarkable capabilities in language understanding and generation tasks y2022large ; gandhi2023understanding ; nam2024using ; du2024haloscope ; liu2025llm4gen . Under instruction-based paradigms, LLMs can generate informative, contextually relevant responses grounded in domain knowledge, enabling natural language interaction with users. They have been successfully applied to various real-world domains, including legal consultation cheong2024not ; guha2023legalbench ; bignotti2024legal and clinical decision support li2024mediq ; kweon2024ehrnoteqa ; singhal2023large . In the field of crop science, preliminary efforts have explored the use of LLMs for tasks such as answering pest and disease-related questions or solving agricultural examination problems zhang2024empowering ; shaikh2024role ; kuska2024ai . Despite these developments, current LLMs still exhibit significant limitations in handling agricultural-specific terminology, multilingual capabilities, and external knowledge integration. Their performance in multimodal agricultural scenarios involving both images and text remains suboptimal. To bridge this gap, the emergence of LVLMs has brought significant advances in the field of multimodal learning. Models such as GPT-4o hurst2024gpt , Flamingo alayrac2022flamingo , Llava-onevision li2024llava , and Qwen2.5-VL bai2025qwen2 adopt unified architectures that integrate visual and textual processing, achieving impressive results in open-domain tasks such as image captioning, visual question answering, and multi-turn dialogue deng2024enhancing ; hu2024matryoshka ; liu2024convbench . However, their applications in the agricultural domain remain limited. These models are predominantly trained on general-purpose datasets and lack effective modeling of agriculture-specific visual concepts and expert knowledge. As a result, they face significant challenges in fine-grained disease classification, precise lesion localization, and domain-specific question answering within agricultural contexts. In this work, we introduce AgriDoctor , a domain-adapted, modular, and extensible multimodal framework specifically designed for intelligent crop disease diagnosis and agricultural knowledge interaction. As a pioneering attempt to apply agent-based multimodal reasoning in the agricultural domain, AgriDoctor integrates a unified pipeline comprising five core components: a router for intent classification and task dispatching, a disease classifier, a lesion detector, a knowledge retriever, and a LLM that integrates upstream visual outputs with textual inputs for context-aware reasoning and response generation. The model supports both pure textual queries and multimodal inputs, and produces multimodal outputs, including classification results, lesion visualizations, and textual explanations. Crucial to the effectiveness of AgriDoctor is the AgriMM dataset, a large-scale, high-quality multimodal agricultural benchmark we construct to support fine-grained reasoning. AgriMM includes 400,000 expert-annotated disease images across 29 crops and 138 disease types, bounding-box annotations for lesion localization, a structured knowledge base covering 831 pest and disease entries in both English and Chinese, and 300,000 bilingual prompt examples for intent-oriented tool selection. This rich and well-structured dataset enables AgriDoctor to learn fine-grained visual patterns, domain-specific knowledge, and task-aware reasoning strategies essential for real-world agricultural applications. Through comprehensive experiments, we demonstrate that AgriDoctor, when trained with AgriMM, significantly outperforms general-purpose LVLMs on a variety of agricultural tasks, including disease classification, lesion detection, and knowledge-grounded question answering. Our results highlight the importance of domain adaptation, modular reasoning, and high-quality agricultural supervision for advancing multimodal AI in agriculture. We hope that AgriDoctor and AgriMM together will serve as strong foundations for future research on domain-specific LVLMs and practical AI applications in sustainable farming. The key contributions of this work are summarized as follows: • We propose AgriDoctor, a pioneering agent-style multimodal framework that combines image understanding and natural language reasoning through a modular and extensible design, enabling intelligent agricultural assistance across diverse input modalities. • We construct AgriMM, a comprehensive and large-scale benchmark for multimodal agricultural tasks, comprising 400,000 annotated disease images, 831 expert-curated entries of agricultural pest and disease knowledge, and 300,000 prompts for intent-driven tool selection. • We perform extensive experiments demonstrating that AgriDoctor, trained on AgriMM, outperforms state-of-the-art LVLMs on fine-grained agricultural tasks, setting a new standard for intelligent farming applications. 2 Related Work 2.1 Tool Use in Multimodal Agents Tool learning has become essential for enhancing agents’ abilities to handle complex multimodal tasks. Current approaches fall into two main paradigms: retriever-based and LLM-based tool selection qu2024tool . Retriever-based methods match tools via rules or vector retrieval, while LLM-based approaches dynamically select tools based on semantic understanding. Recent work shows tool learning’s promise in multimodal contexts. For example, LLAVA-PLUS liu2025llava parses user instructions to orchestrate tool use, and CLOVA gao2024clova introduces a three-stage framework (inference, reflection, learning) to support continuous tool adaptation. To evaluate such capabilities, benchmarks like T-Eval chen2023t , API-Bank li2023api , and CARP zhang2024evaluating provide standardized criteria across dimensions such as reasoning, planning, and API interaction. However, relying on LLMs as tool selectors introduces challenges: 1) hallucinations can cause incorrect or conflicting tool use wang2023mint ; gou2023critic ; liu2025llava ; gao2024clova ; 2023toolformer ; 2023toolkengpt ; 2) performance heavily depends on model quality and prompt design; and 3) the inference cost of large models (e.g., Qwen2.5-7B takes 420ms per call) limits applicability in real-time domains like agriculture. To mitigate these issues, we propose AgriDoctor. 2.2 Agricultural LLMs In recent years, LLMs have been increasingly applied to agricultural knowledge services, incorporating expert feedback and domain-specific tuning to address agricultural challenges silva2023gpt . Techniques such as Retrieval-Augmented Generation and Ensemble Refinement have further improved generative performance. Instruction-tuned LLaMA 2 models, trained on over 1.5 million plant science papers, enhance understanding of plant