Are vision-language models ready to zero-shot replace supervised classification models in agriculture?

Earl Ranario, J. Mason Earles · 2025

Read Are vision-language models ready to zero-shot replace supervised classification models in agriculture? by Earl Ranario, J. Mason Earles (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-15977.txt.

Introduction

Earl Ranario and Mason Earles benchmark vision language models for farm image classification in a 2025 paper. Such models are proposed as general purpose visual tools, yet their reliability for agricultural decisions was poorly tested. The team uses 27 data sets from the AgML collection, spanning 162 classes and 248,000 images.

Tasks cover plant disease, pest and damage, and species identification. Zero shot vision language models fall well short of a supervised YOLO11 baseline on every task. The best model, Gemini-3 Pro, reaches about 62 percent average accuracy under multiple choice prompting, while open ended answers score far lower. Plant and weed species prove easiest, and pest and damage hardest. The verdict: assistive use only, for now.

Catalog record

CREATOREarl Ranario, J. Mason Earles
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-15977

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Are vision-language models ready to zero-shot replace supervised classification models in agriculture? Earl Ranario, J. Mason Earles Abstract Vision-language models (VLMs) are increasingly proposed as general-purpose solutions for visual recognition tasks, yet their reliability for agricultural decision support remains poorly understood. We benchmark a diverse set of open-source and closed-source VLMs on 26 agricultural image classification datasets from the AgML ( https://project-agml.github.io/ ) collection, spanning 121 classes and 248,000 images across plant disease, pest and damage, and plant and weed species identification. Across all tasks, zero-shot VLMs substantially underperform a supervised task-specific baseline (YOLO11), which consistently achieves over 95% accuracy per dataset. Under multiple-choice prompting, the best-performing VLM (Gemini-3 Pro) reaches approximately 62% average accuracy, while open-ended prompting yields substantially lower performance, with all models falling below the naive majority-class baseline. Applying LLM-based semantic judging increases open-ended accuracy for top models (e.g., from 21% to 30% for GPT-5 and from 22% to 30% for Gemini-3 Pro) and alters model rankings, demonstrating that evaluation methodology meaningfully affects reported conclusions. Among open-source models, Qwen-VL-72B performs best, approaching closed-source performance under constrained prompting. Few-shot in-context learning further improves performance, with Qwen3-VL-8B reaching 52% accuracy under MCQA 3 and 42% under open-ended prompting at 5-shot, surpassing the zero-shot performance of larger closed-source models. Task-level analysis shows that plant and weed species classification is consistently easier than pest and damage identification, which remains the most challenging category across models. Overall, these results indicate that current off-the-shelf VLMs are not yet suitable as standalone agricultural diagnostic systems, but show promise as assistive components when paired with constrained label ontologies, calibrated prompting strategies, and domain-aware evaluation protocols. Figure 1 : Model performance was evaluated using 26 datasets spanning 121 classes from the AgML collection, totaling 248,000 images. These datasets were collected from a wide range of geographic locations and plant species. The tasks derived from these datasets include plant disease identification tasks, pest and damage detection, and plant or weed species classification. 1 Introduction Global agriculture currently faces challenges of climate extremes, limited technological access, and a lack of publicly available, standardized data [ 1 ] . Therefore, there is a need for climate-resilient and nutrient-stable crops [ 6 ] . While researchers study crops to identify genetic or environmental drivers of disease or yield, for example, acquiring high-quality phenotypic data across large-scale experiments remains manually intensive and costly [ 23 ] . To aid the improvement of high-throughput phenotyping, researchers have integrated artificial intelligence into their experiments. However, expert or model-based visual interpretations, whether from visual guides or vision models, often fail to generalize across different geographic locations. This is due to a “domain shift”, where models trained in one location collapse when deployed in “in-the-wild” environments [ 10 ] . As a result, this creates a knowledge gap between researchers and farmers where each group lacks information of each other’s domain, failing to identify causal stressors and provide the correct management advice. Artificial intelligence, specifically Vision-Language Models (VLMs), addresses these challenges by serving as “virtual agronomists” by integrating visual perception with deep semantic reasoning. These multi-modal tools democratize expert knowledge, allowing users to upload a single image to receive candidate diagnoses, causal insights, and real-time management recommendations. At the same time, agricultural image understanding presents challenges that differ substantially from those in general-purpose benchmarks [ 11 ] . Tasks such as disease diagnosis, pest and damage identification, and fine-grained species recognition often involve subtle visual cues, high intra-class variability, and strong dependence on contextual information such as crop type, phenological stage, geography, and management history. Historically, these challenges have been addressed using supervised, task-specific models trained on curated datasets [ 14 ] . While effective within constrained settings, such approaches require substantial annotation effort and often struggle to generalize across crops, regions, and imaging conditions [ 18 , 31 , 29 ] . However, it remains unclear whether current VLMs are reliable enough for agricultural classification tasks, particularly in settings where misclassification can lead to incorrect or costly management actions. Existing agricultural benchmarks have begun to explore this question, often revealing substantial performance gaps between general-purpose VLMs and domain-specific requirements [ 13 , 3 , 24 ] . At the same time, reported results vary widely depending on evaluation protocol, prompting format, and scoring methodology, making it difficult to draw consistent conclusions about model readiness or comparative performance. In this work, we present a large-scale, systematic benchmark of modern open-source and closed-source vision–language foundation models across 26 agricultural classification datasets from the AgML collection [ 13 ] , spanning 121 classes across plant disease, pest and damage, and plant and weed species identification tasks. Crucially, we do not treat benchmarking as a purely descriptive exercise. Instead, we use this evaluation to interrogate how and under what conditions VLMs succeed or fail in agricultural settings. Specifically, our study reveals five high-level findings that frame the contributions of this paper. • First, we provide an empirical comparison between state-of-the-art zero-shot VLMs and supervised, task-specific baselines. Our results identify a significant performance gap, highlighting the current limitations of general-purpose models in fine-grained agricultural recognition. • Second, we demonstrate that constraining the VLM output space through multiple-choice prompting substantially improves performance over open-ended generation. This finding establishes the importance of explicit label ontologies in reducing semantic ambiguity for specialized domains. • Third, we analyze the impact of evaluation metrics on model rankings, showing that LLM-based semantic judging captures nuances of correctness that traditional string-matching fails to detect, while also identifying the necessary assumptions for this approach. • Fourth, task-level differences indicate that pest and damage identification remains particularly challenging relative to species classification, underscoring the limits of single-image diagnosis without clear contextual priors. • Finally, we investigate the efficacy of few-shot in-context learning, demonstrating how the inclusion of a limited set of labeled contextual priors can significantly enhance model predictions. Table 1: Models were evaluated using both multiple-choice question answering (MCQA) and open-ended question (OEQ) formats, with task-specific prompts curated accordingly. Prompts were structured to evaluate a model’s selective behavior and how it leverages species-level cues. Format Tasks Prompt MCQA 1 All “Classify this image into one of the following categories: {across species classes}. Respond with ONLY the category name, nothing else.” MCQA 2 Disease, Pest, Damage “Classify this image into one of the following categories: {within species classes}, None of the above. Respond with ONLY the category name, nothing else.” MCQA 3 Disease, Pest, Damage “Classify this image of a {plant type} plant into one of the following categories: {within species classes}, None of the above. Respond with ONLY the category name, nothing else.” MCQA 4 Disease, Pest, Damage “Classify this image into one of the following categories: {across species classes}, None of the above. Respond with ONLY the category name, nothing else.” OEQ Disease, Pest, Damage “Respond in one sentence: What disease, pest, damage type, or other stress, if any, is exhibited in this image of a {plant type} plant?” OEQ Plant, Weed Species “Respond in one sentence: What plant or weed species, if any, is shown in this image?” Together, these findings have direct implications for the deployment, evaluation, and future development of vision-language systems in agriculture. Rather than positioning VLMs as drop-in replacements for supervised

Read the complete text