Vision-Language Foundation Models for Comprehensive Automated Pavement Condition Assessment
Blessing Agyei Kyem, Joshua Kofi Asamoah, Anthony Dontoh, Armstrong Aboah · 2026
Read Vision-Language Foundation Models for Comprehensive Automated Pavement Condition Assessment by Blessing Agyei Kyem, Joshua Kofi Asamoah, Anthony Dontoh, Armstrong Aboah (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-08212.txt.
Introduction
Kyem and three colleagues ask whether tuning on domain instructions can enable full pavement review with vision-language models. General-purpose vision-language models perform strongly in daily domains but struggle with expert fields that demand exact terms, ordered reasoning, and building standards. PaveInstruct, with 278,889 image-response pairs across 32 task types, merges labels from nine varied road sets. PaveGPT, a pavement base model trained on it, was tested against top models on sight, grasp, and reasoning tasks.
Catalog record
| CREATOR | Blessing Agyei Kyem, Joshua Kofi Asamoah, Anthony Dontoh, Armstrong Aboah |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 104536 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.08212
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Vision-Language Foundation Models for Comprehensive Automated Pavement Condition Assessment Blessing Agyei Kyem, Joshua Kofi Asamoah, Anthony Dontoh, Armstrong Aboah Abstract General-purpose vision-language models demonstrate strong performance in everyday domains but struggle with specialized technical fields requiring precise terminology, structured reasoning, and adherence to engineering standards. This work addresses whether domain-specific instruction tuning can enable comprehensive pavement condition assessment through vision-language models. PaveInstruct, a dataset containing 278,889 image-instruction-response pairs spanning 32 task types, was created by unifying annotations from nine heterogeneous pavement datasets. PaveGPT, a pavement foundation model trained on this dataset, was evaluated against state-of-the-art vision-language models across perception, understanding, and reasoning tasks. Instruction tuning transformed model capabilities, achieving improvements exceeding 20% in spatial grounding, reasoning, and generation tasks while producing ASTM D6433-compliant outputs. These results enable transportation agencies to deploy unified conversational assessment tools that replace multiple specialized systems, simplifying workflows and reducing technical expertise requirements. The approach establishes a pathway for developing instruction-driven AI systems across infrastructure domains including bridge inspection, railway maintenance, and building condition assessment. 1 Introduction Recent advances in large open-source vision-language models (VLMs), including QwenVL [ 3 , 53 , 4 ] , LLaVA [ 26 , 25 ] have demonstrated strong capability in multimodal perception and language-guided reasoning tasks. While these models exhibit notable performance in open-domain environments, their direct application to specialized technical fields remains limited. Although proprietary models such as ChatGPT, Gemini, and Grok demonstrate strong multimodal capabilities in some technical fields, their limited transparency, inability to fine-tune, and data privacy concerns make them unsuitable for infrastructure assessment where regulatory compliance and reproducibility are essential. In high-precision domains such as medicine and autonomous driving, general-purpose VLMs have shown difficulty reasoning over domain-specific semantics, adhering to specialized terminology, and following structured expert protocols. These limitations have motivated the creation of domain-tailored instruction datasets such as Path-VQA [ 12 ] and VQA-RAD [ 19 ] for clinical imaging, and nuScenes-QA [ 41 ] , DriveLM [ 47 ] , and BDD-X [ 14 ] for autonomous driving, leading to dedicated VLMs such as LLaVA-Med [ 21 ] and Med-PaLM [ 48 ] . These developments underscore a broader conclusion: general-purpose VLMs are insufficient for expert-level reasoning in domain-critical applications, and performance gains require domain-aligned instruction datasets and supervision strategies. Pavement condition assessment presents similar challenges because it involves fine-grained distress identification, severity quantification aligned with engineering standards, precise spatial localization, and structured reporting to support maintenance decision-making. Existing instruction-tuning datasets such as LAION-5B [ 43 ] , Conceptual Captions [ 45 ] , COCO Captions [ 6 ] , Visual Genome [ 15 ] , and LLaVA-Instruct-150K [ 24 ] which are used to train these general-purpose VLMs contain minimal infrastructure content and do not encode pavement engineering terminology or reasoning processes. Consequently, current VLMs often misinterpret pavement distresses, provide nonspecific responses, and fail to follow standardized pavement evaluation procedures especially when tested in zeroshot scenarios. Although existing pavement datasets such as CrackSeg9k [ 17 ] , DeepCrack [ 28 ] , Crack500 [ 57 ] , Pavementscapes [ 51 ] , SVRDD [ 42 ] , PaveDistress [ 29 ] , and recent captioning datasets [ 33 , 18 ] have advanced detection, segmentation, and classification, these datasets are useful for unimodal vision-only tasks and do not provide text or instruction-response supervision required for multimodal technical reasoning, step-wise PCI estimation, or ASTM standards-compliant distress communication. Thus, while some recent works have begun experimenting with VLMs for pavement analysis, they remain constrained by the lack of instruction-grounded, standards-aligned datasets. Recent work has applied VLMs to pavement tasks like zero-shot crack detection and few-shot damage assessment, revealing their potential [ 58 ] . RoadBench [ 54 ] , for instance, offers a benchmark of synthetically captioned road images and introduces RoadCLIP, a non-generative, dual-encoder CLIP model for zero-shot classification and retrieval. However, these efforts face key limitations. First, they rely on prompting general-purpose models rather than fine-tuning domain-specific ones. Second, the captions in the dataset lack instruction-following, conversational structure and are not aligned with ASTM D6433 standards and guidelines. In addition, the models focus on narrow tasks such as classification or semantic localization. As a result, they fall short of the structured reasoning and instruction-following required for comprehensive pavement assessment, including distress detection, severity rating, PCI estimation, and maintenance suggestions. Bridging this gap requires instruction datasets grounded in domain standards and specialized multimodal models capable of end-to-end reasoning across all pavement tasks. To address this gap, this study introduces PaveInstruct , a unique multimodal instruction-following dataset for pavement condition assessment, and PaveGPT , a domain-specialized foundation model trained on this dataset. PaveInstruct integrates pavement imagery with engineering-aligned prompts and structured responses covering distress identification and localization, ASTM-based severity assessment, chain-of-thought PCI estimation, formatted condition reporting, and maintenance recommendations. By explicitly aligning model supervision with pavement engineering workflows, PaveGPT enables technical dialogue, evidence-grounded reasoning, and standards-compliant interpretation of pavement conditions. This work establishes the foundation for instruction-driven pavement intelligence and introduces a unique language-native model designed for automated pavement evaluation and decision support. The main contributions of the proposed approach are summarized below: 1. PaveInstruct, a comprehensive instruction-following dataset containing 278,889 image-instruction response pairs spanning 32 task types across five major categories, is introduced and will be made publicly available for research purposes to advance vision-language models in infrastructure domains. 2. A systematic pipeline for integrating heterogeneous pavement datasets is developed, addressing annotation format unification, coordinate system harmonization, and task-specific instruction generation while preserving semantic richness and engineering validity across nine diverse data sources. 3. PaveGPT, a domain-specialized vision-language foundation model, is presented and demonstrates strong performance across perception, understanding, and reasoning tasks while maintaining computational efficiency suitable for practical deployment in pavement management systems. 4. Comprehensive empirical evidence is provided showing that domain-specific instruction tuning transforms general-purpose VLMs into capable pavement assessment tools, achieving improvements exceeding 20% in spatial grounding, reasoning, and generation tasks, with consistent gains across different model architectures and sizes. 2 Related Works This section reviews existing pavement datasets and instruction-following multimodal datasets to establish the gap that PaveInstruct addresses. While pavement datasets have advanced distress detection through classification, detection, and segmentation tasks, they lack the natural language supervision required for training conversational assessment models, which instruction-following datasets have successfully enabled in domains such as medicine and autonomous driving. 2.1 Existing Pavement datasets Numerous pavement datasets have been developed for distress detection, primarily targeting single computer vision tasks with fixed annotation formats. This section reviews classification, detection, and segmentation datasets, identifying their limitations for vision-language model training. Classification Datasets. Early efforts produced image-level crack datasets to train classifiers distinguishing cracked vs. intact surfaces. For example, Özgenel et al. [ 38 ] compiled the Concrete Crack Images for Classification dataset with 40,000 227×227 pixel images (20k with cracks, 20k without) derived from concrete surfaces on a university campus data. Similarly, Maguire et al. [ 8 ] released the SDNET2018 dataset containing 56,000 annotated images of concrete cracks and non-cracks on bridge decks, walls, and pavements. These large image-level datasets provided plentiful data for training deep classifiers, but they only offer coarse labels (crack present or not) without any spatial localization of distress within the image. Object Detection Datasets. To enable spatial localization