CAD2DMD-SET: Synthetic Generation Tool of Digital Measurement Device CAD Model Datasets for fine-tuning Large Vision-Language Models

João Valente, Atabak Dehban, Rodrigo Ventura · 2025

Read CAD2DMD-SET: Synthetic Generation Tool of Digital Measurement Device CAD Model Datasets for fine-tuning Large Vision-Language Models by João Valente, Atabak Dehban, Rodrigo Ventura (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-21732.txt.

Introduction

João Valente, Atabak Dehban, and Rodrigo Ventura built a tool that generates synthetic training data for teaching vision-language models to read digital measurement devices. Even strong multimodal models fumble this simple task under clutter, occlusions, odd viewpoints, and motion blur. Such conditions are common in head-mounted cameras and augmented reality.

Their tool, CAD2DMD-SET, renders labeled images from 3D CAD models with realistic composition. They also release DMDBench, one thousand annotated real-world images for validation. Fine-tuning three leading models on the synthetic data improved scores substantially; one model's score rose by two hundred percent without degrading other tasks. The tool is planned as open-source.

Catalog record

CREATORJoão Valente, Atabak Dehban, Rodrigo Ventura
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-21732

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CAD2DMD-SET: Synthetic Generation Tool of Digital Measurement Device CAD Model Datasets for fine-tuning Large Vision-Language Models João Valente, Atabak Dehban, Rodrigo Ventura Abstract Recent advancements in Large Vision-Language Models (LVLMs) have demonstrated impressive capabilities across various multimodal tasks. They continue, however, to struggle with trivial scenarios such as reading values from Digital Measurement Devices (DMDs), particularly in real-world conditions involving clutter, occlusions, extreme viewpoints, and motion blur; common in head-mounted cameras and Augmented Reality (AR) applications. Motivated by these limitations, this work introduces CAD2DMD-SET, a synthetic data generation tool designed to support visual question answering (VQA) tasks involving DMDs. By leveraging 3D CAD models, advanced rendering, and high-fidelity image composition, our tool produces diverse, VQA-labelled synthetic DMD datasets suitable for fine-tuning LVLMs. Additionally, we present DMDBench, a curated validation set of 1,000 annotated real-world images designed to evaluate model performance under practical constraints. Benchmarking three state-of-the-art LVLMs using Average Normalised Levenshtein Similarity (ANLS) and further fine-tuning LoRA’s of these models with CAD2DMD-SET’s generated dataset yielded substantial improvements, with InternVL showcasing a score increase of 200% without degrading on other tasks. This demonstrates that the CAD2DMD-SET training dataset substantially improves the robustness and performance of LVLMs when operating under the previously stated challenging conditions. The CAD2DMD-SET tool is expected to be released as open-source once the final version of this manuscript is prepared, allowing the community to add different measurement devices and generate their own datasets. 1 Introduction 1.1 Motivation Despite the impressive progress of the current state-of-the-art large vision-language models (LVLMs) and their broad applicability in various domains, their implementation in real-world applications can present unforeseen challenges [ 1 ] . One example is reading from measurement devices, which is effortlessly performed by humans. The examples below, shown in Fig. 1 , exemplifies these difficulties. Figure 1: DMDBench Challenging Examples for InternVL (a) and LLaVa (b). Green text is correct answer, red text shows wrong answer. (a)Question: What information is being conveyed by the digital display in this image? (b)Question: Identify the device and summarise what it is displaying. Ground-Truth Answer: The digital display conveys a TEMPO reading of 30 BPM. The image shows a digital multimeter with a reading of 0.022 V. Non-fine-tuned Model Answer: The digital display shows the number ’ 30 ’, which could indicate a timer setting , a countdown , or a measurement reading, depending on the context of the device’s use. The device is a digital clock , and it is displaying the time as 00:22 . Fine-tuned Model Answer: The digital display conveys a TEMPO reading of 30 BPM. The device is a digital multimeter , and it is displaying 0.022 A . One contributing factor is the scarcity of open-source datasets expressly designed for measurement-device reading. Most large image–text corpora scraped from the internet are product-grade, displaying high-quality photos with minimal background clutter and frequently showing the same reading, often zero. A related problem was studied by [ 2 ] . These conditions contrast with real-world scenes, where motion blur, visual clutter, and diverse (non-zero) readings prevail. When DMD readings are captured by head-mounted cameras ( e.g. in AR applications), oblique viewpoints, and continuous motion blur intensify these difficulties [ 3 ] . 1.2 Objectives and Contributions Enabling LVLMs to read measurements from DMDs poses distinct challenges due to the diversity of display types (LED, LCD, etc.), measurement units, and frequent use of symbols and abbreviations. The absence of a dedicated dataset has also hindered progress in this domain. This research aims to bridge that gap, offering a foundation for advancing OCR and VQA methods tailored to digital measurement interpretation. To summarise, this work has four main contributions: • We provide CAD2DMD-SET, a flexible synthetic data generation tool targeted at creating a large dataset of DMD; • We provide a labelled real validation set (DMDBench) of multiple DMDs and benchmark various state of the art open-weight LVLMs; • We created a large synthetic dataset to fine-tune multiple LVLMs using CAD2DMD-SET; • We experimentally demonstrate the applicability of the tool in improving the performance of LVLMs in reading DMDs from unconstraint view points and analyse the sources of improvements. 2 Literature Review 2.1 Photogrammetry-based 3D Reconstruction Photogrammetry-based 3D reconstruction is a technique used to create 3D models of real-world objects or environments from a series of 2D photographs. It works by analyzing multiple images of the same object or scene taken from different angles and then using algorithms to triangulate the 3D coordinates of points in space. Several applications allow for such reconstruction and our two main options consisted of KIRI Engine , from KIRI Innovations and Object Capture , from Apple. KIRI Engine, designed for ease of use across mobile and desktop platforms, offers an accessible and cost-effective alternative to traditional scanning methods by relying solely on image data, supporting high-resolution texture mapping and mesh generation. Object Capture, which is part of the RealityKit framework on macOS, relies on structure-from-motion (SfM) and multi-view stereo (MVS) techniques to estimate camera poses and reconstruct dense point clouds, which are subsequently meshed and textured to produce high-fidelity 3D assets. Compared to traditional photogrammetry pipelines, which often require specialised software and substantial manual tuning, Object Capture emphasises ease of use and automation. Unlike KIRI Engine, the framework is optimised for Apple’s hardware ecosystem only, making use of GPU-accelerated computation and native integration with macOS APIs. Furthermore, it achieves results competitive with established photogrammetry tools. Although Object Capture uses SfM techniques for 3D asset reconstructions, some methods use Guassian Splatting [ 4 ] . We decided not to pursue such methods, however, because these 3D models do not have a mesh or a surface to unwrap, making UV mapping (a critical step in CAD2DMD-SET) unfeasible. While KIRI Engine is compatible with multiple platforms, Object Capture offers a free , open-source alternative that produced superior overall model quality. Its output featured cleaner, more organised meshes, which facilitated easier and quicker manual refinement of our 3D models, when needed. As such, our final choice for a photogrammetry-based 3D reconstruction tool was Object Capture. 2.2 3D Modeling and Rendering Today, a broad selection of free 3D modeling and rendering software is readily accessible. Fundamentally, modeling tools enable the accurate creation of shapes and structures, while rendering engines replicate the effects of lighting and materials to generate realistic or stylised visuals. These tools are extensively employed across various industries, and technological advancements have significantly enhanced their capabilities and availability. Two main options were initially considered, Unity’s Perception Package [ 5 ] and Blender . However, Unity’s Perception package was officially discontinued and is now community-maintained only, which makes it a less reliable choice. Blender , on the other hand, is a comprehensive, open-source software suite for 3D modeling, animation, rendering, and simulation. It was originally developed as an in-house application by NeoGeo and later released to the public under the GNU General Public License. Overall, Blender has evolved into a robust and widely adopted platform within both academic and industrial domains. Its open-source nature not only ensures cost-effectiveness but also allows for extensive customisation and community-driven enhancements, making it particularly suitable for research and development purposes. Furthermore, Blender supports a wide range of functionalities including mesh modeling, UV mapping, texturing, real-time

Read the complete text