Robust and Safe Traffic Sign Recognition using N-version with Weighted Voting Journal: Reliability Engineering & System Safety

Linyun Gao, Qiang Wen, Fumio Machida · 2025

Read Robust and Safe Traffic Sign Recognition using N-version with Weighted Voting Journal: Reliability Engineering & System Safety by Linyun Gao, Qiang Wen, Fumio Machida (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-06907.txt.

Introduction

Linyun Gao, Qiang Wen, and Fumio Machida propose a safer way to read traffic signs in a 2025 paper. Self-driving cars rely on sign recognition, yet small attacks can fool a single model and endanger driving. Their framework runs several model versions and combines their answers by weighted soft voting.

Weights are set with a safety analysis method that scores failure modes and their effects. Riskier failure patterns thus count more in the final vote. The goal is recognition that stays robust when one model is attacked, a practical layer of defense for autonomous vehicles.

Catalog record

CREATORLinyun Gao, Qiang Wen, Fumio Machida
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE63 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-06907

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Robust and Safe Traffic Sign Recognition using N-version with Weighted Voting Journal: Reliability Engineering & System Safety Linyun Gao, Qiang Wen, Fumio Machida Abstract Autonomous driving is rapidly advancing as a key application of machine learning, yet ensuring the safety of these systems remains a critical challenge. Traffic sign recognition, an essential component of autonomous vehicles, is particularly vulnerable to adversarial attacks that can compromise driving safety. In this paper, we propose an N-version machine learning (NVML) framework that integrates a safety-aware weighted soft voting mechanism. Our approach utilizes Failure Mode and Effects Analysis (FMEA) to assess potential safety risks and assign dynamic, safety-aware weights to the ensemble outputs. We evaluate the robustness of three-version NVML systems employing various voting mechanisms against adversarial samples generated using the Fast Gradient Sign Method (FGSM) and Projected Gradient Descent (PGD) attacks. Experimental results demonstrate that our NVML approach significantly enhances the robustness and safety of traffic sign recognition systems under adversarial conditions. 1 Introduction In recent years, machine learning (ML) has become a fundamental technology in safety-critical domains, including autonomous driving. In particular, computer vision (CV) systems powered by ML models play a crucial role in enabling autonomous vehicles to perceive and interpret their surroundings [ 1 ] . However, many state-of-the-art CV models rely on complex, unexplainable black-box ML models such as deep neural networks (DNNs). The uncertain outputs raise concerns about the reliability of such systems, especially in high-risk environments where incorrect predictions can lead to severe consequences. Therefore, developing robust methods to enhance the reliability of ML systems remains an urgent research challenge. The N-version Machine Learning (NVML) system leverages the traditional N-version programming technique by incorporating diverse ML models within the same system to enhance output reliability [ 2 ] . Prior research has shown that NVML can improve system reliability in tasks such as image classification [ 3 ] and steering control [ 4 ] . Since different models in an NVML system may produce varying predictions, an effective decision-making mechanism is required to ensure reliable outputs. For example, in a three-version ML-based image classification system, three independent models generate independent predictions, and the final result is determined through majority voting [ 5 ] . However, majority voting has inherent limitations. A major drawback is its inability to resolve cases where all models produce distinct predictions, resulting in a tie with no clear consensus. For example, if three models predict classes A, B, and C, the system is unable to determine a definitive outcome. A simple majority vote may not always yield the most accurate result. In such scenarios, prioritizing the prediction from the model with the highest historical accuracy could be a more effective strategy, as it is more likely to provide a reliable classification for difficult cases. In this work, we explore more refined voting mechanisms for NVML systems in safety-critical applications. We investigate multiple decision strategies in the NVML image classification systems, including majority voting, weighted voting, soft voting, and weighted soft voting. In particular, we emphasize the weighted soft voting approach, which leverages the confidence levels of model predictions to assign appropriate weights to each model. By incorporating confidence scores and reliability of different models, this method has the potential to enable the system to make more informed decisions, ultimately enhancing system reliability in safety-critical applications. To determine safety-aware weights in safety-critical applications, we introduce a metric to evaluate the safety of ML-based traffic sign recognition systems for autonomous vehicles. We utilize the Failure Mode and Effects Analysis (FMEA) method to systematically examine potential failures arising from recognition errors during autonomous driving. By identifying and assessing these failure modes, we assign risk scores based on severity and likelihood. These scores guide the weighting process, ensuring that models associated with lower risks have a greater influence on the system’s final output. We evaluate the effectiveness of various voting mechanisms, including weighted soft voting, through experiments on a traffic sign recognition system comprising three distinct ML models: AlexNet [ 6 ] , VGG16 [ 7 ] , and EfficientNetB0 [ 8 ] . System performance is assessed using metrics such as accuracy, safety score, and the number of high-severity misclassifications. These metrics provide a comparative analysis of different voting strategies. Our results demonstrate that the proposed weighted soft voting method not only enhances accuracy but also significantly improves the safety and reliability of the NVML system by reducing severe misclassification instances. Additionally, we introduce an LLM-based weight assignment approach, where ChatGPT generates model weights based on a severity matrix and misclassification probabilities. However, preliminary evaluations indicate that this method produces suboptimal solutions, highlighting the need for more specialized LLMs and improved prompt engineering. The existing study [ 9 ] has demonstrated the advantage of safety-aware weighted voting for N-version traffic sign recognition systems; we extend this work by analyzing the effectiveness of NVML systems in adversarial environments. The NVML system with the proposed voting mechanism can improve the reliability even under adversarial attack conditions. Adversarial attacks pose a significant challenge to the reliability of ML systems, particularly in safety-critical applications such as autonomous vehicles. These attacks exploit vulnerabilities in ML models by introducing imperceptible perturbations, leading to potentially severe consequences. While traditional single-model systems often lack sufficient robustness against such attacks, NVML systems leverage model redundancy and diversity to enhance robustness. We evaluate the robustness of both single-model ML systems and a proposed three-version NVML system comprising AlexNet, VGG16, and EfficientNetB0, using adversarial samples generated with the Fast Gradient Sign Method (FGSM) [ 10 ] and Projected Gradient Descent (PGD) [ 11 ] attacks. The results reveal that while single-model systems experience significant performance degradation, the NVML system demonstrates superior robustness, particularly when employing advanced voting mechanisms such as safety-aware weighted soft voting. These findings underscore the critical role of NVML systems in ensuring safety and reliability in real-world, even under adversarial environments. The contributions of this paper can be summarized as follows: 1. We introduce a safety-aware weighted soft voting mechanism for NVML systems. 2. We develop a safety metric based on the FMEA method to evaluate and improve the safety of ML-based traffic sign recognition systems. 3. We demonstrate the effectiveness of the proposed safety-aware weighted voting mechanism through the experiments using a real traffic sign dataset. 4. We examine the potential LLM-based approach for weight assignment. 5. Our extensive robustness experiments show that the proposed NVML system, especially when using weighted soft voting, significantly enhances reliability and resilience under adversarial attacks, outperforming single-model systems in maintaining high accuracy and safety. The rest of the paper is organized as follows: Section 2 reviews related work. Section 3 provides background information on traffic sign recognition and the FMEA approach. Section 4 describes the proposed voting mechanisms. Section 5 presents the proposed safety metric. Section 6 presents the experimental setup and results, and Section 7 concludes the paper. 2 Related work Recent advancements in multi-version ML techniques have aimed at enhancing the reliability of ML systems. Xu et al. explored the potential of developing fault-tolerant deep learning systems through model redundancy (i.e., NV-DNN [ 3 ] ). Experimental results indicate that their approach can improve the fault tolerance of deep learning systems. Unlike NV-DNN, which processes

Read the complete text