Quantifying and Bridging the Fidelity Gap: A Decisive-Feature Approach to Comparing Synthetic and Real Imagery

Danial Safaei 1 , Siddartha Khastgir 1 , Mohsen Alirezaei 2 , Jeroen Ploeg 2 , Son Tong 2 and Xingyu Zhao 1 · 2025

Read Quantifying and Bridging the Fidelity Gap: A Decisive-Feature Approach to Comparing Synthetic and Real Imagery by Danial Safaei 1 , Siddartha Khastgir 1 , Mohsen Alirezaei 2 , Jeroen Ploeg 2 , Son Tong 2 and Xingyu Zhao 1 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-16468.txt.

Introduction

Danial Safaei and six co-authors measure virtual test quality in a paper made public on arXiv in 2025. Self-driving cars are vetted in simulators, but pretty synthetic images can still mislead the system under test. What matters is whether the car bases decisions on the same evidence in fake and real scenes.

Their metric, Decisive Feature Fidelity, uses explainable AI tools to find the features driving a system's outputs on matched real and synthetic image pairs, then compares those features across the pair. Counterfactual explanations supply the estimates, and a calibration scheme uses the metric to improve the simulator itself. On 2,126 matched pairs from KITTI and VirtualKITTI2, the metric exposed gaps that conventional output-based fidelity missed. Calibration guided by it improved feature-level and input-level fidelity without hurting output fidelity, across several systems under test.

Catalog record

CREATORDanial Safaei 1 , Siddartha Khastgir 1 , Mohsen Alirezaei 2 , Jeroen Ploeg 2 , Son Tong 2 and Xingyu Zhao 1
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-16468

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Quantifying and Bridging the Fidelity Gap: A Decisive-Feature Approach to Comparing Synthetic and Real Imagery Danial Safaei 1 , Siddartha Khastgir 1 , Mohsen Alirezaei 2 , Jeroen Ploeg 2 , Son Tong 2 and Xingyu Zhao 1 Abstract Virtual testing using synthetic data has become a cornerstone of autonomous vehicle (AV) safety assurance. Despite progress in improving visual realism through advanced simulators and generative AI, recent studies reveal that pixel-level fidelity alone does not ensure reliable transfer from simulation to the real world. What truly matters is whether the system-under-test (SUT) bases its decisions on the same causal evidence in both real and simulated environments—not just whether images “look real” to humans. This paper addresses the lack of such a behavior-grounded fidelity measure by introducing Decisive Feature Fidelity (DFF), a new SUT-specific metric that extends the existing fidelity spectrum to capture mechanism parity —the agreement in causal evidence underlying the SUT’s decisions across domains. DFF leverages explainable-AI (XAI) methods to identify and compare the decisive features driving the SUT’s outputs for matched real–synthetic pairs. We further propose practical estimators based on counterfactual explanations, along with a DFF-guided calibration scheme to enhance simulator fidelity. Experiments on 2126 matched KITTI–VirtualKITTI2 pairs demonstrate that DFF reveals discrepancies overlooked by conventional output-value fidelity. Furthermore, results show that DFF-guided calibration improves decisive-feature and input-level fidelity without sacrificing output value fidelity across diverse SUTs. I Introduction Scenario-based virtual testing with synthetic data is a standard for AV safety evaluation [ 13 , 2 ] , enabling the recreation of complex driving situations. Substantial resources are being directed toward the development of high-fidelity simulators [ 6 ] and generative-AI techniques [ 19 , 26 ] to enhance realism to better reflect actual driving conditions. Most existing efforts have concentrated on improving “pixel-level realism”, that is, aligning synthetic image data with real image data in a way that is agnostic to the system-under-test (SUT). For brevity we focus on image-based data and refer to pixels when discussing input-level fidelity, but the proposed approach extends to other modalities such as LiDAR by replacing pixel correspondence with point-cloud correspondence. However, recent studies suggest that pixel-level, SUT-agnostic fidelity does not assure reliable transfer of AV assessment conclusions between the virtual and real domains. What truly matters is whether the AV grounds its behaviors or decisions on the same underlying principles in both real and synthetic domains [ 4 , 22 , 14 , 28 ] . This emerging consensus has shifted attention from pixel-level realism of the data towards behavioral consistency of the SUT as a more meaningful indicator of simulation validity [ 18 , 17 , 24 , 15 ] . Fig. 1: Top: three levels of the fidelity spectrum (IV, LF, OV) from [ 4 ] . Bottom: our extension that adds decisive-feature fidelity (DFF) and a calibrator to adjust the generator. Safety-aware fidelity (SA) is defined in [ 4 ] but is not explicitly visualized here. The top part of Fig. 1 recalls the input/latent/output/safety-aware fidelity spectrum of [ 4 ] . The bottom part illustrates our extension: an XAI explainer extracts decisive features from the SUT for each paired real and synthetic input, the distance between decisive maps yields a DFF estimate, and this estimate is fed back to a calibrator that adjusts generator knobs to close mechanism gaps. The previous work [ 4 ] systematically categorized synthetic-data fidelity into four levels, as shown in Fig. 1 : input-level (IV) fidelity, latent-feature (LF) fidelity, output-value (OV) fidelity, and safety-aware (SA) fidelity. As the names suggest, these correspond respectively to the fidelity measured in the SUT’s input space (e.g., the perception neural network’s raw inputs), latent feature space, output vector space, and safety-critical output space. While this spectrum of fidelity spans the entire processing pipeline of a neural network, it remains too coarse to capture why the SUT exhibits the same or different behaviors when presented with paired real and synthetic inputs representing the same scenario. For example, an AV perception module may output the same steering command, say a 20-degree left turn, for both the real and synthetic inputs, implying high OV-fidelity. Yet the underlying reasons could differ : the real image might trigger the action due to a detected dog ahead, whereas in the synthetic version of the same scenario, the same decision arises from detecting a right-turn sign next to the dog (and the dog is misdetected, which is concerning). In this case, the system produces the same decision for different reasons, i.e., based on different causal evidence. Notably, the LF-fidelity defined in [ 4 ] cannot fully capture such discrepancies, as it simply compares all extracted features without distinguishing those that actually influence the SUT’s decision. This reveals a gap: existing metrics tell us what the SUT outputs, but not why . We need a metric that checks whether the SUT uses the same visual evidence (e.g., road edges, lane markings) in both domains. This gap directly motivates DFF. Throughout this paper, we evaluate and calibrate synthetic data for testing a fixed SUT; we do not train or adapt the SUT itself. To bridge this gap, we propose an explainable-AI (XAI)-based fidelity metric that quantifies the differences in the decisive features influencing the SUT’s outputs for a given pair of real and synthetic data. Specifically, we first formalize a new SUT-specific fidelity metric, termed Decisive Feature Fidelity (DFF), which extends the fidelity spectrum defined in [ 4 ] . Unlike the LF-fidelity in [ 4 ] , which compares all extracted features, DFF uses recent XAI methods to estimate and compare only the (subset of) features that truly influence the SUT’s output. We then design practical estimators based on counterfactual explanations to quantify DFF and use these estimates as feedback signals to recalibrate the synthetic data generator, thereby improving fidelity. In our experiments we instantiate G with a diffusion-based generator, but the proposed calibration objective is general and applies equally to reconfigurable simulators such as CARLA or PreScan. To validate our approach, we conduct experiments on 2126 matched KITTI–VirtualKITTI2 pairs across three SUTs (steering, YOLOP drivable-area, and YOLOP lane-line), demonstrating the utility of DFF for both assessment and calibration. In summary, the main contributions include: • A new SUT-specific fidelity metric : We formalize DFF, a metric that measures mechanism parity. We contrast it with existing metrics, arguing that focusing on the decisive-feature subspace is more meaningful than comparing raw inputs (IV), entire latent spaces (LF), or just final outputs (OV) of the SUT. • An XAI-based DFF estimator : We develop a practical estimator, using XAI methods such as counterfactual explainers (CF-XAI). The decisive maps for real and synthetic inputs are computed independently and then compared within this subspace. • A DFF-guided calibration objective : We formulate an optimization objective that utilizes DFF estimates as feedback signals to fine-tune the synthetic data generator. Validation on a held-out split demonstrates that this approach improves decisive-feature and input-level fidelity. Crucially, it preserves behavioral (OV) fidelity, ensuring that the SUT’s task performance does not degrade beyond a strictly defined tolerance (a pre-declared non-inferiority margin). II Related Work Virtual testing offers a scalable and risk-free way to assess AV safety across diverse scenarios. However, the fidelity

Read the complete text