xaitimesynth: A Python Package for Evaluating Attribution Methods for Time Series with Synthetic Ground Truth

Gregor Baer [email protected] · 2026

Read xaitimesynth: A Python Package for Evaluating Attribution Methods for Time Series with Synthetic Ground Truth by Gregor Baer [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-06781.txt.

Introduction

Baer presents xaitimesynth, a Python package for testing time series attribution against known ground truth. Real datasets rarely show which time points drive a prediction, so studies rebuild synthetic test data from scratch each time. This package reuses that setup: each sample sums a background signal and a local class-discriminating feature, with the feature window tracked as a truth mask. A fluent generation API and YAML configs support univariate and multivariate series, plus standard localization metrics.

Catalog record

CREATORGregor Baer [email protected]
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE14092
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.06781

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

xaitimesynth: A Python Package for Evaluating Attribution Methods for Time Series with Synthetic Ground Truth Gregor Baer [email protected] Abstract Evaluating time series attribution methods is difficult because real-world datasets rarely provide ground truth for which time points drive a prediction. A common workaround is to generate synthetic data where class-discriminating features are placed at known locations, but each study currently reimplements this from scratch. We introduce xaitimesynth , a Python package that provides reusable infrastructure for this evaluation approach. The package generates synthetic time series following an additive model where each sample is a sum of background signal and a localized, class-discriminating feature, with the feature window automatically tracked as a ground truth mask. A fluent data generation API and YAML configuration format allow flexible and reproducible dataset definitions for both univariate and multivariate time series. The package also provides standard localization metrics, including AUC-PR, AUC-ROC, Relevance Mass Accuracy, and Relevance Rank Accuracy. xaitimesynth is open source and available at https://github.com/gregorbaer/xaitimesynth . † † shortheadings: xaitimesynth: Evaluating Attribution Methods with Synthetic Ground Truth / Baer † † firstpage: 1 keywords explainable AI, time series classification, synthetic benchmarks, XAI evaluation, attribution methods 1 Introduction Explainable AI (XAI) methods aim to make machine learning predictions interpretable, commonly by producing attributions that indicate how much each input feature contributed to a prediction. As deep learning models see growing adoption for time series classification ( 10 ) , attribution methods are increasingly adapted to temporal data (for a review, see 26 ). Yet producing attributions is only half the problem: we also need to evaluate them. XAI methods are commonly assessed through functional evaluation, where computational metrics quantify desirable properties of explanations ( 18 ) . One central property is correctness : how accurately an explanation reflects the model’s decision-making process. A widely used family of correctness metrics is perturbation-based, measuring how model output changes when supposedly important features are altered. However, these metrics can be sensitive to implementation choices ( 23 ; 7 ; 25 ) and yield systematically different results across classes ( 5 ; 6 ) , raising questions about their reliability as standalone evaluation tools. Localization-based metrics offer a complementary approach. Instead of perturbing inputs, they measure the spatial agreement between attributions and known ground truth feature locations. Since real-world datasets rarely provide such ground truth, researchers generate synthetic data where class-discriminating features occupy known regions ( 15 ; 27 ; 8 ; 24 ; 20 ; 5 ; 19 ; 14 ) . This strategy can serve as a sanity check ( 1 ) . Synthetic evaluation has been critiqued for not reflecting the model’s learned decision boundary ( 22 ) . A common assumption, also recognized by 18 , is that if the only class-discriminating information in the data is the simulated feature and a model achieves high accuracy, it likely relies on that feature. This assumption can break down through shortcut learning: if the generation process introduces unintended artifacts, such as systematic differences in signal statistics between classes, the model may exploit those instead. This is a risk researchers should consider when designing synthetic evaluation setups. Despite these limitations, synthetic data gives researchers full control over the data-generating process, allowing them to investigate attribution behavior in settings where the ground truth is known by construction. In practice, however, each study reimplements synthetic data generation independently. We introduce xaitimesynth , a Python package that standardizes this evaluation workflow for time series attribution methods. Following the same principle as CLEVR-XAI ( 3 ) , which provides ground truth feature locations for evaluating visual explanations, the package generates synthetic time series where each sample combines a background signal with a localized, class-discriminating feature, and records the feature location as a ground truth mask. A fluent data generation API and YAML configuration format support reproducible dataset definitions for both univariate and multivariate time series, with fine-grained control over feature placement across channels. The package also provides standard localization metrics, including AUC-PR, AUC-ROC, Relevance Mass Accuracy, and Relevance Rank Accuracy. 2 Related Work Several packages produce post-hoc attributions. Captum ( 16 ) is a widely used library for feature attribution in PyTorch, applicable across data modalities; it also includes perturbation-based evaluation metrics such as infidelity and sensitivity. Since general-purpose explanation libraries typically require adaptation for temporal data, packages like TSInterpret ( 13 ) and Time Interpret ( 9 ) extend Captum and other backends to provide unified interfaces for time series attribution methods. For evaluation, Quantus ( 12 ) offers a comprehensive collection of explanation quality metrics, including localization metrics such as those we implement, but targets primarily image data and does not support time series generation. AIX360 ( 4 ) includes evaluation metrics such as faithfulness and monotonicity, but not localization-based metrics or synthetic data generation. Among existing packages, Time Interpret ( 9 ) comes closest to supporting localization-based evaluation for time series: it provides both perturbation-based and localization metrics alongside its attribution methods. However, it does not include synthetic data generation utilities. On the data generation side, TimeSynth ( 17 ) generates synthetic time series from configurable signal and noise processes, and GluonTS ( 2 ) includes some synthetic data utilities for forecasting. Neither tracks ground truth feature locations, as they are not designed for XAI evaluation. In practice, researchers who need localization-based evaluation for time series often implement their own synthetic data pipelines. These implementations vary in complexity and are not always reusable across studies. xaitimesynth fills this gap by combining configurable synthetic time series generation, with automatic ground truth tracking, and standard localization metrics in a single package. It complements existing attribution and evaluation libraries by providing the reusable data generation infrastructure they lack. Table 1 summarizes the capabilities of the different packages. Table 1: Comparison of related packages. xaitimesynth combines synthetic time series generation with localization metrics for evaluating attribution correctness. Package XAI methods Perturbation metrics Localization metrics Synth. time series TSInterpret ✓ — — — Captum ✓ ✓ — — AIX360 ✓ ✓ — — Quantus — ✓ ✓ — Time Interpret ✓ ✓ ✓ — TimeSynth — — — ✓ GluonTS — — — ✓ xaitimesynth — — ✓ ✓ 3 Package Design The package has two main components: a data generation module that produces synthetic time series with tracked ground truth masks, and an evaluation module that scores attributions against those masks. We describe each below. 3.1 Data Generation Following the additive generation approach used in 5 , each synthetic time series \mathbf{x}=\mathbf{n}+\mathbf{f} is constructed as the sum of a background signal \mathbf{n} and a class-specific feature \mathbf{f} , where \mathbf{n} represents a background pattern (e.g., Gaussian noise, a random walk, or a seasonal signal) and \mathbf{f} contains the class-discriminating pattern within a designated time window, with zeros elsewhere. The package records this window as a binary ground truth mask for every sample. For multivariate time series, each channel can carry independent signals and features, with optional alignment of feature windows across channels. The builder API lets the user declaratively specify signals and features for each class; the package handles ground truth tracking, normalization, and output formatting internally. The following example in Listing 1 defines a two-class dataset, generates train and test splits, and

Read the complete text