DT 2 : Decision-Targeted Digital Twins

Harry Amad, Mihaela van der Schaar · 2026

Read DT 2 : Decision-Targeted Digital Twins by Harry Amad, Mihaela van der Schaar (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-25923.txt.

Introduction

Harry Amad and Mihaela van der Schaar question how digital twins, virtual replicas used to test decisions before acting, are usually trained. Standard training minimizes one-step prediction error. The pair prove that with limited model capacity, a twin trained that way can rank candidate policies in the wrong order, even though ranking policies by reward is the twin's actual job. Experiments confirm the flaw appears in practice, not just in theory.

Decision-targeted training is their fix, packaged as DT squared. Policy values are first estimated from logged data with fitted Q-evaluation, giving proxy ground truth. The twin then learns to produce rollouts that keep the same ranking between policy pairs, using a loss that works with any model architecture. Across settings, the method improves policy ranking and cuts decision regret for both familiar and unseen policies, while simulation accuracy stays respectable.

Catalog record

CREATORHarry Amad, Mihaela van der Schaar
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE83 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.25923

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

DT 2 : Decision-Targeted Digital Twins Harry Amad, Mihaela van der Schaar Abstract A digital twin (DT) is a virtual model of a real-world system that can assist decision-making by simulating scenarios induced by different policies. However, typical machine learning-based DTs do not optimise for this use case. We prove that, when model capacity is limited, training DTs to minimise one-step transition errors can produce suboptimal models for ranking sets of policies according to a reward function. We further show that this holds empirically, even with expressive model classes. To address this, we introduce DT 2 , a decision-targeted DT training paradigm. Firstly, DT 2 uses fitted Q-evaluation to estimate values of candidate policies from offline data. A DT is then trained to generate rollouts that preserve pairwise policy rankings derived from these proxy ground-truth values with an architecture-agnostic loss function. We empirically demonstrate the efficacy of our method across a range of settings and architectures. DT 2 consistently improves policy ranking and reduces decision regret during policy selection relative to conventional DT training, both for policies used during training and for unseen policies, while maintaining a good level of raw simulation fidelity. 1 Introduction A digital twin (DT) is a virtual model of a real-world system that can simulate its dynamics. DTs are appealing in many domains, such as finance ( 59 ) , climate science ( 71 ) , manufacturing ( 54 ) , energy ( 28 ) , agriculture ( 49 ) , robotics ( 41 ) , and medicine ( 30 ) , due to their ability to guide decision-making processes. This typically manifests with a user deliberating over DT-generated simulations of potential policies ( 63 ; 10 ) to select the best one, or establish a preference ordering. Much of the DT literature highlights this critical use case, asserting that a useful DT is one that “aids decision making” ( 72 ) , and “[informs] decisions that realize value” ( 44 ) . To enable such decision support, DTs must generate realistic simulations that allow for user interrogation and verification, while ensuring the learned dynamics lead to correct rankings of relevant policies. In machine learning (ML)-based DTs, however, this goal of decision support is often overlooked. Typically, ML-based DTs involve a model that is trained to minimise a measure of one-step transition error, such as mean squared error (MSE) or negative log likelihood (NLL), across all variables and time points in a dataset of observed state-action trajectories ( 56 ; 33 ; 27 ; 6 ; 40 ; 3 ) . While this can lead to high fidelity simulators, such direction-, variable-, and timepoint-agnostic objectives do not explicitly focus on correctly ranking policies, and they can misalign DTs with their primary downstream purpose by under‑emphasising variables or temporal slices that are especially crucial for discriminating between policies. For example, consider a medical DT, used to help clinicians treat septic shock. Despite large data availabilities in ICU settings, clinicians typically focus on only a few key indicators to make timely decisions ( 47 ) , like blood pressure and lactate levels for sepsis. A DT trained with a typical, variable-agnostic, simulation loss considers non-critical dimensions as equivalent to such key indicator variables. By wasting model capacity on largely irrelevant features, such a DT could provide suboptimal decision support, jeopardising patient health. For a visual example of how simulation training can lead to poor decisions due to being direction-agnostic, see Figure 1 . In this work, we emphasise that, for human-in-the-loop decision support, a DT need not be a perfect replica of previously observed reality, and we contend that DT construction should move away from such context-agnostic optimisation, towards what is best for decision support. Optimal glucose True rollouts DT rollouts \hat{\Phi}_{1} Time Glucose \hat{\Phi}_{2} Time Glucose Figure 1: We visualise two diabetes DTs, \hat{\Phi}_{1} (top) and \hat{\Phi}_{2} (bottom), under treatment plans with high , low , and adaptive insulin levels, and we consider the ‘value’ of a treatment plan as the time it spends in the optimal glucose region. \hat{\Phi}_{1} has lower MSE from the true rollouts, yet it leads to an incorrect treatment plan preference ordering ( high \approx low \succ adaptive ). In contrast, \hat{\Phi}_{2} leads to a correct preference ordering ( adaptive \succ high \approx low ), despite its higher MSE, as its rollouts traverse the correct glucose regions. For this purpose, we introduce DT 2 : a d ecision- t argeted DT training paradigm that balances between seeking fidelity in general, and in those parts of the transition distribution that are most relevant for ranking candidate policies according to a known reward function. Since ground-truth rankings amongst candidate policies are not generally known a priori , we firstly estimate policy values using off-policy evaluation (OPE) methods, which can estimate scalar policy values from offline data ( 20 ) but, distinct from DTs, offer little in the way of interpretability, limiting their application in high-stakes domains ( 55 ) . We use these OPE estimates to construct proxy ground-truth pairwise rankings amongst candidate policies, and introduce a differentiable ranking loss function that operates alongside standard simulation objectives to encourage DTs to generate simulations that preserve these proxy rankings. By doing so, we distil the ranking ability of OPE into the interpretable structure of a DT. A tunable hyperparameter, \lambda , modulates the trade-off between raw simulation accuracy and reconstruction of OPE policy rankings. See code here https://github.com/harrya32/decision-targeted-dt and here https://github.com/vanderschaarlab/decision-targeted-dt . In summary, our contributions are: 1. We prove that standard DT training can be theoretically suboptimal for decision-making when the DT hypothesis space is limited, establishing motivation for decision-targeted training ( \lx@sectionsign 3 ). 2. We propose DT 2 , an architecture-agnostic decision-targeted DT training paradigm that aligns simulations with proxy policy rankings ( \lx@sectionsign 4 ). We use a smooth approximation of Kendall’s rank correlation coefficient ( 31 ) ( \lx@sectionsign 4.1 ) to enable supervision with pairwise preference targets derived from OPE policy value estimates ( \lx@sectionsign 4.2 ). 3. We empirically demonstrate the alignment gap predicted by our theory ( \lx@sectionsign 6.1 ), and show that this persists even with highly expressive hypothesis spaces ( \lx@sectionsign 6.2 ). Across a suite of continuous control tasks, DT 2 reduces decision regret by more than 50\% at an MSE cost of only 17\% , demonstrating capacity for better decision support. Furthermore, we show that this advantage extends to out-of-distribution settings, achieving better preference orderings amongst policies never seen during training ( \lx@sectionsign 6.3 ). 2 Formalism We now formalise the problem of decision support that we aim to address with DTs. 2.1 The Decision-Making Goal Let a real-world system be modelled as a Markov Decision Process (MDP) denoted by \mathcal{M}:=(\mathcal{X},\mathcal{A},\Phi,r,\gamma) . Here, \mathcal{X}\subseteq\mathbb{R}^{d_{\mathcal{X}}} is a d_{\mathcal{X}} -dimensional state space, \mathcal{A}\subseteq\mathbb{R}^{d_{\mathcal{A}}} is a d_{\mathcal{A}} -dimensional action space, \Phi:\mathcal{X}\times\mathcal{A}\rightarrow\mathcal{P}(\mathcal{X}) is the transition distribution, where \mathcal{P}(\mathcal{X}) denotes the set of probability measures over \mathcal{X} , r:\mathcal{X}\times\mathcal{A}\times\mathcal{X}\rightarrow\mathbb{R} is a bounded reward function, and \gamma\in[0,1) is a discount factor. At time t , given the current state \mathbf{x}_{t}\in\mathcal{X} and action \mathbf{a}_{t}\in\mathcal{A} , the next state is sampled according to \mathbf{x}_{t+1}\sim\Phi(\cdot|\mathbf{x}_{t},\mathbf{a}_{t}) . A policy \pi:\mathcal{X}\rightarrow\mathcal{P}(\mathcal{A}) can interact with \mathcal{M} in an initial state \mathbf{x}_{0} , yielding a

Read the complete text