Structure-Aware Set Transformers: Temporal and Variable-Type Attention Biases for Asynchronous Clinical Time Series
Joohyung Lee, Kwanhyung Lee, Changhun Kim, Eunho Yang · 2026
Read Structure-Aware Set Transformers: Temporal and Variable-Type Attention Biases for Asynchronous Clinical Time Series by Joohyung Lee, Kwanhyung Lee, Changhun Kim, Eunho Yang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-06605.txt.
Introduction
Lee and team restore time and variable priors in the STAR Set Transformer for digital health records. For uneven time series, input layout matters as models split events, not slicing time. STAR adds soft attention biases: a temporal nearness penalty with timescales and a field-type affinity from a feature-fit matrix, tested across fusion schedules. On three intensive care prediction tasks, STAR-Set beats prior set baselines and regular-grid and event-time grid approaches.
Catalog record
| CREATOR | Joohyung Lee, Kwanhyung Lee, Changhun Kim, Eunho Yang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 22020 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.06605
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Structure-Aware Set Transformers: Temporal and Variable-Type Attention Biases for Asynchronous Clinical Time Series Joohyung Lee, Kwanhyung Lee, Changhun Kim, Eunho Yang Abstract Electronic health records (EHR) are irregular, asynchronous multivariate time series. As time-series foundation models increasingly tokenize events rather than discretizing time, the input layout becomes a key design choice. Grids expose time × \times variable structure but require imputation or missingness masks, risking error or sampling-policy shortcuts. Point-set tokenization avoids discretization but loses within-variable trajectories and time-local cross-variable context (Fig. 1 ). We restore these priors in ST ructure- A wa R e ( STAR ) Set Transformer by adding parameter-efficient soft attention biases: a temporal locality penalty − | Δ t | / τ -|\Delta t|/\tau with learnable timescales and a variable-type affinity B s i , s j B_{s_{i},s_{j}} from a learned feature-compatibility matrix. We benchmark 10 depth-wise fusion schedules (Fig. 2 ). On three ICU prediction tasks, STAR-Set achieves AUC/APR of 0.7158/0.0026 (CPR), 0.9164/0.2033 (mortality), and 0.8373/0.1258 (vasopressor use), outperforming regular-grid, event-time grid, and prior set baselines. Learned τ \tau and B B provide interpretable summaries of temporal context and variable interactions, offering a practical plug-in for context-informed time-series models. 1 Introduction Electronic health records (EHR) are irregularly sampled, asynchronous multivariate time series. Unlike images or text, there is no canonical discretization of time; consequently the input layout presented to a neural encoder is a key design choice. Common layouts include (i) discretized regular grids ( 6 ; 4 ; 11 ) , (ii) event-time (sparse/irregular) grids ( 1 ; 2 ) , and (iii) point-set tokenizations over observed measurement events ( 3 ; 8 ) . Unless stated otherwise, we use representation to denote this input layout. In a regular-grid layout, time is binned into fixed-width intervals (e.g., hourly) and measurements are aggregated per bin, yielding a dense tensor in \mathbb{R}^{T\times V} ( 6 ) . Because most variables are unobserved in most bins, grid models typically require a filling step, ranging from heuristic imputation (e.g., LOCF) to learned interpolation/imputation modules (e.g., mTAN) ( 6 ; 7 ) . The event-time grid instead indexes time by the union of observed timestamps \tau and stores values in a sparse \mathbb{R}^{|\tau|\times V} array; however, asynchrony still produces substantial missingness, motivating masks, time-gap features, or additional imputation ( 1 ) . Attention-based sequence models ( 10 ) further popularized event-time layouts because they can incorporate irregular timestamps via positional/continuous-time encodings and ignore padding through masking. Yet missingness is a double-edged signal in EHR: observation patterns reflect both physiology and care processes, so models can over-rely on missingness indicators and learn sampling-policy shortcuts that fail under domain shift (e.g., different measurement practices across hospitals) ( 1 ) . Point-set representations avoid discretization by treating each observed event as a token (e.g., value, time, and variable identity) and applying permutation-invariant encoders ( 3 ) . This sidesteps populating unobserved grid entries and reduces dependence on heuristic imputation, but it also removes the two structural axes grids provide by construction: within-variable trajectories (“columns”) and contemporaneous cross-variable relations (“rows”). As illustrated in Figure 1 , these inductive biases become implicit, forcing attention to recover them from data alone. We address this gap by augmenting point-set EHR encoders with simple, parameter-efficient attention biases that re-introduce grid-like structure while preserving set flexibility. We propose (i) a variable-type bias that encourages interactions among tokens of the same measurement type and (ii) a temporal bias that favors interactions among temporally nearby tokens (Figure 1 ). We then study where to inject these biases across Transformer depth and identify an effective layer-fusion strategy via systematic ablations over bias type and insertion schedule (Figure 2 ). More broadly, our approach echoes factorization choices in video modeling, where separating temporal dynamics from channel interactions improves efficiency ( 9 ) . Our contributions are: 1. Biasing set attention for irregular EHR: We introduce STAR-Set Transformer (vt–vt in Figure 2 ), which augments point-set EHR encoders with simple additive attention biases to recover grid-like inductive structure without discretization. 2. Two complementary, parameter-efficient biases: We combine (i) a temporal bias that favors temporally proximal interactions via a learnable time-distance penalty, and (ii) a variable-type bias parameterized by a learnable type-compatibility matrix. 3. Systematic layer-wise ablations and consistent gains: We evaluate bias injection schedules across transformer depth and identify effective layer-fusion strategies, yielding consistent improvements on multiple ICU prediction tasks over grid and set-based baselines. Figure 1: EHR input layouts and biasing set attention. (a) Irregular, asynchronous EHR events. Grid and sparse time \times variable layouts (b,c) make within-variable trajectories (red) and time-local cross-variable relations (blue) explicit (sparse relies on missingness masks), whereas set tokenization (d) obscures both axes. We restore these inductive priors with a variable-type bias (e), favoring same-variable interactions, and a temporal bias (f), favoring temporally proximal interactions. 2 Method 2.1 Set representation and its embedder for EHR We represent an EHR episode as an irregular, asynchronous set of events ( 3 ) . Given a mini-batch of size B , we pack episodes into a tensor {\bm{\mathsfit{X}}}\in\mathbb{R}^{B\times L\times 3} , where each event is a triplet {\bm{\mathsfit{X}}}_{b,i,:}=\big(t_{b,i},\,v_{b,i},\,s_{b,i}\big), (1) with timestamp t_{b,i}\in\mathbb{R} , observed value v_{b,i}\in\mathbb{R} , and variable/type index s_{b,i}\in\{0,\dots,F-1\} . Each episode has an effective length \ell_{b}\leq L ( input_lengths \in\mathbb{N}^{B} ); indices i>\ell_{b} are padding. We use ITE ( 8 ) as the set embedder and denote its output token sequence by {\bm{H}}^{(0)}\in\mathbb{R}^{B\times S\times D} with S=L+2 (two special tokens plus L event tokens). The two special tokens are a learned [CLS] and a demographic token ( demo ) from a Linear–Tanh projection of age/sex. 2.2 Transformer encoder with soft attention biases STAR-Set Transformer encodes {\bm{H}}^{(0)} with an L_{\mathrm{enc}} -layer (4 in this study) pre-norm Transformer encoder ( H heads, GELU activations). Our core modification is to add soft additive biases to the attention logits, injecting (i) a temporal locality prior and (ii) a variable-type prior that are explicit in grid/sparse layouts but not in set tokenizations (Figure 1 ). See Appendix A.1 for more details. Figure 2: Layer-wise fusion strategies for soft attention biases in the set encoder. Each panel illustrates a bias schedule applied across Transformer encoder layers (stacked blocks from early/lower to late/upper) on top of the set embedder. We ablate no bias (nb), temporal bias (tb), variable-type bias (vb), and their combination (vt). The shorthand “x–y” denotes using bias x in the lower layers and bias y in the upper layers: (a) nb–nb, (b) tb–tb, (c) vb–vb, (d) nb–tb, (e) tb–nb, (f) nb–vb, (g) vb–nb, (h) vb–tb, (i) tb–vb, and (j) vt–vt. We denote vt–vt as our proposed STAR Set Transformer. Temporal bias. For each layer \ell and head h , we learn a timescale and define an additive time-distance penalty between query token i and key token j : b^{(\ell,h)}_{\mathrm{time}}(b,i,j)\;=\;-\frac{\big|\tilde{t}_{b,i}-\tilde{t}_{b,j}\big|}{\tau_{\ell,h}},\ \ \tau_{\ell,h}=\exp(\omega_{\ell,h})+\varepsilon. (2) Variable-type bias. For each layer \ell and head h , we learn a type affinity matrix {\bm{B}}_{\ell,h}\in\mathbb{R}^{F\times F} . The variable-type bias is b^{(\ell,h)}_{\mathrm{var}}(b,i,j)\;=\;\begin{cases}({\bm{B}}_{\ell,h})_{\tilde{s}_{b,i},\,\tilde{s}_{b,j}},&\tilde{s}_{b,i},\tilde{s}_{b,j}\in\{0,\dots,F-1\},\\ 0,&\text{otherwise}.\end{cases} (3) Biased attention logits. Let d_{h}=D/H , {\bm{Q}}^{(\ell,h)},{\bm{K}}^{(\ell,h)},{\bm{V}}^{(\ell,h)} be the standard projections, and m^{\mathrm{pad}}_{b,j} be the key padding mask (Appendix A ). We modify the attention logits as \displaystyle A^{(\ell,h)}_{b,i,j} \displaystyle=\frac{\langle{\bm{q}}^{(\ell,h)}_{b,i},{\bm{k}}^{(\ell,h)}_{b,j}\rangle}{\sqrt{d_{h}}}\;+\;\bm{1}\!\left[\texttt{bias\_type}[\ell]\in\{\texttt{tb},\texttt{vtb}\}\right]\,b^{(\ell,h)}_{\mathrm{time}}(b,i,j) \displaystyle\quad+\;\bm{1}\!\left[\texttt{bias\_type}[\ell]\in\{\texttt{vb},\texttt{vtb}\}\right]\,b^{(\ell,h)}_{\mathrm{var}}(b,i,j)\;+\;m^{\mathrm{pad}}_{b,j}. (4) We then apply the usual \mathrm{softmax} over j and compute