Rethinking input complexity in transformer-based clinical prediction: implications for feature dimensionality and sequence length in longitudinal electronic health record data Wansu Chen, Botao Zhou, Robert S Zeiger, William Crawford, Michael Schatz, Eric Puttock, Stanley Xu, Matthew T Slaughter, Fagen Xie Abstract AbstractObjectivesTransformer-based models for clinical prediction using longitudinal electronic health record (EHR) data are often developed with large feature sets and long patient histories under the assumption that more data improves performance. However, high-dimensional inputs and long sequences increase computational burden, potentially limiting scalability. We evaluated how feature dimensionality and sequence length affect predictive performance, calibration, risk stratification, and computational efficiency in EHR prediction.Materials and MethodsUsing longitudinal EHR data from adults with mild asthma in an integrated healthcare system, we evaluated input representation design for predicting acute asthma exacerbation. Feature dimensionality was reduced using Integrated Gradients attribution scores, univariate performance-based selection, and clinically guided selection strategies. Sequence length was varied using percentile-based truncation of patient histories. Performance was assessed using discrimination, calibration, high-risk classification, threshold-based event capture, and computational efficiency across regions.ResultsModels using fewer features achieved discrimination comparable to the 80-feature reference model, with area under the receiver operating characteristic curve values ranging from 0.843 to 0.864 vs 0.870 for the full model. Moderate sequence-length truncation reduced training time by more than 70% with minimal loss in discrimination. Although reduced-dimensional models showed attenuation of predicted risk at the upper tail, they identified similar high-risk populations and captured comparable proportions of asthma exacerbation events at clinically relevant thresholds.ConclusionTransformer-based prediction models maintained strong performance across reduced feature sets. While dimensionality reduction modestly affected calibration at the highest risk levels, moderate sequence-length reduction substantially reduced computational burden with limited change in overall discrimination. These findings highlight tradeoffs between input complexity, predictive performance, and computational efficiency. Introduction Deep learning models are increasingly used for clinical risk prediction using longitudinal electronic health record (EHR) data. Transformer-based architectures, in particular, enable modeling of temporally ordered clinical events and have demonstrated strong performance across a range of prediction tasks by capturing complex dependencies within patient histories. As a result, there has been growing interest in applying transformer-based models to large-scale EHR data to support risk stratification and population-level clinical decision-making. In practice, these models are often developed under the implicit assumption that more data leads to better performance, including the use of large feature sets and long longitudinal histories. However, high-dimensional input representations introduce substantial practical challenges. Large numbers of features increase data engineering complexity and require stable availability of heterogeneous data elements across sites, time periods, and clinical workflows, which may limit model portability and robustness after deployment., Similarly, long patient histories increase computational cost for both training and inference, particularly for transformer-based models where complexity scales with sequence length., These factors can hinder scalability, delay model updates, and create barriers to routine use in real-world health systems. Despite these challenges, there is limited empirical guidance on how input design choices, specifically feature dimensionality and sequence length, affect clinically relevant model behavior. Prior work in traditional machine learning has emphasized dimensionality reduction to improve generalizability and interpretability, whereas deep learning approaches often rely on representation learning without explicit feature selection., In the context of EHR-based modeling, studies examining dimensionality reduction, sequence compression, or truncation strategies have primarily emphasized optimization-oriented metrics such as training loss and discrimination, whereas comparatively less attention has been devoted to calibration, clinical utility, and threshold-based identification of high-risk patients. As a result, it remains unclear how much input complexity is necessary to achieve robust and clinically useful performance. In this study, we address this gap by systematically evaluating feature dimensionality and sequence length as key design parameters in transformer-based clinical risk prediction using longitudinal EHR data. Using acute asthma exacerbation (AAE) in adults with mild asthma as a representative use case, we compare multiple feature selection strategies and sequence-length configurations under controlled conditions while holding model architecture constant. We assess the impact of these design choices on discrimination, calibration, risk stratification, threshold-based event capture, and computational efficiency. By providing a comprehensive evaluation of input representation tradeoffs, this work aims to inform the development of scalable and deployable deep learning models for clinical prediction in real-world healthcare settings. Methods Study overview We conducted a retrospective methodological evaluation to examine how input dimensionality influences predictive performance and computational efficiency in deep learning–based clinical risk prediction models. Acute asthma exacerbation in patients with mild asthma was used as a representative use case within a previously established EHR cohort of adults with mild asthma., The study design held the study population, outcome definition, temporal encoding of EHR data, and transformer architecture constant while systematically varying feature dimensionality, feature-selection strategy, and maximum sequence length (). Experiments were conducted under controlled conditions to isolate the effects of these input representation design choices on discrimination, calibration, risk stratification, threshold-based event capture, and computational efficiency. Data source, study population, and outcome The analytic dataset was derived from structured EHR data from Kaiser Permanente Southern California, including diagnoses, medications, laboratory results, and healthcare encounters. Cases were defined as adults aged 18–85 years who experienced an AAE during 2013–2019, with the AAE date serving as the index date. Eligible cases required at least one asthma diagnosis and ≥12 months of continuous enrollment before the index date and met criteria for mild asthma, defined by low-intensity controller therapy, limited short-acting beta-agonist use, and absence of markers of moderate-to-severe disease, including asthma-related hospitalization and selected comorbid conditions in the past 12 months. Patients could contribute multiple eligible case events during the study period. Controls were selected from eligible patients without AAE during the study period and assigned case-derived index dates to align predictor assessment windows. Each control patient contributed only one observation to the analytic dataset. Outcome definition Acute asthma exacerbation was defined using structured EHR criteria including asthma-related hospitalization, emergency department or observation encounters, or systemic corticosteroid treatment associated with asthma-related diagnoses. Baseline transformer architecture and longitudinal input encoding Predictor variables were ascertained from the 24 months preceding the index date. Structured EHR data were transformed into temporally ordered longitudinal sequences representing clinical events occurring during the observation window (). Each clinical concept (e.g., diagnosis, medication, laboratory result, or healthcare encounter) was represented as an individual token, with multiple tokens permitted on the same calendar day. Demographic characteristics were encoded as static patient-level variables. Clinical events were arranged chronologically to form the longitudinal input sequence for transformer-based modeling. The transformer-based architecture and training framework were held constant across all experiments. Model inputs consisted of embedded longitudinal token sequences with positional encoding to preserve temporal order. Training procedures and hyperparameters were held constant across experiments to isolate the effects of feature dimensionality and sequence length. Dimensionality reduction strategies and feature subsets To evaluate the effect of dimensionality reduction, we applied three complementary feature-selection strategies to the 80-feature baseline input space (). For all experiments, reduced-dimensional models were trained using the same transformer architecture and training settings as the full-feature reference model, with only the input feature set modified. For the two data-driven strategies, feature rankings were derived once from the full dataset to define fixed feature subsets that were applied consistently across all cross-validation experiments, thereby enabling direct comparison of identical input representations across model configurations. Integrated Gradients (IG)–based feature selection quantified each feature’s contribution to model output using attribution scores derived from the trained reference transformer model. Integrated Gradients is a gradient-based attribution method that estimates feature importance by integrating gradients from a baseline input to the observed input and has been widely used to interpret deep neural network predictions., Attribution values were computed across the training set and summarized using the mean absolute IG score for each feature. Features were then ranked by importance, and reduced feature subsets were defined using the top 10, top 20, and top 40 ranked features. Features included in each IG-derived subset are listed in . Univariate area under the curve (AUC)-based feature selection ranked features according to their individual discriminatory ability for predicting AAE. Each feature was evaluated independently using its area under the receiver operating characteristic curve (AUROC) within the training data. Features were subsequently ranked by AUROC, and reduced feature subsets were defined using the top 10, top 20, and top 40 ranked features. Features included in each AUC-derived subset are listed in . Clinically guided feature selection used a fixed subset of 20 features curated a priori based on clinical expertise and guideline-informed reasoning (). This subset was intended to reflect clinically interpretable variables commonly available in real-world healthcare settings. Feature selection was informed by prior evidence on risk factors for AAE, including findings from our previously published work. Sequence-length and runtime design To evaluate the impact of maximum sequence length on predictive performance and computational efficiency, we systematically varied the maximum sequence length (max_len) applied to longitudinal input sequences while holding all other model components constant. A maximum sequence length of 512 tokens was selected as the reference configuration based on the empirical distribution of sequence lengths (). This threshold retained most longitudinal histories while limiting excessive computational burden associated with very long sequences. Sequences exceeding this length were truncated to retain the most recent tokens, whereas shorter sequences were padded. Alternative maximum sequence length settings were defined using percentile-based cutoffs derived from the empirical sequence-length distribution, including the 98th, 95th, 90th, 85th, 80th, 75th, 70th, 65th, 60th, 55th, and 50th percentiles. For each percentile threshold, max_len was defined such that the corresponding proportion of patient sequences could be retained without truncation, while longer sequences were truncated to preserve the most recent longitudinal information. Because sequence lengths demonstrated a pronounced right-skewed distribution, percentile-based truncation enabled systematic evaluation of tradeoffs between retained longitudinal information and computational efficiency. Evaluation was limited to the 50th percentile to avoid excessive truncation of patient histories. All sequence-length experiments were conducted using the IG-selected 20-feature model. Feature-selection strategy, feature count, transformer architecture, and training procedures were held constant to isolate the independent effect of sequence length. Batch size was fixed at 32 across experiments. Model training and evaluation Cross-site validation design Model development and evaluation were conducted using six-fold cross-validation defined by geographically distinct Medical Service Areas (MSAs). In each iteration, data from five MSAs were used for training and the remaining MSA was held out for evaluation. Performance metrics were averaged across folds. This design approximates an internal–external validation framework by evaluating model performance across geographic and operational variation. Discrimination performance Model discrimination was assessed using AUROC and the area under the precision–recall curve (AUPRC), calculated on held-out evaluation data within each cross-validation fold. Calibration and risk stratification across feature set Calibration and risk stratification analyses evaluated how feature dimensionality influenced predicted risk distributions, patient-level risk assignment, and high-risk identification. Shared risk bins derived from the full-feature reference model were applied unchanged across all reduced-feature models. The mean predicted risk and observed event rates were calculated within each bin to assess calibration across the risk distribution (). To evaluate patient-level risk assignment, predicted risks from reduced-feature models were compared against predictions from the full-feature reference model among patients with predicted risk ≥0.8 under the reference model (). For operational risk stratification analyses, a fixed predicted-risk threshold derived from the reference model was used to classify approximately 20% of patients as high risk (). Event capture was assessed as the proportion of observed exacerbations occurring among patients classified as high risk (). Computational efficiency Computational efficiency was quantified using total training time, inference time per patient record, and throughput, defined as the number of patient records processed per second. Runtime comparisons were conducted under standardized hardware and software conditions. Sensitivity analysis Sensitivity analyses were conducted using batch sizes of 32, 64, and 128 while holding all other model parameters constant. Discrimination and runtime metrics were evaluated across cross-validation folds. Sensitivity of risk stratification to selection thresholds was additionally evaluated using cumulative event capture curves. Patients were ranked by predicted risk, and the cumulative proportion of observed events captured was calculated as progressively larger fractions of the population were selected (). This analysis complements fixed-threshold evaluation across broader population strata. Software All analyses were performed in Python. Deep learning models were implemented using PyTorch and the Hugging Face Transformers library. Data preprocessing and evaluation were conducted using NumPy, pandas, and scikit-learn. Ethics approval This study was approved by the Kaiser Permanente Southern California Institutional Review Board with a waiver of informed consent for retrospective analysis. Results Patient characteristics A total of 395 941 observations were included in the analytic dataset, comprising 86 424 cases with AAE and 309 517 controls without. These observations correspond to eligible index events, and individual patients could contribute multiple events. The mean (±SD) age was 44.7 (±17.9) years, and 63.9% were female. Baseline characteristics stratified by case status are summarized in . Compared with controls, cases were older (49.2 vs 43.4 years) and more likely to be female (72.4% vs 61.6%). Indicators of asthma-related disease activity and treatment intensity were consistently more prevalent among cases, including higher use of short-acting β-agonists (87.7% vs 72.2%), asthma controller medications (66.3% vs 38.9%), combination ICS/LABA therapy (22.6% vs 9.6%), and systemic corticosteroids (72.7% vs 38.5%). Cases also had a higher prevalence of comorbid atopic conditions, including allergic rhinitis (27.9% vs 20.9%) and chronic sinusitis (26.5% vs 14.9%), as well as greater prior healthcare utilization, reflected by a higher proportion with asthma-related emergency department or urgent care visits (36.3% vs 20.2%). Elevated eosinophil counts were more common among cases than controls (38.0% vs 32.8%), consistent with greater inflammatory disease burden. Selected feature sets across dimensionality-reduction strategies Feature-selection strategies yielded overlapping but non-identical subsets of variables, with attribution- and performance-based methods selecting features based on model-derived importance metrics, while the clinically guided subset emphasized interpretability and implementation feasibility (). These selected feature sets formed the basis for all reduced-dimensional model evaluations. Discrimination performance under fixed-length sequence design Under the fixed-length sequence design, reduced-feature models demonstrated discrimination performance comparable to the full-feature reference model despite substantial reductions in input dimensionality (). Performance was evaluated on held-out MSAs in each fold and averaged across all six folds. The reference model achieved a mean AUROC (±SD) of 0.870 (±0.007) and an AUPRC of 0.718 (±0.034), with only modest differences across reduced-feature configurations. Among IG-selected models, the AUROC values ranged from 0.843 to 0.864 across 10-, 20-, and 40-feature configurations, with corresponding AUPRC values between 0.664 and 0.698 (). Discrimination differences remained modest despite substantial reductions in feature count. Models using univariate AUROC-based selection showed similar patterns, although discrimination was consistently slightly lower than that of IG-based models at comparable feature counts. The clinically guided 20-feature model achieved an AUROC of 0.836 (±0.005) and an AUPRC of 0.646 (±0.034), retaining most of the discriminatory performance of higher-dimensional models despite its more parsimonious feature set. The total training time and inference latency were similar across models under the fixed-length sequence design, indicating that reductions in feature dimensionality did not materially affect computational performance in this setting (). Impact of maximum sequence-length truncation on runtime and discrimination Sequence-length truncation was associated with substantial reductions in computational cost while preserving most discriminatory performance across a wide range of truncation thresholds (). Using the IG-selected 20-feature model as a representative configuration, setting the maximum sequence length at the 98th percentile of the empirical sequence-length distribution reduced total training time by approximately 38% relative to the fixed-length reference configuration, with minimal change in AUROC. Progressively smaller maximum sequence lengths yielded larger gains in computational efficiency. The maximum sequence lengths corresponding to the 80th and 75th percentiles reduced training time by more than 70% compared with the reference configuration, while AUROC declines remained modest across cross-validation folds. In contrast, maximum sequence lengths below the 65th percentile were associated with more pronounced degradation in discrimination, suggesting diminishing returns when the allowed sequence length becomes overly restrictive (). These relationships are illustrated in , where the training time decreased monotonically as the maximum sequence length was reduced, whereas the AUROC remained largely stable across a wide range of truncation thresholds, with only modest declines observed at the most restrictive settings. shows the empirical distribution of sequence lengths, which was highly right-skewed, with most sequences relatively short and a small proportion substantially longer. The median sequence length was approximately 67, with the 80th, 90th, 95th, and 98th percentiles corresponding to sequence lengths of approximately 133, 186, 243, and 328, respectively. Calibration and risk re-stratification across feature sets Calibration and risk stratification analyses evaluated the effects of feature dimensionality on predicted risk distributions, patient-level risk assignment, and high-risk identification (). Using shared risk bins defined by quintiles of predicted risk from the full-feature reference model, calibration plots demonstrated systematic differences between the reference and reduced-feature models in higher-risk strata (). Across reduced-feature configurations, the mean predicted risk and observed event rates were lower in the highest-risk quintiles, indicating attenuation of risk estimates with increasing dimensionality reduction. Differences between models were minimal in lower-risk groups. Patient-level comparisons further illustrated differences in risk assignment (). This analysis was restricted to patients with predicted risk ≥ 0.8 in the reference model and is shown for three representative 20-feature configurations (IG-20, AUC-20, and Clinical-20), based on a random sample of 1000 patients per model. Across all reduced-feature models, predicted risks were systematically lower than those of the reference model, as indicated by deviation below the identity line. The degree of attenuation varied across feature-selection strategies, with IG-20 showing closer alignment and AUC-20 demonstrating greater downward deviation. To assess the operational implications of these differences, a fixed high-risk threshold defined by the reference model (top 20% of patients) was applied across all models (). The proportion of patients classified as high risk remained close to 20% across configurations. For example, IG-10 and IG-20 identified slightly fewer high-risk patients (19.2% and 19.4%, respectively), whereas IG-40 identified a proportion similar to the reference model. Using the same threshold, the proportion of observed asthma exacerbation events captured among high-risk patients was also similar across models (). IG-40 and IG-20 captured proportions of events most comparable to the reference model, with only modest differences observed across reduced-feature configurations. Sensitivity analysis In sensitivity analyses, discrimination performance was highly consistent across batch-size configurations (). The mean AUROC values increased modestly with larger batch sizes, from 0.8479 at batch size 32 to 0.8495 at batch size 128, with differences ≤0.002 across folds. Computational efficiency improved with increasing batch size. The average total training time decreased from 1.32 hours at batch size 32 to 1.16 hours at batch size 128, corresponding to an approximately 12% reduction in runtime (). In a separate sensitivity analysis, the cumulative event capture was evaluated across a range of risk-selection thresholds (). Differences in event concentration across models were minimal at the operational threshold used in the main analysis () and became more apparent only when broader fractions of patients (approximately 25%–35%) were considered. Even under this worst-case ranking-based scenario, differences in cumulative event capture across models remained modest. Discussion In this study, we provide a systematic evaluation of input representation design in transformer-based clinical risk prediction using longitudinal EHR data. Across multiple feature-selection strategies and sequence-length configurations, we demonstrate that substantially more parsimonious input representations can preserve most discriminatory performance while markedly improving computational efficiency. Importantly, our findings show that reductions in feature dimensionality and sequence length alter predicted risk distributions, particularly at the upper tail, without materially affecting clinically relevant risk stratification or event capture at operational decision thresholds. These results highlight a critical distinction between global performance metrics and deployment-relevant behavior. By quantifying tradeoffs between input complexity, computational burden, and clinically meaningful performance, this study provides practical guidance for scalable and deployable transformer-based models in real-world healthcare settings. Longitudinal EHR-based prediction models are frequently constructed using large numbers of heterogeneous features and extended temporal histories in an effort to maximize predictive signal.,, However, this approach introduces important practical challenges that are often underemphasized in model development. High-dimensional feature spaces increase data engineering complexity and may reduce portability across sites and workflows, while long input sequences substantially increase computational cost for transformer-based architectures.,, Our findings suggest that much of this input complexity may be unnecessary for achieving strong and clinically meaningful predictive performance. Reduced feature sets and shorter sequences preserved discrimination and threshold-based risk stratification while substantially improving computational efficiency, supporting more parsimonious input design as a viable strategy for real-world implementation. Prior studies examining dimensionality reduction in deep learning models for longitudinal EHR data have largely emphasized optimization-oriented metrics such as training loss, accuracy, and discrimination.,,, In contrast, our study evaluated calibration, patient-level risk assignment, threshold-based risk stratification, and event capture under controlled comparisons of both feature dimensionality and sequence length. This integrated framework provides practical insight into how input representation choices influence both predictive performance and deployment-relevant behavior in transformer-based EHR models. highlights an important distinction between distribution-level calibration and threshold-based clinical performance., Reduced-feature models exhibited attenuation of predicted risk and modest calibration differences at the upper end of the risk spectrum, particularly within the highest-risk group (). Despite these differences, performance at clinically relevant decision thresholds remained largely unchanged. When a fixed threshold identifying the top 20% of patients as high risk was applied, reduced-feature models identified a similar proportion of high-risk patients and captured a comparable fraction of observed asthma exacerbation events relative to the full-feature reference model (). Across feature-selection strategies, IG-based models (IG-40 and IG-20) showed performance most closely aligned with the reference model, whereas more aggressive reduction strategies demonstrated slightly greater attenuation. These findings suggest that dimensionality reduction introduces modest attenuation in calibration at the extreme upper tail of predicted risk, but with limited practical impact on threshold-based patient identification and event capture. Although this study was not designed to evaluate clinical interventions, the observed differences in risk stratification across model configurations have important implications for how prediction models are operationalized in clinical settings. In practice, such models are typically deployed using fixed decision thresholds to identify patients for enhanced monitoring or preventive action., Reduced-feature models, particularly when combined with moderate maximum sequence length designs, may offer a favorable balance between computational efficiency, data availability, and clinical performance in settings with constrained resources or frequent retraining requirements. Conversely, higher-dimensional models with longer effective input histories may better differentiate patients at the extreme upper end of the risk distribution, albeit with greater computational and operational complexity. These considerations are particularly relevant for health systems seeking to operationalize deep learning models under real-world constraints of data availability, computational resources, and model maintenance., Longitudinal EHR data are characterized by highly skewed distributions of observed sequence length, with a small proportion of patients contributing disproportionately long histories. By empirically characterizing this distribution and evaluating alternative maximum sequence length settings, we demonstrate that reducing the allowed input length yields substantial gains in computational efficiency. Moderate maximum sequence length settings reduced training time by more than half while preserving discrimination, whereas more restrictive maximum sequence lengths resulted in diminishing returns and larger performance degradation. Importantly, sequence-length selection was guided by empirical data characteristics rather than arbitrary architectural constraints, enhancing the practical relevance of the modeling framework. The observed effects of dimensionality reduction and sequence-length design are likely dependent on the prediction task and the underlying data. Reduced-feature models may have less capacity to distinguish patients at the extremes of risk when variables encoding healthcare utilization intensity or complex nonlinear relationships are excluded. Similarly, the impact of sequence-length truncation will depend on the temporal structure of the prediction task and the distribution of sequence lengths of the underlying data. Accordingly, model design choices should be evaluated within the context of the specific clinical application. Sequence-length truncation may have important implications for preserving longitudinal clinical information. Because sequence truncation removed the earlier portion of longer longitudinal EHR sequences, historical clinical information may have been excluded for patients with more extensive clinical histories. Potentially affected information could include historical exacerbations, prior medication exposures, treatment changes, or patterns of healthcare utilization that may provide important context regarding long-term disease severity. Consequently, patients with longer or more complex longitudinal histories may be more susceptible to information loss than patients with shorter histories. Although overall model discrimination remained stable across moderate sequence-length reductions, these aggregate performance metrics do not exclude the possibility that truncation could have a greater impact within specific patient subgroups. Accordingly, future implementation studies should evaluate model performance beyond population-level discrimination metrics, as aggregate performance measures may mask differences among patients with longer or more complex longitudinal histories. Subgroup validation stratified by historical sequence length or healthcare utilization may help determine whether sequence truncation disproportionately affects these patients before clinical deployment. This study has several limitations that warrant consideration. First, we evaluated a limited set of predefined feature-selection strategies, including IG, univariate AUROC ranking, and a clinically guided feature subset. Other approaches, such as adaptive or dynamically learned feature selection methods, may further optimize the tradeoff between computational efficiency and predictive performance. Second, all analyses were conducted within a single integrated healthcare system. Although this setting enables consistent longitudinal data capture and outcome ascertainment, generalizability to other healthcare systems may be limited. Third, computational evaluations were performed under controlled experimental conditions with fixed model architectures and training procedures. Real-world deployment may introduce additional constraints related to data latency, missingness, evolving feature availability, and retraining frequency that were not explicitly modeled in this study. In addition, we did not characterize the specific types of longitudinal clinical information removed by truncation or evaluate whether truncation differentially affected patients with longer or more complex disease histories. In summary, transformer-based clinical risk prediction models can achieve strong performance using substantially more parsimonious input representations than is commonly assumed. Although reductions in feature dimensionality and sequence length introduce modest changes in calibration and risk distribution, these effects have limited impact on clinically relevant risk stratification and event capture at operational decision thresholds. These findings suggest that input complexity can be strategically evaluated and reduced to improve computational efficiency while maintaining strong overall predictive performance.