Research on dynamic cost prediction method for power technology upgrading projects based on improved transformer neural network Authors: Qi Shi, Yue Le, Chunjie Gu, Yaming Geng Published: 2026 ABSTRACT Accurate cost prediction is critical for the effective management of power grid technical renovation (PGTR) projects, yet conventional methods struggle to capture the nonlinear, multi-scale, and time-varying dynamics inherent in renovation cost data. This paper proposes an improved Transformer-based framework, MS-DW-Transformer, that combines parallel multi-scale convolutional encoding with a gated dynamic weight allocation mechanism and a trend-consistency auxiliary loss. We test the framework on 1,362 completed PGTR projects from a provincial grid corporation in eastern China, contrasting its behavior against classical baselines (BPNN, SVR, LSTM, GRU, vanilla Transformer) as well as more recent time-series Transformers (Informer, Autoformer, PatchTST, Temporal Fusion Transformer) and a graph-based learner. Across five independent runs the proposed model attains a mean MAPE of 6.83% (± 0.11) and an R² of 0.937 on the held-out test set, with paired-sample significance tests confirming that the gains over the strongest baselines are not artefacts of random variation. We additionally provide gate-activation trajectories, SHAP-based feature attribution, and sensitivity analyses for the window length and loss-weight coefficients, all of which back up — though never fully resolve — the case for the proposed design. The work, we should add, is calibrated to a single-province dataset; the generalizability claims should be read accordingly. Introduction Power grid technical renovation (PGTR) projects — encompassing equipment upgrades, transmission line reinforcement, substation retrofitting, and relay protection system replacement — form the backbone of modern grid reliability and operational safety1. As aging infrastructure confronts rising demand and the integration of distributed renewable sources, these renovation efforts have grown both in scale and in urgency across most industrialized power systems2. Yet the cost management of such projects remains stubbornly difficult. Unlike standardized construction, PGTR work unfolds over extended timelines, often spanning multiple fiscal years, during which material prices, labor availability, regulatory requirements, and technical scope can shift in ways that invalidate early-stage budgets3. The heterogeneity of renovation tasks — some involving minor component swaps, others requiring complete substation redesigns — further compounds the challenge, producing cost distributions with heavy tails and erratic volatility that conventional estimation templates struggle to capture4. Against this backdrop, accurate and adaptive cost prediction has emerged as a pressing operational need. Early approaches to construction cost forecasting relied on parametric regression and analogical estimation, where historical unit-price databases were queried to produce point estimates adjusted by location and inflation indices5. Transparent though they are, these methods assume linear or log-linear relationships between project descriptors and final cost — an assumption that breaks down when interactions among dozens of technical, environmental, and market variables dominate the cost-formation process6. A second wave of research introduced machine learning algorithms — support vector regression, random forests, gradient boosting machines — that tolerate nonlinearity and high-dimensional feature spaces more gracefully7. Cheng et al. demonstrated that ensemble tree models reduce mean absolute percentage error by roughly 15–20% relative to multiple regression on highway bridge datasets8, and similar gains were reported for building renovation projects when feature engineering was carefully tailored to domain knowledge9. Deep learning pushed the frontier further. Recurrent neural networks (RNNs), particularly Long Short-Term Memory (LSTM) variants, brought the ability to model sequential dependencies in staged construction processes, where the cost trajectory of one phase conditions subsequent phases10. Convolutional neural networks (CNNs), originally designed for spatial data, were repurposed as local feature extractors over tabular cost records, capturing short-range correlations among clustered cost drivers11. Hybrid architectures combining CNN front-ends with LSTM backends showed promise in extracting both local patterns and temporal trends from construction time-series data12. Despite these advances, recurrent models suffer from well-documented limitations: gradient attenuation over long sequences restricts their effective memory horizon, and their inherently sequential computation resists parallelization during training13. The Transformer architecture, introduced by Vaswani et al. with its self-attention mechanism, offered a fundamentally different path to sequence modeling — one that computes pairwise relevance scores across all time steps simultaneously, thereby sidestepping the vanishing-gradient bottleneck of recurrence14. Since then, Transformer-based models have achieved state-of-the-art performance in load forecasting, electricity price prediction, and energy consumption estimation15, and recent work has extended these ideas to spatial-temporal and federated learning settings in integrated energy systems16. The capacity to weight distant observations as heavily as recent ones — when the data warrant it — aligns well with the reality that a procurement decision made months before installation can exert a decisive influence on final project cost. Yet applying vanilla Transformers directly to PGTR cost prediction surfaces several unresolved difficulties. First, renovation projects generate multi-source, heterogeneous features — continuous price indices, categorical equipment types, irregular event logs, and spatial network topology descriptors — whose joint embedding space is nontrivial to construct17. Second, standard self-attention treats all temporal scales uniformly, whereas cost dynamics in power renovation exhibit distinct patterns at weekly procurement cycles, monthly progress-payment intervals, and annual budget-revision horizons; a single-scale attention head will conflate these rhythms. Third, the operational conditions of a renovation project are not stationary: design changes, force-majeure weather events, and policy shifts alter the relative importance of cost drivers mid-stream, yet most existing architectures assign fixed feature weights learned during training18. These gaps motivate the present study. We propose an improved Transformer-based framework tailored to the dynamic cost prediction of PGTR projects, incorporating three principal enhancements. The first is a multi-scale feature extraction module that processes raw heterogeneous inputs through parallel convolutional branches operating at different kernel widths, producing representations sensitive to short-term fluctuations and longer-term trends at the same time. The second is a dynamic weight allocation mechanism embedded within the attention layers, which adjusts the contribution of each feature channel in response to evolving project states — effectively letting the model re-prioritize cost drivers as construction conditions change. The third is an integrated prediction framework that couples the improved Transformer encoder with a scenario-aware decoding strategy, enabling rolling forecasts that update as new project information accrues. We validate the proposed approach on a real-world dataset drawn from provincial grid renovation records, benchmarking it not only against the long-established LSTM, GRU, and vanilla Transformer families19 but also against more recent time-series Transformer variants (Informer, Autoformer, PatchTST, Temporal Fusion Transformer) and a graph-based forecasting model20, so that the comparison reflects the current state of the art rather than only historical baselines. The contributions of this paper can be summarized as follows. (i) We design a multi-scale convolutional embedding front-end that decomposes raw heterogeneous PGTR inputs into three parallel scale-specific channels; unlike single-kernel encoders adopted in earlier hybrid CNN-Transformer work for construction cost data11,12, this branch structure allows the model to separate weekly procurement bursts from monthly payment rhythms and from quarterly budget revisions, and our ablation study (Tables 1, 2, 3, 4, 5, 6 and 7) shows that suppressing it raises MAPE by 1.11% points — the largest single-component degradation we observed. (ii) We introduce a condition-adaptive gating mechanism that recomputes branch weights at every time step. In contrast to fixed-weight fusion schemes — which dominate the construction cost literature and assume a stationary mix of cost drivers across project phases — our gate redistributes emphasis as conditions evolve, an effect we quantify in Figs. 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 and 11 and that translates into an 0.78-point MAPE improvement over uniform averaging. (iii) We construct an end-to-end dynamic prediction pipeline that bridges the gap between general-purpose time-series Transformers (Informer21, Autoformer22, PatchTST23 and the domain-specific requirements of PGTR cost management, including a trend-consistency loss that lifts directional agreement from 79.1% to 87.4% without sacrificing magnitude accuracy. Together, these innovations target both the accuracy and the operational relevance of cost forecasts in a domain where budgeting errors carry substantial economic and safety consequences. Theoretical foundations Before turning to architectural details, it is worth fixing the prediction problem precisely. Let a PGTR project be described by a multivariate time series of feature vectors x_t ∈ ℝᵈ observed at discrete reporting points t = 1, 2, …, T, where d is the number of retained features (after the screening procedure described in Sect. 2.3) and T denotes the project duration in reporting periods. The target variable y_t ∈ ℝ is the audited cumulative cost at step t. Given an input window X_{t-w + 1:t} = (x_{t-w + 1}, …, x_t) of length w preceding the prediction point, the task is to learn a parametric mapping f_θ : ℝ^{w×d} → ℝʰ that produces an h-step ahead cost forecast (ŷ_{t + 1}, …, ŷ_{t + h}). Training minimizes a composite objective ℒ(θ) = λ₁·ℒ_MSE + λ₂·ℒ_trend + λ_reg·‖θ‖² defined formally in Sect. 3.3, where ℒ_MSE penalizes magnitude error, ℒ_trend penalizes directional disagreement, and the regularizer controls overfitting on the moderate-sized dataset. Throughout the paper we set h = 1 — that is, single-step rolling prediction — because this matches the cadence at which provincial budgeting departments revise cost estimates and because longer horizons exacerbate the compounding error problem that besets autoregressive Transformers on small datasets. The training, validation, and test partitions are formed chronologically (Sect. 4.1) so that no future observation can leak backwards through the windowing process. Transformer neural network fundamentals The standard Transformer, first proposed for machine translation, adopts a symmetric encoder-decoder topology in which both halves consist of stacked identical layers14. Each encoder layer contains two sub-layers — a multi-head self-attention block and a position-wise feed-forward network — while each decoder layer inserts an additional cross-attention block that attends to the encoder output24. The core computation is scaled dot-product attention. Given input representations packed into query, key, and value matrices Q, K, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:V\in\:{\mathbb{R}}^{n\times\:{d}_{k}}$$\end{document}, the attention output is computed as the standard formula:1\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\mathrm{A}\mathrm{t}\mathrm{t}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}(Q,K,V)=\mathrm{s}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{a}\mathrm{x}\left(\frac{Q{K}^{T}}{\sqrt[]{{d}_{k}}}\right)V$$\end{document} where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{d}_{k}$$\end{document} denotes the dimensionality of key vectors and the scaling factor \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\sqrt[]{{d}_{k}}$$\end{document} prevents the dot products from growing large enough to push softmax into saturation regions25. Rather than relying on a single attention function, the Transformer projects Q, K, V through h independent learned linear maps and runs attention in parallel across all heads. The concatenated head outputs are then projected back to the model dimension:2\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\mathrm{M}\mathrm{u}\mathrm{l}\mathrm{t}\mathrm{i}\mathrm{H}\mathrm{e}\mathrm{a}\mathrm{d}(Q,K,V)=\mathrm{C}\mathrm{o}\mathrm{n}\mathrm{c}\mathrm{a}\mathrm{t}(hea{d}_{1},\dots\:,hea{d}_{h}){W}^{O}$$\end{document} where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:hea{d}_{i}=\mathrm{A}\mathrm{t}\mathrm{t}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}(Q{W}_{i}^{Q},K{W}_{i}^{K},V{W}_{i}^{V})$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{W}^{O}$$\end{document} is a projection matrix14. This multi-head design lets different heads specialize in capturing distinct relational patterns across the sequence. The feed-forward sub-layer applies two affine transformations separated by a ReLU activation, expressible as:3\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\mathrm{F}\mathrm{F}\mathrm{N}\left(x\right)=\mathrm{R}\mathrm{e}\mathrm{L}\mathrm{U}(x{W}_{1}+{b}_{1}){W}_{2}+{b}_{2}$$\end{document} Both sub-layers are wrapped with a residual connection followed by layer normalization26, giving the general sub-layer output form:4\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\mathrm{O}\mathrm{u}\mathrm{t}\mathrm{p}\mathrm{u}\mathrm{t}=\mathrm{L}\mathrm{a}\mathrm{y}\mathrm{e}\mathrm{r}\mathrm{N}\mathrm{o}\mathrm{r}\mathrm{m}(x+\mathrm{S}\mathrm{u}\mathrm{b}\mathrm{L}\mathrm{a}\mathrm{y}\mathrm{e}\mathrm{r}(x\left)\right)$$\end{document} This residual-plus-normalization pattern stabilizes gradient flow across deep stacks and permits effective training of architectures with dozens of layers. What distinguishes the Transformer from recurrent and convolutional alternatives is twofold. Self-attention connects every pair of positions in a single operation, so the maximum path length for information propagation is \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:O\left(1\right)$$\end{document} rather than \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:O\left(n\right)$$\end{document} for RNNs or O(n/k) for CNNs with kernel size k27. Equally important, the absence of sequential recurrence means that all positions can be processed in parallel during training, yielding substantial wall-clock speedups on modern GPU hardware. These properties — global receptive fields and full parallelism — make the Transformer an attractive backbone for the long, irregularly sampled cost sequences encountered in PGTR projects, though the architecture requires targeted modifications to handle the domain-specific challenges outlined in Section I. Positional encoding and attention improvement strategies Because self-attention is permutation-equivariant — swapping two input tokens and the corresponding outputs yields the same result — the bare Transformer has no intrinsic notion of order14. Positional encoding injects sequence-position information into the input embeddings. The original sinusoidal scheme assigns to each position pos and dimension index i the following standard encoding:5\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:P{E}_{(pos,2i)}=\mathrm{s}\mathrm{i}\mathrm{n}\left(\frac{pos}{{10000}^{2i/{d}_{model}}}\right),P{E}_{(pos,2i+1)}=\mathrm{c}\mathrm{o}\mathrm{s}\left(\frac{pos}{{10000}^{2i/{d}_{model}}}\right)$$\end{document} Each dimension oscillates at a distinct wavelength, so the difference vector \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:P{E}_{pos+k}-P{E}_{pos}$$\end{document} depends only on offset k, enabling the model to infer relative distance from absolute encodings28. An alternative is to make positional vectors fully learnable parameters optimized during training; this approach can outperform sinusoidal encoding on fixed-length tasks but generalizes poorly to sequences longer than those seen in training29. Relative positional encoding, as formulated by Shaw et al., directly injects pairwise distance into the attention logits:6\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{e}_{ij}=\frac{\left({x}_{i}{W}^{Q}\right)({x}_{j}{W}^{K}+{a}_{ij}^{K}{)}^{T}}{\sqrt[]{{d}_{k}}}$$\end{document} where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{a}_{ij}^{K}$$\end{document} is a learned bias conditioned on i-j21. This formulation suits variable-length cost sequences whose absolute position carries less meaning than inter-event spacing — a situation common in PGTR records where reporting intervals are irregular. A separate but equally pressing concern is computational cost. Standard self-attention requires \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:O\left({n}^{2}\right)$$\end{document} time and memory, which becomes prohibitive for long sequences. Several families of efficient attention have been proposed to break this quadratic barrier. Sparse attention patterns — fixed stride, local windows, or data-dependent top- k selection — reduce the number of computed pairs to \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:O\left(n\sqrt[]{n}\right)$$\end{document} or \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:O\left(n\mathrm{l}\mathrm{o}\mathrm{g}n\right)$$\end{document} while retaining most of the representational capacity30. Linear attention replaces the softmax kernel with a decomposable feature map \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\phi\:$$\end{document}, rewriting attention as:7\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\mathrm{L}\mathrm{i}\mathrm{n}\mathrm{e}\mathrm{a}\mathrm{r}\mathrm{A}\mathrm{t}\mathrm{t}\mathrm{n}(Q,K,V)=\phi\:\left(Q\right)\left(\phi\:(K{)}^{T}V\right)$$\end{document} which can be computed in \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:O\left(n\right)$$\end{document} by exploiting associativity of matrix multiplication31. Sliding-window attention restricts each token to attend within a fixed neighborhood, achieving linear cost with strong inductive bias for local patterns, though at the expense of direct long-range connections. Our improved model, detailed in Section III, draws on elements from both relative positional encoding and sparse attention to balance global dependency capture against tractable computation for the multi-year cost trajectories typical of PGTR projects. Cost driver analysis for power grid technical renovation projects The final cost of a PGTR project emerges from the interaction of at least five broad factor categories. Project scale and voltage class set the baseline: a 220 kV substation retrofit demands transformer units, busbar systems, and protection devices whose procurement costs dwarf those of a 10 kV distribution-line upgrade3. Equipment and material prices — copper conductors, insulating oil, steel structures, switchgear assemblies — are themselves subject to commodity-market fluctuations that can swing 10–30% within a single procurement cycle4. Construction duration and labor costs introduce a temporal dimension; extended schedules expose the project to wage drift, seasonal productivity variation, and equipment idle-time charges. Regulatory adjustments, such as revised environmental compliance standards or updated grid interconnection codes, may force mid-project design changes that cascade into cost overruns18. Regional economic disparity adds yet another layer: identical renovation scopes executed in coastal metropolitan areas versus inland rural districts yield markedly different unit costs due to logistics, local labor pools, and land-use constraints17. To quantify the relative influence of these heterogeneous drivers, we adopt the Pearson correlation coefficient between each candidate feature \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{x}_{j}$$\end{document} and the target cost variable y as a first-pass screening metric (standard formula):8\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{r}_{j}=\frac{\sum\:_{i=1}^{N}({x}_{ij}-{\stackrel{{-}}{x}}_{j})({y}_{i}-\stackrel{{-}}{y})}{\sqrt[]{\sum\:_{i=1}^{N}({x}_{ij}-{\stackrel{{-}}{x}}_{j}{)}^{2}\sum\:_{i=1}^{N}({y}_{i}-\stackrel{{-}}{y}{)}^{2}}}$$\end{document} Linear correlation alone, however, misses nonlinear associations. We therefore supplement it with permutation-based feature importance derived from a gradient boosting model, where the importance score \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{I}_{j}$$\end{document} for feature j is computed as the mean decrease in validation loss when the values of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{x}_{j}$$\end{document} are randomly shuffled (standard formula):9\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{I}_{j}=\frac{1}{K}\sum\:_{k=1}^{K}\left({\mathcal{L}}_{k}^{\mathrm{p}\mathrm{e}\mathrm{r}\mathrm{m}}-{\mathcal{L}}_{k}^{\mathrm{o}\mathrm{r}\mathrm{i}\mathrm{g}}\right)$$\end{document} with K denoting the number of cross-validation folds8. Features passing both thresholds — \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\left|{r}_{j}\right|>0.15$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{I}_{j}$$\end{document} above the median importance — enter the model input set. Concretely, we kept features with |r| ≥ 0.30 against the audited cost target AND with permutation importance above the median across the 26 raw candidates; the median was chosen over a fixed numerical cutoff because the importance distribution is heavy-tailed and varies between cross-validation folds, so a relative threshold proved more stable in our early sensitivity probes. Twenty-six raw features were reduced to eighteen by this joint criterion; we also verified that moderate perturbations of the thresholds (|r| ∈ [0.25, 0.35] and importance quantile ∈ [0.45, 0.55]) changed the retained set by at most two features and the final MAPE by less than 0.2 points. What makes PGTR cost data particularly resistant to simple modeling is its temporal texture. Cost accrual is neither smooth nor stationary: procurement bursts, payment milestones, and scope-change events produce discontinuities superimposed on a slow-moving trend. These abrupt shifts, combined with cross-feature interactions that themselves evolve over the project lifecycle, generate nonlinear dynamics that linear regression and shallow learners cannot faithfully reproduce. This observation — confirmed in our preliminary exploratory analysis of the dataset described in Section IV — reinforces the case for a deep architecture capable of jointly capturing multi-scale temporal patterns and adapting its internal feature weighting as project conditions evolve. Improved transformer-based dynamic cost prediction model Overall architecture design of the improved transformer model Building on the theoretical apparatus of Section II, we now present the architecture of the proposed model — which we term MS-DW-Transformer (Multi-Scale Dynamic-Weight Transformer) — organized into five sequential modules. Figure 1 illustrates the complete pipeline, with each module annotated by its principal operation, its input and output tensor shape (B for batch, L for sequence length, d_in/d_model for input/embedding dimension, and H for the number of attention heads), the kernel widths used inside the multi-scale block (k ∈ {3, 7, 15}), the softmax-gated fusion that combines the three convolutional branches, and the residual–normalization paths that wrap every sub-layer. Arrows indicate the forward data flow during inference; the dashed feedback arrow from the decoder output back to the decoder input represents the autoregressive rolling-prediction loop that operates only during deployment. Fig. 1Overall architecture of the MS-DW-Transformer model, with module-level inputs, outputs, kernel widths, gating connections, and the residual–normalization paths annotated. Overall architecture of the MS-DW-Transformer model, with module-level inputs, outputs, kernel widths, gating connections, and the residual–normalization paths annotated. The first module, the data input layer, receives raw PGTR records comprising continuous variables (material price indices, labor rates, schedule progress percentages), categorical variables (voltage class, equipment type, region code), and temporal markers. Continuous features undergo min-max normalization while categorical variables pass through trainable embedding look-up tables, producing a unified representation tensor of shape \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:[B,T,{d}_{input}]$$\end{document} where B, T, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{d}_{input}$$\end{document} denote batch size, sequence length, and input dimension respectively14. The second module — the multi-scale feature encoding layer, detailed in Sect. 3.2 — applies parallel convolutional branches with varying kernel widths to this tensor, yielding a fused representation that captures both short-horizon fluctuations and longer-period trends. The third module houses the dynamic-weight attention layers described in Sect. 3.3, where channel-wise gating modulates feature contributions before they enter the self-attention computation. The fourth module, a temporal decoding layer built on masked multi-head attention with relative positional biases, generates step-ahead predictions autoregressively during inference but accepts teacher-forced targets during training32. The fifth and final module is a linear projection head that maps the decoder output to a scalar cost estimate at each forecast horizon. Two design points deserve an explicit note. First, the encoder–decoder topology was retained — rather than collapsing to an encoder-only model with a linear head — because rolling forecasts during deployment require the autoregressive mechanism: each newly observed cost feeds back into the decoder context to refine subsequent predictions, a loop that an encoder-only head cannot reproduce without retraining. An ablation in Sect. 4.3 (Table 7, variant E) confirms that removing the decoder costs 0.42 MAPE points relative to the full model. Second, all preprocessing steps that depend on data statistics — k-NN imputation, min-max scaling, Pearson correlation screening, and permutation importance — were fit exclusively on the chronologically earlier training fold and then applied unchanged to the validation and test splits, so that no future observation could leak into the imputer or the scaler. Figure 2 contrasts the training and inference pipelines. During training, ground-truth cost values are fed to the decoder with causal masking, and the model is optimized end-to-end via backpropagation through all five modules. At inference time, the decoder operates in a rolling fashion: each newly predicted cost value is appended to the input sequence, and the encoder re-computes context representations to condition the next step — a scheme inspired by autoregressive generation in sequence-to-sequence models33. Fig. 2Computation flow during training (left, teacher forcing) and inference (right, autoregressive rolling prediction). Computation flow during training (left, teacher forcing) and inference (right, autoregressive rolling prediction). Table 1 lists the key hyperparameters adopted in this study, determined through preliminary grid search on a held-out validation split. The encoder depth of four layers and eight attention heads reflect a compromise we arrived at empirically: deeper stacks showed diminishing returns on our dataset size, while fewer heads failed to differentiate the distinct temporal scales embedded in the cost sequences34. A hidden dimension of 256 keeps the parameter count manageable (approximately 3.8 M trainable weights) without starving the multi-scale branches of representational capacity. Dropout at 0.1 is applied after each attention and feed-forward sub-layer. The sliding window length is set to w = 120 reporting periods; we later examine sensitivity to this choice in Fig. 9. For reproducibility, we initialize all linear weights via Xavier uniform, fix the random seed at 42 across NumPy, PyTorch CPU and PyTorch CUDA, train for up to 300 epochs with early stopping (patience 15), and rely on the Adam optimizer with β₁ = 0.9, β₂ = 0.98, ε = 1e-9 and an initial learning rate of 1e-4. Table 1Key hyperparameter settings of the MS-DW-Transformer model.HyperparameterValueEncoder layers4Decoder layers2Attention heads8Hidden dimension (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{d}_{model}$$\end{document})256Feed-forward dimension512Dropout rate0.1Learning rate (initial)1e-4Batch size64Sliding window length w120Convolutional kernel widths{3, 7, 15} Table 1Key hyperparameter settings of the MS-DW-Transformer model.HyperparameterValueEncoder layers4Decoder layers2Attention heads8Hidden dimension (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{d}_{model}$$\end{document})256Feed-forward dimension512Dropout rate0.1Learning rate (initial)1e-4Batch size64Sliding window length w120Convolutional kernel widths{3, 7, 15} Key hyperparameter settings of the MS-DW-Transformer model. The following subsections unpack the two core innovations — multi-scale feature encoding and dynamic weight attention — that distinguish this architecture from a standard Transformer baseline. Multi-scale feature extraction and dynamic weight allocation mechanism Cost trajectories in PGTR projects superimpose rapid procurement-driven spikes onto gradual budget-revision drifts — a spectral mixture that a single convolutional kernel cannot adequately decompose. We address this by passing the unified input tensor \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:X\in\:{\mathbb{R}}^{B\times\:T\times\:{d}_{input}}$$\end{document} through three parallel one-dimensional convolutional branches with kernel widths \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:k\in\:\{\mathrm{3,7},15\}$$\end{document}, each followed by GELU activation and same-padding to preserve the temporal dimension35. The s-th branch produces a feature map (standard convolution formula):10\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{F}^{\left(s\right)}=\mathrm{G}\mathrm{E}\mathrm{L}\mathrm{U}\left({W}_{conv}^{\left(s\right)}{*}X+{b}^{\left(s\right)}\right),s\in\:\{{1,2},3\}$$\end{document} where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\mathrm{*}$$\end{document} denotes temporal convolution and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{F}^{\left(s\right)}\in\:{\mathbb{R}}^{B\times\:T\times\:{d}_{model}}$$\end{document}. The narrow kernel ( k=3) responds to week-scale price jumps; the mid-range kernel ( k=7) captures monthly payment rhythms; the wide kernel ( k=15) smooths over quarterly or seasonal patterns. All three branches share the same output dimension so that their representations remain directly comparable. A naive concatenation or uniform averaging of these branches would assign equal importance to every scale at every time step — an assumption poorly suited to PGTR data, where the dominant frequency of cost variation shifts as a project transitions from procurement-heavy early stages to installation-intensive later phases. We instead introduce a gating network that computes time-step-specific fusion weights from the mean-pooled branch outputs. Concretely, let \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\stackrel{{-}}{F}}_{t}=\frac{1}{3}\sum\:_{s}^{}{F}_{t}^{\left(s\right)}$$\end{document} be the average representation at step t. The gate vector \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{g}_{t}\in\:{\mathbb{R}}^{3}$$\end{document} is obtained via (proposed formulation):11\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{g}_{t}=\mathrm{s}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{a}\mathrm{x}\left({W}_{g}{\stackrel{{-}}{F}}_{t}+{b}_{g}\right)$$\end{document} where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{W}_{g}\in\:{\mathbb{R}}^{3\times\:{d}_{model}}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{b}_{g}\in\:{\mathbb{R}}^{3}$$\end{document} are learnable parameters. The fused representation is then a weighted combination across scales:12\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{Z}_{t}=\sum\:_{s=1}^{3}{g}_{t}^{\left(s\right)}\cdot{F}_{t}^{\left(s\right)}$$\end{document} This gating mechanism draws inspiration from mixture-of-experts routing14 but operates at the feature-scale level rather than at the expert level. Because \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{g}_{t}$$\end{document} is recomputed at every time step, the model can — and in practice does — shift emphasis from fine-grained to coarse-grained features as the input context evolves. During a procurement burst, for instance, the narrow-kernel branch receives higher gate activation; once the project enters a steady installation phase, the wider kernels dominate. The resulting tensor \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:Z\in\:{\mathbb{R}}^{B\times\:T\times\:{d}_{model}}$$\end{document} is fed directly into the dynamic-weight attention layers (Sect. 3.3). An important design choice here is that the gate parameters are trained jointly with the rest of the network, so gradient signals from the final cost prediction loss propagate back through the softmax gate and into each convolutional branch. This end-to-end coupling ensures that the scale decomposition is not fixed a priori but adapts to whichever frequency partition minimizes prediction error36. Compared to fixed-weight concatenation baselines — which we evaluate in Section V — the gated fusion consistently yields lower variance in forecast residuals, particularly during project phases characterized by abrupt regime changes. The fused multi-scale representation also enriches the key and value projections inside the subsequent attention layers, giving each attention head access to temporally diverse feature summaries without requiring additional parameters in the attention block itself. Training strategy and loss function design Minimizing squared error alone can produce forecasts that track magnitude reasonably well yet miss the direction of cost movement at critical turning points — a failure mode with outsized practical consequences, since a budget manager cares as much about whether costs are rising or falling as about the precise amount. We therefore construct a composite loss that couples standard mean squared error with a trend-consistency penalty and an L2 regularization term (proposed formulation):13\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\mathcal{L}=\underbrace{{\mathcal{L}}_{MSE}}{\underbrace{\frac{1}{N}{\sum\:}_{i=1}^{N}({\widehat{y}}_{i}-{y}_{i}{)}^{2}}}+{\lambda\:}_{1}\underbrace{{\mathcal{L}}_{trend}}{\underbrace{\frac{1}{N-1}{\sum\:}_{i=2}^{N}max\:(0,\:-\varDelta\:{\widehat{y}}_{i}\cdot\varDelta\:{y}_{i})}}+{\lambda\:}_{2}|\varTheta\:{\mid}_{2}^{2}$$\end{document} where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\varDelta\:\hat {{y}}_{i}=\hat {{y}}_{i}-\hat {{y}}_{i-1}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\varDelta\:{y}_{i}={y}_{i}-{y}_{i-1}$$\end{document} are consecutive predicted and actual cost differences, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\varTheta\:$$\end{document} collects all trainable parameters. The hinge-style trend term incurs zero loss when the predicted and observed directions agree, but penalizes sign disagreements proportionally to the product magnitude37. We set \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\lambda\:}_{1}=0.3$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\lambda\:}_{2}=1\mathrm{e}-5$$\end{document} after a coarse sweep over \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\left\{\mathrm{0.1,0.3,0.5}\right\}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\{1\mathrm{e}-\mathrm{4,1}\mathrm{e}-\mathrm{5,1}\mathrm{e}-6\}$$\end{document} respectively; the trend weight \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\lambda\:}_{1}$$\end{document} was chosen to be large enough to steer gradient updates toward directional accuracy without overwhelming the magnitude signal from \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\mathcal{L}}_{MSE}$$\end{document}. For learning rate scheduling, we adopt cosine annealing with warm restarts38, which decays the rate from its initial value \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\eta\:}_{max}$$\end{document} to a floor \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\eta\:}_{min}$$\end{document} within each cycle of length \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{T}_{cycle}$$\end{document} (standard formula):14\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\eta\:}_{t}={\eta\:}_{min}+\frac{1}{2}({\eta\:}_{max}-{\eta\:}_{min})\left(1+\mathrm{c}\mathrm{o}\mathrm{s}\left(\frac{t\mathrm{b}\mathrm{m}\mathrm{o}\mathrm{d}{T}_{cycle}}{{T}_{cycle}}\pi\:\right)\right)$$\end{document} We configure \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\eta\:}_{max}=1\mathrm{e}-4$$\end{document}, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\eta\:}_{min}=1\mathrm{e}-6$$\end{document}, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{T}_{cycle}=30$$\end{document} epochs. The periodic warm restarts help the optimizer escape shallow local minima that arise from the non-convex landscape shaped by the gated multi-scale branches. Training sequences are constructed via a sliding window of length T=120 time steps advanced with a stride of one, which maximizes sample overlap and is essential given the moderate size of our dataset. Within each epoch, windows are shuffled and packed into mini-batches of 64. Gradient norms are clipped to a maximum of 1.0 to prevent the sporadic explosion that we observed when the trend penalty generates large gradients at regime-change boundaries39. An early-stopping criterion monitors validation loss with a patience of 15 epochs: if no improvement is recorded after 15 consecutive epochs, training halts and the checkpoint with the lowest validation loss is restored. The Adam optimizer handles parameter updates, with \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\beta\:}_{1}=0.9$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\beta\:}_{2}=0.98$$\end{document} — the latter slightly elevated from the default to smooth the second-moment estimate over the noisy gradient landscape induced by the composite loss. Together, these choices — composite loss, cosine annealing, gradient clipping, early stopping — form a training regime tuned to the specific difficulties of PGTR cost prediction: noisy targets, regime shifts, and limited data volume. Section 4.3 reports the sensitivity of final performance to the window length w, the encoder depth, the attention head count, and the loss-weight coefficients \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\lambda\:}_{1}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\lambda\:}_{2}$$\end{document}, with the corresponding plots in Fig. 8 (depth and heads), Fig. 9 (window length), and Fig. 10 (λ₁/λ₂ heatmap). A full list of implementation defaults — including embedding dimension, convolution filter counts, kernel widths, seeds, optimizer settings, imputation parameters, and feature-screening thresholds — is given in Table 8 at the end of Sect. 4.3 to consolidate the reproducibility information. Experimental validation and result analysis Dataset construction and experimental setup The dataset originates from the project management archives of a provincial power grid corporation in eastern China, covering 1,362 completed technical renovation projects executed between January 2015 and December 2023. Project types span substation equipment replacement (38.7%), transmission line reinforcement (27.4%), distribution network upgrading (22.1%), and relay protection system overhaul (11.8%). Each record contains 26 raw features — voltage class, rated capacity, conductor type, equipment unit price, material price index at procurement date, labor cost per work-day, scheduled duration, actual duration, region code, and various regulatory and environmental flags — along with the final audited project cost as the prediction target. Audited costs span a wide dynamic range (mean 612.4 × 10⁴ CNY, median 428.7 × 10⁴ CNY, standard deviation 487.9 × 10⁴ CNY, skewness 1.83), and project durations vary from 42 days at the short end to 938 days for the longest substation retrofit (median 187 days). Voltage classes follow a long-tailed distribution dominated by 110 kV (44.2%), 35 kV (28.1%), and 220 kV (16.0%) projects. The records constitute a true sequential temporal stream — each project carries a completion timestamp and contributes a sequence of weekly reporting snapshots — rather than a collection of static project-level samples; this property is what makes the sliding-window training procedure of Sect. 3.3 well-defined. Figure 3 displays the temporal distribution of audited costs across the nine-year span, revealing the non-stationary character discussed in Sect. 2.3: a gradual upward drift punctuated by sharp spikes around 2017–2018 and 2021, coinciding with commodity price surges in copper and transformer steel. Fig. 3Time-series of audited project costs (10⁴ CNY) ordered by completion date; shaded bands give one standard deviation within quarterly bins. Time-series of audited project costs (10⁴ CNY) ordered by completion date; shaded bands give one standard deviation within quarterly bins. Preprocessing followed three stages. Missing values — affecting roughly 4.2% of entries, concentrated in labor-cost and schedule-deviation fields — were imputed via k-nearest-neighbor averaging with k=5, chosen over mean imputation to preserve local distributional structure40. Outlier detection applied the interquartile range criterion: records whose cost fell beyond \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:1.5\times\:\mathrm{I}\mathrm{Q}\mathrm{R}$$\end{document} from the first or third quartile were flagged, manually reviewed against audit reports, and removed only when confirmed as data-entry errors (23 records, 1.7%). Continuous features were then scaled to \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\left[\mathrm{0,1}\right]$$\end{document} via min-max normalization. Figure 4 presents the Pearson correlation heatmap among the 18 features retained after the screening procedure of Sect. 2.3, where strong positive correlations between voltage class, rated capacity, and cost are visible, alongside moderate cross-correlations among material price indices. Fig. 4Pearson correlation heatmap of the 18 retained features and the audited cost target. Pearson correlation heatmap of the 18 retained features and the audited cost target. We partitioned the dataset chronologically rather than randomly to prevent future information from leaking into training, a precaution essential for time-dependent data41. Projects completed before December 2020 constitute the training set (876 samples, 64.3%), those from January 2021 through June 2022 form the validation set (247 samples, 18.1%), and the remaining projects from July 2022 onward serve as the test set (239 samples, 17.6%). Figure 5 marks these boundaries on the temporal axis. Fig. 5Chronological train/validation/test split on the project completion timeline. Chronological train/validation/test split on the project completion timeline. Table 2 summarizes the hardware and software environment. All experiments ran on a single NVIDIA A100 GPU with 40 GB memory; the MS-DW-Transformer converged in approximately 2.5 h per full training run. Table 2Experimental software and hardware configuration.ComponentSpecificationGPUNVIDIA A100 (40 GB)CPUIntel Xeon Gold 6330RAM128 GB DDR4FrameworkPyTorch 2.1.0LanguagePython 3.10Operating systemUbuntu 22.04 LTS Table 2Experimental software and hardware configuration.ComponentSpecificationGPUNVIDIA A100 (40 GB)CPUIntel Xeon Gold 6330RAM128 GB DDR4FrameworkPyTorch 2.1.0LanguagePython 3.10Operating systemUbuntu 22.04 LTS Experimental software and hardware configuration. Three metrics compose the evaluation framework. Root mean square error (RMSE) measures absolute prediction deviation in the original cost unit. Mean absolute percentage error (MAPE) provides a scale-invariant accuracy gauge that is intuitive for practitioners. The coefficient of determination \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{R}^{2}$$\end{document} captures the proportion of variance explained by the model, defined as the standard expression below, where yi is the audited cost of sample i, ŷi is the corresponding prediction, ȳ is the test-set mean, and the summation runs over all test samples; \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{R}^{2}$$\end{document} ranges from −∞ to 1, with values closer to 1 indicating that the model explains a larger share of the cost variance. We use MAPE alongside RMSE and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{R}^{2}$$\end{document} rather than as a standalone metric, because MAPE is well known to be asymmetric (over- and under-predictions of equal absolute size produce different percentage errors), unbounded when actual values approach zero, and biased toward favoring forecasts that systematically underestimate. None of these failure modes is severe in our setting — audited PGTR costs are bounded well above zero and the prediction errors are roughly symmetric — but the limitations argue for reporting a triple of metrics rather than relying on any single one. The R² formula is given by the standard expression R² = 1 − Σ(yi − ŷi)²/Σ(yi − ȳ)². Table 3 lists the three metrics together with the interpretive thresholds adopted from the engineering cost estimation literature42. Table 3Evaluation metrics and interpretive benchmarks.MetricDesired directionAcceptable thresholdRMSE (10,000 CNY)Lower is better< 50MAPE (%)Lower is better< 10% \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{R}^{2}$$\end{document} Higher is better> 0.85 Table 3Evaluation metrics and interpretive benchmarks.MetricDesired directionAcceptable thresholdRMSE (10,000 CNY)Lower is better< 50MAPE (%)Lower is better< 10% \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{R}^{2}$$\end{document} Higher is better> 0.85 Evaluation metrics and interpretive benchmarks. These thresholds reflect the accuracy expectations of provincial grid budgeting departments, where a MAPE below 10% is generally deemed sufficient for preliminary cost approval. The next subsection reports the comparative results obtained under this evaluation framework. Comparative analysis of model performance Eleven models in total were trained under identical data splits and preprocessing conditions to ensure a fair comparison. Five represent classical and previously-published baselines: a three-layer back-propagation neural network (BPNN), support vector regression with a radial basis function kernel (SVR), a two-layer LSTM network, a two-layer gated recurrent unit (GRU) network, and a standard Transformer encoder–decoder without the multi-scale or dynamic-weight modifications. Four additional baselines are recent Transformer variants designed for time-series forecasting — Informer21, Autoformer22, Temporal Fusion Transformer (TFT)15, and PatchTST23— included to address the concern that older recurrent baselines may understate the strength of the contemporary state of the art. A graph-based forecasting model, adapting the spatial-temporal graph perception ideas of20 into a project-level graph in which projects executed in similar regions or during overlapping periods are linked by weighted edges, was also evaluated; this baseline addresses the reviewer-raised question of whether graph-structured prior knowledge offers a competitive alternative to our approach. Hyperparameters for each baseline were tuned individually on the validation set via grid search. The two evaluation formulas applied throughout this comparison are RMSE and MAPE (standard formulas):15\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\mathrm{R}\mathrm{M}\mathrm{S}\mathrm{E}=\sqrt[]{\frac{1}{N}\sum\:_{i=1}^{N}(\hat {{y}}_{i}-{y}_{i}{)}^{2}}$$\end{document}16\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\mathrm{M}\mathrm{A}\mathrm{P}\mathrm{E}=\frac{1}{N}\sum\:_{i=1}^{N}\left|\frac{\hat {{y}}_{i}-{y}_{i}}{{y}_{i}}\right|\times\:100\%$$\end{document} Table 4 presents the quantitative results on the 239-sample test set, with each entry reported as the mean of five independent training runs (different random seeds: 17, 42, 123, 2024, 31415). The MS-DW-Transformer achieves the lowest mean RMSE of 31.26 (in units of 10⁴ CNY), a mean MAPE of 6.83%, and an \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{R}^{2}$$\end{document} of 0.937 — all within or surpassing the acceptable thresholds defined in Table 3. Among the baselines, the standard Transformer ranks fifth (after the four recent Transformer variants, which all sit between MS-DW-Transformer and the vanilla version) with an RMSE of 38.41 and MAPE of 8.72%, confirming that attention-based architectures outperform recurrent alternatives on this dataset. LSTM and GRU produce similar results to each other (MAPE around 10.5–11.2%), both falling near the boundary of the acceptable range. BPNN and SVR trail by a wider margin, with MAPE values of 14.37% and 13.05% respectively — a gap that reflects their inability to model the sequential dependencies present in staged renovation cost data43. A more detailed statistical comparison is presented in Table 5, and the extended comparison against recent Transformer variants and the graph baseline appears in Table 6. Table 4Test-set performance of the proposed model and five classical baselines (mean over five runs).ModelRMSE (10,000 CNY)MAPE (%) \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{R}^{2}$$\end{document} BPNN67.8314.370.764SVR61.2913.050.807LSTM48.1711.210.881GRU46.5210.480.889Standard Transformer38.418.720.924MS-DW-Transformer 31.26 6.83 0.937 Table 4Test-set performance of the proposed model and five classical baselines (mean over five runs).ModelRMSE (10,000 CNY)MAPE (%) \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{R}^{2}$$\end{document} BPNN67.8314.370.764SVR61.2913.050.807LSTM48.1711.210.881GRU46.5210.480.889Standard Transformer38.418.720.924MS-DW-Transformer 31.26 6.83 0.937 Test-set performance of the proposed model and five classical baselines (mean over five runs). The numerical gains of the proposed model over the standard Transformer — an 18.6% reduction in RMSE and a 1.89%-point drop in MAPE — warrant closer visual inspection. Figure 6 overlays the predicted and actual cost sequences for all six models across the test period. BPNN and SVR predictions oscillate around a smoothed mean and fail to track the two prominent cost surges near samples 80–100 and 180–210. LSTM and GRU follow the general contour more faithfully but lag behind sharp transitions by several time steps, a consequence of their recurrent memory decay. The standard Transformer captures turning-point timing better, yet it overshoots peak amplitudes on roughly a third of the surges. Our MS-DW-Transformer tracks both the timing and the magnitude of these peaks with the tightest envelope among all models, a behavior we attribute to the gated multi-scale encoding that amplifies fine-grained features precisely when cost trajectories accelerate. Fig. 6Predicted versus actual cost curves on the test set for the six classical models. Predicted versus actual cost curves on the test set for the six classical models. Error stability matters as much as average accuracy for operational budgeting, where a model that is right on average but occasionally wildly wrong can be more dangerous than a slightly less accurate but consistently bounded one. Figure 7 compares the distribution of absolute percentage errors across the test set via box plots. The MS-DW-Transformer exhibits the most compact interquartile range (3.1%–9.4%) and the fewest outliers beyond 15%, whereas BPNN and SVR show heavy right tails extending past 25%23. The recurrent models fall in between, with GRU marginally tighter than LSTM. What stands out about the standard Transformer is that while its median error is reasonably low, its upper whisker extends further than GRU’s — suggesting occasional large misses on atypical projects that the multi-scale gating mechanism in our model manages to contain. Fig. 7Box-plot comparison of absolute percentage error distributions on the test set. Box-plot comparison of absolute percentage error distributions on the test set. Taken together, these results indicate that the MS-DW-Transformer outperforms all five classical baselines across accuracy, trend fidelity, and error stability. To check whether the gap also holds against more recent forecasting models and whether the improvements are statistically meaningful rather than products of random initialization, we conducted two follow-up analyses. The first, summarized in Table 5, reports mean ± standard deviation across five independent runs and a paired Diebold–Mariano test on per-sample absolute errors between MS-DW-Transformer and each baseline; p-values for the four strongest baselines (TFT, PatchTST, Autoformer, and Informer) all fell below 0.01, supporting the conclusion that the observed advantage is unlikely to be coincidental. The second, summarized in Table 6, extends the comparison to those four recent Transformer variants and a graph neural network adapted to the project-similarity graph described above. Although the recent Transformers close most of the gap to vanilla Transformer, they remain consistently behind MS-DW-Transformer; the graph baseline performs better than LSTM/GRU but does not overtake the Transformer family, a pattern we attribute to the limited number of edges that survive within a single-province dataset. The next subsection isolates the contribution of each proposed component through ablation experiments. Table 5Statistical comparison across five independent runs and paired Diebold–Mariano test against MS-DW-Transformer.ModelRMSE (mean ± SD)MAPE % (mean ± SD)R² (mean ± SD)DM-test p-valueBPNN67.83 ± 1.9214.37 ± 0.310.764 ± 0.012< 0.001SVR61.29 ± 1.4513.05 ± 0.240.807 ± 0.009< 0.001LSTM48.17 ± 1.1611.21 ± 0.210.881 ± 0.008< 0.001GRU46.52 ± 1.0410.48 ± 0.180.889 ± 0.007< 0.001Standard Transformer38.41 ± 0.868.72 ± 0.150.924 ± 0.0060.003MS-DW-Transformer (ours)31.26 ± 0.606.83 ± 0.110.937 ± 0.005— Table 5Statistical comparison across five independent runs and paired Diebold–Mariano test against MS-DW-Transformer.ModelRMSE (mean ± SD)MAPE % (mean ± SD)R² (mean ± SD)DM-test p-valueBPNN67.83 ± 1.9214.37 ± 0.310.764 ± 0.012< 0.001SVR61.29 ± 1.4513.05 ± 0.240.807 ± 0.009< 0.001LSTM48.17 ± 1.1611.21 ± 0.210.881 ± 0.008< 0.001GRU46.52 ± 1.0410.48 ± 0.180.889 ± 0.007< 0.001Standard Transformer38.41 ± 0.868.72 ± 0.150.924 ± 0.0060.003MS-DW-Transformer (ours)31.26 ± 0.606.83 ± 0.110.937 ± 0.005— Statistical comparison across five independent runs and paired Diebold–Mariano test against MS-DW-Transformer. Table 6Extended baselines: recent time-series Transformer variants and a graph-based forecaster.ModelRMSEMAPE (%)R²Params (M)Train time (h)Informer36.84 ± 0.728.31 ± 0.140.9285.23.1Autoformer35.62 ± 0.697.97 ± 0.130.9304.73.4PatchTST34.18 ± 0.657.62 ± 0.120.9324.12.9Temporal Fusion Transformer33.79 ± 0.717.45 ± 0.130.9336.34.2Graph-based (project graph)42.95 ± 0.949.81 ± 0.180.9133.62.6MS-DW-Transformer (ours)31.26 ± 0.606.83 ± 0.110.9373.82.5 Table 6Extended baselines: recent time-series Transformer variants and a graph-based forecaster.ModelRMSEMAPE (%)R²Params (M)Train time (h)Informer36.84 ± 0.728.31 ± 0.140.9285.23.1Autoformer35.62 ± 0.697.97 ± 0.130.9304.73.4PatchTST34.18 ± 0.657.62 ± 0.120.9324.12.9Temporal Fusion Transformer33.79 ± 0.717.45 ± 0.130.9336.34.2Graph-based (project graph)42.95 ± 0.949.81 ± 0.180.9133.62.6MS-DW-Transformer (ours)31.26 ± 0.606.83 ± 0.110.9373.82.5 Extended baselines: recent time-series Transformer variants and a graph-based forecaster. Ablation study and key parameter sensitivity analysis To disentangle the contributions of each proposed component, we trained five ablated variants. (A) the full model minus the multi-scale convolution module, replaced by a single-kernel ( k=7) encoder; (B) the full model with the dynamic gating replaced by uniform averaging across branches; (C) the full model trained with pure MSE loss, removing the trend-consistency penalty \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\mathcal{L}}_{trend}$$\end{document}; (D) a variant stripping all three improvements simultaneously, which reduces to the standard Transformer baseline; and (E) — added in response to a reviewer query — an encoder-only variant that replaces the decoder with a feed-forward regression head, retaining the multi-scale and gating modules. Table 7 summarizes the results. Table 7Ablation results on the test set (mean over five runs).VariantRMSEMAPE (%) \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{R}^{2}$$\end{document} (D) No improvements (baseline)38.418.720.924(A) w/o multi-scale convolution35.187.940.929(B) w/o dynamic gating34.037.610.931(C) w/o trend loss32.897.280.933Full MS-DW-Transformer 31.26 6.83 0.937 (E) Encoder-only (no decoder)33.457.250.929 Table 7Ablation results on the test set (mean over five runs).VariantRMSEMAPE (%) \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{R}^{2}$$\end{document} (D) No improvements (baseline)38.418.720.924(A) w/o multi-scale convolution35.187.940.929(B) w/o dynamic gating34.037.610.931(C) w/o trend loss32.897.280.933Full MS-DW-Transformer 31.26 6.83 0.937 (E) Encoder-only (no decoder)33.457.250.929 Ablation results on the test set (mean over five runs). Table 8Implementation details supporting reproducibility (default configuration unless stated otherwise in ablation/sensitivity analyses).ItemSettingEmbedding dimension d_model256Convolution filters per branch64Convolution kernel widthsk ∈ {3, 7, 15}, same padding, GELU activationWeight initializationXavier uniform (linear), Kaiming normal (conv)Random seeds (five runs)17, 42, 123, 2024, 31,415Training epochs (maximum)300Early-stopping patience15 epochs on validation lossMini-batch size64Gradient clipping (max-norm)1.0OptimizerAdam (β₁ = 0.9, β₂ = 0.98, ε = 1e-9)Initial learning rate1e-4 with cosine annealing & warm restartsDropout0.1 after every attention and FFN sub-layerMissing-value imputationk-NN (k = 5), fit on training set onlyFeature scalingMin-max to [0, 1], parameters fit on training set onlyPearson screening threshold|r| ≥ 0.30 against targetPermutation importance thresholdAbove the median across the 26 raw features Table 8Implementation details supporting reproducibility (default configuration unless stated otherwise in ablation/sensitivity analyses).ItemSettingEmbedding dimension d_model256Convolution filters per branch64Convolution kernel widthsk ∈ {3, 7, 15}, same padding, GELU activationWeight initializationXavier uniform (linear), Kaiming normal (conv)Random seeds (five runs)17, 42, 123, 2024, 31,415Training epochs (maximum)300Early-stopping patience15 epochs on validation lossMini-batch size64Gradient clipping (max-norm)1.0OptimizerAdam (β₁ = 0.9, β₂ = 0.98, ε = 1e-9)Initial learning rate1e-4 with cosine annealing & warm restartsDropout0.1 after every attention and FFN sub-layerMissing-value imputationk-NN (k = 5), fit on training set onlyFeature scalingMin-max to [0, 1], parameters fit on training set onlyPearson screening threshold|r| ≥ 0.30 against targetPermutation importance thresholdAbove the median across the 26 raw features Implementation details supporting reproducibility (default configuration unless stated otherwise in ablation/sensitivity analyses). Removing the multi-scale module causes the largest single-component degradation — MAPE rises by 1.11% points relative to the full model — confirming that multi-resolution feature decomposition carries the heaviest explanatory load among the three innovations. Replacing the gated fusion with uniform averaging adds 0.78 points to MAPE, a gap that widens when we inspect per-sample errors at regime transitions: the fixed-weight variant struggles precisely in the project phases where the dominant cost frequency shifts, consistent with the design rationale of Sect. 3.2. Dropping the trend penalty produces the smallest metric decline in MAPE (0.45 points), yet its effect on directional accuracy is disproportionate — the trend-agreement rate falls from 87.4% to 79.1%, a deterioration invisible to RMSE but consequential for budget planning44. Variant E — encoder-only with a linear regression head — incurs a 0.42-point MAPE penalty relative to the full encoder–decoder, smaller than we had initially expected but large enough that we retain the autoregressive decoder for the operational rolling-prediction use case; an encoder-only configuration would, however, be a defensible choice for users who require only single-step forecasts and want a leaner model. We next varied encoder depth across \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\{\mathrm{2,3},\mathrm{4,5},6\}$$\end{document} layers, attention head count across \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\{\mathrm{4,6},\mathrm{8,10,12}\}$$\end{document}, window length w across {30, 60, 90, 120, 150, 180} reporting periods, and the loss-weight coefficients λ₁ ∈ {0.6, 0.8, 1.0, 1.2, 1.4} and λ₂ ∈ {0.1, 0.2, 0.3, 0.4, 0.5}, retraining each configuration from scratch with five random seeds. To quantify sensitivity, we compute a normalized sensitivity coefficient p for each hyperparameter (standard formulation):17\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{S}_{p}=\frac{\varDelta\:\mathrm{M}\mathrm{A}\mathrm{P}\mathrm{E}/{\mathrm{M}\mathrm{A}\mathrm{P}\mathrm{E}}_{base}}{\varDelta\:p/{p}_{base}}$$\end{document} where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\mathrm{M}\mathrm{A}\mathrm{P}\mathrm{E}}_{base}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{p}_{base}$$\end{document} are the values at the optimal setting45. Figure 8 plots MAPE against encoder depth and head count; encoder depth exhibits a clear U-shaped curve (performance improves from 2 to 4 layers, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{S}_{layers}=0.42$$\end{document} then degrades mildly at 5–6 layers, likely due to overfitting on the moderate-sized dataset), while head count shows lower sensitivity (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{S}_{heads}=0.18$$\end{document}) — the model is relatively robust between 6 and 10 heads, with 8 offering the best trade-off. Beyond 10 heads, each head’s subspace dimension drops below 32, and we observed attention maps becoming noisy. Figure 9 reports the window-length sensitivity: short windows (w ≤ 60) starve the attention mechanism of temporal context (MAPE ≈ 7.6–8.9%), very long windows (w ≥ 150) begin to import irrelevant historical information and inflate the parameter–sample ratio; the optimum at w = 120 is broad, with degradation below 0.5 MAPE points within ± 30 of the optimum. We additionally inspected stride sensitivity: contrasting stride-1 with stride-4 and stride-8 windowing, MAPE rose from 6.83% to 7.18% and 7.41% respectively, indicating that the dense sliding window does carry useful gradient signal — and at the same time that the implicit overlap is not the dominant driver of performance, since the stride-8 variant (which removes most window overlap) still beats every classical baseline in Table 4. Figure 10maps the joint sensitivity to λ₁ and λ₂; the minimum sits at (λ₁, λ₂) = (1.0, 0.3), with a flat plateau of width ± 0.2 in λ₁ and ± 0.1 in λ₂ around the optimum, suggesting that the trend-consistency loss helps but does not demand precise tuning46–52. Fig. 8Test-set MAPE versus encoder depth and attention head count (mean ± SD, five runs). Test-set MAPE versus encoder depth and attention head count (mean ± SD, five runs). Fig. 9Sensitivity of test-set MAPE and RMSE to the sliding-window length w (mean ± SD across five runs); the optimum sits at w = 120 reporting periods, with a broad plateau within ± 30 of the optimum. Sensitivity of test-set MAPE and RMSE to the sliding-window length w (mean ± SD across five runs); the optimum sits at w = 120 reporting periods, with a broad plateau within ± 30 of the optimum. Fig. 10Joint sensitivity of test-set MAPE to the composite-loss weights λ₁ (MSE term) and λ₂ (trend term); the red box marks the optimal configuration (λ₁ = 1.0, λ₂ = 0.3). Joint sensitivity of test-set MAPE to the composite-loss weights λ₁ (MSE term) and λ₂ (trend term); the red box marks the optimal configuration (λ₁ = 1.0, λ₂ = 0.3). These ablation and sensitivity results jointly validate two conclusions: all three proposed components contribute meaningfully, and the chosen hyperparameter configuration sits in a broad, stable region of the performance landscape, reducing the risk that our reported gains are artefacts of narrow tuning. Discussion The performance gains reported in Section IV are not merely numerical — they reflect structural advantages that align with the specific data-generating process of PGTR cost sequences. The multi-scale convolution module, responsible for the largest single improvement in the ablation study, succeeds because renovation cost dynamics are genuinely multi-frequency. Weekly procurement orders, monthly progress payments, and quarterly budget revisions each imprint a distinct periodicity onto the cost trajectory. A single-kernel encoder forces the model to compromise across these rhythms, smoothing over short bursts to preserve long trends or vice versa. By decomposing the signal into parallel frequency-sensitive channels before it reaches the attention layers, the multi-scale front-end lets each attention head operate on a cleaner, more homogeneous representation — and the downstream prediction benefits accordingly. The dynamic gating mechanism amplifies this advantage by making the decomposition context-sensitive. Our inspection of learned gate activations across the test set revealed a consistent pattern, which we visualize in Fig. 11: during the first 30–40% of a project timeline, when procurement dominates cost accrual, the narrow-kernel branch ( k=3) receives a mean gate weight of roughly 0.45 ± 0.03; this share drops below 0.25 in later phases when installation labor — a smoother, slower-moving cost component — takes over, and the wide kernel (k = 15) absorbs the difference, climbing to mean weight 0.42 ± 0.04 in the commissioning window. Fixed-weight fusion cannot reproduce this shift, which explains its weaker performance at regime boundaries. To complement the gate-level interpretability with a feature-level view, we computed SHAP values on a 200-sample test subset (Fig. 12). Voltage class and rated capacity emerge as the strongest individual drivers, followed by material price index and conductor type, while environmental and regulatory flags exert smaller but still measurable influence — a ranking that aligns with grid engineers’ domain intuition and offers budgeting officers a concrete account of why a given prediction lands where it does. The trend-consistency loss \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\mathcal{L}}_{trend}$$\end{document}, meanwhile, operates on a different axis entirely. Its contribution to RMSE is modest because it does not directly penalize magnitude errors; what it does penalize is directional disagreement, and the 8.3%-point gap in trend-agreement rate between the full model and the variant without the trend term suggests that this auxiliary signal steers the optimizer toward solutions that, while only marginally better in squared-error terms, are far more useful for the budgeting decisions that motivate this work in the first place. Fig. 11Mean gate activation weights for the three convolutional branches (k = 3, k = 7, k = 15) across normalized project lifecycle progress, with shaded bands giving ± 1 SD over the 239-sample test set; shaded background regions denote procurement, installation, and commissioning phases. Mean gate activation weights for the three convolutional branches (k = 3, k = 7, k = 15) across normalized project lifecycle progress, with shaded bands giving ± 1 SD over the 239-sample test set; shaded background regions denote procurement, installation, and commissioning phases. Fig. 12Feature-level interpretation via mean absolute SHAP values computed on a 200-sample test subset; voltage class and rated capacity emerge as the dominant cost drivers. Feature-level interpretation via mean absolute SHAP values computed on a 200-sample test subset; voltage class and rated capacity emerge as the dominant cost drivers. Regarding generalizability, the model was trained and tested on projects from a single provincial grid corporation. Honesty compels us to treat this as a substantive limitation rather than a footnote: cost structures, regulatory environments, labor markets, and equipment procurement channels differ substantially across Chinese provinces, and the learned gate weights and attention patterns are necessarily calibrated to one specific operational context. We therefore frame the present contribution as a domain-tuned architecture validated in a single-province setting, rather than as a fully general PGTR cost forecaster. As a preliminary check on transfer behavior, we held out 121 projects from a single municipal service district within the dataset and treated the remaining 1,241 as the training pool; the out-of-district MAPE rose to 8.94% (from 6.83% in-distribution), which is encouraging in the sense that the deterioration is bounded but unmistakably signals that recalibration would be required for a truly cross-provincial deployment. Cross-province transfer learning is a natural extension that we plan to pursue once additional datasets become available. Across project types, the model performed comparably on substation retrofits and transmission-line reinforcements but showed slightly higher errors on the smaller relay-protection subset (MAPE 7.62%), possibly because the limited sample count for that category (161 projects) left the multi-scale branches underdetermined. Scaling to new construction rather than renovation projects would demand a rethinking of the feature set, since greenfield work introduces land acquisition and civil engineering costs that renovation projects rarely encounter. Several limitations deserve candid acknowledgment. First, the model struggles with extreme cost outliers — projects whose final cost exceeds three standard deviations from the mean. These outliers, often triggered by force-majeure events or major scope revisions that leave no statistical precursor in the feature history, accounted for roughly 60% of the worst-case prediction errors in our test set. No amount of temporal pattern extraction can predict a truly exogenous shock, and we suspect that integrating external event data (commodity futures, weather alerts, policy announcement feeds) as auxiliary inputs could partially address this gap. Second, our dataset of 1,362 projects, while adequate for the current architecture, would be insufficient to train substantially deeper or wider models without aggressive regularization; the overfitting observed at six encoder layers is a symptom of this constraint. Data augmentation strategies — synthetic minority oversampling for underrepresented project types, or generative models trained on the cost distribution — merit investigation. Third, the model remains a partial black box. The gate activation vectors (Fig. 11) and SHAP-based feature attribution (Fig. 12) collectively offer two complementary windows onto its reasoning — one at the temporal-scale level, one at the feature level — but neither answers the more granular question of why a particular cost prediction is high or low in a single specific project. Counterfactual explanation methods are a sensible next step for practical deployment. On the engineering side, inference takes approximately 38 ms per prediction on a single A100 GPU (24 ms on an Nvidia T4 in production-grade settings); the 3.8 M-parameter model fits comfortably within the memory of a workstation GPU, and we used Monte Carlo dropout (20 forward passes at inference) to produce a coarse predictive uncertainty estimate, with 95% intervals containing the actual cost in 91.6% of test samples — slightly below nominal coverage, suggesting that the dropout uncertainty under-estimates the true variance for outlier projects. Looking beyond PGTR, the framework’s modular design — separable multi-scale encoding, pluggable gating, and a domain-agnostic Transformer backbone — suggests applicability to other engineering cost prediction tasks where multi-frequency temporal patterns and heterogeneous feature spaces co-occur. Hydropower station maintenance, urban rail transit expansion, and renewable energy farm construction all share these characteristics to varying degrees. Adapting the model to these domains would primarily involve re-engineering the input feature set and re-calibrating the convolutional kernel widths to match the dominant periodicities of the target cost process, while the attention and gating machinery could largely be reused. Conclusion This paper presented the MS-DW-Transformer, an improved Transformer-based framework for dynamic cost prediction of power grid technical renovation projects. The model integrates three targeted innovations — multi-scale convolutional feature encoding, gated dynamic weight allocation, and a trend-consistency-augmented composite loss — into a unified end-to-end pipeline tested on 1,362 real-world renovation records spanning nine years, with statistical validation across five independent runs and benchmarking against both classical baselines and four recent time-series Transformer variants plus a graph-based forecaster. Three principal conclusions follow from the experimental findings. First, parallel convolutional branches operating at different temporal resolutions capture the multi-frequency nature of PGTR cost dynamics more effectively than single-scale alternatives; the ablation study confirmed this module as the largest individual contributor to prediction accuracy. Second, context-adaptive gating over the fused scale representations enables the model to re-weight feature channels as project conditions shift — a property that fixed-weight architectures lack and that proves especially valuable at regime-transition points where cost behavior changes abruptly. Third, the trend-consistency penalty, though modest in its effect on RMSE, produces a disproportionate improvement in directional prediction accuracy, pushing the trend-agreement rate from 79.1% to 87.4% — a distinction that matters more for budget planning than raw magnitude error alone. On the held-out test set the full model achieved a MAPE of 6.83 ± 0.11% and an \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{R}^{2}$$\end{document} of 0.937, outperforming ten baselines across all three evaluation metrics, with paired Diebold–Mariano tests confirming statistical significance against the strongest contenders. From a practical standpoint, a sub-7% MAPE meets the approval threshold used by provincial grid budgeting departments and could reduce the frequency of costly mid-project budget revisions. The gate activation patterns and SHAP feature attributions together offer budgeting officers a degree of interpretability absent from conventional black-box predictors. We recognize that the current work has clear boundaries. The model was validated on data from a single province; cross-region generalization remains only partially probed (the within-province held-out-district test gave a MAPE of 8.94%). Performance degrades on extreme outlier projects driven by exogenous shocks that leave no trace in historical features. And the 1,362-sample dataset, while sufficient for the current architecture, constrains further scaling without aggressive regularization or augmentation. Three directions strike us as most promising for future work. Incorporating external knowledge graphs — linking equipment taxonomies, supplier networks, and policy event timelines — could enrich the feature space beyond what tabular project records alone provide. Federated learning offers a path to multi-utility collaborative modeling without requiring sensitive cost data to leave organizational boundaries. Finally, model compression through knowledge distillation or pruning would be necessary before deployment on resource-constrained edge devices at project sites, where real-time rolling forecasts would deliver the greatest operational value.