Contextual Gating within the Transformer Stack: Synergistic Feature Modulation for Enhanced Lyrical Classification and Calibration
Marco Gameiro · 2025
Read Contextual Gating within the Transformer Stack: Synergistic Feature Modulation for Enhanced Lyrical Classification and Calibration by Marco Gameiro (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-02053.txt.
Introduction
Sorting song lyrics by type or theme is a text classification job, and extra structural features can help if they are woven in well. Marco Gameiro's 2025 study builds the SFL Transformer, which feeds auxiliary structural features directly into the attention mechanism of a pre-trained BERT encoder.
Earlier designs fused such features only at the final output layer, after the text had already been processed. Gameiro's contextual gating instead modulates the hidden states inside the encoder stack, letting structure steer reading as it happens. The study also examines calibration, how well the model's confidence matches its accuracy, a trust issue for any classifier.
Catalog record
| CREATOR | Marco Gameiro |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 12 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-02053
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Contextual Gating within the Transformer Stack: Synergistic Feature Modulation for Enhanced Lyrical Classification and Calibration Marco Gameiro Abstract This study introduces a significant architectural advancement in feature fusion for lyrical content classification by integrating auxiliary structural features directly into the self-attention mechanism of a pre-trained Transformer. I propose the SFL Transformer, a novel deep learning model that utilizes a Contextual Gating mechanism (an Intermediate SFL) to modulate the sequence of hidden states within the BERT encoder stack, rather than fusing features at the final output layer. This approach modulates the deep, contextualized semantic features ( H seq H_{\text{seq}} ) using low-dimensional structural cues ( F struct F_{\text{struct}} ). The model is applied to a challenging binary classification task derived from UMAP-reduced lyrical embeddings. The SFL Transformer achieved an Accuracy of 0.9910 0.9910 and a Macro F1 score of 0.9910 0.9910 , significantly improving the state-of-the-art established by the previously published SFL model (Accuracy 0.9894 0.9894 ). Crucially, this Contextual Gating strategy maintained exceptional reliability, with a low Expected Calibration Error ( ECE = 0.0081 \text{ECE}=0.0081 ) and Log Loss ( 0.0489 0.0489 ). This work validates the hypothesis that injecting auxiliary context mid-stack is the most effective means of synergistically combining structural and semantic information, creating a model with both superior discriminative power and high-fidelity probability estimates. \jvol vv \jnum nn 2025 \dochead Long Paper \pageonefooter Action editor: {action editor name}. Submission received: DD Month YYYY; revised version received: DD Month YYYY; accepted for publication: DD Month YYYY. \affilblock 1 Introduction The original challenge of integrating complex deep semantic features with interpretable structural cues for lyrical analysis remains an active area of research Tavares and Ayres 2025 ; Wu and Wan 2025 . While Transformer architectures, like BERT Devlin et al. 2019 , excel at sequence modeling, the optimal method for leveraging non-textual metadata (such as popularity or linguistic metrics) within these models is still debated Gao et al. 2020 . Previous efforts, including our own Synergistic Fusion Layer (SFL) model Gameiro 2025 , demonstrated that non-linear gating is vastly superior to simple concatenation for maximizing model reliability and calibration (achieving \text{ECE}=0.0035 vs. 0.0500 for Random Forest). The previous SFL model, however, operated outside the core sequence architecture, fusing a final Sentence-BERT embedding ( F_{\text{deep}} ) with the structural features ( F_{\text{struct}} ) at the classification head. While effective, this approach treats the semantic embedding as a static vector, missing the potential for the structural context to influence the *internal contextualization* performed by the Transformer’s self-attention mechanism. I address this limitation by introducing the SFL Transformer with an Intermediate SFL (ISFL) module. This architecture relocates the contextual gating mechanism inside the Transformer encoder stack. The ISFL uses the structural features ( F_{\text{struct}} ) to generate a gate that modulates the sequential hidden states ( \mathbf{H}_{\text{seq}} ) after a pre-determined encoder layer, directly influencing the subsequent layers’ attention computations. This work makes three principal contributions: 1. Novel Architecture: We propose the Intermediate SFL (ISFL) module for Contextual Gating, which non-linearly injects structural context by modulating the hidden states mid-stack, providing a superior method for synergistic feature fusion. 2. Performance Enhancement: The SFL Transformer achieves a new state-of-the-art performance for this task ( \text{Accuracy}=0.9910 , \text{Macro F1}=0.9910 ), surpassing the performance of the non-contextual SFL architecture ( \text{Accuracy}=0.9894 ). 3. Architectural Validation: We demonstrate that applying the structural context directly to the sequence representation improves discriminative power while maintaining the model’s high-fidelity calibration ( \text{ECE}=0.0081 ), validating the hypothesis that structural context is optimally utilized as an internal modulator rather than an external feature. 2 Methods 2.1 Data and Feature Preparation The data acquisition, preprocessing, and the custom lyrical structure features ( F_{\text{struct}} ) remain consistent with the prior work Gameiro 2025 . The four structural features are: popularity_norm , rhyme_density , lexical_diversity (TTR), and pronoun_ratio . These features are standardized using StandardScaler ( \mathbf{X}_{\text{aux}} ). 2.2 Label Creation and Data Split The binary classification target (Class 0: Dominant Archetype, Class 1: Alternative Archetypes) is generated identically: Sentence-BERT embeddings, UMAP reduction ( \text{n\_components}=20 ) McInnes, Healy, and Melville 2018 , and HDBSCAN clustering Campello, Moulavi, and Sander 2013 . This ensures direct comparability of results across model generations. The data is split using a StratifiedShuffleSplit ( \text{test\_size}=0.2 ). 2.3 The SFL Transformer Architecture The SFL Transformer is built upon the BERT-base-uncased model. Unlike standard fine-tuning, which only uses the final classification head, our model modifies the internal encoder structure. 2.3.1 Intermediate SFL (ISFL) Module (Contextual Gating) The IntermediateSFL (ISFL) module replaces the previous external dense layer SFL and is integrated directly within the Transformer’s encoder layers. This module allows structural context to modulate the sequence representations at an intermediate step. • Gating Vector Generation: The low-dimensional structural features \mathbf{F}_{\text{struct}}\in\mathbb{R}^{4} are mapped to a high-dimensional Gating Vector \mathbf{g}\in\mathbb{R}^{D_{\text{hidden}}} via a two-layer mapping: \mathbf{g}=\sigma(\mathbf{W}_{\text{gate}}\cdot\mathbf{F}_{\text{struct}}+\mathbf{b}_{\text{gate}}) where D_{\text{hidden}}=768 (BERT’s hidden size), and \sigma is the Sigmoid activation, ensuring gate values are between 0 and 1. • Sequence Modulation: The Gating Vector \mathbf{g} is broadcast across the sequence dimension and applied to the hidden states \mathbf{H}_{\text{seq}} output by the preceding Transformer block ( L_{i} ): \mathbf{H}^{\prime}_{\text{seq}}=\mathbf{H}_{\text{seq}}\odot\mathbf{g} where \odot is the Hadamard (element-wise) product. This selectively scales the features of every token in the sequence based on the global structural context provided by F_{\text{struct}} . Figure 1 : Architecture of the Intermediate Synergistic Fusion Layer (ISFL) Module. The structural cues ( F_{\text{struct}} ) are used to generate a global Gating Vector ( \mathbf{g} ), which is broadcast and non-linearly modulates the contextual sequence embeddings ( \mathbf{H}_{\text{seq}} ) output by a preceding Transformer block via element-wise multiplication ( \odot ). 2.3.2 Insertion Point The ISFL module is strategically inserted into the BERT encoder after the \mathbf{6^{th}} Transformer block ( \mathbf{insert\_layer\_index=6} ). This placement allows the first six layers to develop preliminary contextual representations before they are structurally modulated, and the final six layers to re-contextualize the modulated sequence. 2.4 Training and Evaluation The lyric text is tokenized using the BertTokenizer with a \mathbf{MAX\_LEN} of 128 . Training is performed using the AdamW optimizer with \mathbf{lr=2e-5} for 3 epochs, following standard practices for fine-tuning pre-trained Transformers. Evaluation focuses on the same critical reliability metrics: Log Loss , Brier Score Loss , and Expected Calibration Error (ECE) Naeini, Cooper, and Hauskrecht 2015 ; Guha et al. 2024 . The ECE is calculated across 10 bins. 3 Results 3.1 Comparative Performance: SFL Transformer vs. Previous Models Table 1 presents the performance of the new SFL Transformer alongside the previous best-performing models. Table 1 : Comparative Analysis: SFL Transformer (Contextual Gating) vs. Previous Architectures Model Configuration Accuracy Macro F1 MCC Brier Score Loss Log Loss ECE Score SFL Transformer (Contextual Gating) 0.9910 0.9910 0.9821 0.00849 0.04885 0.00808 Previous SFL Model (Gated Fusion) Gameiro 2025 0.9894 0.9894 0.9787 0.00796 0.03045 0.00351 RF Baseline (Concatenated) Gameiro 2025 0.9868 0.9868 0.9736 0.01589 0.07720 0.05000 The SFL Transformer achieved the highest discriminative performance across all standard metrics ( \mathbf{\text{Accuracy}=0.9910} , \mathbf{\text{Macro F1}=0.9910} , \mathbf{\text{MCC}=0.9821} ). This represents an improvement in Accuracy of 0.16\% and 0.32\% over the previous SFL model and the RF baseline, respectively. The superior MCC score confirms the ISFL’s ability to maximize predictive separation between the two classes. 3.2 Reliability and Calibration Analysis While