SURGeLLM : Rethinking Multi-Task Evaluation through Task-Aware Feature Gating with Class-Balanced Normalization
Noor Islam S. Mohammad, Uluğ BayazıtDept. of Computer Science, Istanbul Technical University {islam23, ulugbayazit}@itu.edu.tr · 2026
Read SURGeLLM : Rethinking Multi-Task Evaluation through Task-Aware Feature Gating with Class-Balanced Normalization by Noor Islam S. Mohammad, Uluğ BayazıtDept. of Computer Science, Istanbul Technical University {islam23, ulugbayazit}@itu.edu.tr (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-24259.txt.
Introduction
Mohammad and Bayazıt re-engineer fine-tuned language encoders, so one model can serve many text tasks without each task corrupting the others. Mixed deployments face mismatched inductive biases per task. Class imbalance warps feature statistics. No channel exists for outside lexical knowledge. Their 2026 framework, SURGeLLM, answers each problem with a small light module inside a unified transformer.
A surgical feature gate learns a per-dimension weighting over chosen lexical indicators, collapsing to the identity when those features carry no signal. Task-conditioned prefix tokens announce the task and binned feature values at every input. Instance-Weighted Normalization strips class-prior bias from the gate's statistics. Across four tasks — sentiment, multi-hop retrieval, prompt attribution, and authorship detection — the full variant posts the best average scores, with the largest jump on authorship detection. A control using random vocabulary confirms the gains come from genuine lexical signal, not extra parameters.
Catalog record
| CREATOR | Noor Islam S. Mohammad, Uluğ BayazıtDept. of Computer Science, Istanbul Technical University {islam23, ulugbayazit}@itu.edu.tr |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 72 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.24259
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SURGeLLM : Rethinking Multi-Task Evaluation through Task-Aware Feature Gating with Class-Balanced Normalization Noor Islam S. Mohammad, Uluğ BayazıtDept. of Computer Science, Istanbul Technical University {islam23, ulugbayazit}@itu.edu.tr Abstract Fine-tuned encoders deployed across heterogeneous NLP tasks face three compounding problems: mismatched inductive biases, class-imbalance corruption of feature statistics, and no mechanism to condition attention on external lexical knowledge. We introduce SURGeLLM , a unified transformer framework that addresses each with a dedicated lightweight module: a surgical feature gate (learned per-dimension sigmoid over curated lexical indicators and [CLS] ; provably degenerates to identity when features are uninformative), task-conditioned prefix tokens (quantized feature values and task identity prepended to every input), and Instance-Weighted Normalization (IWN; removes class-prior bias from gate statistics). We prove an excess-risk bound linking gate benefit to surgical feature alignment . Across four tasks, SST-2, multi-hop retrieval, LLM-prompt attribution, and authorship detection, covering 17,830 examples and eleven model variants over three seeds, the IWN variant achieves macro-F1 0.940 ( + 0.036 +0.036 over the strongest non-IWN baseline; + 0.130 +0.130 on authorship detection). A random-vocabulary control ( − 0.028 -0.028 avg. F1) confirms gains are lexical, not parametric. Code, vocabularies, and a 99.5 % 99.5\% -recovery auto-extraction recipe are released. 1 Introduction Pre-trained encoders fine-tuned per task incur real costs: parameter duplication, no amortized inference, and no shared linguistic structure. Multi-task learning ( 5 ; 29 ; 35 ) addresses this in principle, but structurally heterogeneous tasks—differing in vocabulary, label space, and register—interfere destructively ( 47 ; 9 ; 16 ) in ways that near-isotropic benchmarks like GLUE ( 46 ) do not expose. We study the hard case: a single encoder handling (a) movie-review sentiment, (b) multi-hop retrieval QA, (c) LLM-prompt attribution, and (d) human/LLM authorship—tasks sharing a backbone but drawing on largely disjoint surface signals. Two observations motivate explicit feature injection beyond end-to-end fine-tuning. First, stylometric surface statistics remain discriminative even after fine-tuning ( 14 ; 34 ) , suggesting the encoder does not always exploit them optimally. Second, sequence truncation destroys global statistics (pronoun rates, marker densities) that cannot be recovered from a partial view ( 13 ) . We address both with a surgical vocabulary , ten curated lexical indicator groups yielding a 16-dimensional feature vector \mathbf{s}\in\mathbb{R}^{16} computed on the full untruncated text—fused with the [CLS] representation via a learned per-dimension sigmoid gate and simultaneously injected as task-conditioned prefix tokens. Global standardization \mathbf{s} is contaminated by class prior under severe skew (our authorship corpus: 9.3{:}1 ), causing the gate to learn a sub-optimal fusion. Instance-Weighted Normalization (IWN) replaces global with class-balanced per-dimension statistics at training time, with no test-time labels required, yielding +0.130 an absolute F1 on authorship detection, the largest single gain in our study. Contributions. Framework (§ 3 ): a unified multi-task encoder with per-dimension feature gates, task-conditioned prefix tokens, and IWN; plug-compatible with any HuggingFace encoder. Theory (§ A ): excess-risk bound (Theorem 1 ) linking gate benefit to surgical feature alignment \rho_{k} ; degeneracy result (Proposition 2 ) proving the gate is safe when features are uninformative. Empirics (§ 6 – 7 ): eleven variants across four encoder backbones and T5-base over three seeds; IWN achieves an aggregate macro-F1 of 0.940 ( +0.036 over the strongest non-IWN baseline); random-vocabulary control ( -0.028 avg. F1) confirms gains are lexical, not parametric. Auto-extraction (Appendix E ): Log-odds plus embedding clustering recovers 99.5\% manual curation performance, enabling transfer to new domains. 2 Related Work Multi-task and feature-augmented Transformers. MT-DNN ( 29 ) , Muppet ( 1 ) , T5 ( 35 ) , and mixture-of-experts models ( 40 ; 15 ) all assume near-homogeneous task structure. Injecting handcrafted features into neural encoders ( 14 ; 34 ) and shallow-feature scalar gating ( 42 ; 18 ) are the closest precedents. SURGeLLM differs on three axes: (i) structurally heterogeneous tasks; (ii) a per-dimension, instance-conditioned cross-modal gate (versus scalar intra-modal gating in highway networks and GLUs ( 11 ) ); (iii) explicit class-imbalance remediation via IWN. LLM-text Detection and Stylometry. Detection methods span token-level probability signals ( 17 ) , curvature-based zero-shot tests ( 32 ) , and watermarking ( 21 ) . Classical stylometry ( 22 ; 43 ) shows surface features reliably signal authorship; our surgical vocabulary inherits this tradition and integrates it as an encoder prior. Class imbalance in loss-side ( 27 ) and sampling-side ( 7 ; 10 ) corrections are standard. IWN is a feature-statistics correction—class-balancing the standardization of \mathbf{s} before-gate projection—orthogonal to both and, to our knowledge, novel in feature-augmented NLP gating. 3 The SURGeLLM Framework 3.1 Problem Formulation Let \mathcal{T}=\{t_{1},t_{2},t_{3},t_{4}\} be a fixed set of tasks, each associated with a label space \mathcal{Y}_{t_{k}} of cardinality n_{c,k} . The multi-task corpus is \mathcal{D}=\bigcup_{k=1}^{|\mathcal{T}|}\mathcal{D}_{k} where \mathcal{D}_{k}=\{(x_{i},y_{i},t_{k})\}_{i=1}^{N_{k}} . We seek a single parametric model f_{\theta}:\mathcal{X}\times\mathcal{T}\to\bigcup_{k}\mathcal{Y}_{t_{k}} that minimizes the multi-task empirical risk: \mathcal{L}(\theta)=\sum_{k=1}^{|\mathcal{T}|}\frac{w_{k}}{|\mathcal{D}_{k}|}\sum_{(x,y,t_{k})\in\mathcal{D}_{k}}\ell\!\left(f_{\theta}(x,t_{k}),\,y\right), (1) where \ell is the cross-entropy loss and \{w_{k}\} are non-negative task weights. We use w_{k}=1 throughout and rely on per-task batch sampling for balance; alternative schedules ( 44 ; 39 ; 28 ) are compatible with our framework. What is shared and what is task-specific. Of the model’s parameters, the encoder \mathcal{E}_{\phi} ( 66 M– 220 M depending on backbone), the surgical feature projection (\mathbf{W}_{s},\mathbf{b}_{s}) , the gate matrices (\mathbf{W}_{g},\mathbf{b}_{g}) , the task-embedding matrix \mathbf{E}\in\mathbb{R}^{|\mathcal{T}|\times d} , and the prefix-token embeddings are all shared across tasks. Only the per-task heads \{(\mathbf{W}_{1,k},\mathbf{b}_{1,k},\mathbf{W}_{2,k},\mathbf{b}_{2,k})\}_{k=1}^{|\mathcal{T}|} are task-specific. The shared parameters constitute over 99\% of the total parameter count, justifying the multi-task framing in the conventional MT-DNN sense ( 29 ) . 3.2 Encoder Backbone Given an input text x , a pretrained transformer encoder \mathcal{E}_{\phi} (BERT, RoBERTa, DistilBERT, or ALBERT in our experiments) produces a sequence of contextual representations. We extract the [CLS] token embedding: \mathbf{h}=\mathcal{E}_{\phi}(x)_{[0]}\in\mathbb{R}^{d}, (2) where d=768 for all base-scale encoders. A learnable task-embedding matrix \mathbf{E}\in\mathbb{R}^{|\mathcal{T}|\times d} provides per-task offset vectors \mathbf{E}_{t_{k}} that are mixed with \mathbf{h} through a small-coefficient residual addition: \tilde{\mathbf{h}}=\mathbf{h}+\alpha\,\mathbf{E}_{t_{k}},\qquad\alpha=0.1. (3) Why a small mixing coefficient? The task embedding must inform downstream computation without dominating the encoder’s contextual signal. We pick \alpha=0.1 following the residual-norm-preservation argument of 20 : at initialization, the task embedding contributes a perturbation of magnitude \alpha\,\left\lVert\mathbf{E}_{t_{k}}\right\rVert , which is small relative to the encoder output norm \left\lVert\mathbf{h}\right\rVert\approx\sqrt{d}\sigma_{h} for the \sigma_{h}\approx 1 initialization scheme used in modern encoders. Empirically, \alpha\in[0.05,0.2] was stable; \alpha=1 caused the task embedding to dominate during early training and slowed convergence by {\sim}1 epoch. 3.3 Surgical Feature Extraction Let \mathcal{V}=\{v_{1},\ldots,v_{10}\} the ten indicator groups of the surgical vocabulary be (Appendix D contains the complete listing). For an input x with a lowercased form \tilde{x} , the count feature for the j -th group is: s_{j}=\sum_{w\in v_{j}}\mathbf{1}[w\in\tilde{x}],\qquad j=1,\ldots,10, (4) where prefix matching is used for inflectional families (e.g., oscillat* matches oscillation, oscillates, oscillating ). Six surface features are appended: s_{11} (total word count), s_{12} (mean word length in characters), s_{13} (sentence count obtained via splitting on .!? ), s_{14} (question-mark count), s_{15} (exclamation-mark count), and s_{16}=\mathbf{1}[\text{any digit in }\tilde{x}] (indicator for the presence of digits). The full surgical feature vector is \mathbf{s}(x)=[s_{1},\ldots,s_{16}]^{\top}\in\mathbb{R}^{16}_{\geq 0} . 3.4 The Surgical Feature Gate The gate \mathcal{G} fuses the task-conditioned CLS representation \tilde{\mathbf{h}} with a non-linear projection of the surgical-feature vector. We describe each step explicitly. Step 1: Feature