Revisiting Auxiliary Losses for Conditional Depth Routing: An Empirical Study
Qingwei Lin · 2026
Read Revisiting Auxiliary Losses for Conditional Depth Routing: An Empirical Study by Qingwei Lin (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-17228.txt.
Introduction
Lin revisits auxiliary losses for conditional depth routing, where some tokens take a cheap FFN and the rest the full one. Lin compares an MLP gate scoring hidden states with a JEPA-guided gate predicting full-versus-cheap outcomes. With utility and rank supervision, the JEPA-guided gate improves early optimization in 3 of 3 seeds, with lower loss and about 10.3x lower gradient norms. Removing that supervision improves loss and speed for both gates and erases its edge, because the oracle label assumes all later layers run full.
Catalog record
| CREATOR | Qingwei Lin |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 69489 |
| LICENSE | http://creativecommons.org/licenses/by-sa/4.0/ |
| RIGHTS | CC BY SA |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.17228
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Revisiting Auxiliary Losses for Conditional Depth Routing: An Empirical Study Qingwei Lin Abstract Conditional depth execution routes a subset of tokens through a lightweight cheap FFN while the remainder execute the standard full FFN at each controlled layer. The central difficulty is gate training : the gate decision must propagate through many layers before it influences the language modeling (LM) loss, so the resulting gradients are weak and noisy. Auxiliary losses are commonly stacked to stabilise training, yet the interactions among them—particularly between a predictive auxiliary and explicit score supervision—have not been systematically compared under controlled conditions. We evaluate two gate designs under a 157.5M-parameter decoder-only model with controller-only training, 50% full-path budget, and 3-seed runs on a fineweb-edu subset. The MLP gate (G1) maps the current hidden state directly to a utility score; the JEPA-guided gate (G3) introduces an action-conditional predictor that forecasts, in a low-dimensional latent space, the outcome of executing full versus cheap for each token, with alignment enforced against a fixed target projection head. Under the standard recipe that includes oracle-style utility regression and pairwise rank supervision (jointly util/rank ), G3 consistently improves early-to-mid optimisation dynamics relative to G1—lower average LM loss, faster threshold hits, ≈ 10.3 × {\approx}\,10.3{\times} lower training gradient norms—in 3/3 seeds, with the 20k-step endpoint LM within a preset 0.005 heuristic reference threshold. Ablations confirm that JEPA alignment is necessary for predictor non-collapse (A1) and that its weight is insensitive across λ JEPA ∈ { 0.25 , 1.0 , 2.0 } \lambda_{\text{JEPA}}\in\{0.25,1.0,2.0\} (A4). A key finding emerges from ablation A3: simultaneously removing util/rank improves best/average LM and threshold-hit speed in 3/3 seeds for both gate architectures; moreover, the early-to-mid optimisation advantage of G3 over G1 disappears, leaving the two gates indistinguishable on the reported LM metrics (though score-calibration behavior still differs). We trace this to a structural mismatch: the oracle label assumes “current layer forks; all subsequent layers execute full,” whereas under actual gated execution each subsequent controlled layer routes only a fraction ρ \rho of its tokens through full, making the teacher off-policy. Under controller-only training this bias can accumulate, rendering util/rank a net-negative contribution under the current recipe ( λ util = 1.0 \lambda_{\text{util}}=1.0 , λ rank = 0.2 \lambda_{\text{rank}}=0.2 ). As a byproduct, removing util/rank cuts the training-time FLOPs proxy from ≈ 1.53 × {\approx}\,1.53{\times} to ≈ 1.07 × {\approx}\,1.07{\times} the full-only baseline (measured wall-clock on a single V100-32GB: 2.87 h → 1.75 h 2.87\mathrm{h}\!\to\!1.75\mathrm{h} , ≈ 39 % {\approx}\,39\% reduction) while simultaneously improving LM quality. All conclusions are strictly scoped to the studied regime and are not extrapolated to larger scales, joint backbone training, or alternative oracle definitions. 1 Introduction A substantial portion of the inference cost of modern language models arises from the full-activation paradigm, under which every token executes the complete feed-forward network (FFN) at every layer. Conditional depth execution relaxes this constraint: at a subset of controlled layers , some tokens execute the standard full FFN while the remainder execute a cheaper alternative ( cheap FFN ), thereby improving the compute–quality trade-off under a fixed budget. The central difficulty of this framework resides not in the design of the cheap path, but in gate/controller training . Although a budget loss can constrain the average full-path ratio, it cannot directly indicate whether a given token at a given layer warrants full computation. When the gate relies solely on indirect gradients propagated from the LM loss through a long chain of layers, training tends to be slow and unstable. To address this weak-supervision problem, prior work in conditional computation and routing has introduced various auxiliary signals to stabilize gate training, including budget/alive constraints ( 16 ) , router numerical-stability losses ( 22 ) , load-balancing losses ( 8 ) , and, in some cases, explicit router supervision or teacher-style signals. However, the pairwise interactions among these auxiliary signals in the specific setting of conditional depth routing—in particular, whether a predictive auxiliary and explicit score supervision are compatible, and whether their joint application is necessarily beneficial—have not, to our knowledge, been systematically compared under controlled conditions. Scope and positioning. This paper evaluates a JEPA-inspired gate design for conditional depth routing and, through systematic ablation, discovers that the util/rank losses are net-negative under the tested recipe. The main experiments run in a small, controlled regime ( {\approx}\,157.5 M total parameters, {\approx}\,0.83 – 0.85 M trainable controller parameters, with a supplementary G1-costmatch control at {\approx}\,0.90 M; controller-only training, 50% budget; a 25% budget supplement is also reported). Throughout, we use the shorthand “ utility/rank ” for the joint Huber-regression and pairwise-rank losses on the oracle label u^{\star} (defined in § 3.3 ). The two families of auxiliary signals under study are: • Predictive auxiliary (JEPA-inspired). We examine a JEPA-inspired variant that trains a predictor to forecast, in a low-dimensional latent space, the outcome of executing full versus cheap for each token at each controlled layer. A fixed target projection head enforces alignment. This auxiliary shapes the future features on which the gate relies. To our knowledge, instantiating JEPA’s action-conditional latent-space prediction as an auxiliary supervision for conditional depth gates has not been systematically reported in the conditional computation literature; we treat it as a candidate auxiliary to be empirically evaluated , not as a default baseline. • Oracle supervision (utility/rank). During training, a counterfactual trajectory (“current layer forks into full vs. cheap; all subsequent layers execute full”) is run online to construct a token-level utility label u^{\star}=\widetilde{\mathrm{CE}}_{\text{cheap}}-\widetilde{\mathrm{CE}}_{\text{full}} . Huber regression and pairwise rank losses then explicitly pull the gate output toward u^{\star} . This auxiliary shapes the gate’s output score itself . Intuitively, the two families supply stronger training signals from the “feature” and “label” directions, respectively, and combining them would appear to be at least as effective as using either in isolation. We take this additivity expectation as our starting point and subject it to empirical scrutiny. We organise findings along two main lines—the G1 vs. G3 comparison and the A3 ablation discovery—with supporting mechanistic evidence: • Main comparison (G1 vs. G3). Under the standard recipe with util/rank, the JEPA-guided gate (G3) consistently improves early-to-mid optimisation dynamics relative to the MLP gate (G1)—lower avg LM, faster threshold hits, {\approx}\,10.3{\times} lower gradient norms—in 3/3 seeds, with the 20k-step endpoint LM within the 0.005 heuristic reference threshold. JEPA alignment is necessary: without it, the predictor collapses (A1); its weight is insensitive in the range 0.25–2.0 (A4). • Key ablation finding (A3). Simultaneously removing util/rank improves LM in 3/3 seeds for both gate architectures ; moreover, the early-to-mid optimisation advantage of G3 over G1 disappears—per-seed LM differences fall within the 0.005 heuristic reference threshold, making the two gates indistinguishable on the reported LM metrics. Both the direction and the stability of this result contradict the starting intuition that util/rank is a “necessary auxiliary.” We trace this to a structural mismatch: the oracle’s “subsequent-all-full” assumption is off-policy relative to actual gated execution (§ 3.3 ). • Supporting evidence (A2, 25% budget). Token-wise semantic pairing provides measurable but secondary benefit (A2); the G3 vs. G1 endpoint LM falls within the 0.005 margin at 25% budget as well, though with per-seed sign reversal. The contributions of this paper are: 1. A systematic evaluation