CosineGate: Semantic Dynamic Routing via Cosine Incompatibility in Residual Networks

Yogeswar Reddy Thota · 2025

Read CosineGate: Semantic Dynamic Routing via Cosine Incompatibility in Residual Networks by Yogeswar Reddy Thota (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-22206.txt.

Introduction

Yogeswar Reddy Thota introduces CosineGate, a method that lets residual networks skip work they do not need. Deep residual networks normally run every block on every input, even when simply passing the input through would do. CosineGate instead watches how the block's output compares with its input, using a cosine incompatibility ratio as a self-supervised signal for when a block adds nothing.

A Gumbel-Softmax relaxation makes the per-sample, per-block skip decisions trainable end to end, and a progressive penalty on total computation keeps usage in check. On CIFAR-10, settings span a frontier from aggressive savings to conservative ones. A balanced setup reached 91.3 percent accuracy while cutting computation by 28.5 percent, matching a full ResNet-20 at lower cost and without extra supervision or distillation.

Catalog record

CREATORYogeswar Reddy Thota
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-22206

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CosineGate: Semantic Dynamic Routing via Cosine Incompatibility in Residual Networks Yogeswar Reddy Thota Abstract Modern deep residual networks perform substantial redundant computation by evaluating all residual blocks for every input, even when identity mappings suffice. We introduce CosineGate , an end-to-end differentiable architecture for dynamic routing in residual networks that uses cosine incompatibility between identity and residual feature representations as a self-supervised skip signal. CosineGate measures semantic redundancy through the Cosine Incompatibility Ratio (CIR) , defined as 1 − cos ⁡ ( x , F ⁡ ( x ) ) 1-\cos(x,F(x)) , and uses Gumbel-Softmax relaxation to enable per-sample, per-block gating during training. A progressive FLOPs regularization term controls average compute usage without destabilizing optimization. On CIFAR-10, CosineGate systematically spans the accuracy–efficiency Pareto frontier: an aggressive configuration achieves 89.9% accuracy with 24.1% FLOPs savings , a balanced configuration achieves 91.3% accuracy with 28.5% savings at epoch 160 , and a conservative configuration reaches a peak of 93.2% accuracy with minimal compute reduction. These results match or exceed ResNet-20 (91.3%) while reducing computation, without auxiliary supervision, distillation, or task-specific heuristics. Our results demonstrate that simple geometric measures of feature incompatibility provide a principled and effective signal for dynamic residual routing. 1 Introduction 1.1 Motivation: Redundancy and Directionality in Residual Computation Residual networks (ResNets) [ 6 ] form the backbone of modern deep learning systems due to their ability to train deep architectures using identity shortcut connections. Each residual block computes y=x+F(x;\,W), (1) where x denotes the block input and F(x) represents a learned residual transformation. This formulation stabilizes optimization by enabling gradient flow through identity paths and guarantees that the trivial solution F(x)=0 always exists. However, this architectural strength introduces a fundamental inefficiency: every residual block performs full computation for every input , regardless of whether the residual transformation contributes meaningful new information beyond the identity mapping. In practice, many residual blocks operate close to identity for a large fraction of inputs, especially in deeper layers where representations become increasingly abstract. Despite this redundancy, standard ResNets incur a fixed computational cost independent of input complexity. This inefficiency is particularly problematic for deployment in resource-constrained environments such as edge devices, embedded systems, and TinyML platforms, where compute, memory, and energy budgets are severely limited. On such devices, executing redundant convolutional operations can dominate latency and power consumption, limiting scalability to larger models or higher-throughput workloads. Empirical evidence strongly supports the prevalence of redundancy in deep networks. The Lottery Ticket Hypothesis [ 4 ] demonstrates that sparse subnetworks comprising only a small fraction of parameters can match the performance of dense models when trained appropriately. However, most pruning approaches are static, applied post-training, and require iterative retraining procedures [ 5 ] , making them unsuitable for adaptive, real-time inference. These observations motivate a central question addressed in this work: Can residual networks dynamically suppress redundant computation during training and inference using a principled, representation-level signal, while preserving end-to-end differentiability and accuracy? 1.2 From Static Pruning to Dynamic Routing Research on neural network efficiency has evolved through several paradigms. Early methods focused on static structured pruning, permanently removing weights, filters, or channels based on magnitude or sensitivity metrics [ 5 , 11 ] . While effective for model compression, these approaches impose a fixed computation graph and lack input adaptivity. More recent work explores dynamic execution, allowing networks to adjust computation on a per-input basis. SkipNet [ 18 ] introduced stochastic residual block skipping using Gumbel-Softmax relaxation, while reinforcement learning approaches learned layer-wise skip policies [ 9 ] . Although effective, these methods typically rely on learned biases or task-specific heuristics rather than explicit measures of semantic redundancy. Parallel advances in mixture-of-experts and conditional computation architectures [ 2 ] demonstrate the power of selective activation at scale. However, these approaches often incur substantial routing overhead and architectural complexity, making them less suitable for compact convolutional models or edge deployment. A key limitation shared by prior dynamic routing methods is the absence of a geometrically grounded, representation-aware criterion for deciding when a residual computation is necessary. 1.3 Geometric Redundancy and Cosine Incompatibility We observe that redundancy in residual computation can be characterized geometrically. If the residual transformation F(x) produces a representation that is strongly aligned with the identity input x , then the block contributes little new semantic information. Conversely, when F(x) introduces features that are directionally distinct from x , the computation is informative and should be preserved. To formalize this intuition, we introduce the Cosine Incompatibility Ratio (CIR) : \text{CIR}(x)=1-\cos(x,F(x)), (2) where \cos(\cdot,\cdot) denotes cosine similarity between flattened feature representations. CIR provides a scale-invariant, representation-level measure of semantic novelty introduced by a residual block. Low CIR indicates redundancy (identity-dominated behavior), while high CIR indicates complementary refinement. Unlike magnitude-based pruning criteria or learned gating heuristics, CIR is: • Self-supervised : derived directly from internal representations, • Input-adaptive : computed per sample and per block, • Architecturally minimal : requiring no auxiliary networks or supervision. 1.4 Biological and Neuromorphic Motivation The concept of suppressing redundant computation based on directional similarity has strong parallels in biological neural systems. Neurophysiological studies show that cortical neurons exhibit direction-selective suppression , where neurons reduce firing when incoming signals align with existing population activity, while orthogonal or novel stimuli trigger stronger activation [ 13 , 3 ] . This behavior aligns closely with principles of predictive coding, where redundant information is suppressed and only prediction errors propagate through the network [ 14 ] . In this context, cosine similarity serves as a proxy for directional agreement in neural population responses, while cosine incompatibility corresponds to novelty-driven activation. From a neuromorphic perspective, such directional gating is particularly attractive. Neuromorphic hardware and TinyML systems emphasize sparse, event-driven computation to reduce energy consumption [ 1 , 7 ] . A gating mechanism based on cosine incompatibility naturally supports this paradigm by activating computation only when representations deviate meaningfully from identity, enabling scalable deployment of deep models on edge devices. CosineGate can thus be viewed as a step toward biologically inspired, direction-aware computation in artificial neural networks, bridging modern deep learning with principles underlying neuromorphic efficiency. 1.5 Problem Formulation Formally, given a residual network composed of N blocks, we seek to learn input-dependent gates \{g_{i}(x)\}_{i=1}^{N} , with g_{i}(x)\in[0,1] , such that each block computes y_{i}=x_{i}+g_{i}(x)\cdot F_{i}(x_{i}). (3) The gating function must satisfy three criteria: 1. Semantic grounding : decisions reflect representational redundancy, 2. Differentiability : gates are trainable end-to-end, 3. Global efficiency control : overall computation can be regulated. CosineGate satisfies these criteria by parameterizing gates using CIR, a lightweight controller, and Gumbel-Softmax relaxation, combined with a global FLOPs regularization objective. 1.6 Contributions This work makes the following contributions: • We introduce CosineGate , a dynamic residual routing mechanism grounded in cosine-based geometric redundancy. • We propose the Cosine Incompatibility Ratio (CIR) as a universal, self-supervised signal for residual block skipping. • We formulate a stable, end-to-end training objective combining classification accuracy, consistency regularization, and progressive FLOPs control. • We demonstrate that CosineGate spans the full accuracy–efficiency Pareto frontier on CIFAR-10, matching or exceeding ResNet-20 accuracy while reducing FLOPs by up to 28.5%. 1.7 Paper Organization Section 2 reviews related work in dynamic routing, pruning, and neuromorphic computation. Section 3 presents the CosineGate methodology,

Read the complete text