Activation-Free Backbones for Image Recognition: Polynomial Alternatives within MetaFormer-Style Vision Models

Jeffrey Wang, Jonathan Gregory, Grigorios G Chrysos · 2026

Read Activation-Free Backbones for Image Recognition: Polynomial Alternatives within MetaFormer-Style Vision Models by Jeffrey Wang, Jonathan Gregory, Grigorios G Chrysos (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-20839.txt.

Introduction

Jeffrey Wang, J. Gregory, and G. Chrysos show that pointwise activations and exponential softmax are not needed in MetaFormer-style vision models. They design activation-free polynomial options for three core parts: MLPs, convolutions, and attention. Hadamard products replace standard nonlinearities in their design. Their PolyNeXt models match or beat standard rivals on ImageNet, ADE20K segmentation, and out-of-distribution robustness tests. The work shows a simpler path to strong vision results.

Catalog record

CREATORJeffrey Wang, Jonathan Gregory, Grigorios G Chrysos
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE62347
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.20839

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Activation-Free Backbones for Image Recognition: Polynomial Alternatives within MetaFormer-Style Vision Models Jeffrey Wang, Jonathan Gregory, Grigorios G Chrysos Abstract Modern vision backbones treat pointwise activations (e.g., ReLU, GELU) and exponential softmax as essential sources of nonlinearity, but we demonstrate they are not required within MetaFormer-style vision backbones. We design activation-free polynomial alternatives for three core primitives (MLPs, convolutions, and attention), where Hadamard products replace standard nonlinearities to yield polynomial functions of the input. These modules integrate seamlessly into existing architectures: instantiated within MetaFormer, a modular framework for vision backbones, our PolyNeXt models match or exceed activation-based counterparts across model scales on ImageNet classification, ADE20K semantic segmentation, and out-of-distribution robustness. We also substantially outperform prior polynomial networks at reduced computational cost, showing that polynomial variants of standard modules beat complex custom architectures. Our code is available at https://github.com/jjwang8/PolyNeXt . 1 Introduction Modern vision architectures rely on pointwise activations (ReLU, GELU, SiLU) in feedforward layers and exponential functions inside softmax attention ( 40 ; 15 ) . These activation functions were introduced to provide nonlinearity, enabling neural networks to approximate complex functions. However, activation functions are not the only source of nonlinearity. Polynomials offer a fundamental and well-understood alternative: by the Stone-Weierstrass theorem, any continuous function on a compact domain can be uniformly approximated by polynomials. Unlike activation functions, whose approximation properties depend on architectural depth and specific functional forms, polynomials provide a natural basis for function approximation. The Hadamard product (elementwise multiplication) of two learned linear projections provides a simple mechanism for constructing polynomial representations. Given an input vector \bm{x} , computing (\bm{W}_{a}\bm{x})*(\bm{W}_{b}\bm{x}) yields a second-degree polynomial in the input features, where \bm{W}_{a} and \bm{W}_{b} are learned weight matrices representing linear projections and * denotes elementwise multiplication. By composing such operations across layers, networks can represent polynomials of arbitrary degree without any activation functions. This approach has theoretical appeal: the degree grows predictably with depth, and the resulting networks inherit the approximation guarantees of polynomial function spaces. Figure 1 : ImageNet-1K accuracy comparison. Our polynomial backbones (CPolyNeXt, APolyNeXt) consistently outperform MetaFormer instantiations (ConvFormer, CAFormer) and prior polynomial networks (MONet, DTTN) across model scales. Curves show accuracy vs. (a) parameters and (b) FLOPs. A growing body of work has explored polynomial networks that replace activations with multiplicative interactions ( 7 ; 8 ) . Part of this research has been motivated by Fully Homomorphic Encryption (FHE), an encryption paradigm where standard networks incur substantial overhead due to their non-polynomial activations ( 2 ) . Recent architectures such as MONet ( 4 ) and DTTN ( 31 ) have demonstrated that polynomial networks can approach the performance of activation-based models. However, these methods introduce custom architectures with polynomial modules tightly coupled to their specific designs. This limits their practical utility: adopting them requires committing to an entirely new architecture, their custom designs cannot easily incorporate future improvements to standard layers (e.g., efficient attention variants, better normalization), and when evaluated end-to-end, they still underperform activation-based counterparts. We take a different approach: rather than designing a new architecture from scratch, we develop polynomial alternatives to the standard modules found in modern vision backbones. Our key insight is that by preserving the input-output interface of conventional components, polynomial modules remain compatible with orthogonal architectural improvements. We target three primitives: MLPs for channel mixing, convolutions for local spatial mixing, and self-attention for global spatial mixing. Here, mixing refers to operations that combine information across a particular dimension, either channels (features at each spatial location) or spatial positions. These operations contain nearly all nonlinearities in contemporary backbones: MLPs apply pointwise activations between linear projections, separable convolutions interleave spatial filtering with activations, and attention uses the exponential function in softmax ( 15 ; 29 ; 28 ) . Further, by removing all activations, our work makes progress toward FHE-amenable architectures that can be used for privacy and security tasks. The polynomial modules we introduce are: PolyMLP replaces the activation in feedforward networks with a Hadamard product of two parallel linear projections: \text{PolyMLP}(\bm{x})=\bm{W}_{o}\left((\bm{W}_{a}\bm{x})*(\bm{W}_{b}\bm{x})\right). (1) PolyConv fuses parallel convolutional branches with different receptive fields via elementwise multiplication, replacing the activation in separable convolutions. PolyAttn replaces the exponential in softmax attention with a polynomial kernel, yielding an activation-free attention mechanism. Because our changes target specific operations (activations, exponentials) rather than module structure, they remain compatible with architectural improvements such as windowed attention ( 28 ) , sparse patterns ( 5 ) , or efficient attention variants. We validate our modules within the MetaFormer framework ( 46 ; 47 ) . MetaFormer provides a modular template for vision backbones with pluggable mixing operations. Its instantiations ConvFormer and CAFormer use exactly the primitives we target: separable convolutions, self-attention, and gated MLPs. This modular structure allows us to test each polynomial replacement independently under controlled conditions. Because these primitives are widely adopted ( 34 ; 16 ; 50 ) , we expect our polynomial replacements to transfer beyond MetaFormer to other frameworks. Figure 2 : Polynomial module primitives. We introduce activation-free replacements for three core vision operators: (a) PolyMLP for channel mixing, which forms a second-degree polynomial by Hadamard-multiplying two linear projections (with LayerNorm) before an output projection; (b) PolyConv for convolutional spatial mixing, which replaces the activation in separable convolutions with a Hadamard fusion of a coarse (dilated) and fine depthwise branch (with channel reversal) followed by consolidation and output projection; and (c) PolyAttn for attention-based spatial mixing, which shares the Q/K projection, injects local context via depthwise convolutions on Q,K,V , and replaces softmax with a polynomial attention kernel (s\cdot QK^{\top}+1)^{p} followed by \ell_{1} normalization, where s=\sigma(\lambda) is a learnable scale and p is the polynomial degree. Layer-by-layer comparisons to standard MLP/Conv/Attention blocks are deferred to Appendix Fig. S4 . Realizing the full potential of polynomial modules requires addressing stability challenges. Unlike ReLU which doesn’t amplify outputs, Hadamard products of two large values produce even larger values. This multiplicative amplification compounds across layers, and we observed training instabilities in early experiments without proper controls. To address these challenges and enable stable training of networks reaching 200 layers, we introduce lightweight stabilization mechanisms: Sigmoid-Scale bounds residual contributions via a sigmoid-parameterized scalar, and multi-input skip connections (following NASNet ( 51 ) ) provide each cell with features from two preceding cells, improving gradient flow. We also find that polynomial networks benefit from a depth-over-width design: narrower layers stacked deeper outperform wider, shallower configurations at matched parameter counts ( Section 3.5 ). Prior polynomial networks lack these mechanisms, instead using shallower, wider designs that limit polynomial degree growth. To evaluate our polynomial modules, we instantiate them within a MetaFormer-style architecture ( 47 ) , adopting its four-stage hierarchical structure while adding our stabilization mechanisms. We introduce two model variants: CPolyNeXt , which uses polynomial convolutions throughout, and APolyNeXt , which combines polynomial convolutions with polynomial attention. This design enables direct comparison against MetaFormer instantiations (ConvFormer, CAFormer) at matched scales. Our PolyNeXt backbones match or exceed these activation-based counterparts on both standard and robustness benchmarks, while outperforming prior polynomial networks at lower computational cost. Ablations confirm that reintroducing activations consistently degrades performance. Our contributions are: • Polynomial alternatives to core vision primitives. We introduce PolyMLP, PolyConv, and PolyAttn, which substitute standard nonlinearities with Hadamard products while preserving

Read the complete text