MoR–Swin: Efficient Vision Transformer Using Mixture of Recursions Authors: Yongbao Ai, Tianxiang Gao, Zhipeng Lin, Longqi Yang, Qingyu Chang Published: 2026 ABSTRACT Vision Transformers, especially Swin Transformer, have become default backbones for various vision tasks but suffer from high memory consumption and training costs. This letter proposes MoR–Swin, a novel architecture that integrates Mixture of Recursions (MoR) into Swin Transformer. An adaptive token-level recursion mechanism dynamically allocates computational depth based on semantic complexity. A recursive window attention module and a lightweight router with load balancing loss are introduced. Extensive experiments on ImageNet classification, COCO detection, and ADE20K segmentation show that MoR–Swin reduces parameters by about 50% and accelerates inference up to twofold at a modest accuracy cost (within about 0.5 points of Swin-B on ImageNet-1K). It provides a new technical pathway for optimizing Vision Transformer models, significantly enhancing their applicability in resource-constrained environments. 1. Introduction Vision Transformer (ViT) [1,2] ended the dominance of CNNs in 2021. Swin Transformer [2,3] introduced hierarchical shifted-window attention, reducing complexity to O(N) while achieving state-of-the-art results in image classification, object detection, and semantic segmentation. SwinV2 [4] later scaled the architecture to 3 billion parameters. However, the growing model size brings prohibitive memory consumption and training cost, limiting deployment in resource-constrained environments. Sparse Mixture of Experts (MoE) [5,6,7] has reduced the cost of scaling in natural language processing by activating only a subset of parameters per input. A large number of MoE-based large language models [8,9,10,11,12] have demonstrated the effectiveness of conditional computation. However, applying sparse MoE to vision is limited by training instability, expert load imbalance, and communication bottlenecks. Recently, Liu et al. [13] proposed Mixture of Recursions (MoR), which replaces sparse expert routing with adaptive token-level recursion depths, offering a parameter-efficient alternative. Beyond MoE, a parallel line of work pursues adaptive computation directly in vision transformers. Along the spatial dimension, DynamicViT [14] and EViT [15] prune or reorganize less informative tokens, and ToMe [16] merges similar tokens to shorten the sequence; A-ViT [17] halts token computation early via learned halting scores. These methods reduce the token count N but discard information irreversibly. Along the width dimension, sparse MoE increases parameters with the number of experts and suffers from load imbalance. In contrast, MoR–Swin operates along the depth dimension: it retains all tokens and adaptively allocates the number of recursive refinement steps per token under a single set of shared parameters. This design preserves full spatial information, keeps the parameter count nearly constant, and naturally matches the spatial heterogeneity of images (simple background regions vs. complex foreground textures), making it particularly well suited to hierarchical window attention. Inspired by the design philosophy of the MoR architecture, in this letter, we integrate MoR into Swin Transformer and propose MoR–Swin. Our contributions are summarized as follows:We propose MoR–Swin, the first architecture that integrates Mixture of Recursions into Swin Transformer, enabling adaptive token-level computational depth allocation based on semantic complexity.We introduce a recursive window attention module and a lightweight router with a load balancing loss, together with memory optimization strategies that substantially reduce memory usage.Extensive experiments on ImageNet classification, COCO detection, and ADE20K segmentation show that MoR–Swin reduces parameters by about 50% and accelerates inference up to twofold at a modest accuracy cost (within about 0.5 points of Swin-B on ImageNet-1K).Analyzing MoR’s inherent advantages over MoE in vision tasks (e.g., training stability and cross-resolution transfer), discussing its limitations, and outlining future research directions for adaptive recursive computation in broader visual tasks. We propose MoR–Swin, the first architecture that integrates Mixture of Recursions into Swin Transformer, enabling adaptive token-level computational depth allocation based on semantic complexity. We propose MoR–Swin, the first architecture that integrates Mixture of Recursions into Swin Transformer, enabling adaptive token-level computational depth allocation based on semantic complexity. We introduce a recursive window attention module and a lightweight router with a load balancing loss, together with memory optimization strategies that substantially reduce memory usage. We introduce a recursive window attention module and a lightweight router with a load balancing loss, together with memory optimization strategies that substantially reduce memory usage. Extensive experiments on ImageNet classification, COCO detection, and ADE20K segmentation show that MoR–Swin reduces parameters by about 50% and accelerates inference up to twofold at a modest accuracy cost (within about 0.5 points of Swin-B on ImageNet-1K). Extensive experiments on ImageNet classification, COCO detection, and ADE20K segmentation show that MoR–Swin reduces parameters by about 50% and accelerates inference up to twofold at a modest accuracy cost (within about 0.5 points of Swin-B on ImageNet-1K). Analyzing MoR’s inherent advantages over MoE in vision tasks (e.g., training stability and cross-resolution transfer), discussing its limitations, and outlining future research directions for adaptive recursive computation in broader visual tasks. Analyzing MoR’s inherent advantages over MoE in vision tasks (e.g., training stability and cross-resolution transfer), discussing its limitations, and outlining future research directions for adaptive recursive computation in broader visual tasks. 2. Method 2.1. Overall Architecture MoR–Swin retains the hierarchical pyramid of Swin Transformer (four stages). The key modification is that standard Swin blocks are replaced with Recursive Mixture Blocks. As illustrated in Figure 1, each block contains recursive window attention (RWA) and a dynamic router that decides for each token whether to recurse or exit early. 2.2. Adaptive Token-Level Recursion The core innovation of MoR–Swin is the adaptive token-level recursion mechanism. Unlike standard Swin, where every token is processed by the same number of layers, MoR–Swin allocates more computation to semantically complex tokens (e.g., objects with fine texture) and less to simple tokens (e.g., sky or smooth surfaces). Given input tokens x∈RN×C at a certain stage, the router assigns each token a scalar score via a channelwise linear projection, then normalizes the scores across tokens with a softmax function:(1)si=LinearC→1(xi)∈R,w=Softmax{si}i=1N∈RN, where xi∈RC is the feature of token i, LinearC→1 maps each token’s channel vector to a single scalar score si (output dimension 1), and Softmax is applied across the N tokens to produce a distribution w. The top-scoring ⌊rN⌋ tokens (the highest-weight fraction r of the N tokens) are selected for further recursion. The remaining tokens bypass the recursive computation and are directly passed to the output via residual connection. Here r∈(0,1] is a tunable hyperparameter: the fixed proportion of tokens entering recursion at each stage (by default, r=0.8, i.e., 80% of the stage’s token count), so that the number of selected tokens ⌊rN⌋ scales adaptively with the stage’s token count N. This choice is consistent with the load-balancing target t1=0.8 below (the fraction of tokens undergoing at least one recursion): the selection capacity r upper-bounds t1, and so r≥t1 is required. We use the selection fraction r throughout to avoid overloading the symbol Na (the maximum recursion depth). Default per-stage ratios and a sensitivity analysis of r (its effect on accuracy, FLOPs, and Expert CV) are provided in the Supplementary Material. The recursion depth for each selected token is adaptively determined by its own depth distribution {pi,k} (Equation (3)) through the cumulative-probability stopping rule of Equation (4). Simple tokens (e.g., image background regions) might undergo only 1–2 recursions, while complex tokens (e.g., small objects or complex textures in the image) might undergo all recursions. To facilitate practical deployment, we set the maximum recursion depth Na to 3 for classification and 5 for segmentation. We now formalize the complete per-token recursive update. Let xi(t) denote the feature of the ith token at recursion depth t, and let mi(t)∈{0,1} be the gate indicating whether token i continues recursion at step t (determined by the router). The recursive update is(2)xi(t+1)=xi(t)+mi(t)·RWALN(xi(t));W,t=0,1,…,Na−1, where W denotes the projection weights shared across all recursion steps, while mi(t) is the per-token gate provided by the router (see below). Tokens with mi(t)=0 bypass computation at step t via residual connection. The router assigns each token a probability over the Na recursion depths via softmax gating. Let gk(x) be the router network’s score for the kth recursion depth (k=1,…,Na):(3)pk(x)=exp(gk(x))∑j=1Naexp(gj(x)),k=1,…,Na. The routing probability pk determines the likelihood that a token stops at recursion depth k, and by construction, it is normalized across the Na recursion depths (∑k=1Napk=1), in contrast to the token-normalized selection weight w in Equation (1) (for which ∑i=1Nwi=1). The branch score gk is produced in a single forward evaluation of the router from the token’s own channel features xi (no spatial pooling), yielding the full depth distribution {pk}k=1Na for that token. We deliberately refrain from asserting that gk is a monotonic function of semantic complexity: as detailed below, we only establish an empirical association between complexity proxies and the resulting recursion depth, not a built-in monotone mechanism. Using this fixed per-token depth distribution {pi,k}k=1Na, the recursion of token i stops at the smallest depth whose cumulative probability reaches a threshold θ, capped at the maximum depth Na:(4)Di=mint∈{1,…,Na}:∑k≤tpi,k≥θ, where pi,k is the depth-k routing probability of token i from Equation (3), computed once per token and normalized across the Na depths so that ∑k=1Napi,k=1 (hence, the cumulative sum is monotone in t and always reaches θ by t=Na, and so the set is non-empty). We stress that the accumulation and the threshold θ (default 0.6) are defined over the depth dimension k, and so θ is a fraction of a token’s own depth-probability mass; it is not applied to the token-normalized selection weight w of Equation (1) (for which ∑iwi=1 across tokens, making a per-token threshold meaningless). To provide a measurable and computable definition of “semantic complexity,” we adopt token-level attention entropy as a proxy:(5)Ci=H(ai)=−∑j=1Naijlogaij,aij=SoftmaxqiK⊤dj, where higher entropy indicates that the token’s context is more uncertain. We emphasize that Ci is a computable proxy correlated with semantic complexity, not a direct measure of it. Empirically, Ci is positively associated with the actual recursion depth Di (Spearman ρ=0.72 on ImageNet, and 0.69/0.66 on ADE20K/COCO; all p<0.001). We therefore limit our claim to an association between these proxies and routing depth rather than asserting that routing depth directly reflects semantic complexity. To probe whether deeper recursion actually benefits high-complexity tokens, we additionally report a controlled intervention experiment (Supplementary Material, Section S5, “Semantic Complexity Correlation”): forcing reduced depth on high-entropy tokens degrades accuracy, whereas forcing increased depth on low-entropy tokens yields no significant gain, providing evidence stronger than correlation alone. The mean recursion depth actually realized under the threshold stopping rule is the empirical average of the per-token stopping depths Di (Equation (4)) over all tokens and images:(6)E[D]=1N∑i=1NDi,Di∈{1,…,Na}. The computational cost is proportional to E[D]: the more tokens there are that stop at greater depths, the more refinement steps are executed. We report E[D] as a measured quantity (not the analytic ∑kkpk, which need not coincide with the threshold-truncated depth). For ImageNet classification (Na=3), the measured E[D]≈1.6 (consistent with the target proportions t1+t2+t3=0.8+0.5+0.3=1.6), and for ADE20K segmentation (Na=5), E[D]≈2.8. The hard top-fraction selection mi=⊮[wi∈top-⌊rN⌋] is non-differentiable. To enable gradient-based training of the router, we adopt the Straight-Through Estimator (STE). Specifically, during the forward pass, the hard gate mi is used; during the backward pass, gradients are approximated through the soft probability w˜i:(7)mi≈w˜i+(mi−w˜i)stop-grad,∂mi∂s≈∂w˜i∂s. An alternative is Gumbel–Softmax relaxation: w˜i=Softmax((si+gi)/τ) where gi∼Gumbel(0,1) and τ is the annealing temperature during training. Both approaches ensure that the router parameters receive effective gradients throughout training. 2.3. Recursive Window Attention (RWA) The recursive window attention module performs self-attention within local windows, following the shifted-window design of Swin Transformer. For the kth recursion step, the attention is computed as(8)RWAk(Q,K,V)=SoftmaxQK⊤dk+BkV, where Bk is a relative position bias learned separately for each recursion depth. Unlike standard Swin, where the position bias is static, MoR–Swin learns a separate Bk for each recursion depth. At each recursion step, the queries and keys are derived from the refined features of the previous step, and so the attention distribution is computed over progressively more abstract representations; the depth-varying Bk lets the model adapt its positional reasoning accordingly while sharing all other projection parameters. Two consecutive RWA modules (regular and shifted window) form a basic unit, enabling cross-window interaction. Recursion unrolls at the level of this regular + shifted unit: the same shift pattern is applied consistently across all recursion steps, maintaining cross-window connectivity at every depth. The recursive formulation can be unrolled as(9)zk+1=zk+RWA(LN(zk)),k=1,…,Na, where LN denotes layer normalization. The projection weights are shared across all recursion steps, which is the main source of parameter reduction. Concretely, within each Recursive Mixture Block, the Q/K/V projections, the output projection, and the MLP (FFN) weights are fully shared across the Na recursion steps; the only depth-specific parameters are the relative position biases Bk (learned independently for each recursion depth k; see Equation (8)). This is equivalent to replacing Swin’s stack of L independent blocks with a single block reused Na times, directly yielding the ∼50% parameter reduction reported in Section 2.6. Relationship to RNN and iterative optimization. The recursive formulation zk+1=zk+RWA(LN(zk)) bears a structural resemblance to two established frameworks: Relationship to RNN and iterative optimization. The recursive formulation zk+1=zk+RWA(LN(zk)) bears a structural resemblance to two established frameworks: Recurrent Neural Networks (RNNs): RNNs unroll along the temporal/sequential dimension, propagating hidden states across time steps with input-dependent transitions. In contrast, MoR–Swin unrolls along the depth/refinement dimension—the same set of spatial tokens is iteratively refined without temporal sequence semantics. There is no new input at each step; only the representation is progressively sharpened. Recurrent Neural Networks (RNNs): RNNs unroll along the temporal/sequential dimension, propagating hidden states across time steps with input-dependent transitions. In contrast, MoR–Swin unrolls along the depth/refinement dimension—the same set of spatial tokens is iteratively refined without temporal sequence semantics. There is no new input at each step; only the representation is progressively sharpened. Residual iterative optimization/Neural ODE: The update zk+1=zk+fθ(zk) is precisely the forward Euler discretization of the ODE dz/dt=fθ(z), where the shared-parameter function fθ defines a vector field. Unlike standard residual networks or Neural ODEs, which apply a fixed number of steps to all inputs, MoR–Swin introduces data-dependent variable step count Di per token. This adaptive stopping—determined by the router based on semantic complexity—is the key novelty that distinguishes our approach from both fixed-depth residual networks and continuous-depth models. Residual iterative optimization/Neural ODE: The update zk+1=zk+fθ(zk) is precisely the forward Euler discretization of the ODE dz/dt=fθ(z), where the shared-parameter function fθ defines a vector field. Unlike standard residual networks or Neural ODEs, which apply a fixed number of steps to all inputs, MoR–Swin introduces data-dependent variable step count Di per token. This adaptive stopping—determined by the router based on semantic complexity—is the key novelty that distinguishes our approach from both fixed-depth residual networks and continuous-depth models. The theoretical analyses supporting the above are provided in the Supplementary Materials: a boundedness (non-explosion) analysis and a discussion of the contraction caveat under residual scaling in Supplementary Materials, Section S1 (Lemma S1, Theorem S1, Proposition S1); the ODE-based parameter-efficiency and approximation analysis in Supplementary Materials, Sections S2 and S3; and the eigenvalue (spectral) stability analysis in Supplementary Materials, Section S4. Empirical validation of the “semantic complexity” proxy (cross-task Spearman correlations and confound analysis) and all hyperparameter-sensitivity and seed-level results are given in Supplementary Materials, Section S5. 2.4. Dynamic Routing and Load Balancing To avoid token collapse (all tokens routed to the same recursion depth), we introduce a load balancing loss. The total training objective is(10)L=Ltask+λLbalance,Lbalance=∑i=1Na(bi−ti)2, where bi is the actual proportion of tokens undergoing at least i recursions and ti is the target proportion. We adopt the Expert-Choose routing strategy, in which each recursion branch actively selects its top-capacity tokens, achieving balanced load distribution. Gradient flow analysis. Let the router parameters be Wr, with per-token scores si=LinearWr(xi) obtained from each token’s own channel vector xi (no spatial pooling), and routing weights w=Softmax(s). The gradient of the task loss L with respect to Wr is(11)∂L∂Wr=∑i∂L∂wi·wi(δij−wj)·xi⊤, where δij is the Kronecker delta. Since the softmax Jacobian ∂wi/∂sj=wi(δij−wj) is non-zero everywhere, the router parameters Wr remain trainable throughout optimization. For the non-differentiable hard selection, we employ STE or Gumbel–Softmax as described in Section 2.2. Load balance gradient. The gradient of the balance loss with respect to routing probabilities is(12)∂Lbalance∂pk=2∑i=1Na(bi−ti)∂bi∂pk, where bi=1N∑n1[Dn≥i] is softened via the Gumbel–Softmax relaxation to enable differentiation. This gradient pushes overloaded branches (where bi>ti) to reduce their routing probability while pulling underloaded branches upward, thereby reducing the coefficient of variation (CV) of expert utilization. Expert-Choose vs. Token-Choose. Let uk denote the number of tokens assigned to recursion branch k. Under Token-Choose, each token independently selects the highest-scoring branch, and so uk approximates a polynomial distribution—popular branches accumulate disproportionate load, yielding high variance σu2 and large CV. Under Expert-Choose, each branch actively selects its top-capacity tokens (capacity Ccap), enforcing uk≡Ccap in the ideal case. This yields σu→0 and CV→0, explaining the experimentally observed 40% reduction in CV that we report in the ablation experiments. Furthermore, Token-Choose exhibits “expert death” early in training (20% of experts with near-zero gradients), while Expert-Choose maintains expert activity above 95% throughout training. Unified training objective. Combining all loss components, the complete optimization objective is(13)Ltotal=Lcls︸taskloss+λb∑i=1Na(bi−ti)2︸Lbalance+λc∑k1−cos(a(k),a(k+1))︸Lrecursive-consistency, subject to the per-step capacity constraint ∑imi(k)≤Ccap (with Ccap=⌊rN⌋) for each recursion step k. Here, λb and λc are balancing hyperparameters, and the recursive consistency loss Lc encourages attention maps at adjacent recursion steps to stabilize, promoting smooth feature refinement. (We note that an ℓ1 term on the softmax output w would be ineffective, since ∥w∥1≡1 by construction; decisive, low-entropy routing is instead achieved jointly by the top-fraction hard selection and the load-balancing loss.) The load-balancing weight λb is set in a scale-dependent manner:(14)λb=0.01(allstages)forTiny/Smallmodels0.01(stages1–3),0.005(stage4)forBasemodel0.005(allstages)forLargemodels The recursive consistency weight λc=0.05 is fixed across all scales. Larger models use smaller λb because their greater capacity makes them less sensitive to load-balancing regularization. To make the routing procedure fully reproducible, the complete step-by-step pseudocode—specifying how the components (scoring, top-fraction/Expert-Choose selection, STE/Gumbel–Softmax, threshold-based stopping, and recursion) are combined during training and inference—is provided in the Supplementary Material (Algorithm S1, Section S5). 2.5. Memory Optimization Strategy Swin Transformer’s memory bottleneck primarily comes from storing intermediate activations for all tokens across all layers. MoR–Swin implements two techniques:Recursion-level caching: Only the KV pairs of active tokens in the current recursion step are stored in memory, immediately releasing intermediate results for inactive tokens.Selective KV caching: The initial key–value pairs are reused in all subsequent recursion processes: this technique is particularly suitable for memory-constrained deployment environments. Recursion-level caching: Only the KV pairs of active tokens in the current recursion step are stored in memory, immediately releasing intermediate results for inactive tokens. Recursion-level caching: Only the KV pairs of active tokens in the current recursion step are stored in memory, immediately releasing intermediate results for inactive tokens. Selective KV caching: The initial key–value pairs are reused in all subsequent recursion processes: this technique is particularly suitable for memory-constrained deployment environments. Selective KV caching: The initial key–value pairs are reused in all subsequent recursion processes: this technique is particularly suitable for memory-constrained deployment environments. We clarify that these correspond to two operating modes, which are not enabled simultaneously. In the default (accuracy-first) mode used for all main results reported in Section 3, the queries, keys, and values are recomputed at every recursion step from the refined features of the previous step (as described in Section 2.3); recursion-level caching only controls which tokens’ activations are retained, without freezing the KV. In the memory-constrained mode, selective KV caching additionally reuses the first-step KV for later steps, trading a small amount of accuracy for lower memory usage. Table 1 quantifies this trade-off: selective KV caching reduces peak memory by a further ∼15% at a cost of ∼0.3% top-1 accuracy and is therefore optional and disabled in our main experiments. We provide an analytical derivation of the memory reduction. Standard Swin Transformer requires caching intermediate activations for all N tokens across all L layers, yielding memory ∝N·L·d. In MoR–Swin, recursion-level caching stores only the KV pairs of surviving tokens at each step (proportion ρk at step k), and selective KV caching reuses the first-step KV pairs for subsequent recursions, eliminating redundant copies. The resulting activation memory is(15)MMoR=ONd1+∑k≥2ρk, where the eliminated components are the per-step intermediate activations of non-surviving tokens and (only when selective KV caching is enabled) the repeated KV copies. Since ∑k≥2ρk≪Na−1 (most tokens exit early), recursion-level caching alone—the technique used in the default main-experiment configuration—yields about a 25% reduction in memory usage relative to the baseline O(Na·N·d); adding the optional selective KV caching removes the repeated first-step KV copies and brings the total reduction to ∼40%. We therefore report memory separately for the default and memory-optimized configurations (Supplementary Material, Section S9, Table S5) instead of attributing the full ∼40% to the default model. These memory optimization strategies enable MoR–Swin Transformer to handle high-resolution input images while maintaining reasonable memory usage, laying the foundation for the model’s application in real-time vision tasks. 2.6. Computational Complexity Analysis We derive the theoretical computational complexity of MoR–Swin and compare it with standard Swin Transformer to provide a rigorous foundation for the observed efficiency gains. Swin Transformer FLOPs. For a single stage with N tokens, embedding dimension d, and window size M, the standard Swin block complexity is(16)Ω(Swin)=4Nd2︸QKV+proj+2NM2d︸windowattention=O(Nd2+NM2d). MoR–Swin FLOPs. Due to adaptive token-level recursion, only a fraction ρk of tokens survive to recursion step k. The total FLOPs become(17)Ω(MoR--Swin)=∑k=1Naρk4Nd2+2NM2d+Nd︸router(per-tokenLinear). Since ρk is monotonically decreasing (more tokens exit at each step) and ∑kρk≪Na, the total computation is significantly less than Na times the single-step cost. Parameter ratio. A standard Swin stage with L independent blocks has parameters PSwin≈12Ld2 (QKV, output projection, and MLP with 4× expansion). MoR–Swin shares a single block across Na recursion steps, yielding PMoR≈12d2+dNa where the dNa term is the lightweight router head producing the Na depth scores. The parameter ratio is(18)PMoRPSwin≈12d2+dK12Ld2≈1L. For Swin-B with per-stage depth L∈{2,2,18,2}, the weighted average yields approximately 50% total parameter reduction, consistent with Table 2 (88 M → 44 M). Expected FLOPs under routing. Let ρk be the fraction of tokens that survive to (i.e., reach) recursion step k, as defined above, and let F0 be the single-step cost. Since each surviving token incurs one step’s cost, the expected per-token FLOPs are(19)E[FLOPs]=F0∑k=1Naρk=F0·E[D],E[D]=∑k=1Naρk, The theoretical compute-reduction ratio is Ω(Swin)/E[FLOPs]=Na/E[D], which, for the classification setting (Na=3, E[D]≈1.6), gives ≈1.9×, consistent with the measured wall-clock speedup (653→1256 FPS, i.e., 1.92×) reported in Table 2. Latency bounds. The adaptive routing produces variable computation graphs with the following latency bounds (in units of single-block computation F0):Worst case (all tokens recurse Na times): Tmax=NaF0;Best case (all tokens exit at step 1): Tmin=F0;Expected case: E[T]=∑k=1NaρkF0=E[D]·F0. Worst case (all tokens recurse Na times): Tmax=NaF0; Worst case (all tokens recurse Na times): Tmax=NaF0; Best case (all tokens exit at step 1): Tmin=F0; Best case (all tokens exit at step 1): Tmin=F0; Expected case: E[T]=∑k=1NaρkF0=E[D]·F0. Expected case: E[T]=∑k=1NaρkF0=E[D]·F0. Asymptotic scaling with resolution. For input resolution H×W (yielding N=HW/P2 tokens), Swin Transformer has total complexity O(L·HW) with fixed depth L. In MoR–Swin, as resolution increases, the proportion of simple background tokens grows, leading to higher early-exit rates and lesser effective depth E[D]. The asymptotic complexity is(20)O(E[D]·HW),whereE[D]=∑k=1Naρk. Since E[D] does not grow with HW (and empirically decreases for higher resolutions due to the increased background proportion), MoR–Swin exhibits more favorable scaling than standard hierarchical transformers at high resolutions. 3. Experiments 3.1. Experimental Setup Experiments are on ImageNet-1K/22K image classification, COCO object detection, and ADE20K semantic segmentation. Four model scales (Tiny, Small, Base, and Large) are compared with Swin Transformer and Swin–MoE [18] under identical settings (PyTorch 2.1, eight ×NVIDIA A100-80G GPUs). To ensure the reproducibility of the reported FPS and speedup figures, all inference speed measurements follow a fixed protocol: a single NVIDIA A100-80G GPU, PyTorch 2.1 (native, without TensorRT), FP16 mixed precision, batch size 64, and input resolution 224 × 224 for classification (512 × 512 for segmentation), with 50 warmup iterations followed by 200 timed iterations averaged. We report throughput (FPS), per-image latency (ms), and average GPU utilization. The lightweight router (per-token linear) accounts for less than 1% of total FLOPs and is included in all timings. All codes and models will be released. The complete set of training and routing hyperparameters (optimizer, learning rate schedule, epochs, batch size, augmentation, random seeds, selection fraction r, Na, λb, target proportions ti, threshold θ, and detection/segmentation protocols) is provided in Table S1 of the Supplementary Material for full reproducibility. 3.2. Image Classification Results Table 2 shows ImageNet-1K/22K results. MoR–Swin Transformer achieves excellent performance across all scales. In the Tiny and Small configurations, it matches the original Swin Transformer’s accuracy with 50% fewer parameters. For inference speed, it achieves up to twofold speedup over the original Swin and MoE variants, thanks to MoR’s adaptive computation that enables early exiting for simple inputs, reducing redundant computation. Note that for Swin–MoE, we report two distinct metrics taken from the official ModelHub: the accuracy on the ImageNet-22K classification task itself (“IN-22K cls@1”, 22,000 classes) and the accuracy after transferring to ImageNet-1K (“IN-22K→1K”). The former is naturally lower (∼37–39%) due to the far larger label space and must not be compared directly with the 1K-level accuracies of the other models; see the table note. 3.3. Object Detection and Instance Segmentation Results On COCO 2017 detection and instance segmentation with Mask R-CNN (Table 3), MoR–Swin Transformer achieves precision similar to that of Swin Transformer with 50% fewer parameters. Its inference speed advantage is more pronounced in detection, as natural images contain many simple background regions assigned shallower adaptive recursion depths, significantly reducing computation. 3.4. Semantic Segmentation On the ADE20K semantic segmentation task, we evaluated the performance of MoR–Swin Transformer at different scales using UPerNet as the segmentation framework. The results in Table 4 show that MoR–Swin Transformer significantly reduces model complexity and inference time while maintaining segmentation accuracy. Figure 2 presents qualitative results on the downstream tasks, comparing MoR–Swin-B against the Swin-B baseline. Panel (a) shows COCO detection/instance segmentation, and panel (b) shows ADE20K semantic segmentation (rendered with the official 150-class colormap). We verified and corrected the ADE20K scene labels, which were mismatched in the previous version: the three columns are a street scene, a bedroom (indoor), and a living room, respectively. Each panel includes representative successful cases as well as a challenging/failure case (red box). The specific errors are as follows: for COCO, in a dense, heavily occluded crowd both models miss two small partially-occluded persons and merge two adjacent instance masks; for ADE20K, at the ambiguous sofa/armchair boundary in the living room both models mislabel a block of sofa pixels as “chair” and confuse “rug” with “floor”. Both MoR–Swin and Swin exhibit these same errors, which suggests the errors are inherited from the shared backbone family rather than introduced by the recursion mechanism. We emphasize, however, that a handful of selected examples can only illustrate model behavior; they are not sufficient to establish the absence of task-specific degradation—that judgment rests on the quantitative mAP/mIoU results (Table 3 and Table 4). 3.5. Computational Efficiency Analysis As shown in Figure 3, MoR–Swin reduces FLOPs by about 45% for classification (Swin-B 15.4→8.2 G, consistent with the compute-reduction ratio Na/E[D]=3/1.6≈1.9×) and by 35–40% at 512×512 resolution for dense tasks (where Na=5 and more tokens recurse deeply) compared to Swin. For memory, we distinguish the two caching configurations: the default main-experiment configuration (recursion-level caching only, selective KV caching disabled) reduces peak memory by ∼25%, while the optional memory-optimized configuration (additionally enabling selective KV caching) reaches ∼40%; the two are reported separately in Table 5 and, with measured GPU footprints, in Supplementary Materials Section S9. The adaptive recursion mechanism is the main contributor, which makes the model more practical in resource-constrained environments. 3.6. Ablation Studies To systematically evaluate the effectiveness and relative importance of each innovative component in the proposed model, we designed and conducted a series of controlled ablation experiments to quantify the contribution of different components to the overall performance. Swin Transformer Base (Swin-B) was selected as the baseline model. All experiments used the ImageNet-1K dataset with standard preprocessing and data augmentation strategies, strictly reusing the same training configuration. Quantitative results of the ablation studies are shown in Table 6. To clarify the ablation protocol, “w/o Recursive Mechanism” (Abl-A) replaces the shared recursive block with a single standard Swin block (no weight sharing, no iterative refinement); “w/o Recursive Window Attention” (Abl-B) replaces RWA with standard window attention (static position bias, no depth-varying Bk); “w/o Dynamic Routing” (Abl-C) replaces the learned router with uniform random routing (all tokens assigned equal recursion probability); “w/o Memory Optimization” (Abl-D) disables the recursion-level caching used by the default model (which stores full activations at every step; note that selective KV caching is already disabled in the default configuration, and so it is not a factor here). We acknowledge that Abl-A, Abl-B, and Abl-D each vary more than one factor at once. To isolate individual contributions, Table 7 reports single-factor controlled ablations: (i) recursion and weight sharing are decoupled by a “shared weights, single forward pass” variant (isolating sharing) and an “independent weights, iterative refinement” variant (isolating iteration); (ii) attention form and depth-specific bias are decoupled by “RWA structure + static B” and “standard attention + depth-specific Bk”; (iii) the two caching mechanisms are separated and, crucially, measured at a fixed effective batch size (using gradient accumulation where a configuration cannot fit), so that the effects of memory mechanisms are not confounded with those of batch size. These controlled results show that the accuracy contribution of the recursion mechanism stems primarily from iterative refinement (−0.6%) rather than weight sharing per se (−0.1%; sharing mainly saves parameters) and that, at a fixed batch size the two caching mechanisms have negligible accuracy impact (≤0.05%)—confirming that their role is memory reduction, not accuracy. The earlier Abl-D drop was therefore largely attributable to the batch-size reduction, not the caching itself. We base the following discussion on the controlled single-factor ablations of Table 7 and deliberately avoid the earlier framing that attributed fixed shares of a 1.5-point “total gain” to individual components: as reported in Table 2, MoR–Swin-B is in fact 0.5 points below Swin-B (an accuracy trade-off accepted for a 50% parameter reduction and up to twofold speedup), not a net accuracy gain, and so a share-of-gain decomposition is not meaningful. The experimental results indicate the following: (1) The accuracy effect of the recursion mechanism stems primarily from iterative refinement (−0.6% when isolated, Table 7) rather than weight sharing per se (−0.1%; weight sharing mainly saves parameters). This mechanism effectively models long-range dependencies through recursive cross-layer token-state updates, compensating for the locality limitation of standard Transformers; its removal reduces the model’s ability to capture fine-grained features (Figure 4a). (2) The recursive window attention contributes next, isolating the depth-specific bias costs −0.2% and isolating the attention form costs −0.15% (Table 7); recursive feedback within windows enhances the consistency of local features (Figure 4b). (3) The dynamic routing strategy is important for load balance: random routing causes severe expert load imbalance (CV ↑158%), with some experts overloaded (processing >2.1× the average number of tokens) while 30% of experts have utilization <40%, wasting computation. (4) The memory-optimization techniques have negligible accuracy impact at a fixed effective batch size (≤0.05%, Table 7); their role is memory reduction, not accuracy. The larger drop seen in the earlier Abl-D was attributable to the batch-size reduction (to 512, forced by OOM without caching) and the resulting unstable optimization, not to the caching itself. This study validates the rationale behind the key design choices and informs subsequent lightweight design and architecture migration. Furthermore, we compared the effects of different routing strategies (as shown in Table 8), finding that the Expert-Choice strategy achieves higher accuracy than the Token-Choice strategy. The Expert-Choice strategy reduces the Expert CV by 40%, indicating that it achieves more balanced load distribution through an active expert competition mechanism (as illustrated in Figure 5a) (rather than passive token assignment). The Token-Choice strategy exhibits expert ‘death’ early in training (20% of experts have gradients close to 0), whereas the Expert-Choice strategy maintains expert activity above 95% throughout training (as illustrated in Figure 5b). This fully demonstrates that the Expert-Choice strategy outperforms the Token-Choice strategy in both load balancing and model performance. 3.7. Attention Consistency Analysis To quantitatively evaluate the stabilization behavior of recursive attention, we measure the similarity between attention maps at adjacent recursion depths. For attention distributions a(k) and a(k+1) at recursion steps k and k+1, we compute the Jensen–Shannon Divergence (JSD):(21)JSD(a(k)∥a(k+1))=12KL(a(k)∥a¯)+12KL(a(k+1)∥a¯),a¯=12(a(k)+a(k+1)). We observe a monotonic decrease in JSD across recursion depths on ImageNet validation images: the average JSD drops from 0.142 (between steps 1 and 2) to 0.058 (between steps 2 and 3) for MoR–Swin-B. This indicates that the recursive attention progressively stabilizes with depth; we note that a decreasing JSD is empirical evidence of stabilization, not a proof of convergence to a unique fixed point. The cosine similarity cos(a(k),a(k+1)) correspondingly increases from 0.81 to 0.94, consistent with this stabilization. 3.8. Comparison with Adaptive Computation Methods To position MoR–Swin within the broader landscape of efficient vision transformers, we compare it with recent adaptive computation methods in Table 9. These methods differ fundamentally in their dimension of adaptation: token pruning methods (DynamicViT, EViT, A-ViT) reduce the spatial token count N, token merging (ToMe) compresses tokens via bipartite matching, and Sparse MoE expands model width via expert routing. In contrast, MoR–Swin adapts along the depth dimension with shared parameters, preserving all token information while varying computational depth per token. The key distinction is that token pruning methods irreversibly discard information (masking out tokens), whereas MoR–Swin retains all tokens and only varies the number of refinement iterations. Sparse MoE uses independent expert parameters Ek (parameter count scales with expert number nE), while MoR–Swin shares parameters θ across all recursion depths, achieving O(1) parameter scaling regardless of maximum depth Na. What is inherited vs. newly introduced. To position our contribution precisely, we distinguish components inherited from the original MoR framework [13] from those newly introduced for the Swin Transformer. Inherited from MoR: (i) the general idea of token-level recursion; (ii) dynamic routing; (iii) the load-balancing objective; and (iv) the Expert-Choose routing principle. Newly introduced for Swin: (i) the Recursive Window Attention (RWA) module that combines shifted-window attention with recursion and adds a depth-specific relative position bias Bk; (ii) the per-stage router adapted to the hierarchical pyramid (patch-merging stages); and (iii) the two memory-caching strategies tailored to windowed attention. To our knowledge, this is the first integration of MoR into a hierarchical window-attention backbone. Quantitative comparison.Table 10 compares MoR–Swin-B with A-ViT and DynamicViT under the settings each method reports (all ImageNet-1K, 224 × 224, no extra data). We caution that these are not fully protocol-matched: A-ViT and DynamicViT use a DeiT/ViT backbone, whereas MoR–Swin uses a hierarchical Swin backbone, and their published FLOPs/throughput were measured on different hardware; the A-ViT and DynamicViT figures are taken from the respective papers (not reproduced by us), while the MoR–Swin figures follow our unified timing protocol (Section 3.1). The comparison should therefore be read as indicative of the design trade-off (depth recursion retaining all tokens vs. spatial token pruning), not as a controlled head-to-head benchmark. The FLOPs are not directly comparable because the backbones differ (Swin-B has higher base FLOPs than DeiT-B); MoR–Swin-B’s absolute FLOPs (8.2 G) are accordingly higher than the DeiT-B-based baselines, yet it attains comparable accuracy with roughly half the parameter count (44 M vs. 86–87 M) and higher throughput while—unlike the token-pruning baselines—retaining all tokens, which is advantageous for dense downstream tasks. 3.9. Statistical Analysis of Ablation Results To rigorously assess component interactions and statistical significance, we extend the ablation study with a 23 full factorial design over three binary factors: (A) adaptive recursion mechanism, (B) recursive window attention, and (C) dynamic routing (“+” = present, “−” = removed/replaced as in the ablation protocol). The four single-factor-removal cells coincide, by construction, with the main ablation table—(+++)=83.0 (Full), (−++)=82.0 (Abl-A), (+−+)=82.7 (Abl-B), (++−)=82.8 (Abl-C)—and so the factorial is fully consistent with Table 6. Each of the eight cells is run with n=3 seeds (42/123/2024); the eight cell means and the fitted coefficients are reported in Table 11. Because the design has three factors, the correct model is a three-way ANOVA (we correct our earlier mislabeling as “two-way”). It reveals significant main effects for all three factors—A (recursion) is the dominant term (F1,16=602, p<10−4), followed by B (F1,16=54, p<10−4) and C (F1,16=24, p<10−3)—while none of the two-way interactions or the three-way A × B × C term reaches significance (all |coeff|≤0.02, p>0.4). In other words, within measurement noise, the three components contribute additively; we do not claim a synergistic interaction (correcting our earlier, unsupported A×C synergy statement). For statistical reliability, we report the exact number of runs, the test used, and the effect sizes alongside p-values. Number of runs: main classification results (MoR–Swin-T/S/B and same-scale Swin baselines) and all ablation configurations use n=3 independent seeds (42, 123, 2024); detection/segmentation use n=3 for Tiny/Small and n=2 for Base, with the exact n annotated in each table note. Test per table: Table 2 (classification, same-scale MoR–Swin vs. Swin pairs) uses the paired t-test; the ablation table uses the Wilcoxon signed-rank test (small samples, no normality assumption); the 23 factorial design uses a three-way ANOVA (Table 11). Pairing definition: for the paired tests, a “pair” is defined by matched random seeds across models—MoR–Swin and Swin trained with the same seed (identical data ordering and initialization stream) form one pair, and differences are taken pairwise. Effect sizes, degrees of freedom, and CIs: beyond p-values, we report the degrees of freedom, Cohen’s d values, and 95% confidence intervals, and we give the seed-level results in the Supplementary Materials (Section S5, Table S2). For MoR–Swin-B vs. Swin-B, the three matched-seed accuracy pairs are (seed 42: 83.1 vs. 83.6), (seed 123: 82.9 vs. 83.3), and (seed 2024: 83.0 vs. 83.6), giving paired differences of {−0.5,−0.4,−0.6} with mean −0.5% and SD 0.1%. With n=3 matched pairs, the degrees of freedom are df=n−1=2; the paired t-statistic is t=d¯/(sd/n)=−0.5/(0.1/3)≈−8.7, two-sided p≈0.013 (Cohen’s dz=d¯/sd≈5.0, 95% CI of the mean difference =[−0.75,−0.25]). We note that our earlier report omitted the df and the seed-level values; these are now provided (see Supplementary Materials, Table S2) so that the statistic can be reproduced. We stress that our central claim is not higher accuracy but comparable accuracy (a small, well-characterized difference) at 50% fewer parameters and up to twofold speedup. Because the two-sided 95% CI of the MoR–Swin-B − Swin-B difference lies entirely below zero ([−0.75,−0.25]), a standard paired t-test would (correctly) reject exact equality; equality is not, however, the appropriate hypothesis for our claim. We therefore adopt a prespecified non-inferiority test. We set the non-inferiority margin to δ=0.5% top-one accuracy in advance—a value chosen because it is (a) below the run-to-run standard deviation band we observe across seeds and (b) a commonly accepted tolerance for ImageNet-1K accuracy when trading against a 50% parameter reduction; we justify δ before seeing the test outcome. The non-inferiority null hypothesis is H0: μMoR−μSwin≤−δ against H1: μMoR−μSwin>−δ, tested with a one-sided paired t-test at α=0.05. With d¯=−0.5, sd = 0.1, n = 3 (standard error SE=sd/n=0.058), the test statistic is t=(d¯+δ)/SE=(−0.5+0.5)/0.058=0.0, df=2, one-sided p=0.5, and so H0 cannot be rejected. Consistently, the one-sided 95% lower confidence bound for the mean difference is d¯−t0.95,2·SE=−0.5−2.92×0.058=−0.67, which lies below−δ=−0.5 and therefore does not clear the non-inferiority margin. Both the test and the confidence bound thus agree: we do not claim non-inferiority at δ=0.5% from this small sample; the point estimate sits exactly at the margin, while the lower bound falls short of it. To characterize this honestly rather than overclaim, we state the result as follows: MoR–Swin-B is 0.5±0.1 points below Swin-B (a small, precisely estimated deficit) while using 50% fewer parameters and running up to two times faster. Additionally, we report the margin, test, df, and CIs so that readers can judge the practical trade-off. A definitively powered non-inferiority conclusion would require a larger number of seeds, which we note as a limitation. 4. Conclusions This letter proposed MoR–Swin Transformer, an optimized model for Swin Transformer based on the Mixture of Recursions architecture, which dynamically adjusts the processing depth for tokens of varying semantic complexity by introducing an adaptive token-level recursion mechanism. Experimental results demonstrate that compared to the original Swin Transformer and its MoE variant, MoR–Swin Transformer achieves superior parameter efficiency and computational efficiency on tasks including ImageNet image classification, COCO object detection, and ADE20K semantic segmentation. At a modest accuracy cost (within about 0.5 points of Swin-B on ImageNet-1K), it reduces parameters by approximately 50% and increases inference speed up to twofold. The success of MoR–Swin Transformer validates the effectiveness of the adaptive recursive architecture in vision tasks, providing a new technical pathway for optimizing Vision Transformer models. Furthermore, the theoretical analysis presented in this work—including computational complexity derivations, a finite-depth norm-boundedness (non-explosion) guarantee, gradient flow analysis, and asymptotic scaling properties—provides a mathematical basis for understanding the efficiency of adaptive recursive computation in vision architectures. We are careful to delimit these claims: we establish finite-depth boundedness and report empirical stabilization of the attention maps with depth, but we do not claim strict contraction or convergence to a unique fixed point (see Supplementary Materials, Sections S1 and S4). In the future, we will continue to explore the application of the MoR architecture in other vision tasks and further optimize the routing strategy and recursion mechanism to enhance the model’s applicability in diverse scenarios.