Geometrically Constrained and Token-Based Probabilistic Spatial Transformers
Johann Schmidt, Tom Siegl, Martin Becker, Sebastian Stober · 2025
Read Geometrically Constrained and Token-Based Probabilistic Spatial Transformers by Johann Schmidt, Tom Siegl, Martin Becker, Sebastian Stober (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-11218.txt.
Introduction
Johann Schmidt and 3 co-authors rebuild the spatial transformer idea on modern foundations in this 2025 paper. Rotated or scaled objects hide the shape cues a classifier needs. A model for high-stakes use should resist such changes, explain any correction it makes, and admit when an input is ambiguous. Strictly symmetric architectures guarantee some of this but limit flexibility and cost heavily.
Their framework uses a transformer to estimate the affine change applied to an input image. The change is broken into readable parts and fitted under adjustable geometric limits, which avoids the degenerate warps that used to destabilize training. Sharing weights with the classifier keeps the cost low. Tests on insect and medical images beat rivals under varied transformations.
Catalog record
| CREATOR | Johann Schmidt, Tom Siegl, Martin Becker, Sebastian Stober |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 57 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-11218
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Geometrically Constrained and Token-Based Probabilistic Spatial Transformers Johann Schmidt, Tom Siegl, Martin Becker, Sebastian Stober Abstract Spatial transformations such as rotation and scale obscure the morphological cues needed for accurate image classification. Careful consideration is required for reliable use in high-stakes settings. A model should stay robust under such transformations, expose why a correction was applied, and signal when its input is ambiguous. While geometrically equivariant architectures provide a mathematically grounded solution, they often limit model flexibility through strict symmetry constraints and incur significant computational overhead. Spatial Transformer Networks (STNs) offer a data-driven, flexible alternative for learning pseudo-equivariances to affine transformations. However, STNs have historically been restricted to convolutional architectures and suffer from training instability. To address this, we introduce a novel STN framework. It leverages the global modeling capabilities of transformers to regress the affine transformation acting on the input. For this, we decompose affine transformations into interpretable primitives, regressed under adaptable geometric constraints, thereby preventing the training instability typically caused by degenerate transformations. By sharing weights between the localization network and the classification backbone, the framework requires minimal computational overhead. Extensive experiments on challenging insect biodiversity and medical imaging benchmarks demonstrate that our approach achieves superior predictive performance under diverse spatial transformations while maintaining high efficiency. Code is available at https://github.com/johSchm/TokenSTN . Figure 1 : Our method integrates a Spatial Transformer module directly into the Vision Transformer pipeline. Unlike traditional STNs, we regress geometric primitives (e.g., rotation, scale) individually under adaptable constraints to ensure training stability and prevent degenerate transformations. By sharing weights between the localization network and the classification backbone, the framework eliminates feature redundancy and maintains high computational efficiency. 1 Introduction Vision Models are Brittle Deep vision models exhibit notorious brittleness when facing spatial variations [ 13 ] . Because distinct spatial transformations of a single object project onto vastly different points in the pixel manifold, the resulting signals can disrupt prediction consistency. As illustrated in Fig. 1 , while both images depict the same moth, the right, rectified version presents a canonical pose that is intuitively easier to label. Unlike humans, who naturally prefer such rectified views, neural networks must infer this preference entirely from their training distribution. Consequently, classifiers that view these transformed versions as distinct inputs often fail to learn consistent label functions, necessitating explicit mechanisms to ensure robustness against spatial deformation [ 13 ] . Limitations of Data Augmentation and Invariant Models A theoretically grounded solution is the use of inductive biases to render the classifier inherently invariant—ensuring the model predicts the same class regardless of the input’s transformation. Convolutional Neural Networks gain translation invariance through spatial pooling [ 54 ] , but remain highly sensitive to rotation and scale. To achieve broader invariance, filters can be lifted to a filter bank over which pooling is applied [ 6 ] , or the predictions of any model over all (or many) transformed inputs can be pooled [ 41 ] . However, strictly invariant methods often trade flexibility for efficiency. They face significant limitations regarding computational cost, increased memory footprints, and the difficulty of engineering constraints for complex transformation groups [ 5 ] . Distinct from strict invariance, pseudo-invariant methods achieve robustness in a data-driven fashion, a paradigm that has driven recent breakthroughs in computer vision [ 51 ] . The simplest solution is to augment the classifier’s input with transformed input samples. However, this forces the classifier to either learn replicated features for every transformation [ 40 ] or learn “fuzzy” averages that obscure detail [ 3 , 7 ] . Spatial Transformer Networks A more efficient data-driven alternative is the Spatial Transformer Network (STN) [ 24 ] , which decomposes geometric variance handling (localization network) from the classification task (downstream classifier). Acting as differentiable rectifiers, they employ a localization network to regress affine transformation parameters to minimize the downstream classification objective. This provides a mechanism for learning input-dependent rectifications without introducing the architectural rigidity of invariant models, allowing the downstream classifier to focus purely on the rectified features. While there has been a long line of further developments [ 31 , 48 , 47 ] , some open problems still prevail. STNs are notoriously difficult to train due to training instability and computational redundancy. The standard approach involves regressing an entire affine matrix [ 24 ] . For training STNs without explicit constraints, the optimization landscape is riddled with degenerate solutions, such as collapsing the image into a line. As a result, training becomes unstable because degenerate solutions eliminate semantic information, leaving the classifier to operate on effectively meaningless inputs. One solution is to strictly enforce invertibility [ 10 ] , which increases computational cost and complicates training due to additional integration steps and regularization terms. Another is to limit the regression to specific subgroups like rotation [ 52 , 44 ] , which are naturally invertible. This, however, sacrifices the geometric flexibility required for different applications. Regressing affine transformations depends on the classifier’s inferred preference from the training distribution [ 46 ] and the object class itself. This can lead to computational redundancy, as effective rectification requires semantic understanding, forcing the localization network to duplicate the feature extraction effort of the downstream classifier. As a solution, [ 15 ] proposed different weight-sharing schemes between networks. These approaches are limited to Convolutional Neural Networks, while transformers (dominating computer vision) have not been used to the best of our knowledge. A final issue is the regression of point estimates for transformation parameters, which causes brittle predictions in noisy scenes (common in natural images). To address this, [ 47 ] treats transformation parameters as latent variables with distributions rather than point estimates. This allows estimating the confidence of a prediction and thus identifying uncertainty. However, the solution requires a hierarchical prior, which induces a significant engineering effort. Contributions In summary, due to their flexibility STNs have the potential to vastly outperform more restricted specializations. However, current solutions to make this feasible strongly restrict this flexibility, impact trainability, are computationally expensive, or require substantial engineering effort. To unlock the full potential of data-driven invariance, we propose a modernized STN framework that reconciles geometric flexibility with training stability 1 1 1 The stability provided by our bounded regression should be understood with respect to the transformed and resampled signal itself, rather than the gradient directly. Specifically, our bounding strategy prevents degenerate transformation extremes (such as excessive zoom-out that renders the target object invisible) while avoiding the substantial computational overhead associated with diffeomorphic integration [ 10 ] . and computational efficiency. Our contributions are: • To prevent the degenerate solutions common in unconstrained STNs without the computational cost of diffeomorphic integration, we introduce a bounded geometric decomposition. By regressing individual primitives (e.g., rotation, scale) under lightweight constraints, we ensure invertible and interpretable transformations. • We extend the STN paradigm to transformers through a weight-sharing scheme between the localization network and the classifier, eliminating architectural redundancy and lifting spatial rectification to state-of-the-art vision architectures. • To improve robustness for noisy scenes, we simplify the variational objective of [ 47 ] by employing independent Gaussian posteriors for each transformation primitive, achieving competitive results. Together, robustness and geometric interpretability, achieved at minimal cost, target central requirements of trustworthy vision