Zero-Shot Test-Time Canonicalization using Out-of-Distribution Scoring
Dominik Lindner (✉), Johann Schmidt, Tom Siegl, Martin Becker, Sebastian Stober · 2026
Read Zero-Shot Test-Time Canonicalization using Out-of-Distribution Scoring by Dominik Lindner (✉), Johann Schmidt, Tom Siegl, Martin Becker, Sebastian Stober (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-24178.txt.
Introduction
Lindner and colleagues fix tilted, stretched inputs for pretrained vision models without retraining anything. Classifiers often fail on rotated, scaled, or sheared images even though the object's class is unchanged. Test-time canonicalization instead straightens each input back toward the training distribution before classification, leaving the classifier untouched. Previous canonicalizers leaned on a narrow family of logit-based energy scores and handcrafted searches.
Reframing canonicalization as out-of-distribution detection lets any such score serve as the energy to minimise over transformations. Lindner's team systematically evaluates around twenty scores and nine search algorithms across handwritten characters, sketches, natural images, and 3D point clouds. Distance-based scores paired with random search plus local refinement perform best overall. Because straightening an already-aligned input can hurt, a gate transforms inputs only when needed. The 2026 sweep maps a design space earlier work left unexplored, and names a dependable default recipe.
Catalog record
| CREATOR | Dominik Lindner (✉), Johann Schmidt, Tom Siegl, Martin Becker, Sebastian Stober |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 83 KB |
| LICENSE | http://creativecommons.org/licenses/by-sa/4.0/ |
| RIGHTS | CC BY SA |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.24178
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Zero-Shot Test-Time Canonicalization using Out-of-Distribution Scoring Dominik Lindner (✉), Johann Schmidt, Tom Siegl, Martin Becker, Sebastian Stober Abstract Pretrained vision models often misclassify inputs that are rotated, scaled, or sheared, even though these affine transformations leave the object class unchanged. Robustness is usually restored either by building equivariance into the architecture or by retraining with augmentation, both of which require changing or retraining the model. Test-time canonicalization instead leaves the classifier untouched. It undoes the transformation of each input, mapping it to a canonical form near the training distribution before classification. Existing canonicalizers, however, rely on a narrow set of logit-based energy scores and bespoke search procedures, leaving the design space of scoring functions and optimizers unexplored. We reframe canonicalization as out-of-distribution (OOD) detection, which lets any OOD score serve as the energy minimized over transformations. Across benchmarks ranging from handwritten characters and sketches to natural images and 3D point clouds, we systematically evaluate around twenty OOD scores and nine search algorithms, finding that distance-based scores paired with random search and local refinement perform best overall. Because canonicalizing an already-aligned input can hurt accuracy, we add a gated mechanism that transforms an input only when its OOD score indicates this is needed, preserving most in-distribution accuracy while retaining the robustness gains on transformed inputs. Code is available at github.com/johschm/its . Figure 1 : During test-time in-distribution (ID) samples are passed directly to the downstream model, while out-of-distribution (OOD) samples are passed to a zero-shot canonicalization module. This module uses a search heuristic to minimize the OOD score of the sample by applying affine transformations. Only if this decreases the OOD score the result is passed to the downstream model. 1 Introduction For a model to be reliable, its predictions should remain stable under class-preserving transformations of the input. A rotated, rescaled, or sheared image of an object still depicts the same object. Such transformations are common at deployment time, for instance in microscopy, aerial imagery, or sketch and document analysis, where the upright pose seen during training cannot be assumed. Yet many vision models are not robust to them [ 17 , 62 ] , and retraining a deployed model for every such setting is often impractical. A strong form of robustness is invariance, where the predicted class remains unchanged under input transformations [ 27 ] . The most widely used method to increase robustness without altering the model architecture is data augmentation , which adds semantically redundant, transformed samples to the training set to encourage learning robust filters [ 25 ] . Invariance can be guaranteed by equivariant models [ 7 ] using group-pooling [ 8 ] , which constrain the architecture through explicit inductive biases. This, unfortunately, is very constraining, as such an inductive bias has to be designed for the specific transformation and the model has to be trained from scratch. The downstream prediction can be made invariant without such constraints by averaging over all transformations [ 59 ] . Subsequent works reduce this to an average over a subspaces [ 43 ] , but the computational cost remains prohibitive. This is closely related to Test-time Augmentation (TTA), where predictions are also averaged over transformed inputs. However, they assume a heuristic distribution over transformations and do not guarantee invariance [ 30 ] . An alternative approach to spatial robustness is canonicalization , which seeks to transform inputs into a canonical form by “undoing” group transformations prior to inference. Crucially, canonicalization does not constrain the downstream architecture, which can therefore be treated as fixed.Early canonicalizers learned to regress transformation parameters directly from data [ 26 ] . However, this can lead to fragile predictions even with modern backbones [ 48 ] . Later approaches formalized canonicalization as the selection of a canonical representation for an input sample by minimizing an equivariant scoring function over group transformations [ 29 , 41 ] . These methods typically rely on equivariant features or architectures for the canonicalizer [ 8 ] to ensure consistent scoring across transformed inputs. More recent work relaxes the requirement of equivariant canonicalizers by introducing training procedures for non-equivariant networks [ 42 ] , but such approaches still require substantial training effort. Many learning-based canonicalizers further assume or benefit from training data that exhibits consistent alignment or a canonical pose distribution [ 41 ] . This assumption is often violated in real-world datasets, where object pose may vary significantly. Recent work addresses this by progressively re-aligning the training data during learning [ 51 ] , but at the cost of additional training complexity. This raises a practical question: can canonicalization be obtained without training at all? Recent methods therefore formulate canonicalization as a test-time optimization problem, where a transformation is selected by minimizing a score defined directly on the downstream classifier’s outputs [ 50 ] . However, existing canonicalization methods are limited by their narrow choice of energy functions, reliance on bespoke and costly search procedures, and a tendency to degrade in-distribution (ID) accuracy when inputs are already aligned. Existing approaches employ logit-based energies [ 50 ] or heuristic criteria [ 52 ] that favor transformations producing confident ID predictions. We propose the hypothesis that OOD scores constitute suitable energy functions for canonicalization. The underlying assumption is that a canonical form lies in a high-density region of the training distribution and therefore attains a low OOD score, while transformed inputs move off this manifold and score higher. Minimizing an OOD score over the transformation group should thus recover a canonical form. This is a necessary rather than a sufficient condition. A low score favors transformations that appear typical, which need not coincide with the correct semantic pose, especially for classes with inherent pose ambiguity. We therefore treat it as an empirical hypothesis and test it across a broad range of scores and datasets. If it holds, the extensive body of OOD detection literature can be leveraged directly, broadening the design space for canonicalization. Moreover, the optimization strategy used to search over transformations is equally important. Prior work relies on bespoke search procedures tailored to specific scoring functions [ 50 , 53 ] , including Lie-algebra-based methods for continuous transformation groups [ 52 , 30 ] . While effective, these approaches introduce additional complexity. Based on prior work, it remains unclear whether simpler global optimization strategies, such as simulated annealing or random search, which have shown benefits in other domains [ 47 ] , can achieve competitive performance for common transformations such as affine transformations. Another issue of current canonicalization frameworks is the degradation of ID accuracy when inputs are already aligned [ 50 ] , as incorrect or unnecessary transformations can distort features. This presents a practical challenge, since most real-world inputs are typically close to the training distribution and only a minority exhibit large transformations. Our main contributions to these problems are: 1. We systematically benchmark OOD scores as energy functions for canonicalization, addressing the limited exploration of energy functions in prior work. We find that the best-performing score varied across datasets. 2. We show that prior search procedures can be replaced by simpler global optimizers (e.g., random search, simulated annealing) without sacrificing performance, with random search with local refinements working best overall. 3. We introduce a