On the rankability of visual embeddings
Ankit Sonthalia, Arnas Uselis, Seong Joon Oh · 2025
Read On the rankability of visual embeddings by Ankit Sonthalia, Arnas Uselis, Seong Joon Oh (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-03683.txt.
Introduction
Sonthalia, Uselis, and Oh ask whether image embeddings line up neatly with graded traits, in a 2025 machine learning paper. An embedding turns an image into a list of numbers. The authors call a model rankable for a trait if a straight line through that space orders images by the trait, be it age, crowd size, or head pose. They test seven popular encoders on nine datasets.
Many embeddings turned out to be rankable already, with no extra training. Even two extreme examples, the youngest and oldest faces, say, often sufficed to find the right direction. Full labeled datasets were rarely needed. The finding opens cheap ways to rank images inside vector databases. It also raises fresh questions about how such tidy structure arises during training.
Catalog record
| CREATOR | Ankit Sonthalia, Arnas Uselis, Seong Joon Oh |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 48 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-03683
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
On the rankability of visual embeddings Ankit Sonthalia, Arnas Uselis, Seong Joon Oh Abstract We study whether visual embedding models capture continuous, ordinal attributes along linear directions, which we term rank axes . We define a model as rankable for an attribute if projecting embeddings onto such an axis preserves the attribute’s order. Across 7 popular encoders and 9 datasets with attributes like age, crowd count, head pose, aesthetics, and recency, we find that many embeddings are inherently rankable. Surprisingly, a small number of samples, or even just two extreme examples, often suffice to recover meaningful rank axes, without full-scale supervision. These findings open up new use cases for image ranking in vector databases and motivate further study into the structure and learning of rankable embeddings. Our code is available at https://github.com/aktsonthalia/rankable-vision-embeddings . 1 Introduction Visual embeddings are widely used for image retrieval. This relies on embeddings forming a metric space, where similar images are placed nearby. Modern visual encoders generally satisfy this property, and many systems depend on it in the form of vector databases. Ranking is another core operation in databases. It allows users to navigate large collections by sorting, paginating, and filtering results. For instance, e-commerce platforms like Amazon benefit from ranking product images by visual quality or certain product-specific attributes ( eg. ranking shoes by how formal they look). In this work, we ask: are visual embeddings also rankable? Retrieval relies on local similarity around a query. Ranking requires a global ordering along an attribute. Prior work has largely addressed the former. The global rankability of embeddings remains underexplored. We define rankability as follows: given an embedding function f and a continuous attribute A (e.g. “age”), we say f is rankable with respect to A if there exists a rank axis v_{A} such that the projection v_{A}^{\top}f(x) preserves the correct order of the target attribute A(x) over a dataset. For instance, if A denotes “age”, this projection would sort face images from youngest to oldest. We examine two questions: (1) Are visual embeddings rankable? (2) How easily can we recover the rank axis for a given attribute? To address (1), we evaluate 7 modern visual encoders, from ResNet to CLIP, across 9 datasets with 7 attributes: age, crowd count, 3 head pose angles (pitch, roll, yaw), image aesthetics, and recency. We find that many embedding spaces are indeed rankable ( section 3 ). To address (2), we estimate the rank axis v_{A} with minimal supervision. The structure of the embedding space makes full-dataset regression unnecessary. In many cases, a handful of annotated samples and, in some cases, a pair of samples x_{l} (low) and x_{h} (high) already recover non-trivial ranking performance. For the latter case, we define the rank axis as v_{A}=(f(x_{h})-f(x_{l}))/\|f(x_{h})-f(x_{l})\|_{2} . This opens up the possibility for fast ordering of new images by arbitrary attributes. For example, a photo app lets users sort selfies by age appearance. It uses CLIP embeddings and two reference images: one of a child and one of an elderly person. The app computes v_{\text{age}} without training. Users scroll from youngest-looking to oldest-looking faces in their album ( section 4 ). Contributions: 1. We define and motivate rankability as a property of visual embeddings, distinct from retrieval. 2. We study rankability across modern encoders and real-valued attributes; results show that current embeddings are rankable. 3. We show that rank axes can sometimes be recovered using only two or a handful of labelled samples. 2 Related work Embeddings for retrieval. Visual encoders are commonly used to index images in vector databases, enabling nearest neighbour search for retrieval tasks [ 38 , 45 , 31 , 49 ] . This setup, known as deep metric learning [ 5 , 6 , 38 ] , predates vision-language models like CLIP [ 45 ] . CLIP and related models shifted the focus to cross-modal similarity modelling, where vision and language share a joint embedding space used for classification [ 45 ] , retrieval [ 63 , 2 ] , and captioning [ 36 , 26 ] . While the majority of work in visual encoders is devoted to the understanding of the local similarity structure, we study how visual embeddings support global ranking instead of just local retrieval. Improving embedding geometry and structure. Prior work has explored ways to improve the geometry of the embedding space. Order embeddings and hyperbolic representations have been used to model hierarchies [ 55 , 21 , 8 , 44 ] . Training disentangled representation [ 60 ] is considered critical for compositionality, where attributes are assigned to certain linear subspaces [ 51 , 3 ] . Others have defined concepts like uniformity and separability of the representations [ 59 ] . In this work, we focus on the analysis of a wide range of visual encoders, rather than introducing recipes for improvements. Analysing embedding geometry and structure. A large body of work has examined the geometry and structure of CLIP’s learned embedding space. CLIP and its derivatives have been studied extensively [ 4 , 46 , 30 , 65 ] . Several works have reported modality gaps between vision and language embeddings [ 12 ] . Some studies point to the absence of certain structures and capabilities in CLIP representations: attribute-object bindings [ 29 , 68 , 25 ] , or the association of attributes to corresponding instances. Others argue that much information is already present in CLIP representations, including parts-of-speech and linguistic structure [ 39 ] , attribute-object bindings [ 24 ] , and compositional attributes [ 53 ] . The platonic representation hypothesis further suggests that models converge to similar internal structures [ 19 ] . In this work, we analyse the embedding geometry and structure for modern visual embeddings from the novel perspective of rankability. Linearly probing an embedding. Linear probing is a fast and widely used method to test for the presence of concepts in visual embeddings [ 23 , 18 , 54 ] . It measures the accuracy of a linear classifier trained on intermediate-layer features, effectively testing whether a hyperplane can separate embeddings containing a concept from those that do not. This technique has been used to study the geometry of CLIP’s embeddings [ 28 ] and to probe for specific information such as attribute-object bindings and compositionality [ 24 , 53 ] . While effective for binary separability, linear probes are limited in capturing non-binary, ordinal, or relational structures [ 1 ] . Prior work has not directly analysed how continuous attributes are laid out in the embedding space. Our work extends this line of research by moving beyond concept detection to characterise the internal structure of embeddings along ordinal axes, introducing rankability as a new property not captured by prior probing methods. Ordinal information in embeddings. Early works on relative and ordinal attributes explored how continuous visual attributes could be inferred and ranked [ 43 , 34 , 71 , 14 , 64 ] . However, these efforts were limited to smaller models and datasets. Recent studies have begun to examine ordinal signals in large pre-trained models. These include aligning CLIP with ordinal supervision [ 58 ] , and applying CLIP or general VLMs to specific tasks such as aesthetics