Escaping the big data paradigm in self-supervised representation learning

Carlos Vélez-García, Miguel Cazorla, Jorge Pomares · 2025

Read Escaping the big data paradigm in self-supervised representation learning by Carlos Vélez-García, Miguel Cazorla, Jorge Pomares (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-18056.txt.

Description

Carlos Vélez García, Miguel Cazorla, and Jorge Pomares published a paper in 2025 on learning good image features without giant data sets. Self-supervised vision models usually demand millions of images and heavy computing, shutting out small labs and niche fields. The authors ask whether clever design can replace sheer scale.

Their tokenizer, SCOTT, is a shallow convolutional front end that gives a vision transformer a built-in sense of local pattern. Training pairs masked image modeling with a predictive method that works in feature space rather than pixels. On three small, fine-grained sets, covering flowers, pets, and objects, their frozen models beat fully supervised training. They also rivaled systems pre-trained on far larger data, at a fraction of the cost.

Catalog record

CREATORCarlos Vélez-García, Miguel Cazorla, Jorge Pomares
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE70 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-18056

Browse the full library · About the Tower of Babel