TinySSL: Distilled Self-Supervised Pretraining for Sub-Megabyte MCU Models

Bibin Wilson · 2026

Read TinySSL: Distilled Self-Supervised Pretraining for Sub-Megabyte MCU Models by Bibin Wilson (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-08241.txt.

Description

Bibin Wilson brings self-supervised learning down to microcontroller scale. At that scale, models hold fewer than half a million parameters. They must fit under a megabyte. Three obstacles bite hard. The projection head dominates. The representation bottlenecks. And results swing with augmentation choices.

The framework learns from a frozen DINO teacher through asymmetric distillation. It adds multi-scale feature matching. It also ramps augmentation strength in a set sequence. On a small MobileNet backbone pretrained on CIFAR-100, it beat a tiny SimCLR baseline by eighteen points. That score is for linear-probe accuracy. It reached ninety-four percent of the supervised ceiling, with no labels and no text supervision.

Catalog record

CREATORBibin Wilson
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE62 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.08241

Browse the full library · About the Tower of Babel