TinySSL: Distilled Self-Supervised Pretraining for Sub-Megabyte MCU Models
Bibin Wilson · 2026
Read TinySSL: Distilled Self-Supervised Pretraining for Sub-Megabyte MCU Models by Bibin Wilson (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-08241.txt.
Description
Bibin Wilson brings self-supervised learning down to microcontroller scale. At that scale, models hold fewer than half a million parameters. They must fit under a megabyte. Three obstacles bite hard. The projection head dominates. The representation bottlenecks. And results swing with augmentation choices.
The framework learns from a frozen DINO teacher through asymmetric distillation. It adds multi-scale feature matching. It also ramps augmentation strength in a set sequence. On a small MobileNet backbone pretrained on CIFAR-100, it beat a tiny SimCLR baseline by eighteen points. That score is for linear-probe accuracy. It reached ninety-four percent of the supervised ceiling, with no labels and no text supervision.
Catalog record
| CREATOR | Bibin Wilson |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 62 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.08241