What DINO saw: ALiBi positional encoding reduces positional bias in Vision Transformers
Moritz Pawlowsky, Antonis Vamvakeros, Alexander Weiss, Anja Bielefeld, Samuel J. Cooper *, Ronan Docherty · 2026
Read What DINO saw: ALiBi positional encoding reduces positional bias in Vision Transformers by Moritz Pawlowsky, Antonis Vamvakeros, Alexander Weiss, Anja Bielefeld, Samuel J. Cooper *, Ronan Docherty (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-16840.txt.
Description
Pawlowsky and co-authors probe positional bias in vision transformers. Models like DINOv2 learn rich image features, but layout choices can add bias tied to position, not content. That hurts fields like materials science, where images have no preferred direction. With linear probing, the team finds bias across many objectives and encodings. They then fine-tune models to use ALiBi relative positional encoding. The authors report that bias drops while general semantics stay intact. The cleaned features also work well for segmenting complex microscopy images.
Catalog record
| CREATOR | Moritz Pawlowsky, Antonis Vamvakeros, Alexander Weiss, Anja Bielefeld, Samuel J. Cooper *, Ronan Docherty |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40419 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.16840