What DINO saw: ALiBi positional encoding reduces positional bias in Vision Transformers

Moritz Pawlowsky, Antonis Vamvakeros, Alexander Weiss, Anja Bielefeld, Samuel J. Cooper *, Ronan Docherty · 2026

Read What DINO saw: ALiBi positional encoding reduces positional bias in Vision Transformers by Moritz Pawlowsky, Antonis Vamvakeros, Alexander Weiss, Anja Bielefeld, Samuel J. Cooper *, Ronan Docherty (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-16840.txt.

Description

Pawlowsky and co-authors probe positional bias in vision transformers. Models like DINOv2 learn rich image features, but layout choices can add bias tied to position, not content. That hurts fields like materials science, where images have no preferred direction. With linear probing, the team finds bias across many objectives and encodings. They then fine-tune models to use ALiBi relative positional encoding. The authors report that bias drops while general semantics stay intact. The cleaned features also work well for segmenting complex microscopy images.

Catalog record

CREATORMoritz Pawlowsky, Antonis Vamvakeros, Alexander Weiss, Anja Bielefeld, Samuel J. Cooper *, Ronan Docherty
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40419
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.16840

Browse the full library · About the Tower of Babel