RefAM : Attention Magnets for Zero-Shot Referral Segmentation

Anna Kukleva, Enis Simsar, Alessio Tonioni, Muhammad Ferjad Naeem, Federico Tombari, Jan Eric Lenssen, Bernt Schiele · 2025

Read RefAM : Attention Magnets for Zero-Shot Referral Segmentation by Anna Kukleva, Enis Simsar, Alessio Tonioni, Muhammad Ferjad Naeem, Federico Tombari, Jan Eric Lenssen, Bernt Schiele (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-22650.txt.

Description

Anna Kukleva and 6 co-authors find that filler words in a prompt act like magnets for attention. Referring segmentation asks a model to outline the object a text phrase points to. Most methods fine-tune or chain several models. The team instead reuses attention scores inside diffusion transformers, with no training and no architecture change. Stop words soak up surplus attention, and filtering them cuts noise.

Deeper layers grow global attention sinks that blur the maps. Suppressing or redirecting those sinks sharpens the grounding. Appending stop words splits background activity into small clusters, which sharpens maps further. Their framework, RefAM, led training-free methods on image and video benchmarks.

Catalog record

CREATORAnna Kukleva, Enis Simsar, Alessio Tonioni, Muhammad Ferjad Naeem, Federico Tombari, Jan Eric Lenssen, Bernt Schiele
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE54 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-22650

Browse the full library · About the Tower of Babel