ITSELF: Attention Guided Fine-Grained Alignment for Vision–Language Retrieval
Tien-Huy Nguyen, Huu-Loc Tran, Thanh Duc Ngo · 2026
Read ITSELF: Attention Guided Fine-Grained Alignment for Vision–Language Retrieval by Tien-Huy Nguyen, Huu-Loc Tran, Thanh Duc Ngo (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-01024.txt.
Introduction
Finding a person from a written sketch of how they look is a hard search task. Tien-Huy Nguyen, Huu-Loc Tran, and Thanh Duc Ngo work on this text-based person search, where a system must match a phrase to the right photo among many similar ones. Past methods often latch onto false clues and lose the fine details that set one person apart.
Their system, called ITSELF, uses the model's own focus maps as a guide. It gathers the image and text parts the model already attends to most, then trains on those parts to sharpen the match. Extra steps merge focus across layers, drop repeated picks, and shift from broad context to fine detail as training goes on. Tests on three standard data sets show top results and strong performance on new data.
Catalog record
| CREATOR | Tien-Huy Nguyen, Huu-Loc Tran, Thanh Duc Ngo |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-01024
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ITSELF: Attention Guided Fine-Grained Alignment for Vision–Language Retrieval Tien-Huy Nguyen, Huu-Loc Tran, Thanh Duc Ngo Abstract Vision Language Models (VLMs) have rapidly advanced and show strong promise for text-based person search (TBPS), a task that requires capturing fine-grained relationships between images and text to distinguish individuals. Previous methods address these challenges through local alignment, yet they are often prone to shortcut learning and spurious correlations, yielding misalignment. Moreover, injecting prior knowledge can distort intra-modality structure. Motivated by our finding that encoder attention surfaces spatially precise evidence from the earliest training epochs, and to alleviate these issues, we introduce ITSELF , an attention-guided framework for implicit local alignment . At its core, Guided Representation with Attentive Bank (GRAB) converts the model’s own attention into an Attentive Bank of high-saliency tokens and applies local objectives on this bank, learning fine-grained correspondences without extra supervision. To make the selection reliable and non-redundant, we introduce Multi-Layer Attention for Robust Selection (MARS), which aggregates attention across layers and performs diversity-aware top-k selection; and Adaptive Token Scheduler (ATS), which schedules the retention budget from coarse to fine over training, preserving context early while progressively focusing on discriminative details. Extensive experiments on three widely used TBPS benchmarks show state-of-the-art performance and strong cross-dataset generalization, confirming the effectiveness and robustness of our approach without additional prior supervision. Our project is publicly available at https://trhuuloc.github.io/itself 1 Introduction Figure 1 : Evolution of text-based person search paradigms. (a) Global matching method uses powerful MLLM to synthesize extra datasets (b) Recent local implicit matching method implicitly reasons over relations among all local tokens. (c) Ours - ITSELF with GRAB: an attention-guided local branch to learn implicitly fine-grained, discriminative features to achieve better alignment. Text-based person search (TBPS) aims to identify, from a large image gallery, the person best matching a textual query [ 16 ] . Solving it requires extracting identity-discriminative cues from both image and text to distinguish individuals with subtle differences. Recent advances in vision language models (VLMs) [ 17 , 48 , 22 , 36 , 23 ] , notably CLIP [ 26 ] , have shown strong potential for tackling these fine-grained challenges. Building on this foundation, TBPS-CLIP [ 3 ] was the pioneer to apply CLIP to TBPS, followed by extensions [ 13 , 24 , 53 , 19 , 43 , 20 ] that further narrow the text–image gap. However, many recent methods [ 52 , 33 , 39 , 53 , 46 , 21 ] rely on costly external resources. For example, using MLLMs to synthesize auxiliary data ( Fig. 1 (a) ), while effective, increase compute and annotation costs, and hinder scalability and robustness. To motivate the limitation, we pose a simple question: How can a TBPS model capture fine-grained, discriminative details on its own without costly external supervision? To minimize this, recent work pursues implicit local alignment ( Fig. 1 (b) ) to explore discriminative cues without burdensome external supervision. Embedding-space correspondence methods [ 9 , 34 , 38 , 45 ] infer region–phrase matches from implicit signals, yet sparse labels leave these alignments weakly constrained and unstable. Fully implicit feature learning [ 30 , 31 , 42 ] optimizes local losses but overlooks the semantics of specific text–region pairs, offering no guarantee of precise correspondences. Masked-modeling–style alignment [ 13 , 11 ] encourages grounding via cross-modal reconstruction, but global-context shortcuts can bypass true dependencies. Across these lines, the core issue persists: locality constraints remain too weak, diffusing supervision, weakening discriminative region selection, and limiting robust local feature learning. Attention is well suited to surface fine-grained cues and strengthen cross-modal correspondence, yet its potential remains underexplored thoroughly in TBPS. We probe this with a simple diagnostic: attention-guided retention masking ( Fig. 2 ). For each image, we compute attention from the image encoder’s last layer, retain the top- k patches, and mask the rest. We then measure the R1 accuracy gap between the original input and its counterpart over early epochs and multiple retention ratios on RSTP-Reid [ 51 ] . This analysis reveals two consistent patterns. First, saliency appears early: by epoch 3, the R1 gap falls below 1 percentage point for every retention setting, indicating that the retained patches already capture nearly all discriminative evidence. Second, attention is spatially precise: selected patches consistently align with semantically meaningful parts and carried objects, providing reliable localization cues for discriminative region–phrase correspondence. Figure 2 : Rank-1 accuracy gap between unmasked and masked images under different mask retention ratios during the early training epochs on RSTP dataset. Building on these findings, we present ITSELF , an attention-guided framework for implicit local alignment in TBPS. ITSELF optimizes a global text–image alignment loss and augments it with Guided Representation with Attentive Bank (GRAB), an attention-driven local branch that builds an Attentive Bank by selecting discriminative tokens from both modalities using model’s own attention without additional supervision ( Fig. 1 (c) ). Unlike previous methods that keep local alignment fully implicit, providing no direct constraints on phrase–region correspondence and thus inviting shortcut learning and spurious correlations that cause misalignment, our method injects an attention-derived locality prior that focuses learning on the most informative regions and aligns them consistently across modalities, thereby suppressing noise and reinforcing global alignment. In essence, our key innovation is an attention-driven implicit locality mechanism that turns internal saliency maps into reliable anchors for fine-grained alignment. Based on analyses [ 1 , 37 , 4 ] showing different attention heads specialize in complementary cues, with distinct patterns emerging at different depths. We introduce Multi-Layer Attention for Robust Selection ( MARS ) inside GRAB to avoid repeatedly picking the same dominant tokens from a single attention map. MARS aggregates attention across layers and performs diversity-aware top- k selection across modalities , ensuring complementary coverage. The selected tokens populate GRAB’s Attentive Bank, where local objectives reinforce both inter- and intra-modal structure. As suggested by ( Fig. 2 ) where the R1 gap between unmasked and attention-retained inputs quickly narrows in early epochs, we further introduce an Adaptive Token Scheduler (ATS) that maintains a larger retention budget at the start to avoid discarding important cues and the resulting training instability, then progressively anneals the budget to focus on high-confidence, fine-grained tokens. This schedule reduces redundancy and false negatives and stabilizes local learning. Finally, following recent practice, we adopt CLIP [ 26 ] as the backbone, allowing ITSELF to transfer pretrained knowledge while continuing to learn cross-modal, implicit local correspondences on TBPS. In summary, our main contributions are as follows: • ITSELF Framework: A novel attention-guided implicit local alignment framework, ITSELF , with GRAB leveraging encoder attention to mine fine-grained cues and reinforce global alignment without additional supervision. • Robust Selection & Scheduling: We propose MARS , which fuses attention across layers and performs diversity-aware top- k selection; and ATS , which anneals the retention budget from coarse to fine over training to stabilize learning and prevent early information loss. • Strong Empirical Results: Extensive experiments establish SOTA performance on 3 widely used TBPS benchmarks and improved cross-dataset generalization, confirming the effectiveness and robustness of our approach. 2 Related Work 2.1 Text-based Person Search (TBPS) Over the last few years, the computer vision community has shown a lot