From Global to Local: Rethinking CLIP Feature Aggregation for Person Re-Identification

Aotian Zheng, Winston Sun, Bahaa Alattar, Vitaly Ablavsky, Jenq-Neng Hwang, [0.5em] Department of Electrical, Computer Engineering, University of Washington, Seattle, WA 98195, [0.3em] Equal contribution · 2026

Read From Global to Local: Rethinking CLIP Feature Aggregation for Person Re-Identification by Aotian Zheng, Winston Sun, Bahaa Alattar, Vitaly Ablavsky, Jenq-Neng Hwang, [0.5em] Department of Electrical, Computer Engineering, University of Washington, Seattle, WA 98195, [0.3em] Equal contribution (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-22190.txt.

Introduction

Zheng and four colleagues propose SAGA-ReID, rebuilding person identity from CLIP patch tokens aligned with text-space anchors. That design stresses spatially stable evidence while muting corrupted or absent regions, with no need for image descriptions. Controlled tests isolate aggregation under synthetic masking, where identity signal is absent, and real human distractors, where an overlapping person adds confusing signal. SAGA's edge over global pooling grows as occlusion increases in both settings. Benchmarks confirm steady gains over CLIP-ReID, up to +10.6 Rank-1 on occluded benchmarks.

Catalog record

CREATORAotian Zheng, Winston Sun, Bahaa Alattar, Vitaly Ablavsky, Jenq-Neng Hwang, [0.5em] Department of Electrical, Computer Engineering, University of Washington, Seattle, WA 98195, [0.3em] Equal contribution
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45464
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.22190

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

From Global to Local: Rethinking CLIP Feature Aggregation for Person Re-Identification Aotian Zheng, Winston Sun, Bahaa Alattar, Vitaly Ablavsky, Jenq-Neng Hwang, [0.5em] Department of Electrical, Computer Engineering, University of Washington, Seattle, WA 98195, [0.3em] Equal contribution Abstract CLIP-based person re-identification (ReID) methods aggregate spatial features into a single global [CLS] token optimized for image–text alignment rather than spatial selectivity, making representations fragile under occlusion and cross-camera variation. We propose SAGA-ReID, which reconstructs identity representations by aligning intermediate patch tokens with anchor vectors parameterized in CLIP’s text embedding space — emphasizing spatially stable evidence while suppressing corrupted or absent regions, without requiring textual descriptions of individual images. Controlled experiments isolate the aggregation mechanism under two qualitatively distinct conditions — synthetic masking, where identity signal is absent, and realistic human distractors, where an overlapping person introduces semantically confusing signal — with SAGA’s advantage over global pooling growing substantially as occlusion increases across both conditions. Benchmark evaluations confirm consistent gains over CLIP-ReID across standard and occluded settings, with the largest improvements where global pooling is most unreliable: up to +10.6 Rank-1 on occluded benchmarks. SAGA’s aggregation outperforms dedicated sequential patch aggregation on a stronger backbone, confirming that structured reconstruction addresses a bottleneck that backbone quality and architectural complexity alone cannot resolve. Code available at https://github.com/ipl-uw/Structured-Anchor-Guided-Aggregation-for-ReID 1 Introduction Person re-identification (ReID) aims to match images of the same individual across disjoint camera views and is a fundamental problem in multi-camera perception systems [ 22 , 20 ] . Recent approaches increasingly adopt vision–language models (VLMs), particularly CLIP [ 13 ] , to leverage large-scale image–text pretraining for improved generalization. Methods such as CLIP-ReID [ 8 ] adapt the image encoder to produce identity-discriminative representations by aligning a global [CLS] token with textual prompts during training. Despite strong performance, this paradigm exhibits a key structural limitation: the deployed model reduces to a standard image encoder whose [CLS] attention pattern — optimized for image–text alignment rather than spatial selectivity — conflates identity-relevant and corrupted regions under occlusion, viewpoint shift, and background clutter, precisely where ReID is hardest. Why existing alternatives fall short. Part-based methods [ 14 , 5 ] impose spatial structure through heuristic partitions or visually learned groupings, but neither approach carries a prior about which regions are identity-relevant. More recent CLIP-based methods introduce richer language signals or inference-time conditioning. CLIP-SCGI [ 4 ] augments training with LLaVA-synthesized captions but discards language at inference. PromptSG [ 19 ] retains language at inference through a fixed prompt augmented by an image-conditioned context token, conditioning visual features on a per-instance basis — but this treats language as instance-level descriptive guidance rather than a shared structural basis for aggregation. Methods that introduce dedicated aggregation mechanisms fare no better without the right inductive bias: CLIMB-ReID [ 21 ] applies sequential patch filtering via a state space model, but the Mamba-specific feature underperforms the global [CLS] token on single-image occluded benchmarks, where temporal coherence — the signal Mamba relies on — is unavailable. The common failure across all these approaches is that none provides a structured basis that governs how patch features are reconstructed at inference — as opposed to conditioned, filtered, or compressed (Figure 1 ). Figure 1: Comparison of feature aggregation paradigms for person ReID. Left: The global [CLS] token aggregates all patch features through self-attention without spatial selectivity. Middle: Existing methods use a text prompt as query in cross-attention over image patches, suppressing non-person patches — effective against background clutter but not human distractor occlusion, where target and distractor patches receive comparable attention weights. Right: SAGA-ReID uses learned structure anchors as keys and values with image patches as queries; target patches align more strongly with the anchors than distractor patches do, and the final representation is constructed by aggregating anchor-reconstructed patch features rather than the original patch features directly. Our approach. We propose SAGA-ReID ( S tructured A nchor- G uided A ggregation for Re-Identification), which uses anchor vectors parameterized in CLIP’s text embedding space as keys and values in a cross-attention module over intermediate patch tokens. Rather than conditioning or filtering features, this performs structured reconstruction : patch features are selectively recombined according to their alignment with a structured anchor basis, emphasizing spatially stable evidence while suppressing regions that are absent, corrupted, or identity-irrelevant. Text-space initialization provides a structured optimization prior that produces more diverse and stable aggregation behavior than unconstrained learned bases — not by enforcing explicit semantic correspondence, which does not hold at the intermediate layers used for aggregation, but by constraining anchors to lie along semantically meaningful directions. Shared structured anchors capture structure across images while image-conditioned domain anchors provide instance-specific adaptation; at inference, the anchor-refined representation is fused with the global [CLS] feature. On Occluded-DukeMTMC, SAGA-ReID achieves 77.8 Rank-1 / 68.3 mAP — a +10.6 / +8.0 improvement over CLIP-ReID and a +6.7 / +4.9 improvement over the strongest comparable baseline — without external supervision. Gains are largest where global pooling is most unreliable. To isolate the aggregation mechanism independently of benchmark construction, we conduct controlled experiments under two qualitatively distinct occlusion conditions: synthetic masking, where identity signal is absent, and realistic human distractors, where an overlapping person introduces semantically confusing signal. SAGA’s advantage over global pooling grows substantially as occlusion increases across both conditions, confirming that structured reconstruction addresses a failure mode that is not specific to any single occlusion geometry or signal degradation type. Critically, SAGA’s anchor-guided aggregation outperforms CLIMB-ReID’s Mamba feature on a stronger backbone, confirming that the benefit is specifically attributable to the reconstruction mechanism rather than backbone quality or architectural complexity. Contributions. • Identifying the aggregation structure gap. We identify that existing CLIP-based ReID methods share a common limitation regardless of whether language is used at training or inference: none provides a structured basis that governs how patch features are reconstructed at inference time. Methods that condition features on language treat it as instance-level descriptive guidance; methods that introduce dedicated aggregation mechanisms lack structured aggregation priors. We show this gap is the primary source of fragility under occlusion and cross-camera variation. • Structured anchor-guided reconstruction. We propose SAGA-ReID, which inverts the standard cross-attention polarity used in vision-language models: rather than using text as query to condition or filter image patches — which cannot distinguish target from distractor when both are person patches — image patches attend to text-space initialized anchors as keys and values. This enables structured feature reconstruction without explicit part labels or textual supervision, and to our knowledge represents the first application of this inverted cross-attention design to single-image ReID aggregation. • Consistent gains under challenging conditions. SAGA-ReID achieves consistent improvements over CLIP-ReID across standard benchmarks with substantially larger gains under occlusion, and outperforms dedicated aggregation mechanisms on stronger backbones. Controlled experiments under synthetic masking and realistic human distractor occlusion confirm that the advantage grows with occlusion severity and generalizes across qualitatively distinct failure modes — demonstrating that structured reconstruction addresses a bottleneck that conditioning, filtering, and architectural complexity alone cannot resolve. 2 Related Work Person re-identification. Early ReID methods focused on supervised metric learning using convolutional backbones [ 6 ] . PCB [ 14 ] introduced part-based pooling to impose spatial structure, and TransReID [ 5 ] extended this to transformer architectures with side-information

Read the complete text