GraSP-VL: Length as a Semantic Granularity Interface for Vision-Language Representations
Zesheng Li, Chengchang Pan, Honggang Qi · 2026
Read GraSP-VL: Length as a Semantic Granularity Interface for Vision-Language Representations by Zesheng Li, Chengchang Pan, Honggang Qi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-17727.txt.
Introduction
Zesheng Li, Chengchang Pan, and Honggang Qi treat embedding length as a dial for meaning in frozen vision-language models. Their GraSP-VL learns one shared, near-orthogonal prefix transform so short prefixes expose coarse ideas like object identity while longer prefixes reveal finer attributes, relations, and caption-level meaning. On a 20,147-example COCO and Flickr30K pool it reaches a staircase score of 53.01 and strong hard-negative selectivity, while barely disturbing the full-dimensional space or its zero-shot accuracy.
Catalog record
| CREATOR | Zesheng Li, Chengchang Pan, Honggang Qi |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 59252 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.17727
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
GraSP-VL: Length as a Semantic Granularity Interface for Vision-Language Representations Zesheng Li, Chengchang Pan, Honggang Qi Abstract Frozen vision-language embeddings contain signals at multiple semantic resolutions, from object identity to attributes, relations, and full-caption meaning, but they expose these signals through a fixed-length vector interface. We study whether embedding length can be turned into a controllable semantic access interface. We propose GraSP-VL , which learns a shared near-orthogonal prefix transform over frozen VLM embeddings. GraSP-VL instantiates a Semantic Matryoshka interface: short prefixes are assigned coarse semantic roles, while longer prefixes progressively expose finer language-grounded distinctions. Because the transform is shared across image and text embeddings and preserves full-dimensional geometry, prefix behavior changes without rewriting the original VLM space. On a 20,147-example COCO/Flickr30K annotation pool, GraSP-VL reaches a staircase score of 53.01 and hard-negative selectivity of 89.76, while keeping full-space drift below 10 − 6 10^{-6} . It also transfers to SugarCrepe-clean with 86.03 object accuracy and 11.96 mean external emergence, and preserves full-dimensional zero-shot CIFAR-100 accuracy. These results show that frozen VLM embeddings can be reorganized into a truncatable semantic prefix interface rather than merely compressed. 1 Introduction Vision-language models (VLMs) such as CLIP make one embedding space support caption retrieval, zero-shot recognition, and prompts at very different semantic resolutions ( 26 ; 14 ; 38 ; 5 ) . This is convenient, but it hides an interface mismatch: sometimes a query only asks for an entity such as dog ; sometimes it asks for an attributed phrase such as brown dog ; sometimes the crucial information is a relation or event such as dog running through snow . A monolithic embedding gives all of these questions the same access pattern. Our premise is that frozen VLM outputs contain multi-granularity semantic signals; the missing piece is a controllable way to access them. Teaser. GraSP-VL reaches the desired corner: high semantic selectivity with near-zero full-space drift. The usual way to make embeddings shorter is to truncate, project, or compress them. These operations may preserve performance, but they do not tell us what the shorter vector is supposed to mean. The first small fraction of a frozen CLIP embedding is not guaranteed to be object-level; PCA may keep strong variance directions, but it does not know that a prefix should represent entities before attributes or relations. In this paper we study a different interface question: can representation length itself indicate semantic granularity? This question is close to, but distinct from, Matryoshka-style representation learning ( 17 ; 34 ; 39 ) , which makes multiple prefix lengths useful under different storage or latency budgets. Our target is not just usefulness under truncation: a short prefix should be coarse on purpose, and a longer prefix should add distinctions that the shorter one ignores. Put differently, length should be a semantic lens, not only a compression budget. We propose GraSP-VL ( Gra nularity- S elective P refixes for V ision- L anguage representations), a method for learning this access interface over frozen VLM embeddings. We use Semantic Matryoshka to denote the resulting prefix-interface principle: shorter prefixes are intentionally coarse, while longer prefixes reveal progressively finer distinctions under an explicit contract. GraSP-VL keeps the VLM frozen and learns a lightweight transform that is shared by image and text embeddings. The key constraint is that the full-dimensional geometry is preserved, while prefix geometry changes. This lets us ask a clean question: can we reorganize access to information already present in the frozen embedding so that short prefixes match coarse language and longer prefixes become increasingly compositional under an explicit interface contract? We do not claim that the chosen dimensional breakpoints are natural semantic constants or that the original coordinate order contains a discoverable semantic hierarchy. Rather, we test whether frozen VLM signals can be reorganized into a controllable prefix interface without rewriting the full embedding space. Our evaluation is built as a falsification chain. If frozen VLMs lack multi-granularity signals, there is nothing to expose; if direct truncation, PCA, or random rotations already produce the desired order, no learned interface is needed; if Matryoshka-style training or a generic adapter gives the same behavior, the effect is ordinary compression or adaptation rather than semantic control. Our contributions are: • We propose GraSP-VL , a shared prefix transform for frozen VLM embeddings that preserves full-space similarity while changing which semantics are exposed by short prefixes. • We formulate Semantic Matryoshka as the access-interface principle implemented by GraSP-VL: embedding length is assigned semantic granularity by an explicit contract rather than treated as a discovered natural boundary. • We introduce a granularity-selective objective that combines prefix capability, cumulative retention, hard-negative discrimination, early-prefix invariance, and full-space preservation. • We design a diagnostic protocol for prefix organization that separates semantic selectivity from raw prefix truncation, PCA compression, Matryoshka-style preservation, generic adaptation, and single-embedding compositional tuning. Figure 1: Sample case. Longer prefixes progressively expose attribute, relation/action, and full-caption distinctions. 2 Related Work Nested representations. Nested representations make one embedding usable at multiple dimensionalities. Matryoshka Representation Learning trains prefixes of the same vector to remain predictive ( 17 ) , and Matryoshka-Adaptor adapts pretrained embeddings for smaller usable dimensions ( 34 ) . Retrieval-specific work makes the compression motivation explicit: MRL-AdANNS uses Matryoshka representations for adaptive web-scale semantic search ( 27 ) , and SMEC re-evaluates MRL for retrieval embedding compression with sequential training, adaptive dimension selection, and cross-batch memory ( 39 ) . Recent multimodal nested systems connect this idea to token-set or multi-vector retrieval ( 20 ; 3 ; 31 ) . These methods target prefix usability and adaptive retrieval budgets; GraSP-VL instead assigns different semantic responsibilities to different prefixes. VLM adaptation. Many methods adapt frozen or partially frozen VLMs with a small number of trainable parameters. Prompt-learning methods such as CoOp, CoCoOp, and MaPLe tune textual or multi-modal prompts ( 43 ; 42 ; 15 ) . Feature-side methods such as CLIP-Adapter and Tip-Adapter modify or cache CLIP features for downstream recognition ( 8 ; 41 ) , while adapters and low-rank adaptation provide general parameter-efficient machinery ( 9 ; 11 ) . GraSP-VL is also lightweight, but it reorganizes prefix access while preserving full-space behavior rather than adapting the VLM to a downstream classifier or domain. Compositional VLMs. Standard retrieval scores often hide whether a VLM understands attributes, relations, or word order. SugarCrepe probes object, attribute, relation, and hard caption distractors outside standard retrieval leaderboards ( 10 ) . Recent tuning methods such as NegCLIP, CE-CLIP, FSC-CLIP, CLIP-CAE, TripletCLIP, CLoVe, AHNPL, and DeGLA improve compositional sensitivity through hard negatives, preservation-aware training, attribution enhancement, model patching, or global-local alignment ( 36 ; 40 ; 24 ; 19 ; 25 ; 4 ; 13 ; 12 ) . Fine-grained retrieval models such as SCAN and FILIP use region-word or token-level interactions rather than a single truncatable vector ( 18 ; 33 ) ; FG-CLIP and FineLIP strengthen CLIP-style full embeddings with fine-grained data ( 32 ; 2 ) . These methods improve compositionality in one representation, while we ask where such sensitivity should appear across prefixes. Hierarchical semantics. Hierarchical semantics have long been central to language and vision. WordNet organizes lexical concepts taxonomically ( 22 ) , while ImageNet and Visual Genome provide