WAVECLIP: Wavelet Tokenization for Adaptive-Resolution CLIP
Moshe Kimhi \sthanks Equal contribution, Erez Koifman, Ehud Rivlin, Eli Schwartz \sthanks Equal supervision, Chaim Baskin · 2025
Read WAVECLIP: Wavelet Tokenization for Adaptive-Resolution CLIP by Moshe Kimhi \sthanks Equal contribution, Erez Koifman, Ehud Rivlin, Eli Schwartz \sthanks Equal supervision, Chaim Baskin (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-21153.txt.
Introduction
Vision-language models like CLIP process every image at one fixed resolution, wasting effort on easy pictures. Moshe Kimhi and four co-authors present WAVECLIP in a 2025 paper on adaptive-resolution inference. Their model swaps the usual patch embedding for a wavelet-based tokenization.
A multi-level wavelet decomposition lets the model read an image from coarse to fine. Inference starts with low-resolution tokens and adds detail only when needed, reusing cached computation across levels. Causal cross-level attention ensures only genuinely new information enters at each step. In zero-shot classification tests, a simple confidence gate triggers early exits. Users gain a dial between speed and accuracy within a single model, instead of maintaining separate models per resolution.
Catalog record
| CREATOR | Moshe Kimhi \sthanks Equal contribution, Erez Koifman, Ehud Rivlin, Eli Schwartz \sthanks Equal supervision, Chaim Baskin |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 16 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-21153
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
WAVECLIP: Wavelet Tokenization for Adaptive-Resolution CLIP Moshe Kimhi \sthanks Equal contribution, Erez Koifman, Ehud Rivlin, Eli Schwartz \sthanks Equal supervision, Chaim Baskin Abstract We introduce WAVECLIP, a single, unified model for adaptive-resolution inference in CLIP, enabled by wavelet-based tokenization. WAVECLIP replaces standard patch embeddings with a multi-level wavelet decomposition, enabling the model to process images coarse-to-fine while naturally supporting multiple resolutions within the same model. At inference time, the model begins with low-resolution tokens and refines only when needed, using key–value caching and causal cross-level attention to reuse computation, effectively introducing to the model only new information when needed. We evaluate WAVECLIP in zero-shot classification, demonstrating that a simple confidence-based gating mechanism enables adaptive early exits. This allows users to dynamically choose a compute–accuracy trade-off using a single deployed model. Our approach requires only lightweight distillation from a frozen CLIP teacher and achieves competitive accuracy with significant computational savings. † † address: 1 Technion – Israel Institute of Technology 2 IBM Research 3 Ben-Gurion University of the Negev † † address: 1 Technion – Israel Institute of Technology 2 IBM Research 3 Ben-Gurion University of the Negev 1 Introduction Contrastive language-image pretraining (CLIP) enables strong zero-shot transfer with a frozen text tower and a Vision Transformer (ViT) image encoder [ 1 , 2 , 3 ] . However, its inference cost scales quadratically with the number of image tokens, leading to very high GFLOPs. While much of the prior work on efficient CLIP reduces or redesigns the vision tower [ 4 , 5 , 6 ] or modifies the training objective [ 3 , 7 , 8 ] , these approaches—though effective for accuracy or efficiency—typically require heavy pre-training on web-scale datasets (e.g., 2-5B examples). Moreover, each model is trained independently, yielding separate models for different scenarios. This makes it hard to dynamically adjust the trade-off at inference time and complicates deployment in systems that need adaptive behavior. We introduce WAVECLIP , a training-light, drop-in alternative to patch tokenization that turns a multi-level discrete wavelet transform (DWT) into a progressive, cacheable inference schedule for CLIP. WAVECLIP processes coarse tokens first (LL band) and exits early when confident; otherwise, it appends level-wise detail tokens (LH/HL/HH). A block-causal cross-level attention mask allows tokens at level \ell to attend to tokens from levels \leq\ell , so key/value states are reused and prior computation is never repeated. A short distillation from a frozen CLIP teacher preserves zero-shot alignment, avoiding full image-text pretraining. Our contributions: (i) A wavelet tokenizer + cross-level attention mask that enables early exits with KV reuse in CLIP, no ViT/text-tower changes. (ii) A light distillation procedure that maintains zero-shot alignment. (iii) Accuracy/compute trade-offs via simple confidence gating, that is controllable at inference time. Figure 1: Accuracy–compute frontier on ImageNet-1k (zero-shot). WAVECLIP spans a smooth trade-off curve by adjusting threshold at inference, while prior methods appear as fixed points—each requiring a separate model for a single compute budget. Figure 2: WAVECLIP overview. A discrete wavelet transform (DWT) decomposes the image and, after patchify, yields multiscale token groups. Inference proceeds coarse-to-fine in a single Transformer: each group uses bidirectional attention internally, while a causal cross-level mask lets later groups attend to earlier tokens only. 2 Related work In CLIP-specific efficiency, most prior work shrinks or redesigns the vision tower itself: TinyCLIP distills large CLIP models into compact students [ 4 ] ; MobileCLIP trains lightweight backbones with multimodal reinforcement [ 5 ] ; and MobileCLIP2 further strengthens distillation/captioning for improved zero-shot at similar runtimes [ 6 ] . These approaches trade parameters and capacity for speed. In contrast, we keep the standard CLIP ViT and text tower intact [ 9 , 1 , 2 , 3 ] and target how much visual evidence is processed per image. Early-exit methods dynamically adjust compute based on input difficulty, by forecasting predictions from earlier layers of the model [ 10 , 11 ] ; this differs from our approach, as deciding to process fewer (coarser) tokens directly reduces GFLOPs when the model architecture stays intact, enabling robust deployment and KV-cache reuse. Multiscale tokenization. Wavelet decompositions provide a principled, dyadic multiresolution representation [ 12 , 13 ] and have recently been explored as drop-in tokenizers for ViTs [ 14 ] . We leverage this structure not only to replace patchify, but to enable progressive, cacheable inference: start from the coarse LL band and only add LH/HL/HH detail when the gate deems it necessary. A block-causal cross-level attention mask and key-value (KV) caching let finer levels reuse keys/values from coarser levels, avoiding recomputation while preserving the standard CLIP architecture. 3 Method Given an image x\in\mathbb{R}^{H\times W\times 3} and a ViT image encoder with B transformer blocks, a standard patch embedder with patch size P yields N=\frac{HW}{P^{2}} spatial tokens (plus a [CLS] token). Self-attention incurs O(N^{2}d) cost per block, so reducing N directly lowers GFLOPs. In CLIP-style zero-shot inference, an image embedding v\in\mathbb{R}^{d} is compared to a bank of text embeddings \{t_{m}\}_{m=1}^{M} via cosine similarity. Figure 3: Comparing the computational cost of using KV cache vs no KV cache reuse. 3.1 Wavelet decomposition as structured tokenization Instead of patchify, we first convert x from RGB to YCbCr and apply an L -level 2D discrete wavelet transform (DWT) independently to each channel , producing channel-wise subbands \{(\mathrm{LL}^{(\ell,c)},\mathrm{LH}^{(\ell,c)},\mathrm{HL}^{(\ell,c)},\mathrm{HH}^{(\ell,c)})\}_{\ell=1}^{L},c\in\{Y,\mathrm{Cb},\mathrm{Cr}\}, with dyadic scaling [ 12 , 13 ] ; \mathrm{LL}^{(L,c)} is the coarsest approximation for channel c , while \mathrm{LH}/\mathrm{HL}/\mathrm{HH} add band-pass detail at progressively finer scales. We tokenize each subband by non-overlapping P\times P patches and then merge tokens from all channels, ordering them coarse-to-fine across levels (and channels), following evidence that wavelet tokenizers can replace patchify in ViTs [ 14 ] . Unlike [ 14 ] , which concatenates DWT subbands along the channel dimension to preserve the spatial grid and thus keep the token count N , we explicitly reduce the number of tokens by processing coarse (LL) tokens first and appending LH/HL/HH detail only when needed, directly lowering the quadratic attention term and GFLOPs. Denote the cumulative token set after \ell refinement steps by \mathcal{X}^{[0]}=\mathrm{patchify}(\mathrm{LL}^{(L)}) \mathcal{X}^{[\ell]}=\mathcal{X}^{[s-1]}\cup\mathrm{patchify}\!\big(\mathcal{D}^{(L-s+1)}\big),\;\ell=1,\dots,L where \mathcal{D}^{(\ell)}=\{\mathrm{LH}^{(\ell)},\mathrm{HL}^{(\ell)},\mathrm{HH}^{(\ell)}\} . Thus \mathcal{X}^{[L]} equals the full-resolution token set. This design lets us begin with very few coarse tokens and add detail only when needed, directly controlling N . Token counts: At level \ell , spatial size \frac{H}{2^{\ell}}\times\frac{W}{2^{\ell}} , yielding \frac{HW}{P^{2}4^{\ell}} tokens. Hence N_{\text{coarse}}=\frac{HW}{P^{2}4^{L}}+1,\qquad N^{[s]}=\frac{HW}{P^{2}}\!\left(\frac{1}{4^{L}}+\sum_{\ell=L-s+1}^{L}\frac{3}{4^{\ell}}\right)+s, where +s accounts for [CLS] . We train the model with the new DWT tokenizer and new [CLS] tokens to align with the original CLIP embedding space using a pretrained CLIP teacher. Given a student embedding v^{[\ell]} for the \ell ’s [CLS] token, and teacher embedding v_{T} , we minimize the cosine distance: \mathcal{L}_{\text{distill}}=\sum_{\ell=1}^{L}1-\frac{\langle v^{[\ell]},\,v_{T}\rangle}{\|v^{[\ell]}\|\|v_{T}\|} 3.2 Causal cross-level attention with KV reuse Let f_{\theta} denote the ViT image encoder. We enforce level causality via a block-lower-triangular (by level) attention mask: tokens introduced at level \ell may attend to tokens from level \leq\ell , but not vice versa. Formally, with cached key/value states \mathcal{KV}^{[\ell-1]} from previous levels, v^{[s]}=f_{\theta}\!\big(\mathcal{X}^{[s]};\,\mathcal{KV}^{[s-1]},\,M_{\leq s}\big), (1) where M_{\leq s} is the cross-level causal mask. This allows incremental computation: upgrading from s{-}1 to s reuses \mathcal{KV}^{[s-1]} and primarily pays for interactions involving the new tokens, avoiding recomputation on earlier tokens. Fig. 2 illustrates the tokenization