PERL: Parameter Efficient Reasoning in CLIP Latent Space

Simone Carnemolla University of Catania [email protected], Salvatore Calcagno University of Catania [email protected] Daniela GiordanoUniversity of Catania [email protected] Concetto SpampinatoUniversity of Catania [email protected] Matteo PennisiUniversity of Catania [email protected] · 2026

Read PERL: Parameter Efficient Reasoning in CLIP Latent Space by Simone Carnemolla University of Catania [email protected], Salvatore Calcagno University of Catania [email protected] Daniela GiordanoUniversity of Catania [email protected] Concetto SpampinatoUniversity of Catania [email protected] Matteo PennisiUniversity of Catania [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-18464.txt.

Introduction

Simone Carnemolla and four co-authors present PERL, a 2026 lightweight adaptation system that augments a frozen CLIP model. A compact shared reasoning module, applied in loops, generates a latent reasoning token at each step. The token enters an intermediate encoder layer, lifting higher-level meaning while keeping CLIP's pretrained structure. Across fifteen benchmarks it takes the best parameter-performance trade-off among compared methods, with only about 6K trainable parameters.

Catalog record

CREATORSimone Carnemolla University of Catania [email protected], Salvatore Calcagno University of Catania [email protected] Daniela GiordanoUniversity of Catania [email protected] Concetto SpampinatoUniversity of Catania [email protected] Matteo PennisiUniversity of Catania [email protected]
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE59705
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.18464

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

PERL: Parameter Efficient Reasoning in CLIP Latent Space Simone Carnemolla University of Catania [email protected], Salvatore Calcagno University of Catania [email protected] Daniela GiordanoUniversity of Catania [email protected] Concetto SpampinatoUniversity of Catania [email protected] Matteo PennisiUniversity of Catania [email protected] Abstract Contrastively trained vision-language models such as CLIP provide strong zero-shot transfer by aligning images and text in a shared embedding space. However, adapting these models to downstream tasks without degrading their open-vocabulary generalization remains challenging. Existing parameter-efficient adaptation methods typically improve task specialization through learned prompts, adapters, or multimodal transformations, where adaptation capacity is primarily expressed through additional trainable parameters. Inspired by recent latent reasoning methods in language models, we investigate a complementary perspective: can adaptation emerge from iterative reasoning on latent representations rather than from increasing parameter count alone? We introduce PERL (Parameter-Efficient Reasoning in CLIP Latent Space), a lightweight adaptation framework that augments a frozen CLIP model with a compact shared reasoning module applied recurrently across refinement steps. At each step, PERL generates a latent reasoning token conditioned on the current representation and injects it into an intermediate encoder layer, progressively refining higher-level semantic representations while preserving CLIP’s pretrained multimodal structure. Across 15 benchmarks spanning base-to-novel generalization, cross-dataset transfer, and out-of-distribution ImageNet variants, PERL achieves the best parameter-performance trade-off among the compared methods under a fast-adaptation few-shot setting, combining strong novel-class accuracy and competitive transfer performance with only about 6K trainable parameters, up to 817x fewer than the largest compared approach. Overall, our results suggest that iterative latent reasoning provides a complementary adaptation mechanism to parameter scaling in discriminative vision-language models. 2 2 footnotetext: Equal Contribution 1 Introduction Adapting CLIP-style vision-language models [ 24 ] to downstream tasks is often framed as adding a small amount of task-specific flexibility while keeping the pretrained image and text encoders fixed. This constraint is attractive because it preserves CLIP’s broad image-text alignment and zero-shot transfer ability, while enabling lightweight downstream specialization. A large body of work explores this trade-off through learned prompts, multimodal prompt transformations, visual perturbations, low-rank updates, cache-based mechanisms, or adapter modules. Prompt tuning methods learn context vectors [ 37 ] ; conditional prompting methods such as CoCoOp generate instance-dependent prompts [ 36 ] ; multimodal prompting methods introduce coordinated visual-language transformations [ 18 ] ; and adapter-based approaches insert compact trainable modules inside the backbone [ 7 , 15 , 16 ] . These approaches have delivered strong gains and are largely complementary. Despite their architectural differences, however, many express adaptation primarily through additional learned parameters, support statistics, or optimization-dependent components. Recent latent reasoning methods in language models suggest a complementary perspective. Approaches such as Coconut [ 11 ] and Quiet-STaR [ 34 ] show that additional capability can emerge not only from increasing model size, but also from iteratively refining hidden representations over multiple computation steps. In these systems, the model effectively “thinks longer” by repeatedly updating its latent state before prediction. While latent reasoning has been studied primarily in autoregressive language models, the broader principle is more general: repeated computation can provide expressivity complementary to parameter scaling. In this work, we investigate whether this principle transfers to frozen discriminative vision-language models. Rather than increasing adaptation capacity primarily through larger or more specialized trainable modules, we ask whether a small shared module can gain effective capacity by being applied recurrently over the model’s evolving latent state. Our central hypothesis is that, in a frozen vision-language model, adaptation need not arise only from storing more task-specific parameters; it can also emerge from progressively steering pretrained representations toward task-relevant decision regions while remaining compatible with CLIP’s original multimodal embedding space. This perspective is especially relevant for few-shot and rapidly deployable specialization of frozen foundation models. In practical settings, a single backbone may support many downstream adaptations across users, domains, tasks, or deployment environments. The cost of adaptation is therefore not only inference latency, but also the cumulative burden of training, storing, selecting, and maintaining many specialized parameter sets. In such settings, trainable parameters must remain small, while a modest increase in per-example computation is often acceptable. Figure 1 : Novel-class accuracy versus trainable parameters for representative CLIP adaptation methods. PERL achieves competitive novel-class generalization with only 6K trainable parameters, suggesting that iterative latent reasoning can complement parameter scaling in CLIP adaptation. We instantiate this idea in PERL ( Parameter-Efficient Reasoning in CLIP Latent Space ), a lightweight adaptation framework in which a compact shared module recurrently refines latent representations over multiple computation steps. Rather than increasing adaptation capacity through additional task-specific parameters, PERL reuses the same small reasoning module iteratively, allowing the model to trade a modest increase in computation for improved task specialization while keeping the CLIP backbone frozen and avoiding test-time gradient updates. Figure 1 situates PERL within the parameter-performance trade-off of representative CLIP adaptation methods. PERL achieves novel-class generalization competitive with substantially larger adaptation frameworks while operating with only 6K trainable parameters, placing it in a favorable low-parameter regime among the compared methods. These results suggest that iterative latent reasoning can complement parameter scaling as a mechanism for adaptation capacity. At the same time, transferring latent reasoning to CLIP is nontrivial. Unlike autoregressive language models, CLIP operates through contrastive alignment between image and text embeddings, and preserving access to this geometry is central to maintaining generalization. Iterative refinement in this setting should therefore not freely rewrite representations, but instead reshape them in a controlled manner that remains compatible with CLIP’s pretrained multimodal structure. We therefore situate our investigation in a constrained few-shot adaptation regime and ask: Can latent reasoning on internal representations provide an adaptation axis complementary to parameter scaling in frozen discriminative vision-language models? PERL is designed around three principles. First, weight sharing across refinement steps lets effective adaptation capacity grow with computation rather than with parameter count. Second, modality-specific reasoning . Third, intermediate-layer insertion allows refinement to influence semantic representations while limiting perturbations to lower-level pretrained features, helping retain CLIP’s pretrained transfer behavior. These design choices naturally motivate evaluation under constrained optimization budgets. A single-epoch few-shot setting reflects deployments where support data, tuning time, and per-task maintenance are limited, while also isolating how much adaptation can be expressed architecturally rather than accumulated through repeated optimization. In this regime, iterative latent reasoning exposes a different trade-off from conventional parameter scaling: a small learned module can be reused multiple times at inference to provide additional computation without storing additional per-step parameters. Empirically, we conduct an extensive evaluation across 15 benchmarks covering base-to-novel generalization, cross-dataset transfer, and out-of-distribution ImageNet variants. Despite using a fully frozen CLIP backbone and only 6K trainable parameters, PERL outperforms common compact prompt-tuning baselines and approaches substantially larger multimodal adaptation methods. Overall, the results indicate that iterative latent reasoning can provide meaningful adaptation capacity even in extremely parameter-constrained regimes. Contributions: • Computation as an adaptation resource. We investigate iterative latent reasoning as an adaptation mechanism for frozen contrastive vision-language models, showing that repeated computation can provide useful task-specific refinement at fixed parameter count. • PERL: a recurrent lightweight adaptation mechanism for CLIP. We introduce a frozen-backbone adaptation framework in which a compact shared projector recurrently refines latent representations through dynamically generated reasoning tokens. • Strong parameter-performance trade-offs under constrained adaptation. Under a single-epoch few-shot adaptation protocol, PERL

Read the complete text