Whispering to a Blackbox: Bootstrapping Frozen OCR with Visual Prompts

Samandar Samandarov, Nazirjon Ismoiljonov, Abdullah Sattorov, Temirlan Sabyrbayev · 2026

Read Whispering to a Blackbox: Bootstrapping Frozen OCR with Visual Prompts by Samandar Samandarov, Nazirjon Ismoiljonov, Abdullah Sattorov, Temirlan Sabyrbayev (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-05276.txt.

Introduction

Samandar Samandarov and the rest of the team present Whisperer, visual prompting learning diffusion preprocessors adapting inputs in pixel space for frozen EasyOCR. Frozen models are stable and lean but lag on mismatched spreads, which Whisperer handles with input boosts. Training clones by-chance found gain policies. A four-stage course grows lucky gains from chance search of a partly trained model. On worn synthetic text images it cuts character error rate a lot, beating hand-made older models like CLAHE, without reinforcement learning pitfalls.

Catalog record

CREATORSamandar Samandarov, Nazirjon Ismoiljonov, Abdullah Sattorov, Temirlan Sabyrbayev
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE30531
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.05276

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Whispering to a Blackbox: Bootstrapping Frozen OCR with Visual Prompts Samandar Samandarov, Nazirjon Ismoiljonov, Abdullah Sattorov, Temirlan Sabyrbayev Abstract In the landscape of modern machine learning, frozen pre-trained models provide stability and efficiency but often underperform on specific tasks due to mismatched data distributions. This paper introduces the Whisperer, a novel visual prompting framework that learns diffusion-based preprocessors to adapt inputs in pixel space, effectively ”whispering” enhancements to frozen downstream models like EasyOCR. By framing the process as behavioral cloning of stochastically discovered improvement policies, our method achieves an 8% absolute (10.6% relative) reduction in Character Error Rate (CER) on a challenging dataset of 300k degraded synthetic text images, surpassing hand-engineered baselines such as CLAHE. The key innovation is a four-stage training curriculum that uses behavioral cloning to amplify ”lucky” improvements discovered through the stochastic exploration of a partially trained diffusion model. This approach is highly sample-efficient and avoids the pitfalls of traditional reinforcement learning. Crucially, we frame this not as naïve reinforcement learning, but as behavioral cloning of an exploration policy: we stochastically sample intermediate diffusion outputs, select those that improve CER by chance, and then train the model to reproduce them. This bootstrapping curriculum (4 stages over 60 GPU-hours) amplifies random successes into a systematic strategy. In summary, by whispering to the frozen OCR through its inputs, we improve an imperfect classifier without touching its weights. 1 Introduction 1.1 The Great Unfreezing The advent of large language models (LLMs) has fundamentally reshaped the landscape of artificial intelligence, introducing a paradigm where immense, pre-trained models can be adapted to a vast array of downstream tasks without altering their core parameters. This process, known as prompting , has emerged as the dominant method for eliciting desired behaviors from these ”frozen” models. By carefully crafting textual inputs, or prompts, users can guide the model’s internal representations to generate specific outputs, solve complex problems, or engage in nuanced dialogue. The power of this approach lies in its efficiency and accessibility; it circumvents the prohibitive computational cost and data requirements of fine-tuning, allowing for rapid iteration and deployment. For instance, appending ”Let’s think step by step” to queries has boosted reasoning in models like GPT-3 by up to 15% on arithmetic and commonsense benchmarks, without altering parameters. This simple yet profound intervention highlights a core principle: the knowledge and capabilities are already latent within the frozen model’s weights. The prompt acts not as a modification of the model itself, but as a key to unlock and direct its existing potential. This approach has proven to be incredibly efficient, circumventing the prohibitive computational and financial costs associated with fine-tuning models with billions of parameters. The prompt, in this context, functions as a librarian in a vast library of knowledge (the frozen model), guiding the model to the correct ”shelf” of information to answer a specific query. This method has been extended to more complex reasoning patterns, such as ”chain-of-thought” prompting , which encourages the model to break down problems into intermediate, sequential steps, further enhancing its problem-solving capabilities. This principle extends beyond simple phrases. Chain-of-thought prompting [ 18 ] elicits intermediate reasoning traces; in-context learning [ 3 ] embeds demonstrations within the prompt; soft prompting [ 10 ] learns continuous embeddings in the input space. All share a common thread: the frozen model’s weights are sacred, but its inputs are malleable . This paradigm shift has democratized access to state-of-the-art AI, allowing researchers and developers to leverage the power of massive models without the need for extensive retraining or specialized hardware. This evolution demonstrates a clear trajectory towards treating the input space as a primary locus/object of control, a principle that has been overwhelmingly validated in the domain of natural language processing. The success of this approach in NLP naturally raises a compelling question: can a similar paradigm be established for other modalities, such as computer vision, where models are also increasingly large, powerful, and often deployed as frozen, cloud-hosted APIs? 1.2 The Gap in Vision The temptation is to force an analogy: if text prompts operate on tokens, perhaps ”visual prompts” should operate on pixels. But the translation is non-trivial. In vision, the dominant adaptation paradigm remains fine-tuning [ 7 ] or, more recently, prompt tuning in embedding space —learned tokens prepended to patch embeddings. These methods require architectural access, gradient flow through the model’s early layers, and a willingness to modify the inference graph. They are not whispering to a black box; they are surgically altering its inner ear. For truly frozen models—production APIs like Google Vision AI, proprietary OCR systems, or legacy classifiers—such access is impossible. This mirrors the challenges that led to the rise of prompting in NLP. The alternative, until recently, has been a reliance on hand-engineered, deterministic preprocessing pipelines: comprising steps like denoising, sharpening, contrast enhancement, and binarization, which are designed based on human perceptual principles, with the goal of making an image ”cleaner” or ”easier to read” for a human observer. However, this approach fundamentally misunderstands the nature of the frozen vision model. The model does not ”see” the world as humans do; its feature extractors and internal representations are shaped by its own training data and architecture, leading to unique ”biases” and ”quirks.” A preprocessing step that enhances an image for human eyes might inadvertently destroy or distort critical features that the model relies on for accurate prediction. This creates a significant gap: we are preparing the data for a human interpreter, not for the specific, non-human ”language” of the frozen model. Our work begins with the recognition of this gap. We ask: Can we prompt a frozen OCR model in its native pixel space, without architectural access, and without the compute ruin of reinforcement learning? The answer, we discovered, requires rethinking what it means to ”prompt.” 1.3 Visual prompting To bridge the gap between human-centric preprocessing and the internal workings of a frozen vision model, we propose Visual Prompting not as a metaphor but as a formal optimization problem. Proposed concept extends the core principle of prompting from the text domain to the visual domain, but with a critical adaptation. In NLP, prompts operate in the discrete token space of the language model. In vision, the analogous interface is the continuous pixel space of the input image. Therefore, a visual prompt is not a textual instruction but a learned transformation applied directly to the input image’s pixels. The transformation is designed to maximize the performance of a frozen downstream model on a specific task, without modifying the model’s weights. This is fundamentally different from traditional transfer learning or adapter-based methods, which typically involve adding new layers or modules to the model architecture. Instead, visual prompting operates entirely in the input space, making it a highly flexible and broadly applicable technique. Interpretation: Given frozen model M: \mathbb{R}^{(H\times W\times C)}\rightarrow\mathbb{R}^{L} (where L is, e.g., the sequence length in OCR), we seek a preprocessor P_{\theta}:\mathbb{R}^{(H\times W\times C)}\rightarrow\mathbb{R}^{(H\times W\times C)} such that: \operatorname{argmax}_{\theta}\mathbb{E}_{x}[M(P_{\theta}(x))]\text{ subject to }||P_{\theta}(x)-x||_{\infty}<\varepsilon The constraint is critical: the prompt must be imperceptible to humans , operating in an L \infty ball of radius \epsilon ( \epsilon =0.1 in practice). This is not an adversarial attack, which seeks to fool the model; it is a constructive whisper

Read the complete text