EVA: Bridging Performance and Human Alignment in Hard-Attention Vision Models for Image Classification
Pengcheng Pan, Yonekura Shogo, Kuniyoshi Yasuo · 2026
Read EVA: Bridging Performance and Human Alignment in Hard-Attention Vision Models for Image Classification by Pengcheng Pan, Yonekura Shogo, Kuniyoshi Yasuo (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-27340.txt.
Introduction
Pan and co-authors introduce EVA, a hard-attention vision model that trades accuracy against human-like looking. Pure accuracy training can push scanpaths away from human patterns. EVA samples a few sequential glimpses through a small fovea-periphery representation. A CNN feature extractor, variance control, and adaptive gating steer those glimpses. On CIFAR-10, scanpath alignment improved while accuracy stayed competitive. Ablations credit the CNN for accuracy and the controls for human-likeness. ImageNet-100 and COCO-Search18 tests confirm the pattern at scale.
Catalog record
| CREATOR | Pengcheng Pan, Yonekura Shogo, Kuniyoshi Yasuo |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 63525 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.27340
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
EVA: Bridging Performance and Human Alignment in Hard-Attention Vision Models for Image Classification Pengcheng Pan, Yonekura Shogo, Kuniyoshi Yasuo Abstract Optimizing vision models purely for classification accuracy can impose an alignment tax, degrading human-like scanpaths and limiting interpretability. We introduce EVA, a neuroscience-inspired hard-attention mechanistic testbed that makes the performance–human-likeness trade-off explicit and adjustable. EVA samples a small number of sequential glimpses using a minimal fovea–periphery representation with CNN-based feature extractor and integrates variance control and adaptive gating to stabilize and regulate attention dynamics. EVA is trained with the standard classification objective without gaze supervision. On CIFAR-10 with dense human gaze annotations, EVA improves scanpath alignment under established metrics such as DTW, NSS, while maintaining competitive accuracy. Ablations show that CNN-based feature extraction drives accuracy but suppresses human-likeness, whereas variance control and gating restore human-aligned trajectories with minimal performance loss. We further validate EVA’s scalability on ImageNet-100 and evaluate scanpath alignment on COCO-Search18 without COCO-Search18 gaze supervision or finetuning, where EVA yields human-like scanpaths on natural scenes without additional training. Overall, EVA provides a principled framework for trustworthy, human-interpretable active vision. 1 Introduction Deploying vision systems in real-world settings requires not only strong task performance but also interpretability and reliability . When humans cannot anticipate why a model reaches a decision, trust and safe integration into high-stakes workflows remain limited. Popular post-hoc explanation tools such as saliency maps and Grad-CAM for CNNs and Vision Transformers can be brittle and occasionally misleading, raising concerns about whether they reflect the model’s actual reasoning process 14 ; 33 ; 27 ; 29 . This motivates a complementary direction: designing models whose internal decision process is inspectable by construction, rather than explained after the fact. Human vision provides a natural blueprint for process-level interpretability. Due to biological constraints, perception is selective. Humans cannot process the entire visual field at high resolution simultaneously. Instead, they actively sample evidence through sequences of saccades and fixations, repositioning the fovea to gather informative glimpses 10 ; 38 ; 26 . Hard-attention models follow this principle by selecting discrete glimpses over time and learning fixation policies, often using reinforcement learning 18 ; 32 . Their trajectories, or scanpaths, form an auditable record of what evidence was requested to support a decision. This makes scanpath behavior a natural axis for interpretability. Modern vision research, however, predominantly optimizes for classification accuracy under full-image, parallel computation. In practice, this can impose an alignment tax . As accuracy improves, scanpath behavior often becomes less human-like, weakening the interpretability promise of active sampling. Rather than treating this trade-off as inevitable, we argue that the performance and human-likeness tension should be treated as a first-class design variable and studied mechanistically. We evaluate scanpath alignment by comparing model fixations to human gaze statistics under established metrics such as NSS and DTW. We then ask which architectural mechanisms can reduce the alignment tax under the standard classification objective. Figure 1 : EVA reduces the alignment tax through mechanism-aware design. Left : a schematic trade-off between accuracy and scanpath similarity to humans. EVA improves the trade-off relative to full-image baselines and prior hard-attention models. Right : EVA uses minimal fovea–periphery sensing with variance control and adaptive gating to regulate active sampling dynamics. We introduce EVA , a lightweight neuroscience-inspired hard-attention mechanistic testbed trained solely with classification labels. EVA is designed to make the alignment tax measurable and adjustable through architectural constraints, while keeping the training objective unchanged. EVA operates with a small number of sequential glimpse requests and combines a minimal fovea–periphery glimpse representation with a CNN-based feature extractor, a neuromodulator-inspired variance control mechanism that stabilizes stochastic attention under uncertainty, and an adaptive gating pathway that regulates information flow. Our goal is not biological fidelity. Instead, these components impose functional priors on how the model acquires evidence over time. We adopt a minimal two-level fovea–periphery design to maintain efficiency and to avoid confounds introduced by complex multi-scale designs 21 . Our evaluation is structured to support mechanistic analysis, scalability testing, and cross-task transfer. On CIFAR-10 , where dense human gaze annotations enable rigorous analysis, we conduct mechanistic ablations to decouple the sources of performance and alignment. We find that stronger CNN-based feature extraction primarily drives accuracy but tends to suppress scanpath human-likeness. In contrast, variance control and gating substantially restore human-aligned fixation trajectories with minimal performance loss. We then validate scalability on ImageNet-100 . Finally, we evaluate scanpath alignment on COCO-Search18 without using its gaze annotations or search supervision. We train EVA on COCO image classification using labels only, optimize the attention policy solely via classification reward, and compare the resulting scanpaths to human search-conditioned scanpaths on COCO-Search18 without finetuning. This setting tests whether an attention policy learned from classification reward can transfer as a human-aligned evidence acquisition strategy in a different behavioral regime. Contributions. First, we formalize and empirically characterize an alignment tax in hard-attention vision. Accuracy-oriented improvements can systematically degrade scanpath human-likeness. Second, we introduce EVA as a mechanistic testbed that treats the performance and human-likeness trade-off as a first-class variable. EVA uses minimal fovea–periphery sensing together with variance control and adaptive gating as interpretable architectural mechanisms. Third, through ablations on CIFAR-10 and evaluations on ImageNet-100 and COCO-Search18, we show that EVA improves scanpath alignment without changing the standard training objective or using gaze supervision. EVA also scales to larger data and supports gaze-supervision-free cross-task scanpath evaluation. By aligning active sampling behavior with human gaze statistics, EVA enables a practical form of interpretability. Users can inspect where the model requested evidence and relate decisions to a human-understandable sequence of glimpses, supporting trustworthy and potentially human-in-the-loop active vision systems. 2 Related Work 2.1 Neuroscience-inspired Vision and Hard Attention Deep CNNs trained for object recognition have been used as functional models of the primate ventral stream, with empirical correspondences to neural responses and benchmarks that quantify neural alignment 17 ; 34 ; 28 . These models capture hierarchical feature processing but do not directly model active evidence acquisition through eye movements. Biological vision relies on recurrent loops and routing mechanisms that modulate information flow during perception, including attentional gating in thalamocortical circuits 12 ; 24 ; 2 ; 3 . Hard-attention models move toward this active-sampling view by making decisions from a sequence of glimpses selected by an attention policy. RAM learns where to look using reinforcement learning signals such as REINFORCE, which introduces high-variance optimization challenges 18 ; 32 . Subsequent work improved representation learning and training strategy to narrow the accuracy gap on more complex benchmarks, including designs that use multi-scale glimpses or stronger feature extractors and systems that scale to ImageNet with more effective training 1 ; 11 . Related active-vision efforts also study uncertainty-driven exploration and multi-level controllers that separate perception and action 30 ; 22 . EVA follows this line of work by introducing neuroscience-inspired components into a hard-attention system, not to claim biological fidelity, but to impose functional priors on how the model acquires evidence over time. 2.2 Human Gaze and Scanpath Alignment Human gaze datasets enable evaluation of where and how models allocate attention. Saliency models and deep saliency networks predict fixation distributions for free viewing, but they often output static maps rather than sequential scanpaths 13 ; 25 . With