Feature-specific predictive processing: What’s in a prediction error? Authors: David Richter, Cem Uran, Martin Vinck, Floris P. de Lange Published: 2025 ABSTRACT AbstractDespite numerous studies reporting sensory prediction errors—a key component of predictive processing theories—the nature of the surprise represented in these errors remains largely unknown. Here, we highlight recent studies, which provide evidence that prediction errors, even in early sensory areas, may reflect high-level surprise, offering new insights into the role of predictive processing in the brain beyond classical accounts of redundancy reduction. The brain has been described as a “prediction machine” that generates expectations about its environment to facilitate perception and decision-making (Friston, 2005; Rao & Ballard, 1999). Studies have demonstrated that neural responses are positively correlated to the amount of surprise that a stimulus elicits, both in the domain of sensory processing such as vision (Alink et al., 2010; den Ouden et al., 2010; Egner et al., 2010; Kaposvari et al., 2018; Kok et al., 2012; Meyer & Olson, 2011; Ramachandran et al., 2016; Richter et al., 2018) and audition (An et al., 2021; Gelens et al., 2024; Han et al., 2019; Lieder et al., 2013; Parras et al., 2017; Vidal et al., 2019), and beyond (Heilbron et al., 2022; Matsumoto et al., 2007; Oemisch et al., 2019); reviews (de Lange et al., 2018; Heilbron & Chait, 2018; Walsh et al., 2020). These studies conventionally compared neural responses between two conditions: predicted and surprising stimuli. In an experiment, these two categories can be well defined, for example, by presenting specific temporal sequences of stimuli that define predicted and surprising stimuli. Indeed, experiments often use simple artificial stimuli (e.g., gratings) and manipulate one specific stimulus feature (e.g., orientation) to create these two conditions (Furutachi et al., 2024). However, natural stimuli are complex constellations of many lower and higher-order features. Consequently, stimulus predictability naturally varies across many dimensions, which, we argue, is critical to investigate for charting the neurocomputational principles underlying perceptual inference. For example, if we move our eyes towards the location where we hear the honking sound of a car, we can strongly predict some features (e.g., specific rectilinear features and object size) while other features are completely unpredictable (e.g., the color of the car). In other words, a stimulus may be both predicted and surprising, depending on the specific features. In our opinion, this poses a fundamental question: What does the visual system predict? Instead of focusing solely on the magnitude of prediction error responses, new research has attempted to expose the contents of prediction errors at different levels of the cortical hierarchy, which can elucidate the brain’s internal models and thereby constrain models of predictive processing. We refer to this new paradigm as feature-specific predictive processing. Using convolutional neural networks (CNN), which have successfully been used to study the encoding of natural images by the visual ventral stream (Yamins et al., 2014), recent studies have started to investigate this by decomposing sensory surprise into lower- and higher-level visual features. First, Uran et al. (2022) explored how spatial predictability in natural images influences neural responses in macaque V1. Spatial predictability here referred to how well an image part could be predicted by its spatial surround, which was quantified using an inpainting algorithm. Then, they quantified the difference in CNN (Fig. 1A) activity patterns between the predicted and actual image patch, at low (early layers of the CNN, tuned to simple features like orientation and contrast) and high (late layers of the CNN, tuned to more complex features, such as objects and complex textures) levels, to determine low- and high-level predictability. They then investigated how low- and high-level predictability modulated distinct aspects of neural activity (Fig. 1B). V1 firing rates were weakly associated with low-level predictability but were strongly (and negatively) correlated with high-level predictability. (A) Object classification CNNs (VGG-16 in Uran et al. (2022), and AlexNet in Richter et al. (2024, 2025)) were used to quantify stimulus predictability at different feature levels. (B) Correlation of macaque V1 neural activity with CNN predictability. Late multi-unit firing rates showed a significant increase in correlation across layers with image unpredictability, while γ-power showed a significant decrease. Figure adapted from Uran et al. (2022). (C) Representational similarity analysis showed a gradient from low-level to high-level visual feature processing along the ventral visual hierarchy during prediction-free contexts (top panel), with early visual cortex (EVC) aligning most with early CNN layers (cold colors) and higher visual areas more with late CNN layers (warm colors). In contrast, prediction error magnitudes increased with high-level visual feature surprise across the visual system, including in EVC (bottom panel). Figure adapted from Richter et al. (2024). Second, Heilbron and de Lange (2025) used a similar analysis approach, investigating spatial predictability in mice. Results showed that visual cortical responses in mouse primary visual areas were again primarily modulated by high-level feature predictability. Moreover, this modulation by high-level predictability was the most pronounced in superficial layers of V1. Third, Richter et al. used fMRI (Richter et al., 2024) and EEG (Richter et al., 2025) in humans to explore how temporal predictability affects the neural response in the visual ventral stream. Participants learned to expect a specific image given a cue. Following learning, participants were occasionally presented with a different, surprising, image, which could be differentially surprising in terms of low-level and high-level features. As expected, the ‘bottom-up’ stimulus feature tuning profile adhered to the classical visual hierarchy: V1 was tuned for simple features, reflected in early layers of the CNN, while higher-order visual regions were tuned to more complex features, reflected in late layers of the CNN (Fig. 1C, upper panel). Interestingly however, all visual areas from early occipital to higher-order ventral regions, including V1, were upregulated as a function of high-level, but not low-level, surprise (Fig. 1C, bottom panel; Richter et al., 2024). Moreover, using EEG, modulations by high-level surprise were observed relatively early, within 200 ms following stimulus onset (Richter et al., 2025). We believe that despite differences in the type of predictions (spatial vs. temporal), populations studied (macaques, mice, humans), and recording methods (multi-unit, Neuropixel probes, fMRI, EEG), the above studies converge on a similar motif, namely that high-level visual surprise strongly drives neural activity, even in the earliest cortical visual areas, potentially as a result of feedback. While this convergence does not necessarily imply that these distinct manipulations and measurements isolate the exact same neurocomputational mechanism, it does suggest a shared operational principle, with related evidence showing that prediction errors in lower macaque face area ML inherit higher-level identity properties after learning (Schwiedrzik & Freiwald, 2017). Next, we discuss how these findings align with several models of predictive processing. (1) Hierarchical predictive coding (HPC). HPC is a major model of cortical function that posits that feedback signals predictions about neural representations in the lower hierarchical level, while feedforward connections convey prediction errors arising from the comparison between top-down predictions and local representations (Rao & Ballard, 1999). Arguably, HPC thereby entails that higher areas signal prediction errors about higher-order features, while prediction errors about lower-order features are signaled in lower areas, because each area generates errors relative to predictions from the next hierarchical level. Thus, we argue that the findings of Uran et al. (2022), Heilbron and de Lange (2025), and Richter et al. (2024) that V1 activity increases for high- rather than low-level surprise may be difficult to reconcile with classic HPC implementations that depend on strict hierarchical error representation. In sum, HPC seems less consistent with the present evidence, given the non-hierarchical error representation evident by increased V1 activity for high-level surprise. (2) Feedback propagation of error signals. An alternative to classic HPC is that error signals are generated in higher hierarchical levels and transmitted to lower levels via feedback connections (Heeger, 2017; also see: Whittington & Bogacz, 2019), where they lead to increased neural activity. V1 receives feedback from many higher-order sensory areas, allowing a direct modulation by top-down signals from many areas (Vezoli et al., 2021). In this scenario, feedback could serve to increase attention (Alink & Blank, 2021), update synaptic weights, aid in learning and novelty detection (Doron et al., 2020), and recruit additional neural resources when surprising events are encountered. This class of models would predict that V1 activity increases for high-level surprise, which we believe is consistent with the observations of the studies summarized above (Heilbron & de Lange, 2025; Richter et al., 2024; Uran et al., 2022). In this case, error signals may act as a scalar surprise signal that gain-modulates V1 responses, rather than subtract specific predictions (Furutachi et al., 2024). We believe that this account provides a parsimonious explanation of the combined findings (Heilbron & de Lange, 2025; Richter et al., 2024, 2025; Uran et al., 2022), as it naturally accounts for V1 upregulations by high-level surprise via feedback. This account is also compatible with ‘opposing process theory’, as proposed by Press et al. (2020), particularly with the feedback-based upweighting of surprising events, following the initial identification of the stimulus. (3) V1 as a comparator circuit for higher-level features. Another possibility, which we consider to be in line with these recent findings, is that low-level areas like V1 may act as a comparator circuit for higher-order features. Local features can provide evidence for objects, and object recognition can be approximated by globally averaging over many local evidence filters (Farahat et al., 2023). For example, a local patch of yellow in a V1 receptive field provides some evidence for a banana, a sun, or an autumn leave, and if feedback into V1 signals the prediction of a blueberry, a higher-order prediction error could result when a local feature is not compatible with the high-level prediction. Thereby, this account explains how high-level error information is reflected in V1, but in contrast to (2) entails a prediction, instead of a scalar-like error signal being relayed via feedback. (4) Dendritic HPC. Interestingly, the study of Uran et al. (2022) showed that while high-level predictability modulated V1 firing rates, low-level prediction errors primarily affected 30–80 Hz oscillatory, narrow-band (not broadband) gamma synchronization within V1. We argue that the dissociation between firing rates and gamma-synchronization fits well within the recently proposed dendritic HPC (dHPC) model (Aizenbud et al., 2025; Mikulasch et al., 2023; Vinck et al., 2025). The dHPC model implements a mathematically equivalent algorithm compared to classic HPC; however, dHPC maps the variables to different neural elements (Aizenbud et al., 2025; Mikulasch et al., 2023; Vinck et al., 2025). In dHPC, the basal dendritic compartment encodes feedforward prediction errors, implemented as interaction between excitatory feedforward inputs and local lateral inhibition. The apical compartment encodes the error of representations in level k with respect to predictions from level k+1, implemented as difference between top-down excitatory inputs and membrane leak currents. Importantly, dHPC entails that low-level predictability gives rise to a tighter balance of excitation and inhibition state, which is known to promote oscillatory gamma-synchronization (Denève & Machens, 2016; Vinck et al., 2025). Thus, we propose that V1 gamma-synchronization reflects the extent to which superficial-layer V1 representations can predict afferent excitatory inputs. The increase in V1 firing rates with higher-order predictability, on the other hand, should reflect the influence of prediction errors on the apical dendritic compartment, for example, via plateau potentials. Thereby, dHPC provides a plausible account of the combined findings (Heilbron & de Lange, 2025; Richter et al., 2024, 2025; Uran et al., 2022), linking low-level predictability to narrow-band gamma synchronization and high-level feedback to increased V1 firing rates. Regardless of these different models, it is important that anatomical findings are in line with the notion that V1 can encode prediction errors about high-level features: V1 receives feedback from many higher-order areas (Vezoli et al., 2021), such that feedback concerning high-level features and error signals may reach it via direct projections. The fact that V1 receives feedback from many hierarchical levels, rather than only from the next level, as assumed in some predictive coding models, in our opinion provides a plausible anatomical basis for the finding that V1 activity signals mismatch signals about high-level features. Does this feature-specific predictive processing strategy make sense from a behavioral standpoint? As the brain may be a “predictive machine” that continuously refines its internal models, a key question is: What are these predictions about? In principle, the brain could predict across all hierarchical levels, including fine-grained details of its sensory input, down to the level of pixels—as classic HPC models may suggest. However, higher-level features are likely more behaviorally relevant and may generally be easier to predict than lower-level features (e.g., rain vs individual drops of rain). Indeed, spatiotemporal prediction tasks like predicting future frames can be learned more effectively at the level of higher-order features rather than at the pixel level (Luc et al., 2017), and high-level predictions may be particularly effective for self-supervised learning of visual representations (Heilbron & de Lange, 2025). Finally, high-level stimulus predictability shows stronger correlations with human perceptual similarity judgements and salience maps (Uran et al., 2022). Thus, learning and behavior may be primarily guided by high-level predictability, and therefore we argue that perhaps the brain’s “prediction game” focuses on high-level, not low-level features in its sensory input. Finally, considerable evidence supports that surprise increases sensory responses (reviews: de Lange et al., 2018; Heilbron & Chait, 2018; Walsh et al., 2020), yet findings are mixed with other studies showing limited or no evidence for such surprise scaling (Den Ouden et al., 2023, 2025; Solomon et al., 2021; Vinken et al., 2018) (review: Feuerriegel et al., 2021). While further work is required to delineate under which circumstances surprise may fail to scale sensory responses (e.g., differences in recording methods and paradigms), the hypothesized emphasis on high-level predictability may be part of the explanation. Contrasts between expected and unexpected inputs with limited variance in high-level features may yield little evidence for sensory responses being driven by surprise, especially when predictions are task-irrelevant and attention is drawn away from the predictable input (Richter & de Lange, 2019). Accordingly, decomposing surprise across levels of abstraction may increase sensitivity by aligning analyses with the features that may be particularly predicted by the sensory brain. Together, these studies demonstrate what kind of predictions the brain makes in richer, more naturalistic, environments. Future challenges include: (1) extending feature-specific prediction error analyses to other sensory modalities, especially audition; (2) charting boundary conditions under which the predominant source of prediction may shift or span across levels of abstraction; (3) determining potential contributions of cortical arousal and pupil dilation associated with high-level surprise to early cortical responses; and (4) better integrating relevant biological implementation principles (e.g. dendritic HPC) with algorithmic and computational goals to generate a unified theory of predictive processing that spans across all of Marr’s levels of analysis.