Spatiotemporal hierarchies of face representation in the human ventral temporal cortex Authors: Sina Salehi, Mohammad Reza A. Dehaqani, Jessica Schrouff, Clara Sava-Segal, Omri Raccah, Sori Baek Published: 2024 ABSTRACT In this study, we examined the relatively unexplored realm of face perception, investigating the activities within human brain face-selective regions during the observation of faces at both subordinate and superordinate levels. We recorded intracranial EEG signals from the ventral temporal cortex in neurosurgical patients implanted with subdural electrodes during viewing of face subcategories (human, mammal, bird, and marine faces) as well as various non-face control stimuli. The results revealed a noteworthy correlation in response patterns across all face-selective areas in the ventral temporal cortex, not only within the same face category but also extending to different face categories. Intriguingly, we observed a systematic decrease in response correlation coupled with an increased response onset time from human face to mammalian face, bird face and marine faces. Our result aligns with the notion that distinctions at the basic level category (e.g., human face versus non-human face) emerges earlier than those at the superordinate level (e.g., animate versus inanimate). This indicates response gradient in the representation of facial images within human face-sensitive regions, transitioning progressively from human faces to non-face stimuli. Our findings provide insights into spatiotemporal dynamic of face representations which varies spatially and at different timescales depending on the face subcategory represented.Supplementary InformationThe online version contains supplementary material available at 10.1038/s41598-024-77895-5. Introduction Humans possess the remarkable ability to recognize objects at different levels of abstraction. This allows us to categorize objects as belonging to broad categories at the superordinate level, such as “living things” (e.g., face, body parts) or “non-living things” (e.g., tools, furniture). At the subordinate level, we can further differentiate within categories, such as discerning between human and primate faces. While much research has focused on object/category representation at the superordinate level, relatively little is known about how we represent objects at the finer-grained subordinate level. Studies employing neuroimaging1 and reversible inactivation techniques2 have demonstrated a topographical organization of object category representations across the inferior temporal cortex (IT). Within object-selective regions, responses to the preferred categories show greater similarity than to non-preferred categories. For instance, face-selective regions exhibit highly correlated responses to face stimuli3. Furthermore, within face-selective regions, individual face-selective neurons response has been shown to be modulated by face subcategories, such as own-race/species versus other-race/species faces4–6. The ventral temporal cortex (VTC) represents conceptual, abstract similarity more prominently than physical similarity based on low-level features7. Prior studies have proposed a continuous representational space for objects in the ventral visual pathway, organized along an animacy scale with animate objects from humans and primates at the top to insects at the bottom, and inanimate objects at the very bottom of this hierarchy8,9. This organization suggests that both perception and neural representation of complex visual stimuli in the VTC occur at different levels of abstraction10 ranging from superordinate level categories (animate versus inanimate)11–13, to basic level categories (faces versus bodies)14–16, and subordinate level categories (human faces versus animal faces.) These complex representations may be explained by the existence of fine functional subregions within the larger cortical regions traditionally thought to represent a single domain (i.e., Fusiform Face Area (FFA)). This suggests that representations operate at different spatial scales ranging from large to fine cortical surfaces1,2,10,17,18. However, technical limitations of fMRI have hampered our ability to investigate these fine spatial representations. Evidence also suggests that representations of visual categories at different levels of abstraction evolve on varying timescales19. Inferior temporal (IT) neurons represent basic level categories earlier than superordinate and subordinate level categories20,21, and their response heterogeneity dynamically increases from early to late phases22. Therefore, the combination of spatial distinctions, temporal dynamics, and interactions across the VTC might be responsible for the evolving representations of complex abstract information. Despite existing knowledge, several gaps remain in our understanding of how we represent objects at the subordinate level. Most studies have focused on the representational code of superordinate categories (face vs. non-face), neglecting very fine-grained animate categories like faces at the subordinate level (e.g., different face species). In our previous work, we demonstrated that face specific information is spatiotemporally distributed, such that responses to faces were significantly faster in posterior sites than in anterior face-selective sites while face selectivity is significantly higher in the anterior sites than in posterior face-selective sites23. In this study, we Used the same dataset of the high temporal and spatial resolution of intracranial EEG (iEEG) by recording across a large area of the human VTC while human subjects viewed numerous stimuli of faces in 4 different subcategories (human, mammal, bird, and marine faces) as well as various non-face stimuli. Our research aims to address these gaps by investigating the spatiotemporal dynamics of category representation at different levels of abstraction. Our findings reveal that the pattern of activity across all face-selective sites in VTC was correlated in response to face stimuli. This correlation was preserved not only for members of the same face species but also for members of other face species categories, enabling clustering of face subcategories based on human VTC responses. While hierarchical representation of objects has been previously demonstrated16,24, we show that this hierarchy also exists in face representation at the subordinate level. Specifically, we demonstrate that response magnitude, spatial pattern of response, and response timing were graded within face-selective regions depending on the face subcategory being represented. Our results suggest that neural responses underlying face representation may exist on a continuum but operate at distinct spatiotemporal scales. Results We recorded iEEG data from subdural electrodes in 8 subjects undergoing epileptic source localization for intractable epilepsy. Five subjects had electrodes implanted in the right hemisphere, while three had implants in the left hemisphere. Figure S1A illustrates electrode coverage across all subjects (n = 357) over the VTC. Out of 357 cortical sites in 8 patients, 145 were visually active (non-selective) and 45 (12%) exhibited face-selective responses (Fig. 1A). Figure 1 shows the average response of face-selective (1 C) and visually active (S1B) VTC sites to face, body and place images. Face subcategories elicited different response profiles. mean responses to human faces were significantly higher than mammal, bird, and marine face categories (human-face = 5.33 ± 3, mammal-face = 4.0 ± 1.97, bird-face = 3.53 ± 1.67, marine-face = 3.65 ± 1.53; human vs. other face p < 0.001). Mean responses to mammal faces were significantly higher than to bird and marine faces (p < 0.01). Among face-selective sites, mean d-prime was 1.4 ± 0.6 for all face vs. non-face stimuli. Mean d-prime for face subcategories vs. non-face were 1.8 for human, 0.95 for mammal, 0.73 for bird and 0.71 for marine face (Fig. 1D). Different face categories also evoked responses with varying onset latencies (Fig. 1E). Mean onset latency was 117 ms for human faces, 132 ms for mammal faces, 154ms for bird faces and 150 ms for marine faces. We observed a systematic difference in evoked response and onset latency between human faces and those of mammals, birds, and marine animals. Human faces elicited the largest responses in face-selective sites, followed by mammal, bird, and marine faces. Response onsets to human faces were significantly shorter than to other face categories (p < 0.001). Although the response onset to mammal faces was shorter than that to bird and marine faces, the difference was not statistically significant (p = 0.1). No significant difference was found between the response onsets to bird and marine faces. These face-selective sites showed a remarkable mid-level preference to all types of faces (relative to other object categories), despite the visual distinctiveness of the face categories (e.g., human versus bird faces). It’s worth noted that in our previous work23 we demonstrated that signal amplitude does not affect our method of response onset calculation. Fig. 1Face selectivity and responses in human ventral temporal cortex (A) Spatial distribution of iEEG face selectivity in human ventral temporal cortex. Each electrode on the cortex is represented by a pie chart showing the relative averaged amplitude elicited by each face category. (B) Experiment design. (C) Peri-stimulus time histogram of average normalized HFB response. Each trace corresponds to the average response of one stimulus category (mean ± SEM). (D) Distribution of d-prime (face vs. non-face) values for all individual face-selective sites. (E) Distribution of response onset of all face-selective sites to different face categories. Horizontal lines in each violin plot indicate the mean of the distributions, white disk is median and the gray bar in each violin indicate 25% and 75% quartile of each data. Face selectivity and responses in human ventral temporal cortex (A) Spatial distribution of iEEG face selectivity in human ventral temporal cortex. Each electrode on the cortex is represented by a pie chart showing the relative averaged amplitude elicited by each face category. (B) Experiment design. (C) Peri-stimulus time histogram of average normalized HFB response. Each trace corresponds to the average response of one stimulus category (mean ± SEM). (D) Distribution of d-prime (face vs. non-face) values for all individual face-selective sites. (E) Distribution of response onset of all face-selective sites to different face categories. Horizontal lines in each violin plot indicate the mean of the distributions, white disk is median and the gray bar in each violin indicate 25% and 75% quartile of each data. Despite the difference observed between the response profiles of face subcategories in face-selective sites, the pattern of response activity to our four face subcategory stimuli was correlated. Using representational similarity analysis, we demonstrated that the responses of face-selective sites to the four face subcategories were more alike than the responses of these sites to the other non-face categories of stimuli at the group-level (Fig. 2A). We calculated Category boundary effect (CBE) to find any category distinction in the cortical representation of categories at super and subordinate levels. We confirmed that the face vs. non-face CBE was significant for face-selective sites (%39.3, p = 9.99e–5), but not for visually active sites (%2.5, p = 0.09). In other words, in face-selective sites, responses to different face stimuli were more similar than did pairs of face and non-face stimuli. In face-selective sites, this effect was also significant for each of the face subcategories separately (Table 1) and could not be explained by the physical similarity/dissimilarity between the images alone (Fig. S2). In the face-selective sites, we observed the mean correlational neural distance gradually increased between the pairs of human/mammal, human/bird, and human/marine faces (mean distance ± STD: human/human = %2 ± 3.7, p = 9.99e-5, n = 300, human/mammal = %12.6 ± 15, p = 9.99e–5, n = 625, human/bird = 19 ± 18.2, p = 9.99e–5, n = 625 and human/marine=%30.7 ± 25, p = 4.9e–4, n = 625, Fig. 2B). These observations suggest that faces, in general, elicit a similar pattern of neuronal activity in the group of face-selective cortical sites despite the diverse physical appearances of faces from different subcategories. Similarity in response patterns across face subcategories indicates a shared neural substrate for representation for faces, regardless of their specific species. Fig. 2Species effect on face response at multisite (group) level. (A) Representational dissimilarity matrix calculated using correlation neural distance and (B) Averaged dissimilarity across categories, similarity values are written inside their corresponding cells. Color indicator around the RDM specify the category, pink = human face, blue = mamal face, green = bird face and green = marine face. Colorbar under the RDM shows the mapping of similarity value to colors in the RDM. In figure A, each pixel corresponds to a similarity value between two individual stimuli and in figure B, each pixel corresponds to a similarity value between two categories. (C) Hierarchical clustering reconstructed from neural response of face-selective sites. (D) Degrees of match between categories and nodes in the clustering tree. The violin plot illustrates the degrees of match between categories and nodes in the clustering tree, measured by the clustering index. The boxplot displays clustering scores derived from shuffled data, while the error bars indicate 25% and 75% quartiles of the shuffled data represented in the boxplot. (E) Multi-Dimensional Scaling (MDS) based on the correlation matrices for face selective sites. Species effect on face response at multisite (group) level. (A) Representational dissimilarity matrix calculated using correlation neural distance and (B) Averaged dissimilarity across categories, similarity values are written inside their corresponding cells. Color indicator around the RDM specify the category, pink = human face, blue = mamal face, green = bird face and green = marine face. Colorbar under the RDM shows the mapping of similarity value to colors in the RDM. In figure A, each pixel corresponds to a similarity value between two individual stimuli and in figure B, each pixel corresponds to a similarity value between two categories. (C) Hierarchical clustering reconstructed from neural response of face-selective sites. (D) Degrees of match between categories and nodes in the clustering tree. The violin plot illustrates the degrees of match between categories and nodes in the clustering tree, measured by the clustering index. The boxplot displays clustering scores derived from shuffled data, while the error bars indicate 25% and 75% quartiles of the shuffled data represented in the boxplot. (E) Multi-Dimensional Scaling (MDS) based on the correlation matrices for face selective sites. Table 1Average distance between face categories.Distance in % (p-value)Category Boundary Effect in % (p-value)Animate/inAnimate–7 (p = 9.9e-05)All-face/Non-face–39.2 (p = 9.9e-05)Human-face/Non-face20.0 (p = 9.9e-05)70.0 (p = 9.9e-05)Mammal-face/Human-face12.6 (p = 9.9e-05)10.6 (p = 9.9e-05)Bird-face/Human-face19.0 (p = 9.9e-05)17 (p = 9.9e-05)Marine-face/Human-face30.7 (p = 4.9e-04)26.6 (p = 9.9e-05)Mammal-face/Non-face12.7 (p = 9.9e-05)43.2 (p = 9.9e-05)Mammal-face/Bird-face16.3 (p = 9.9e-05) 3.5 (p = 0.08) Mammal-face/Marine-face20.9 (p = 9.9e-05)8 (p = 2.9e-04)Bird-face/Non-face17.0 (p = 9.9e-05)36.3 (p = 9.9e-05)Bird-face/Marine-face22.0 (p = 9.9e-05)5 (p = 0.012)Marine-face/Non-face21.2 (p = 9.9e-05)29.5 (p = 9.9e-05)Non Significant values are in [bold].Correlational neural distance and category boundary effect for each binary comparison of face categories, with their respective p-values (FDR corrected). Table 1Average distance between face categories.Distance in % (p-value)Category Boundary Effect in % (p-value)Animate/inAnimate–7 (p = 9.9e-05)All-face/Non-face–39.2 (p = 9.9e-05)Human-face/Non-face20.0 (p = 9.9e-05)70.0 (p = 9.9e-05)Mammal-face/Human-face12.6 (p = 9.9e-05)10.6 (p = 9.9e-05)Bird-face/Human-face19.0 (p = 9.9e-05)17 (p = 9.9e-05)Marine-face/Human-face30.7 (p = 4.9e-04)26.6 (p = 9.9e-05)Mammal-face/Non-face12.7 (p = 9.9e-05)43.2 (p = 9.9e-05)Mammal-face/Bird-face16.3 (p = 9.9e-05) 3.5 (p = 0.08) Mammal-face/Marine-face20.9 (p = 9.9e-05)8 (p = 2.9e-04)Bird-face/Non-face17.0 (p = 9.9e-05)36.3 (p = 9.9e-05)Bird-face/Marine-face22.0 (p = 9.9e-05)5 (p = 0.012)Marine-face/Non-face21.2 (p = 9.9e-05)29.5 (p = 9.9e-05)Non Significant values are in [bold].Correlational neural distance and category boundary effect for each binary comparison of face categories, with their respective p-values (FDR corrected). Average distance between face categories. Non Significant values are in [bold]. Correlational neural distance and category boundary effect for each binary comparison of face categories, with their respective p-values (FDR corrected). To further examine the category structure in neural response, we conducted a cluster analysis of the neural distances in face-selective sites. Hierarchical clustering analysis revealed categorical distinctions between faces and other categories (Fig. 2C). Notably, human faces could be separated from the other face subcategories, and closer examination suggested some distinctions between the four face subcategories. To quantify the representations of face categories in the clustering tree, we used a clustering score to determine whether categories had significantly corresponding nodes in the hierarchical clustering tree. The face category (100 members) and all 4 face subcategories had significant matching nodes (Fig. 2D; Table 2). The clustering score for human faces was significantly greater than the other face subcategories (p < 0.001, human: 0.66 ± 0.044; mammal: 0.57 ± 0.023; bird: 0.56 ± 0.023; and marine: 0.54 ± 0.045). The boxplots in Fig. 2D represent clustering scores from shuffled data. Variations in the shuffled data arise due to biases introduced by the differing number of stimuli across categories and the way the clustering score is defined. Furthermore, Multidimensional Scaling (MDS) plots showed that different face categories are closer together in the MDS space for face-selective sites. These results provide evidence that the similarity of group level response pattern could reconstruct the individual face categories (Fig. 2E). This pattern was specific to face-selective sites and the visually active sites exhibited no pattern of categorical face representations (Fig. S3). It is important to note that this was an unsupervised analysis, and no prior assumption is implied and none of these results were driven by low level features of the stimuli (e.g., frequency information or luminance, see Fig. S2). Table 2Degrees of match between categories and nodes in the hierarchical clustering tree, reconstructed from neural responses.CategoryScoreChanceNo. of stimHuman0.663 ± 0.044***0.467 ± 0.07525Mammal0.573 ± 0.023***0.465 ± 0.07725Bird0.563 ± 0.023***0.464 ± 0.07925Marine0.538 ± 0.045*0.47 ± 0.07525Face0.812 ± 0.032*0.587 ± 0.069100Non-face0.743 ± 0.055***0.677 ± 0.078150Score with the asterisk indicating that the value is significantly larger than the chance. Significance was computed against chance value calculated from random clustering of stimuli by 95% (*) and 99% (***) confidence interval. Table 2Degrees of match between categories and nodes in the hierarchical clustering tree, reconstructed from neural responses.CategoryScoreChanceNo. of stimHuman0.663 ± 0.044***0.467 ± 0.07525Mammal0.573 ± 0.023***0.465 ± 0.07725Bird0.563 ± 0.023***0.464 ± 0.07925Marine0.538 ± 0.045*0.47 ± 0.07525Face0.812 ± 0.032*0.587 ± 0.069100Non-face0.743 ± 0.055***0.677 ± 0.078150Score with the asterisk indicating that the value is significantly larger than the chance. Significance was computed against chance value calculated from random clustering of stimuli by 95% (*) and 99% (***) confidence interval. Degrees of match between categories and nodes in the hierarchical clustering tree, reconstructed from neural responses. Score with the asterisk indicating that the value is significantly larger than the chance. Significance was computed against chance value calculated from random clustering of stimuli by 95% (*) and 99% (***) confidence interval. Fig. 3Time course of face subcategory separability. Time course of the separability index (SI) for the face subcategories versus in-animal stimuli. Shaded areas represent SD, calculated using the bootstrap procedure. The surrounding scatter plots are 3 snapshots at 90, 150 and 350ms time points correspond to human, mammal, bird, and marine faces. Time course of face subcategory separability. Time course of the separability index (SI) for the face subcategories versus in-animal stimuli. Shaded areas represent SD, calculated using the bootstrap procedure. The surrounding scatter plots are 3 snapshots at 90, 150 and 350ms time points correspond to human, mammal, bird, and marine faces. We used separability index in the principal component space (SI; see “Materials and methods”) to measure the strength of face subcategory representation. A higher value of the SI indicates that the tested category had larger between-category distances and smaller within-category scatter. Figure 3 shows the time course of SI for different face subcategories versus in-animate stimuli. SI reached significant values for all face subcategories at some point after the stimulus onset. We observed the onset of SI for human faces significantly earlier than the other faces with marine face was the latest (bootstrap test, p < 0.001; 83 ± 4; mammal: 152 ± 18; bird: 156 ± 11; marine:154 ± 25). We then divided human faces into familiar and non-familiar based on whether the faces were famous (ex: Jennifer Aniston) or not and it was observed that familiar faces had significantly earlier responses (familiar human: 84 ± 5; non-familiar human: 95 ± 7; p < 0.05, bootstrap test). These results emphasized the systematic time difference for representations of face subcategories. As the SI index has no theoretical upper bound, it enabled us to compare the strength of category representations without concerns about ceiling effects. We observed significant segregation of face subcategory representations across different time bins. For instance, at 250 ms after stimulus onset, the SI values were Human: 8.8 ± 1.9; mammal: 2 ± 0.5; bird: 1.5 ± 0.36; marine: 1.03 ± 0.23; familiar human: 7.6 ± 2; and non-familiar human: 4 ± 1.3 (p < 0.001, bootstrap test). To visualize the temporal evolution of face category representations at the group level, we reconstructed 2D representations of stimuli using the first two PCA components computed from the group of VTC site responses. Figure 3 displays three snapshots of stimulus arrangement at 90, 150, and 350 ms, with corresponding SI values included in Table 3. The first two dimensions of the PCA captured 54% of the variance. Table 3The SI values for different face subcategories at three time points:t = 90t = 150t = 350Human1.9 ± 0.46 ± 1.16.4 ± 1.6Mammal0.2 ± 0.10.5 ± 0.22.5 ± 0.6Bird0.2 ± 0.10.4 ± 0.11.6 ± 0.4Marine0.2 ± 0.10.3 ± 0.11.4 ± 0.4Familiar human1.6 ± 0.45.1 ± 1.35.6 ± 1.6Non-familiar human1.3 ± 0.43.5 ± 1.13.2 ± 1.1 Table 3The SI values for different face subcategories at three time points:t = 90t = 150t = 350Human1.9 ± 0.46 ± 1.16.4 ± 1.6Mammal0.2 ± 0.10.5 ± 0.22.5 ± 0.6Bird0.2 ± 0.10.4 ± 0.11.6 ± 0.4Marine0.2 ± 0.10.3 ± 0.11.4 ± 0.4Familiar human1.6 ± 0.45.1 ± 1.35.6 ± 1.6Non-familiar human1.3 ± 0.43.5 ± 1.13.2 ± 1.1 The SI values for different face subcategories at three time points: Temporal dynamics of face responses at the group-level were assessed in a series of RDM’s computed from 100 ms before to 800 ms after the stimulus onset (Fig. 4, and Table 4). In face-selective sites, the correlation between the group-level activity during pairs of human face trials became significant earlier than that between pairs of other types of faces (time of significance: human = 170 ms, mammal = 260 ms, bird = 240ms and marine = 240 ms, Fig. 4A, Table 4a) and was maintained longer (time of non-significance: human = 720ms, mammal = 690ms, bird = 630 ms and marine = 670 ms). Similar results were found when investigating correlations between human versus other faces (time of significance: human/mammal = 210 ms, human/bird = 280 ms and human/marine = 320 ms, time of non-significance: human/mammal = 690 ms, human/bird = 670 ms and human/marine = 660 ms, Fig. 4B, Table 4b). Table 4Temporal evolution of correlation and CBE in face/face and face/non-face comparisons.(a)Start significant correlationPeak correlationEnd significant correlationHuman /Human (face)170300720Mammal/Human (face)210360690Bird/Human (face)280450670Marine/Human (face)320440660Mammal/Mammal (face)260360690Bird /Bird (face)240320630Marine/ Marine (face)240390670(b)Start significant CBEPeak CBEHuman-face/Non-face160 ms360 msMammal/Human (face)160 ms230 msBird/Human (face)160 ms230 msMarine/Human (face)160 ms260 msMammal-face/Non-face230 ms420 msMammal/Bird (face)––Mammal/Marine (face)280 ms–Bird-face/Non-face230 ms430 msBird/Marine (face)––Marine-face/Non-face250 ms410 msAnimate/inAnimate310 ms460 ms(c)Crossing time in CBEMammal-face/Human-faceMammal-face/Non-face290 msBird-face/Human-faceBird-face/Non-face310 msMarine-face/Human-faceMarine-face/Non-face360 msThese tables summarized the information in the time course of face selective sites activity. Table a demonstrated when the correlation between members of a same face category or between different face categories started to significantly rise from baseline, when they peaked and when they fell back to baseline level. Table b demonstrated when the CBE started to be significant and when it peaked. Table c demonstrated when 2 traces of CBE with a shared category crossed, that means the maximum differentiation between the shared category from the 2 others. Table 4Temporal evolution of correlation and CBE in face/face and face/non-face comparisons.(a)Start significant correlationPeak correlationEnd significant correlationHuman /Human (face)170300720Mammal/Human (face)210360690Bird/Human (face)280450670Marine/Human (face)320440660Mammal/Mammal (face)260360690Bird /Bird (face)240320630Marine/ Marine (face)240390670(b)Start significant CBEPeak CBEHuman-face/Non-face160 ms360 msMammal/Human (face)160 ms230 msBird/Human (face)160 ms230 msMarine/Human (face)160 ms260 msMammal-face/Non-face230 ms420 msMammal/Bird (face)––Mammal/Marine (face)280 ms–Bird-face/Non-face230 ms430 msBird/Marine (face)––Marine-face/Non-face250 ms410 msAnimate/inAnimate310 ms460 ms(c)Crossing time in CBEMammal-face/Human-faceMammal-face/Non-face290 msBird-face/Human-faceBird-face/Non-face310 msMarine-face/Human-faceMarine-face/Non-face360 msThese tables summarized the information in the time course of face selective sites activity. Table a demonstrated when the correlation between members of a same face category or between different face categories started to significantly rise from baseline, when they peaked and when they fell back to baseline level. Table b demonstrated when the CBE started to be significant and when it peaked. Table c demonstrated when 2 traces of CBE with a shared category crossed, that means the maximum differentiation between the shared category from the 2 others. Temporal evolution of correlation and CBE in face/face and face/non-face comparisons. Mammal-face/Human-face Mammal-face/Non-face Bird-face/Human-face Bird-face/Non-face Marine-face/Human-face Marine-face/Non-face These tables summarized the information in the time course of face selective sites activity. Table a demonstrated when the correlation between members of a same face category or between different face categories started to significantly rise from baseline, when they peaked and when they fell back to baseline level. Table b demonstrated when the CBE started to be significant and when it peaked. Table c demonstrated when 2 traces of CBE with a shared category crossed, that means the maximum differentiation between the shared category from the 2 others. To investigate the time course of category distinction, we calculated the temporal dynamics of the CBE values for pairs of categories in face-selective sites (Fig. 4C). Human face/non-human face and human face/non-face objects CBE rose nearly simultaneously, suggesting that responses to the human face category can be differentiated from other categories including non-human faces at about 160 ms post stimulus onset. CBE values for human vs. mammal, human vs. bird and human vs. marine faces dropped quickly (between 230 and 260 ms), as a result, pattern of group-level activity for all face categories began to increase (Fig. 4C, Table 4c). Importantly, the intersection of CBE traces with a shared category indicates the timing of maximum differentiation between that shared category and the two others. For example, the crossing point of mammal face vs. non-face and human face vs. mammal face was at 290 ms, showing the timing of maximum response difference to mammal faces relative to non-faces and human faces (Fig. 4C). This intersection occurred at 310 ms for bird faces and at 360 ms for marine faces vs. non-faces and human faces. This finding suggests that the graded responses to face categories were present both in the response magnitude of face-selective sites and in the profile of their group-level activity. The timing of responses followed a similar pattern: both the onset of responses in face-selective sites and the temporal pattern of group-level activity as derived from CBE were sequential for face categories, which means the earliest response and category boundary distinction was to human face, followed by mammals, birds and marines face. Our findings are consistent with the notion that basic level categorical differences (e.g., human face versus non-human face) were distinguishable earlier than the superordinate level categories (e.g., animate versus inanimate). In contrast, correlated group-level activity in visually active sites became significant around 190ms without a difference between face sub-categories (Fig. S4A and B). The CBE for visually active sites was not significant for any pairs of faces and face vs. non-face comparisons. To compare between the fine to coarse category representations in the response pattern of face-selective and all the VTC sites, we also calculated the temporal dynamics of the CBE values for a pair of categories in all the VTC sites (Fig. S4C). The CBE value at the superordinate level increased (animate/inanimate:14, p = 9.9e–05) although the response onset was still delayed (peak CBE = 400ms). The CBE values decreased for the subordinate level categories (human-face/mammal face: 32, human-face/bird-face: 33, human-face/marine-face:35) even though the response onset was early (peak CBE: 230 ms, 230 ms and 240 ms, respectively). Finally, we trained a biologically plausible linear classifier25,26 to examine whether information in the VTC sites can be used to accurately discriminate face subcategories. As expected from category boundary and PCA analyses, temporal patterns of category decoding were sequential for face categories. The human face advantages in terms of onset latencies and classification accuracy were observed compared to non-human faces (P < 0.001; significance level was computed by 99% confidence interval). The earliest time that neural responses could discriminate the face subcategories above chance level compared to non-face stimuli was 83 ± 5, 146 ± 27, 156 ± 18, 177 ± 28 for human, mammal, bird, marine, respectively (Fig. 4E). Late latencies of non-face categories and animate/in-animate comparison was consistent with late representations of unfamiliar and superordinate categories. To compare patterns of classification errors in the learned classifier across different time intervals, we calculated confusion matrices in 4 different time points, 100, 300, 500 and 700 ms (Fig. 4D). The diagonal elements of confusion matrix showed the classification accuracy of face subcategories compared to each other (chance level = 25%). As expected, there was a small rate of classification mistakes in the human face at early phases of neural response compared to other face subcategories (classification accuracy at time 100; Human: 0.83 ± 0.13; Mammal: 0.27 ± 0.17; Brid: 0.29 ± 0.16; Marine: 0.34 ± 0.15) and as time evolved the percentage of diagonal elements for each category started to increase (classification accuracy at: 300; Human: 0.92 ± 0.09; Mammal: 0.54 ± 0.18; Brid: 0.39 ± 0.18; Marine: 0.55 ± 0.17). We also normalized the response of each channel by subtracting the mean and dividing by the SD of response to stimuli (z-score normalization). Representation of face categories was preserved in normalized response condition which indicated that response amplitude do not affect our results (results not shown). Fig. 4Temporal dynamic of group-level activity. (A and B) Average correlation of group-level activity for pairs of face categories in face-selective sites during time course of response. Traces show the mean ± sem of correlation between trials in 100ms time window with 10ms steps and time marker in each data-point shown in the middle of the calculated window. (C) Temporal dynamics of group-level representational differences, the category boundary effect for Face/NonFace and Face/Face categories in face-selective sites. In the last trace, “Anim” stands for animate and included all faces and bodies, and “inAnim” stands for inanimate and included objects and scene categories. Human face was the first category that can be differentiated from others (including other face categories). Quickly after, the difference between human face and other faces started to decrease and the difference between all face categories and non-faces increased. Mammal faces were the second that can be differentiated not only from non-faces, but also from human faces, while still having significant correlation with human faces. Bird and Marine faces were the third and fourth to show response activation. (D) Time courses of classification accuracy using linear SVM classifier, each trace is a result of binary classifier and thus chance level is 50% and below the graph is classifier confusion matrix of the face subcategories in different times. Color bar indicates the percentage of labels assigned to a category (E) and their corresponding onset latencies when the first-time classifier accuracy rose above chance level. Shaded areas are the SE. Temporal dynamic of group-level activity. (A and B) Average correlation of group-level activity for pairs of face categories in face-selective sites during time course of response. Traces show the mean ± sem of correlation between trials in 100ms time window with 10ms steps and time marker in each data-point shown in the middle of the calculated window. (C) Temporal dynamics of group-level representational differences, the category boundary effect for Face/NonFace and Face/Face categories in face-selective sites. In the last trace, “Anim” stands for animate and included all faces and bodies, and “inAnim” stands for inanimate and included objects and scene categories. Human face was the first category that can be differentiated from others (including other face categories). Quickly after, the difference between human face and other faces started to decrease and the difference between all face categories and non-faces increased. Mammal faces were the second that can be differentiated not only from non-faces, but also from human faces, while still having significant correlation with human faces. Bird and Marine faces were the third and fourth to show response activation. (D) Time courses of classification accuracy using linear SVM classifier, each trace is a result of binary classifier and thus chance level is 50% and below the graph is classifier confusion matrix of the face subcategories in different times. Color bar indicates the percentage of labels assigned to a category (E) and their corresponding onset latencies when the first-time classifier accuracy rose above chance level. Shaded areas are the SE. The duration of human face representations was longer than for other face subcategories such that at 700ms after stimulus onset, human face presentation was still represented accurately (classification accuracy at 700 ms; Human:0.53 ± 0.17; Mammal:0.27 ± 0.15; Brid:0.23 ± 0.14; Marine: 0.22 ± 0.15). The temporal dynamics of the confusion matrix showed that most of the confusion happened in mammal and bird face subcategories (‘Marin-Bird misclassification t = 100ms: 0.41 ± 0.15; t = 300: 0.38 ± 0.17; t = 500: 0.40 ± 0.16; t = 700: 0.36 ± 0.17). Furthermore, we observed that face subcategories elicited different response patterns over the cortical surface. To investigate the spatial representations of face subcategories over VTC, we computed the classification performance for 4 face subcategories on each cortical site across all subjects. Then we mapped the pooled performance values on the VTC at MNI space (Fig. 5A). The similarity of spatial representations was measured by a correlation between two cortical maps for a pair of face subcategories. The similarity of sample pairs (Fig. 5B) and the comparison of all pairs are illustrated in Fig. 5C. The most similarities were observed in the pair of human-mammal (r = 0.86), mammal-bird (r = 0.87), and bird-marine (r = 0.84). At the second level, the two pairs of human-bird (r = 0.75) and mammal-marine (r = 0.76) faces show comparable spatial representations and human-marine faces show the lowest level of spatial representations similarity (r = 0.63). Fig. 5The spatial representations of face sub-categories across the VTC. (A) The representations of face sites (measured by SVM classifier) across all subjects are displayed as color-coded fill. The coordinates of the electrodes from left hemisphere subjects were mirrored such that all sites could be displayed on a single hemisphere. (B) Each subpanel shows the scatter plot of 1 face subcategory representations against another one, four pairs, including all human pairs and bird-marine comparison, were plotted as examples. Each point represents 1 site, and the dashed line illustrates the fitted regression line (the correlation coefficient values, spatial similarity, with their P values are shown in each panel). The level of spatial similarity measured by correlation coefficient. (C) Comparison of the face subcategories representations across face sites for all pairs of subcategories. The similarity of spatial representations across the cortex was computed by correlation between face representations pairs. The spatial representations of face sub-categories across the VTC. (A) The representations of face sites (measured by SVM classifier) across all subjects are displayed as color-coded fill. The coordinates of the electrodes from left hemisphere subjects were mirrored such that all sites could be displayed on a single hemisphere. (B) Each subpanel shows the scatter plot of 1 face subcategory representations against another one, four pairs, including all human pairs and bird-marine comparison, were plotted as examples. Each point represents 1 site, and the dashed line illustrates the fitted regression line (the correlation coefficient values, spatial similarity, with their P values are shown in each panel). The level of spatial similarity measured by correlation coefficient. (C) Comparison of the face subcategories representations across face sites for all pairs of subcategories. The similarity of spatial representations across the cortex was computed by correlation between face representations pairs. Control for low-level image features Comparison between neural RDM and luminance re-ordered neural RDM, wavelet RDM and pixel RDM (Fig. S2) revealed a non-significant correlation between the luminance re-ordered and neural RDMs (cor = 0.029, p = 0.24) and a significant reverse correlation between the neural and the pixel RDMs (cor = – 0.067, p = 9.99e–05). This result discards the possibility that luminance drives our results at the population level. There was a significant but weak correlation between the wavelet and neural RDMs (cor = 0.147, p = 9.99e–05). This finding is consistent with previous work, indicating, that low to mid-level features (as carried in the wavelets) are related to the observed neural signals46–48. These results suggest that the patterns observed on face selective and on visually active sites cannot be fully explained by low or mid-level features of the images alone. Additionally, generalized linear model (GLM) analysis allowed us to quantify the impact of wavelet, pixel, and luminance features on the neural response patterns. GLM analysis, which regressed feature-based RDMs (wavelet, pixel, and luminance) onto neural response RDMs, indicated that low- and mid-level visual features influenced category-specific neural responses (Supplementary Table S1). Notably, wavelet features showed significant contributions in multiple category comparisons, for example human vs. mammal faces (β = 0.49, p < 0.00001) and human face vs. marine face (β = 0.49, p < 0.00001). However, the results presented in this paper cannot be fully explained by low or mid-level physical features of the images alone. Discussion In this work, we recorded iEEG signals in 8 neurosurgical patients implanted with subdural electrodes. We found that the pattern of activity across all face-selective sites in VTC was correlated in response to all face subcategories. The response magnitude and response correlation across all face-selective electrodes in VTC gradually decreased between responses to pairs of face subcategories from human to mammal, bird and marine, and operated on different time scales. Specifically, when investigating the time course of category distinctions, we found that basic level categorical differences (human face vs. other faces) are distinguishable earlier than subordinate level categories (animate vs. inanimate). Therefore, our finding suggests that there is a code along the VTC for representing coarse to fine face categories which is spatiotemporally scaled depending on the face subcategory and concept being represented. Spatial dynamics of the face responses It is still a matter of debate how fine object categories are represented in the ventral temporal cortex. In one notable study (Mur et al.27), three models of category representations were proposed: In the first model, neural responses were high for members of the preferred category and low or none for members of non-preferred categories (i.e., in an all-or-none fashion). In the second, the response profile of the neural cluster is graded within the preferred category (i.e., some category members activated the neural cluster more intensely than others). In the third model, the response profile of the neural cluster continuously changed in response to different stimuli/categories, and there was no discontinuity when crossing the category boundary. Mur et al.27 suggested that their results support the second model (category step) because responses within the FFA and parahippocampal place area (PPA) to stimuli in their preferred category (faces and places, respectively) in comparison to outside their preferred category showed a discontinuity when crossing the category boundary. Our findings of category boundaries between representations of face subcategories (significant CBE) are also in line with the second model. While response rank showed a graded response to the four face categories, not all the face-selective sites had a similar response rank to the face subcategories. Notably, Miyakawa et al.22 proposed that well-known category clusters (e.g., face cluster) are homogenously distributed and support coarse category representation, while heterogeneously distributed and smaller clusters in the cortex support fine category representations. Using a high-resolution fMRI, past work showed that there are fine categorical subdomains within the face domain of FFA that suggested that representations could exist at different spatial scales1,28–31. We provided further evidence for this by demonstrating that there is heterogenicity of representations inside the clusters, which then enables fine category representations at different spatiotemporal scales. Grill-Spector and Weiner10 proposed that category representations are hierarchically organized in the VTC such that categories are spatially nested in the region from coarse to fine categories and that this spatially organized nested hierarchy enables the veridical representations of object categories at different abstraction levels. Our findings are consistent with these studies: Between-category response differences in face-selective areas are correlated with the perceptual similarity between faces, rather than with low-level physical features. Responses to different face subcategories formed a gradient within face-selective areas, such that the maximum response was observed for human faces and mammals’ faces, birds’ faces, and marine animals’ faces evoked increasingly weaker responses. Moreover, we found that there were categorical boundaries between these representations. We acknowledge that our clustering based on brain responses was not entirely optimal. For instance, some mammalian faces were grouped with human faces, and certain non-face objects were clustered with birds or marine faces. This may be due to the shape statistics of our image set, which were not adequately controlled across different categories. Although physical features derived from wavelet can explain part of the neural response, as demonstrated in control analysis, this cannot fully explain the neural responses we observed in this paper. Previous studies in monkeys24,48 and humans16 suggest that a portion of the neural code for object categorization, including faces, in the visual cortex is influenced by shape similarity within a category and shape distance between categories. Increasing within-group variability to emphasize semantic features over physical ones introduces ambiguities in neural category representation, as it can result in similar stimuli being grouped into the same category. Therefore, it is expected that animal faces with greater physical similarity to human faces may evoke neural responses more like those of human faces and consequently be clustered within the human face group. In an fMRI study, Connolly et al.8 compared the representational space of six categories (two species from each class of primates, birds, and insects) via multivariate patterns of brain activity (dissimilarity matrices) in the VTC. They found that the pattern of activity across the human VTC correlated with the behavioral judgments of animate categories similarity and suggested that abstracted representational space in the VTC may be on a continuum with primates at the top to insects at the lower end8. Later, this group claimed that they didn’t find evidence for a binary dissociation between animate/inanimate categories in the functional rear of VTC and suggested that abstracted representational space in human vision was related to the perceptual property of objects without the sharp dissociation between animate/inanimate categories9. The same continuous representations of objects, actions and semantics has also been found in non-visual cortices32. However, the distinction between graded representations that we show in this study and the representations continuum cannot easily be explained since each study’s conclusion is limited to the number of categories included. Using distinct categories in a study will inevitably lead to what appears to be category steps in the representations regardless of the continuum. Temporal dynamics of the face responses In this study, we showed that a subordinate level category representation only relied on the activation of selective cortical areas while superordinate level category representations relied on the activation of larger cortical areas, which may account for the delayed representation. Barbeau et al.33 described the spatiotemporal dynamics of face recognition using intracranial EEG data. They showed that there was early onset activity (around 110 ms post stimulus presentation) in the posterior and inferior frontal gyrus followed by a later onset (around 160ms) of activity in the middle and posterior fusiform gyrus. There were two peaks of activation at 240ms and 360ms which involved different areas in the VTC. This late activation was proposed to be a sign of massive parallel processing33. The early onset and the late phase of activity in this study generally corresponded to the onset of correlated activity and peak correlation at group level in our findings. In addition, our findings that mid-level (basic level) categories were represented earlier than the superordinate and subordinate level categories correspond to a comparable monkey study led by Dehaqani et al.21. They investigated the time course of visual object category representations in the IT cortex of macaque monkeys and found that IT neurons represent mid-level categories (e.g., human faces) earlier than superordinate (e.g., animal) and subordinate (e.g., face identity) level categories. They also reported that this temporal order of category representations was observed in both the responses of single neurons and at the population level, complementing our findings regarding the temporal dynamics of the response in VTC face-selective cortical areas. Taken together, our results reveal that the pattern of activity across all face-selective sites in VTC was correlated in response to face stimuli from different species. This correlation was preserved not only for the members of the same face category, but also for members of other face categories, despite the huge diversity in their physical similarity. Specifically, we showed that this correlation within the face-selective regions decreased systematically from human faces to mammal, bird, and marine faces and had different onset and peak time of representation. Our findings demonstrate that the responses to face subcategories (or potentially all animate objects) could be on a continuum and operate at different spatiotemporal scales. These graded representations were observed in the spatiotemporal dynamics of responses across all face selective VTC sites. Materials and methods Data acquisition Eight patients diagnosed with medicine-resistant epilepsy were implanted with subdural electrodes (6 males, 2 females, aged: 23–68 years) to localize the source of seizures as part of their presurgical evaluation at Stanford University Medical Center. Given our previous studies using similar paradigms and recording more than 100 channels, we would need to use 100 channels in all subjects in the target region to have 80% power for detecting 0.05 changes in category representation index (standard deviation of about 50% in normalized scale). All patients provided written informed consent before participating in this study. The experiment was approved by the Stanford Institutional Review Board (IRB) all experiments were performed in accordance with relevant guidelines and regulations and with the Declaration of Helsinki. The location of the grid electrodes was determined purely based on clinical considerations (Fig. S1A; three left hemispheres, and five right). The electrode diameter was either 2.3–1.15 mm with center-to-center spacing of 10–4 mm between adjacent electrodes, respectively. Intracranial EEG signals were recorded at 3052 Hz sampling frequency through a 128-channel Tucker Davis Technologies recording system (www.tdt.com) for subjects one to seven and at 1 kHz sampling frequency with a Nihon-Kohden 512-channel system (www.nihonkohden.com) for the other one subject. Electrodes localization A 3D cortical surface reconstruction was done for each patient using Freesurfer34. A post-implant brain CT scan was taken from the patients and registered to the pre-implant anatomical T1-weighted brain MRI35. Then, the electrode position on the subject’s own reconstructed brain surface was determined using BioImage Suite36 and iELVis MATLAB toolbox37. The electrode positions were also transformed into the MNI space to show electrode coverage across subjects (Fig. 1A and S1A). Experimental paradigm We used Psychtoolbox (http://psychtoolbox.org/) on Mac OSX laptop to run the experiment. The laptop was placed ~ 70 cm from the subject’s eyes and the center of the display was at the subject’s eye level. Each image was subtended 5 visual degrees at its longest dimension. Subjects were instructed to look at the center of the screen while stimuli of different categories were presented in random order. Images were presented for 300ms, followed by a 400 ms inter-stimulus interval during which a blank screen was presented. The images included 11 different categories, each consisting of 25 members. The categories included human faces, human body, mammal faces, mammal body, bird faces, bird body, marine faces, marine body, limbs, objects, and places. Images were repeated one time across experimental runs so the total trials in a run of experiment were 550. During image presentation, the subject was instructed to press a key (‘press 1’) when the pattern of ‘###’ was presented in red (Fig. 1B) on the screen. The onset and offset of each stimulus were marked by a luminance change in the corner of the laptop display that was recorded with a photodiode. Signal preprocessing Preprocessing steps were performed using MATLAB (The MathWorks, Inc.) and the SPM 12 (www.fil.ion.ucl.ac.uk/spm) toolbox in custom routines (https://github.com/LBCN-Stanford/Preprocessing_pipeline). The data was first converted into SPM format, down-sampled to 1000 Hz and filtered for power-line noise (notch filter between 57 and 63 Hz) and its harmonics (117–123 Hz, 177–183 Hz). The electrode signals underwent an automatic quality assessment: sites with variances 5 times larger or smaller than the average variance across all sites were labeled as ‘noisy’. Sites with 3 times more ‘jumps’ (defined as changes in the signal derivative larger than 100 µV) than the average across sites were considered ‘spiky’. Sites marked as either ‘noisy’ or ‘spiky’ or those which showed any epileptic activity, as determined by the neurologist, were excluded from any further analyses. The signal was then re-referenced to the average of the signal over all remaining sites. Each trial was considered in the --100 to 700 ms time window around the stimulus onset and baseline corrected (using the − 100 to 0 ms before stimulus onset). Time-frequency decomposition was then performed using a 7-cycle Morlet wavelet, with frequencies ranging from 70 to 177Hz (high frequency band) with steps of 1Hz and avoiding discarded frequencies from Notch filter. The power in each frequency and time bin was rescaled using the log-power of the baseline activity. The power of the signal was finally averaged within this high frequency band (HFB). HFB power was used as an index of neural population activity because numerous previous human and non-human studies have indicated that HFB power is well correlated with BOLD signal and with the multi-unit activity of a population of neurons in a recorded cortical site38–44. Inclusion criteria Across all sites over all subjects, we used only signals from electrodes located in the Ventral Temporal Cortex (VTC). Neural responses were measured for each stimulus presentation using the mean HFB power in a 450ms window, starting at 50ms after and ending at 500ms after the stimulus onset. Cortical sites which showed a significantly higher response to at least one category compared to their baseline (– 100 to 0 ms) were considered visually active and were included for further analysis (Wilcoxon signed-rank test, p < 0.01). Out of 357 VTC sites in 8 patients, 190 of them were visually active. Visually active sites in which their responses to the four face categories (human, mammal, bird, and marine faces) were significantly higher (Wilcoxon rank-sum test, p < 0.01) than their responses to all other stimuli were deemed “face-selective sites.” The remaining visually active sites were considered non-face sites. Face d-prime Face d-prime was calculated using the following formula: \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\frac{{\left[ {\mu \left( F \right) - \mu \left( {nF} \right)} \right]}}{{\sqrt {\frac{{{\sigma ^2}\left( F \right)+{\sigma ^2}\left( {nF} \right)}}{2}} }}$$\end{document} That is, µ(F) and µ(nF) are the mean site response to face and non-face images, respectively. Similarly, σ2(F) and σ2(nF) are the variance of sites responses to face and non-face images. Response onset latency The response onset for each site was calculated on the averaged response across face trials, which was smoothed with a Gaussian kernel with a 10 ms width. The response onset was considered the first 1 ms of 10 consecutive bins that was 1.9 Standard Deviation (SD) above the mean of the baseline activity. Representational Similarity Analysis (RSA) Representational Dissimilarity Matrices (RDMs) were computed to assess the pattern of neural response within and between the categories at the group level, i.e., across subjects and sites. First, the HFB amplitude was averaged over the [150 500]ms time window for each stimulus and site. For each stimulus, a vector comprised of the average HFB amplitude for all sites was obtained. Those vectors were pooled across subjects. For each stimulus, the responses of all face-selective sites to that stimulus (pooled across subjects) were arranged in a vector. The same was done for visually active sites. Correlation distances were then computed between each pair of response vectors (i.e., 1 minus Pearson correlation). Those values were then assembled into a RDM. Thus, each cell of the matrix compared the response patterns elicited by the two images at the group-level. This analysis was performed separately for face-selective and for visually active sites. Average distances were computed for each category pair (e.g., human/human, human/mammal) and compared to similar distances computed in the baseline time window ([– 100 0]ms before onset). The significance of each distance was then assessed based on random permutations of the stimulus labels, and the obtained p-value was corrected for multiple comparisons using FDR correction. Similarly, distances were compared to each other by computing the category boundary effect and its p-value16. The p-values for category boundary effect were also calculated using a non-parametric permutation test. The RDMs were projected onto a 2-dimensional space using nonmetric Multidimensional Scaling (MDS) for face-selective and visually active sites separately. The categorical structure of the neural data was estimated using a hierarchical cluster analysis based on the dissimilarity matrix of face-selective and visually active sites. We also calculated the category boundary effect (CBE), defined as the absolute difference between the mean neural distance for within-category pairs (all-face vs. all-face) and the mean neural distance for between-category pairs (all-face vs. all non-face). Based on the amplitude of their HFB response averaged in the 150–500 ms time window after onset, the four face subcategories were then compared in both the ‘face-selective’ and ‘visually active’ sites. Permutation tests assessed the significance of potential differences between subcategories (10,000 permutations, FDR corrected for the number of tests (n = 12, 6 binary comparisons for face sites and 6 for non-face sites)). To investigate the temporal dynamics of face responses, we calculated a series of RDM in a 100 ms window with 10ms steps starting from 100 ms before to 800 ms after stimulus onset. Then, in each RDM, a pair of face categories was averaged and the category boundary effect (CBE) for different pairs was calculated to demonstrate the temporal trend of correlated cortical activity between trials and CBE. In each time point, correlation and CBE were assessed as significant using non-parametric permutation tests if their p-value was less than 0.001. Hierarchical clustering analysis To measure the match between intuitive categories and nodes in a tree of hierarchical clustering analysis, we first found the nodes in the reconstructed hierarchy that best matched each category. Then, we computed the average of the following ratios as a hierarchical clustering score defined as the number of category members under the node divided by the total number of members in each category24. To compute how likely a category cluster happens by chance, we shuffled the category labels and repeated the clustering analysis 1000 times. The significance was measured by comparing the category hierarchical clustering score with its chance score using bootstrap confidence interval. Principal component analysis We used principal component analysis (PCA) to illustrate the neural representations of different stimuli in low dimensional space. The eigenvectors of the covariance matrix with the greatest eigenvalues were used to make a transformation matrix and to construct a two-dimensional (2D) representation of the high-dimensional neural response space. We used 100ms sliding window with 1ms shift over neural responses from − 100 to 800 ms after the stimulus onset to calculate PCA and to visualize the temporal dynamics of category representation. Separability Index (SI) To calculate the separability of face subcategories based on the group level responses we used scatter matrices of the face category members as an estimation of covariance in high-dimensional space. We computed within-category and between category scatter, then defined the SI as the ratio of between-category scatter and within-category scatter21. Then, we use a spectral norm to compute the SI in the reduced PC space. This caused SI to be resistant to the inverse scatter problem. The number of PCs covered 90% of the variance of neural responses. Decoding accuracy and confusion matrix We trained Support Vector Machine (SVM) classifiers with a linear kernel using the group of electrodes signals on a randomly selected 70% of stimuli as a training set. We then measured the classification accuracy of the trained classifier on the remaining 30% of stimuli (the test set). The least square method with an equal number of samples for different classes was used to classify stimuli. In the case of more than two classes, the majority voting procedure was applied between pairs of classes. The standard error of classification performance was computed by repeating the calculation 100 times with different sets of training and test trials. To calculate the accuracy of a single class, we extracted the decoding accuracy of the SVM classifier using the confusion matrix. To obtain a confusion matrix, after multiclass training of SVM classifier, we computed the percentage of predicted labels assigned to each category. This percentage was displayed as a color-code in the confusion matrix. The main diagonal elements of the confusion matrix illustrate the classifier performance for each class and the off-diagonal elements show the error made in classification. All the classifier results were cross validated to indicate the classifier performance. Control analysis for low-level image features To ensure that low-level features in the stimulus images did not drive our results, we performed multiple control analyses (Fig. S2). To control the images power spectrum, a spatial frequency power spectrum with rotational average was calculated for each stimulus and averaged across categories. Averaged ​spatial frequency power spectrums were compared using a one-way ANOVA, and post-hoc analysis (Tukey-Kramer) was performed. Power spectral analysis ​returned 153 values for each image, which were then averaged within each category (n = 153). Additionally, we computed the mean luminance across pixels in each category, pooling all non-faces together. To account for the effects of low-level image features on our RSA analysis, we compared the neural RDM to similarity matrices based on the low-level features of the images. To this end, we re-ordered the neural RDM based on the average luminance in each image to obtain a second neural RDM. We computed the physical similarity between stimulus images by performing a multi-level two-dimensional wavelet decomposition of images (using db2 wavelet). The coefficients of images’ wavelet transformations were arranged in vectors and the pairwise Euclidean distance between these vectors was used as an index of physical similarity between images. Similarly, pixel-based similarity was measured by calculating the Euclidean distance between the pixels luminance of two shapes, i and j, within the space of 304 × 305 pixels: \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\left( {{\text{S}}_{{{\text{x}} = {\text{1}}:{\text{3}}0{\text{4}}}} {\text{S}}_{{{\text{y}} = {\text{1}}:{\text{3}}0{\text{5}}}} \left| {{\text{pix}}_{{\text{i}}} \left( {{\text{x}},{\text{ y}}} \right){\text{ }}{-}{\text{ pix}}_{{\text{j}}} \left( {{\text{x}},{\text{ y}}} \right)} \right|^{{\text{2}}} } \right)^{{{\text{1}}/{\text{2}}}}$$\end{document}. Before the comparison, all matrices were converted to percentile rank and the re-ordered neural RDM, wavelet RDM and pixel luminance RDM were correlated with the neural RDM (2D correlation). The neural RDM, re-ordered neural RDM based on luminance, wavelet based RDM, and pixel based RDM are displayed in Fig. S2 (A, C, B and D respectively). Non-parametric permutations (n = 10,000, Mantel test45), assessed the significance of the obtained correlation values. Additionally, to better understand the contributions of low-level features to neural responses, we performed a generalized linear model (GLM) analysis. This model allowed us to quantify the impact of wavelet, pixel, and luminance features on the neural response patterns. We used a GLM to model the relationship between the feature RDMs (predictors) and the neural response RDM (outcome). Each pairwise stimulus was arranged into vectors, and the neural response vector was regressed onto the feature vectors (wavelet, pixel, luminance). \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf{y}={\beta _1}{\mathbf{X}_{{\text{wavelet}}}}+{\beta _2}{\mathbf{X}_{{\text{pixel}}}}+{\beta _3}{\mathbf{X}_{{\text{luminance}}}}+\epsilon$$\end{document} Where y is the vectorized neural response RDM (the dependent variable), Xwavelet​, Xpixel ​, and Xluminance are the predictor matrices derived from the wavelet, pixel, and luminance RDMs. β1​, β2, and β3are the regression coefficients representing the contribution of each feature to the neural response. ∈ is the residual error. After fitting the model, the resulting beta coefficients were normalized to represent the relative contribution of each feature in explaining the neural response. The statistical significance of each predictor’s contribution was evaluated using p-values, which tested the null hypothesis that each β was equal to zero. Electronic Supplementary Material Below is the link to the electronic supplementary material. Supplementary Material 1 Supplementary Material 1