Behaviorally-relevant features of observed actions dominate cortical representational geometry in natural vision Authors: Jane Han, Vassiki Chauhan, Rebecca Philip, Morgan K. Taylor, Heejung Jung, Yaroslav O. Halchenko, M. Ida Gobbini, James V. Haxby, Samuel A. Nastase Published: 2026 ABSTRACT We effortlessly extract behaviorally relevant information from dynamic visual input to understand the actions of others. Here, we compare behavioral judgments of action meaning against other models capturing semantic and visual features. We measured brain activity using fMRI while participants viewed 90 video clips depicting social and nonsocial actions in real-world contexts. Using arrangement tasks, we developed two behavioral models capturing judgments of action meaning (transitive purpose, social content), as well as three control models capturing judgments of visual content (people, objects, scene). We compared these models with semantic models based on word embeddings and visual models based on gaze and motion energy. Behavioral models of action meaning outperformed models of visual content, semantics, and low-level visual features across much of cortex. Transitivity and sociality captured large portions of unique variance throughout the action observation network, extending into regions not typically associated with action perception, such as ventral temporal cortex. Introduction How do we understand the actions of others? From a computational standpoint, an observer must ultimately extract behaviorally relevant information—such as the goal of an action or its social significance—from dynamic visual patterns of bodily movements interacting with other agents or objects1,2. Similar to other domains of vision3–6 and action execution7,8, visual action understanding likely relies on a hierarchy of neural population codes, where information is encoded in the geometric relationships among distributed patterns of neural activity9–11. At each stage of the processing hierarchy, these representational spaces are reshaped so as to disentangle higher-level features and more explicitly represent behaviorally relevant information12,13. The similarity among these neural representations is thought to inform our perception of similarity and ultimately guide behavior14–20. In the current work, we aimed to unravel the representational geometries supporting human action understanding in natural vision. A large body of work in both nonhuman primates21–23 and humans24–29 has charted out a network of cortical areas involved in action observation and understanding. This action observation network appears to unite several major cortical systems. A lateral visual pathway30 proceeding from early visual areas to lateral occipitotemporal (LO) cortex and superior temporal sulcus (STS) is thought to support action understanding and social perception. Subfields of the LO encode visual motion31,32, tool use33–35, faces4,36, body parts37,38, and multi-body configurations39,40, suggesting a pivotal role for LO in action understanding41–46. The posterior superior temporal sulcus (pSTS) in the lateral pathway is implicated in the perception of biological motion47–49 and social interaction50–52, and may interface with broader systems for social cognition53. The lateral pathway is complemented by a parieto-frontal system comprising anterior intraparietal sulcus (aIPS) and ventral premotor (vPM) cortex54. The aIPS is situated near the end of the dorsal “vision for action” pathway55–57 and is thought to encode action goals22,44,58,59. This parietal system is closely intertwined with prefrontal areas, particularly vPM cortex, also associated with motor planning and execution21,23,60–62. Much of this prior work is grounded in experimental contrasts between rudimentary actions—e.g., grasping an object to eat it or place it in a container22—and highly-controlled video stimuli, e.g., depicting only the grasping hand63. These paradigms cannot fully capture the richness and complexity of real-world action understanding64,65. Highly controlled experimental stimuli may artificially constrain neural responses66 and can make it difficult to assess the relative contribution of different action features to neural activity in natural contexts64. The use of dynamic, naturalistic action stimuli, on the other hand, can reveal unexpected results that are otherwise obscured by the selective use of certain stimuli. For example, studies using naturalistic clips of movies has revealed a surprisingly prominent role of ventral temporal (VT) cortex in the perception of dynamic, naturalistic action49,64,67–69. VT is located at the anterior part of the ventral object vision pathway56,57,70,71 and has been historically associated with face and object processing36,64,72,73. However, this understanding of VT may be biased by a historical reliance on highly-controlled, static image stimuli64. More recent work has begun to address these limitations by exploring the neural representations of observed action and social interaction using dynamic and naturalistic stimulus paradigms49,74–80. The transition from highly-controlled experimental manipulations to naturalistic videos of real-world actions marks a significant step forward in the study of action representation. This paradigm shift, however, raises the question of how to quantify the structure of action features as they occur in real-world contexts. For example, recent work has tested bottom-up visual features (e.g., derived from deep neural networks) and human annotations of particular action features (e.g., “is the action directed at an object?”)77,80. To move toward more behaviorally rich models of action understanding, researchers have begun to adopt behavioral arrangement tasks81–83, allowing participants to freely group action stimuli based on their perceived similarity84–87. For example, recent work using behavioral arrangements of naturalistic action clips has highlighted the importance of social-affective features in organizing our understanding of real-world actions88. This approach provides a relatively direct window onto the psychological “space” in which actions are organized20,89, and promises to reveal the most cognitively relevant information underlying the representations of others’ behavior90. In the current study, we pursued three main questions. First, we aimed to quantify whether behavioral models of action meaning/purpose better capture neural representational geometry than models of visual content. This question was motivated by a growing body of work suggesting that action meaning supersedes many other aspects of stimulus content49,67,68 and may have been historically underestimated based on the field’s fixation on static image stimuli64. We adopted sociality and transitivity as plausible candidate models of action meaning/purpose based on prior work43,63,91–93. We expected behavioral judgments of action meaning to outperform other models of visual content throughout the action observation network. Second, we sought to quantify to what extent action representation extends into VT cortex. Historically, VT has not been part of the conversation about action understanding; however, based on a less prominent line of work24,49,63,68,94–98, we hypothesized that VT may encode dynamic, action-related features. Third, we hypothesized that human behavioral judgments may better capture neural representational geometry than other widely-used models like semantic word embeddings74 and low-level visual motion99, inspired by recent work suggesting that multidimensional behavioral goals are an important organizing feature of visual representation100–102. To quantify the relative contributions of different models of action representation during natural vision, we developed a condition-rich, naturalistic paradigm. We presented participants with 90 naturalistic video clips depicting real-world actions from 18 social and nonsocial action categories. This paradigm was designed to broadly sample the space of human actions, with the goal of minimizing experimental bias and fairly evaluating different models. To quantify the neural representational geometries supporting action understanding, we developed nine different representational models, ranging from low-level visual models to verbal annotations to behavioral judgements of the similarity of the visual content of the videos and the purposes of the actions. We focus in particular on representational models derived from behavioral arrangement tasks capturing high-level judgments of the transitive and social goals of actions. Briefly, we find that these more holistic behavioral judgments capture dramatically more variance in neural representational geometry than lower-level models, and extend beyond the canonical action observation network into areas like ventral temporal cortex and precuneus. Results We used fMRI to measure brain activity in 23 participants while they viewed 90 different 2.5 s video clips depicting real-world actions in two scanning sessions (Fig. 1). The 90 clips were sampled from diverse sources and spanned 18 social and nonsocial action categories (see Table S2 for the frequency of corresponding activities in the American Time Use Survey). The fMRI time series were submitted to a participant-level general linear model to estimate response patterns for each of the 90 stimuli, separately for the two different sessions. Using both a surface-based searchlight and targeted regions of interest (ROIs), we computed the Pearson correlation between local response patterns across the two scanning sessions to construct 90 × 90 split-data RDMs (Fig. 2). Prior to computing neural representational geometries, we used hyperalignment to better align cortical-functional topographies across individuals, based on data from a third session in which they viewed a naturalistic movie stimulus (the second half of Raiders of the Lost Ark). We used a whole-brain hybrid hyperalignment algorithm to transform response patterns from the action sessions in each individual into a common space based on both the response time series and functional connectivity during movie-viewing103. These neural RDMs—the neural representational geometries supporting action perception across a variety of cortical regions—serve as the target for modeling in subsequent analyses.Fig. 1Experimental design for fMRI and behavioral data acquisition.A fMRI participants (N = 23) viewed 90 video clips of dynamic, naturalistic actions spanning 18 social and nonsocial action categories in a condition-rich design. Participants were intermittently presented with two action words and were asked to indicate by button press which of the two words more accurately described the action depicted in the previous video (“gyrate” in the depicted example; Table S1). In addition to the two action-viewing fMRI sessions, participants completed a third fMRI session where they viewed a ~1 h naturalistic movie stimulus (the second half of Raiders of the Lost Ark). B A subset of participants (N = 17) completed five additional behavioral sessions where they performed five different multiple-arrangements tasks based on the action purpose (transitivity and sociality) of the video clips and visual content (people, objects, scenes) of still frames from the videos. Representational dissimilarity matrices (RDMs) were calculated based on the Euclidean distances in the arrangements and visualized using multidimensional scaling (MDS). C Participants produced different geometries based on the five different tasks. For example, a video depicting an individual dancer (light blue) was separated from a video depicting a group of dancers (dark blue) when participants arranged the stimuli based on sociality (top left); however, when participants arranged the stimuli based on transitivity, these two videos were clustered together (top right). When arranging static images according to the visual similarity of people, two stimuli depicting a man (in the original stimulus shown to participants; light purple) and a woman (dark purple) with haircare devices were separated (bottom left); when arranging images according to the visual similarity of objects, these two stimuli were grouped together (bottom right). Note that due to copyright reasons, original stimulus images have been replaced by similar images where the photographer and subject have provided consent for the image to be used and publicly shared. Source data are provided as a Source data file.Fig. 2Schematic of pipeline for representational similarity analysis.fMRI data collected while participants viewed the action stimuli were hyperaligned based on responses to a separate 1 h movie. Neural response patterns to the action stimulus were used to compute neural RDMs. A subset of participants performed behavioral arrangement tasks based on action purpose (transitivity, sociality) or visual content (objects, people, scenes); arrangements were converted into behavioral model RDMs. Annotations of videos with verb and nonverb labels were used to assign word2vec semantic embeddings to each action stimulus; verb and nonverb embeddings were used to construct semantic model RDMs. Spearman correlations were computed between different model RDMs and the neural RDMs. Multiple model RDMs were combined to jointly predict neural RDMs using multiple regression. Experimental design for fMRI and behavioral data acquisition. A fMRI participants (N = 23) viewed 90 video clips of dynamic, naturalistic actions spanning 18 social and nonsocial action categories in a condition-rich design. Participants were intermittently presented with two action words and were asked to indicate by button press which of the two words more accurately described the action depicted in the previous video (“gyrate” in the depicted example; Table S1). In addition to the two action-viewing fMRI sessions, participants completed a third fMRI session where they viewed a ~1 h naturalistic movie stimulus (the second half of Raiders of the Lost Ark). B A subset of participants (N = 17) completed five additional behavioral sessions where they performed five different multiple-arrangements tasks based on the action purpose (transitivity and sociality) of the video clips and visual content (people, objects, scenes) of still frames from the videos. Representational dissimilarity matrices (RDMs) were calculated based on the Euclidean distances in the arrangements and visualized using multidimensional scaling (MDS). C Participants produced different geometries based on the five different tasks. For example, a video depicting an individual dancer (light blue) was separated from a video depicting a group of dancers (dark blue) when participants arranged the stimuli based on sociality (top left); however, when participants arranged the stimuli based on transitivity, these two videos were clustered together (top right). When arranging static images according to the visual similarity of people, two stimuli depicting a man (in the original stimulus shown to participants; light purple) and a woman (dark purple) with haircare devices were separated (bottom left); when arranging images according to the visual similarity of objects, these two stimuli were grouped together (bottom right). Note that due to copyright reasons, original stimulus images have been replaced by similar images where the photographer and subject have provided consent for the image to be used and publicly shared. Source data are provided as a Source data file. Schematic of pipeline for representational similarity analysis. fMRI data collected while participants viewed the action stimuli were hyperaligned based on responses to a separate 1 h movie. Neural response patterns to the action stimulus were used to compute neural RDMs. A subset of participants performed behavioral arrangement tasks based on action purpose (transitivity, sociality) or visual content (objects, people, scenes); arrangements were converted into behavioral model RDMs. Annotations of videos with verb and nonverb labels were used to assign word2vec semantic embeddings to each action stimulus; verb and nonverb embeddings were used to construct semantic model RDMs. Spearman correlations were computed between different model RDMs and the neural RDMs. Multiple model RDMs were combined to jointly predict neural RDMs using multiple regression. Models of representational geometry for action understanding To understand the neural representational geometries supporting action understanding, we developed nine different representational models. These model RDMs serve as formal hypotheses about the structure of action representation to be evaluated against the neural RDMs11,104. First, we constructed two low-level visual RDMs (Fig. S1): (1) a motion-energy RDM capturing low-level dynamic visual features of video stimuli99,105,106; and (2) a gaze RDM constructed by computing the Euclidean distance between gaze trajectories acquired in a separate sample of participants (N = 17). Second, we constructed two semantic models from word embeddings derived from a human annotation of (3) nonverbs (nouns and adjectives) and (4) verbs depicted in the video stimuli74 (Fig. S2). For each word, we obtained 300–dimensional word embeddings from the word2vec model107. These embeddings geometrically capture the semantic relatedness of words based on their co-occurrence in large corpora of text. We averaged embeddings for the words assigned to each clip, then computed the pairwise cosine distance between average semantic vectors to construct nonverb and verb RDMs. None of the aforementioned models directly assay human judgments of the relatedness of different actions. To address this, we constructed two additional groups of representational models based on behavioral judgments of similarity in a multiple-arrangements task82,83,86,88. Participants were instructed to arrange the stimuli according to perceived similarity along different criterial dimensions81. First, in three separate tasks, we presented participants with still images representative of each video and asked them to arrange the stimuli according to visual features having to do with the (5) people, (6) objects, or (7) scene depicted in the images (Fig. S3). These tasks were selected to reflect three major features of cortical organization reported in the literature108. In these three tasks, participants were presented with still frames from the videos that could be clicked with the mouse to enlarge the image. Second, in two separate tasks, we presented participants with the video clips and asked them to arrange the stimuli according to two types of action content depicted in the videos (Fig. S4): (8) sociality, capturing the nature of social interactions; and (9) transitivity, capturing the object- and goal-directed nature of the actions63. In these two tasks, participants were presented with the same still frames, but the thumbnails could be clicked to enlarge and play the video clip. In the first trial of each task, all 90 stimuli were presented around the edge of a circle, and participants were asked to move the stimuli into the circle and arrange them such that more similar videos were located nearer to each other, according to the task instructions. In 12 subsequent trials, pseudo-random subsets of 30 stimuli were presented and arranged. These behavioral arrangements were reliable across subjects (average intersubject correlation of 0.473) and the reliability was consistent across models (standard deviation of 0.072 across the five behavioral models; Fig. S5). To visualize the structure of these behavioral judgments, we used multidimensional scaling (MDS)83,109–111. MDS plots of the behavioral RDMs illustrate perceived differences between videos based on different criteria (Fig. 1). For instance, in the transitivity RDM, categories such as “eating”, “tool use”, and “exercise” cluster together regardless of whether the depicted actions are social or nonsocial. By contrast, in the sociality-based behavioral RDM, social and nonsocial action videos are segregated into separate clusters, and distances among the social action videos are larger than distances among nonsocial videos. Related categories of social actions, such as “conversation” and people “eating” together, are grouped into identifiable clusters in the sociality RDM but not in the transitivity RDM. To summarize, in total we tested nine representational models: motion, gaze, nonverbs, verbs, people, objects, scene, sociality, and transitivity. Modeling neural representational geometry We first separately computed the correlation between each model RDM and searchlight-based neural RDMs across cortex. Qualitatively, this analysis revealed that the representational geometries based on the purpose of depicted actions—the transitivity and sociality RDMs—are more strongly correlated with neural RDMs than are the other seven models (Fig. 3 and Fig. S6). Significant correlations with these RDMs map out an extensive cortical system for the representation of agentic actions that includes most of the human visual system. This system includes lateral occipital cortex; temporal cortices in the ventral and inferior gyri and the superior temporal sulcus; parietal cortices in the inferior parietal lobe, intraparietal sulcus, and precuneus; and premotor cortices.Fig. 3Searchlight correlation maps for behavioral and semantic models.Representational geometries for transitivity and sociality—based on behavioral judgments of the actions depicted in the video stimuli—were highly correlated with neural representational geometries throughout the action observation network, including lateral occipitotemporal, ventral temporal, inferior and intraparietal, and premotor cortices (left). Representational geometries for the object, person, and scene content in static images yielded qualitatively lower correlations (upper right), as did semantic geometries based on annotations with descriptive verbs and nonverbs (lower right). See Fig. S6 for gaze and motion-energy maps. Spearman correlation values were computed within each participant, averaged across participants, and thresholded for statistical significance (permutation test, pFDR < 0.001). Source data are provided as a Source data file. Searchlight correlation maps for behavioral and semantic models. Representational geometries for transitivity and sociality—based on behavioral judgments of the actions depicted in the video stimuli—were highly correlated with neural representational geometries throughout the action observation network, including lateral occipitotemporal, ventral temporal, inferior and intraparietal, and premotor cortices (left). Representational geometries for the object, person, and scene content in static images yielded qualitatively lower correlations (upper right), as did semantic geometries based on annotations with descriptive verbs and nonverbs (lower right). See Fig. S6 for gaze and motion-energy maps. Spearman correlation values were computed within each participant, averaged across participants, and thresholded for statistical significance (permutation test, pFDR < 0.001). Source data are provided as a Source data file. The transitivity RDM in particular, and to a lesser extent the object arrangement and verb RDMs, yields strong correlations extending into anterior parietal cortex (aIPS) and premotor areas (vPM). The sociality RDM yields a somewhat more focal map with highest correlations in posterior LO, extending into right pSTS, and bilateral VT. The transitivity RDM and sociality RDM both yield surprisingly strong correlations in VT, given this region’s typical association with face and object processing in experiments using static image stimuli. Transitivity outperforms sociality in frontoparietal areas and anterior VT, whereas sociality outperforms transitivity in superior LO and pSTS, posterior VT, and precuneus/PMC (Fig. S7). Across essentially all of these maps, LO appears to have among the strongest correlations, corroborating its role as a hub of the action observation network that encodes a number of relevant features43. The static-image arrangement RDMs (person, object, scene) and semantic RDMs (verb, nonverb) yield partially overlapping correlation maps, with generally lower correlation values. Of the image arrangement tasks based on visual content, the numerically strongest correlations were observed for the object task, possibly due in part to the collinearity between the objects and the transitive nature of the actions. The analysis of correlations between neural RDMs and the word semantics of annotations showed that this approach produced much weaker correlations. Verb semantics, however, which generally describe the actions, were significantly stronger predictors of neural representational geometry than were nonverb semantics, which consist of nouns and adjectives that generally describe the objects, people, and scenes in the videos (Fig. S7). We replicated the foregoing analysis using neural response patterns extracted from nine ROIs extending from early visual cortex into the action observation network (Fig. 4; see “Regions of interest” in the “Methods” section). As expected, motion energy was the strongest model in early visual cortex, exceeding all other models (all pFDR < 0.001). Transitivity significantly outperformed all other models in VT, including sociality, as well as the behavioral object, person, and scene models (all pFDR < 0.001). Transitivity also significantly outperformed all other models in PPC, AIP, and VPM (all pFDR < 0.001). Behavioral judgments of transitivity and sociality were comparable to each other and significantly outperformed all other models in LO, pSTS (particularly in the right hemisphere), TPJ, and PMC (all pFDR < 0.01). Across all ROIs, behavioral judgments of transitivity and sociality were numerically highest in LO. The performance of transitivity in VT numerically exceeded all other models in all other ROIs, excluding transitivity and sociality in LO. The word embedding model for verb semantics outperformed nonverb semantics across all ROIs except EV (all pFDR < .001). Verb semantics significantly exceeded scene- and person-oriented behavioral arrangements in PPC, AIP, and VPM (all pFDR < 0.001), performing comparably to the object-oriented behavioral model. Verbs also outperformed sociality in AIP (all pFDR < 0.001).Fig. 4Model performance across nine regions of interest.ROIs were selected to span the visual processing hierarchy and encompass the action observation network. Colored markers indicate the mean Spearman correlation across participants between a given model representational geometry and the neural representational geometry in that ROI. The gray null distribution is based on randomly permuting the 90 condition labels when constructing RDMs. Thick colored error bars indicate 95% bootstrap confidence intervals based on resampling participants. Thin gray error bars indicate 95% bootstrap confidence intervals when resampling both participants and stimuli. EV early visual cortex, LO lateral occipitotemporal cortex, VT ventral temporal cortex, pSTS posterior superior temporal sulcus, TPJ temporoparietal junction, PPC posterior parietal cortex, AIP anterior intraparietal sulcus, VPM ventral premotor cortex, PMC posterior medial cortex. Source data are provided as a Source data file. Model performance across nine regions of interest. ROIs were selected to span the visual processing hierarchy and encompass the action observation network. Colored markers indicate the mean Spearman correlation across participants between a given model representational geometry and the neural representational geometry in that ROI. The gray null distribution is based on randomly permuting the 90 condition labels when constructing RDMs. Thick colored error bars indicate 95% bootstrap confidence intervals based on resampling participants. Thin gray error bars indicate 95% bootstrap confidence intervals when resampling both participants and stimuli. EV early visual cortex, LO lateral occipitotemporal cortex, VT ventral temporal cortex, pSTS posterior superior temporal sulcus, TPJ temporoparietal junction, PPC posterior parietal cortex, AIP anterior intraparietal sulcus, VPM ventral premotor cortex, PMC posterior medial cortex. Source data are provided as a Source data file. We also examined the ROIs in terms of their representational similarity with other parts of the brain. First, we correlated a representative searchlight RDM in each ROI with all other searchlight RDMs across the brain and found that each ROI was associated with partially overlapping networks of regions (Fig. S9). Second, we submitted the ROIs RDMs, alongside all nine model RDMs, to multidimensional scaling (MDS) (Fig. S10). We found that VT and LO are both located centrally, with LO located near to the transitivity and sociality models, and VT roughly equidistant from person, scene, transitivity, and sociality models. EV was located peripherally, nearest to the motion and gaze models. AIP and VPM were located farthest from EV, near to each other and to the verbs, object, and transitivity models. Lastly, pSTS, TPJ, and PMC were located relatively near to each other, and near to the sociality model. This visualization recapitulates many of the expected relationships between the ROIs and models, and locates both LO and VT centrally amongst the other ROIs and the models. We next used a multiple regression analysis to combine multiple RDMs into three joint models: (1) a behavioral “action meaning/purpose” model combining the sociality and transitivity video-arrangement RDMs capturing the dynamic, action content of the clips; (2) a behavioral “visual content” model combining the person, object, and scene image arrangement RDMs capturing the static, visual features of the clips; and (3) a semantic model combining verb and nonverb word embeddings derived from co-occurrence statistics in large corpora of text. To evaluate these models, we computed the R2 for the joint model. Together, the behavioral RDMs for sociality and transitivity, accounted for a maximum of 22% of variance (R2 = 0.22) in searchlight neural representational geometries (Fig. 5). By contrast, the behavioral RDMs based on visual content—people, objects, and scenes—accounted for about half as much variance (maximum R2 = 0.12), and the semantic RDMs based on annotations accounted for about a third as much variance (maximum R2 = 0.07). In a direct comparison, the combined transitivity and sociality model outperformed the combined object, person, and scene models across essentially the entire action observation network, as well as in VT (Fig. 5D). The combined transitivity and sociality model also outperformed the combined verb and nonverb (Fig. 5E) and combined gaze and motion models (Fig. 5F).Fig. 5Joint model fit for different families of models.Multiple model RDMs were combined using multiple regression to quantify how much variance in neural representational geometry each family of models explains. Model fit was quantified as the proportion of variance explained (adjusted R2). A Model fit for a joint model comprising transitivity and sociality RDMs based on behavioral arrangement of dynamic video stimuli. B Model fit for a joint model comprising person, object, and scene RDMs based on behavioral arrangement of static images. C Model fit based on a joint model combining verb and nonverb semantic embeddings. Adjusted R2 values were computed within each participant, averaged across participants, and thresholded for statistical significance (bootstrap hypothesis test, pFDR < 0.001). D Comparing the performance of joint models of transitivity and sociality against object, person, and scene. E Comparing the performance of joint models of transitivity and sociality against verb and nonverb. F Comparing the performance of joint models of transitivity and sociality against motion and gaze. Source data are provided as a Source data file. Joint model fit for different families of models. Multiple model RDMs were combined using multiple regression to quantify how much variance in neural representational geometry each family of models explains. Model fit was quantified as the proportion of variance explained (adjusted R2). A Model fit for a joint model comprising transitivity and sociality RDMs based on behavioral arrangement of dynamic video stimuli. B Model fit for a joint model comprising person, object, and scene RDMs based on behavioral arrangement of static images. C Model fit based on a joint model combining verb and nonverb semantic embeddings. Adjusted R2 values were computed within each participant, averaged across participants, and thresholded for statistical significance (bootstrap hypothesis test, pFDR < 0.001). D Comparing the performance of joint models of transitivity and sociality against object, person, and scene. E Comparing the performance of joint models of transitivity and sociality against verb and nonverb. F Comparing the performance of joint models of transitivity and sociality against motion and gaze. Source data are provided as a Source data file. To quantify the amount of reliable variance in neural RDMs (the noise ceiling), we calculated searchlight intersubject correlation (ISC) as the Spearman correlation between each participant’s neural RDM and the mean of other participants (Fig. S11). ISCs were strong in the same cortices that correlated with the action-purpose RDMs, with a maximum r = 0.75, indicating that the action-purpose RDMs accounted for almost 40% of the meaningful variance, as indexed by of the ISC-based noise ceiling in neural representational geometry. The models capturing behavioral judgments of visual content (person, object, science) and the semantic annotation models peaked at 21% and 12% of the meaningful variance, respectively. Variance partitioning analysis Different features of human action—for example, the motion of stirring a pot, the spoon, the pot itself, and the intention of the action—are typically intertwined in everyday life. In the same vein, there are moderate correlations between certain of our model RDMs for our selection of naturalistic video clips (Fig. S12). How can we disentangle these interrelated model features? To quantify how much variance in neural representational geometry can be uniquely explained by a given model RDM, controlling for other model RDMs, we performed a variance partitioning analysis112,113. We used hierarchical regression to estimate the variance explained by a full model containing all model RDMs, and for nested models containing all models except the model (or models) of interest (Fig. 6A). We quantified the unique variance explained as: unique R2 = full R2 – nested R2, where the nested R2 excludes the model(s) of interest. This analysis accounts for correlations amongst model RDMs and identifies variance uniquely explained by a given model, controlling for other models. To rule out alternative explanations, we also included RDMs for the number of people in each stimulus88, the body parts involved80, and action category68 among the control models (Fig. S13 and Table S3). First, we evaluated the unique variance explained by the RDMs derived from behavioral arrangements of dynamic video stimuli based on transitivity and sociality, accounting for all 10 other models (person, object, scene image arrangements, verb and nonverb semantic embeddings, gaze trajectories, motion energy, body parts, number of people, and action category). The combination of these two model RDMs uniquely explained variance throughout LO and VT (Fig. 6B), with a maximum unique R2 = 0.114 in LO. This value is comparable to the maximum non-unique variance explained by the static image arrangement RDMs (R2 = 0.118) and exceeds the semantic RDMs (R2 = 0.068) reported in the previous section; that is, the video arrangement RDMs uniquely explain as much, if not more, variance (accounting for all other models) than other families of RDMs explain in total (including variance correlated with other models).Fig. 6Unique variance explained by transitivity and sociality models.Hierarchical regression was used to quantify unique variance explained by models of interest: unique R2 = full R2 – nested R2, where the nested model excludes the model(s) of interest. The “other” partition comprises the RDMs for person, object, and scene RDMs derived from static-image arrangements, verb and nonverb semantic RDMs, and gaze and motion-energy RDMs, as well as additional control RDMs capturing number of people, the involvement of different body parts, and action category for each stimulus. A Venn diagram for variance partitioning. B Unique variance explained jointly by sociality and transitivity RDMs. C Unique variance explained by sociality RDM. D Unique variance explained by transitivity RDM. Unique R2 values were computed within each participant, averaged across participants, and thresholded for statistical significance (bootstrap hypothesis test, pFDR < 0.001). Source data are provided as a Source data file. Unique variance explained by transitivity and sociality models. Hierarchical regression was used to quantify unique variance explained by models of interest: unique R2 = full R2 – nested R2, where the nested model excludes the model(s) of interest. The “other” partition comprises the RDMs for person, object, and scene RDMs derived from static-image arrangements, verb and nonverb semantic RDMs, and gaze and motion-energy RDMs, as well as additional control RDMs capturing number of people, the involvement of different body parts, and action category for each stimulus. A Venn diagram for variance partitioning. B Unique variance explained jointly by sociality and transitivity RDMs. C Unique variance explained by sociality RDM. D Unique variance explained by transitivity RDM. Unique R2 values were computed within each participant, averaged across participants, and thresholded for statistical significance (bootstrap hypothesis test, pFDR < 0.001). Source data are provided as a Source data file. Finally, we separately evaluated the unique variance explained by the sociality RDM and the transitivity RDM. We found that the sociality RDM explained unique variance in posterior LO, right pSTS, posterior VT, and precuneus, with a maximum unique R2 = 0.060 in LO (Fig. 6C), with all other models factored out. The transitivity RDM explained unique variance across a more diffuse set of areas, including more anterior-inferior LO, AIP and vPM, as well as a more extensive expanse of VT, with a maximum unique R2 = 0.055 in lateral VT/LO (Fig. 6D), with all other models factored out. Overall, these findings indicate that both sociality and transitivity RDMs, based on the video arrangement tasks, capture a considerable amount of unique variance throughout cortex. The transitivity model captures a surprisingly large amount of unique variance in VT areas, including fusiform cortex, not typically associated with the perception of dynamic human actions. In addition to the searchlight variance partitioning analysis, we also quantified the unique variance explained by each model in each ROI (Fig. S14). The results of this analysis were qualitatively similar to the correlation-based ROI analysis (Fig. 4): behavioral models of action meaning capture large portions of variance in LO, pSTS, AIP, and VPM, as well as in VT. The larger range of models does reveal, for example, components of unique variance captured by person and scene behavioral RDMs in VT and by the body part RDM in LO, pSTS, and frontoparietal areas. Representation of action meaning in ventral temporal cortex The prevailing view of VT cortex places it near the culmination of the ventral “what” visual pathway, with a core role in visual object processing and categorization56,72,73,114. VT cortex is characterized by functionally specialized subdomains particularly sensitive to faces36, bodies37, scenes115, and other objects4,34,108,116, as well as dominant representational axes like animacy117,118 and real-world object size119. Based on this literature, we should expect the object, person, and scene static–image arrangement RDMs to provide the best representational models for VT, as these were deliberately constructed to capture behaviorally relevant structure in the domains of visual content typically associated with VT. This prevailing view of VT function, however, is largely based on experimental paradigms that use a restricted set of stimuli—mostly static images of isolated objects. A less prominent line of studies using dynamic stimuli that depict actions, often in naturalistic settings, has shown that VT cortex is strongly activated by meaningful actions, and that the neural representational geometry of VT cortex reflects distinctions and relationships between observed actions24,49,64,68,120–123. In the present paradigm, we find that models capturing behaviorally relevant action meaning—transitivity and sociality—perform surprisingly well in VT, in fact better than the object, person, and scene models (Fig. 5D, pFDR < 0.05). The transitivity model was numerically stronger in VT than all models in all other regions of the action observation network, with the exception of LO (Fig. 4). Transitivity and sociality capture relatively large proportions of unique variance in VT representational geometry even after fully accounting for the object, person, and scene models (Fig. 6). The unexpectedly strong representation of action meaning in VT raises important questions: What drives the representational geometry in VT? Are there distinct representational substructures within VT? In the following, we zoom in on VT cortex in an effort to begin addressing these questions. First, we sought to quantify how many distinct neural representational geometries are needed to describe VT cortex. We used principal component analysis (PCA) to reduce the number of searchlight RDMs from 1851 and 1785 searchlights in left and right VT, respectively, to a number of orthogonal PC RDMs. The first PC RDM explained 60% and 64% of variance across searchlight RDMs in left and right VT, respectively. In both hemispheres, 3 PCs were required to explain 75% of variance across searchlight RDMs; 16 and 15 PCs were required to explain 95% of variance in left and right VT, respectively. These observations indicate that, for the relatively diverse set of naturalistic clips used in our experiment, VT cortex hosts a number of distinct representational geometries, with a handful of RDMs capturing a substantial portion of representational structure. We next quantified the absolute semipartial Spearman correlations between each PC RDM and each of the nine model RDMs, accounting for all other model RDMs (Fig. S15). We found that PC 1 was uniquely correlated with a number of different model RDMs, most notably transitivity and motion. The sociality model yielded relatively large semipartial correlation values across PCs 2, 3, and 4, roughly matching transitivity in PC 2, and exceeding transitivity in PCs 3 and 4. Notably, the image-arrangement RDM for object features yielded negligible semipartial correlations across the first 5 PC RDMs in VT (although the person- and scene-based image-arrangement RDMs did yield moderate correlations). These findings suggest that the dominant representational structures in VT do not cleanly map onto different model RDMs; instead, each PC RDM uniquely correlates with different aspects of multiple model RDMs. Next, we aimed to identify clusters of searchlights based solely on the similarity of their RDMs. For left and right VT, we applied a k-means clustering algorithm with k = 2–10. We selected the solution at k = 4 as having the highest silhouette score in both hemispheres among lower values of k (see the “Representational structure in ventral temporal cortex” section in “Methods” for further details). Due to the overlap between neighboring searchlights, this yielded a parcellation of VT cortex comprising four spatially contiguous clusters (despite no explicit spatial information in the clustering algorithm) that were roughly homologous across left and right hemispheres (Fig. 7A): Cluster 1 comprised lateral VT including fusiform cortex; Clusters 2, 3, and 4 segmented medial VT into three parcels extending from lingual gyrus posteriorly to parahippocampal gyrus anteriorly. Next, we quantified the semipartial Spearman correlation between the centroid RDM for each of these four clusters (Figs. S16, S17) and each of the nine model RDMs, accounting for all other model RDMs (Fig. 7B). The lateral Cluster 1 was most strongly correlated with the transitivity RDM, with moderate correlations for person, scene, and other model RDMs. The posterior-medial Cluster 2 was most strongly correlated with motion energy bilaterally, as well as a strong unique correlation with the sociality RDM in the left hemisphere. The mid- and anterior-medial Clusters 3 and 4 were also uniquely correlated with several different RDMs. Surprisingly, none of these clusters exhibited strong unique correlations with the object or nonverb semantics RDMs. Note that a simple clustering algorithm like k-means imposes hard boundaries between clusters, even in the presence of smooth gradients. In our case, despite finding a reasonable clustering solution, the representational geometries of searchlights in VT appear to vary continuously between clusters (Fig. S18). We do not interpret these clusters as discrete subregions, but rather as one particular view onto the continuum of representational geometries in VT cortex.Fig. 7Clustering searchlight RDMs in VT cortex.We used the k-means algorithm to cluster searchlight RDMs separately in left and right VT cortex. A We selected k = 4 based on the silhouette coefficient. The clusters were spatially contiguous and roughly homologous across hemispheres despite the clustering algorithm only having access to searchlight RDMs. B For each cluster in the left (blue) and right (red) hemisphere, we computed semipartial Spearman correlations between the cluster centroid RDM and each model RDM, controlling for all other model RDMs. Colored markers indicate the mean semipartial Spearman correlations. Error bars indicate 95% bootstrap confidence intervals based on resampling participants. Source data are provided as a Source data file. Clustering searchlight RDMs in VT cortex. We used the k-means algorithm to cluster searchlight RDMs separately in left and right VT cortex. A We selected k = 4 based on the silhouette coefficient. The clusters were spatially contiguous and roughly homologous across hemispheres despite the clustering algorithm only having access to searchlight RDMs. B For each cluster in the left (blue) and right (red) hemisphere, we computed semipartial Spearman correlations between the cluster centroid RDM and each model RDM, controlling for all other model RDMs. Colored markers indicate the mean semipartial Spearman correlations. Error bars indicate 95% bootstrap confidence intervals based on resampling participants. Source data are provided as a Source data file. Discussion Our findings indicate that behavioral judgments of the purpose and social content of actions capture neural representational geometries throughout the action observation network, including ventral temporal (VT) cortex. While the motion-energy model was the strongest model in early visual cortex (Fig. 4), behavioral judgments of transitivity and sociality emerged as the best-fitting models as early as LO, VT, and pSTS, and were superior models across much of cortex (Figs. 3–6). Critically, behavioral judgments of the dynamic action-related content of the video stimuli (transitivity, sociality) outperformed behavioral judgments of the visual content of the stimuli (people, objects, scenes) conveyed by still images from the clips (Fig. S2). Our findings reveal surprisingly strong representation of action meaning in regions not typically associated with action understanding, particularly VT cortex, suggesting that the canonical action observation network24–26,28,43,54 should be expanded. The transitivity model, reflecting behavioral assessments of object- and goal-directed action features, explained unique variance in neural representational geometry across a large swath of areas, including the frontoparietal network, inferior LO, and anterolateral VT (Fig. 6D). Transitivity was the numerically best-performing model in LO, VT, PPC, AIP, and VPM, highlighting privileged cortical representation for object- and tool-oriented actions33–35, 122–125. To account for correlations amongst models, we also performed a hierarchical regression analysis with variance partitioning. When controlling for all other models, including the behavioral model of object features, the transitivity model still captured a large portion of unique variance throughout many of these areas (Fig. 5c). Note that the verb model outperformed nonverbs in most areas, including LO and VT, and performed relatively well across frontoparietal areas41,126,127. The sociality model, on the other hand, captured neural representational geometry in a more localized set of areas, performing comparably to transitivity in several regions of interest (Fig. 4), and significantly exceeding transitivity in focal regions of superior LO, posterior VT, pSTS, and precuneus/PMC in the searchlight analysis (Fig. S2). The sociality model explained a large portion of unique variance in LO (a larger portion than the transitivity model), as well as explaining unique variance in posterior VT, pSTS, and PMC (Fig. 6). Similarly to Wurm and colleagues46,63, we found that the sociality model peaks in superior LO, whereas transitivity predominates in inferior LO (Fig. S7)86. In line with prior work, representation of social action appears to be partially right-lateralized, with strong peaks in right VT, LO, and pSTS76,80. The overall cortical-topographic organization of different aspects of action meaning, like sociality and transitivity, may be constrained by their interdigitation with representations of objects, body parts, and other features typically associated with certain actions46,128,129. Behavioral judgments of action meaning dominate cortical action representation To obtain an assessment of the behaviorally relevant features of actions based on human raters’ intuitions, we asked participants to arrange the video stimuli based on either the purpose (transitivity) or social content (sociality) of the depicted actions84–88. We found that the combined performance of the transitivity and sociality models outstripped all other models tested, including low-level visual models, semantic models, and behavioral judgments of visual content (Fig. S2), capturing a large portion of unique variance throughout the action observation network, particularly in LO and VT (Fig. 5). This finding resonates with recent work by Contier and colleagues101 demonstrating that behavioral judgments of similarity (derived from a triplet odd-one-out task) among static images of objects provide a surprisingly strong model of functional tuning throughout visual cortex. Relatedly, Cichy and colleagues81 found, again using static object images, that behavioral judgments of perceived similarity capture rapidly emerging components of neural representational geometry that are not explained by categorical models or by bottom-up visual features learned by a deep convolutional neural network (DCNN). Taken together, these findings suggest that neural representations of both objects and actions, ranging from putatively low- to high-level cortical areas, are tightly yoked to the behavioral goals that guide judgments of similarity100,102,130,131. The previous argument, however, does not address our observation that behavioral judgments of action meaning (transitivity, sociality) in dynamic videos also better capture neural representational geometry than behavioral judgments of visual content (people, objects, scenes) in static images (Fig. S2). The person, object, and scene behavioral RDMs were expressly constructed to evaluate the contribution of the non-action-related components of our stimuli fairly against behavioral models of action meaning (transitivity, sociality). Part of this effect may be driven by the stimuli themselves: dynamic, naturalistic video stimuli more broadly and robustly engage the visual system than highly-controlled experimental videos or static images64,75,132–135. Certain kinds of meaning can emerge from the dynamic evolution of visual features136 that are not explicit, or present at all, in static images. Behavioral judgments of dynamic clips may better align with neural activity encoding these dynamic features than behavioral judgments of corresponding still frames. That said, we suspect that part of the superiority of the transitivity and sociality models owes to the demands of the corresponding arrangement task. The transitivity and sociality arrangement tasks were designed to focus on two major dimensions of visual action understanding63,88 and orient participants toward the meaning or purpose of the actions depicted in the stimuli—that is, toward the “why” of action observation, that culminates in intentions137–139 and social cognition more broadly140–142. If the adaptive value of action perception is ultimately in understanding the physical and social ramifications of an observed action, then these intentional and social features of action may occupy a privileged role in cortical representation. Action representation in the ventral visual pathway The prevailing model of the ventral visual pathway does not include representation of observed action4,11,56,72,73,143–149. With this in mind, we expected the behavioral arrangement models based on person-, object-, and scene-related visual content to perform best in VT. Surprisingly, although these models were correlated with VT representation, they did not outperform more action-oriented models. The transitivity and sociality models combined significantly outperformed the object, person, and scene models in VT (Fig. 5D) (as well as all other groups of models). Even when fully controlling for the object, person, and scene models (as well as all other models), the transitivity and sociality models each captured a surprisingly large portion of unique variance in VT representational geometry (Fig. 6): the sociality model captured unique variance in posterior VT and focal regions of the fusiform gyrus; the transitivity model captured unique variance in a larger portion of VT extending more anteriorly. (Note that the verb semantic model also outperformed the nonverb semantic model in VT; Fig. S7). VT cortex is not monolithic, and we found variation in searchlight representational geometries across the extent of VT, with transitivity most strongly represented in lateral VT and sociality in left posteromedial VT (Fig. 7). That said, none of the clusters of searchlights we identified in VT cleanly corresponded to particular models, like person or object; each cluster in VT appears to uniquely encode features across a number of different models in varying proportions. Critically, the object, person, and scene models did not dominate any of these clusters in VT, and the object RDM in particular was superseded by other RDMs across all clusters (Figs. 7 and S15). Despite the prevailing view of the ventral visual pathway, these findings reinforce a less prominent line of work showing that VT encodes dynamic qualities of agentic behavior. For example, prior work going back almost 30 years has shown that lateral fusiform cortex responds to simple animations depicting agentic behavior94,96–98 and goal-directed behaviors performed by automated manufacturing tools95. In recent work, Russ and Leopold have shown that homologous face-selective areas in macaque cortex are strongly driven by biological, socially relevant motion49 and are highly sensitive to temporal continuity in dynamic, naturalistic videos69. Relatedly, in a study using naturalistic clips of behaving animals, Nastase and colleagues68 have shown that VT encodes action categories in a way that generalizes across the agents performing them. Recent work in the action observation literature has also reported representation of transitivity and sociality, as well as abstract action categories, in VT cortex63,121,150,151. We contend that, taken together, these results demand a reassessment of VT function to incorporate observed action representation. We are not arguing that features like animacy118, real-world size119, and other object features are not important dimensions of neural representational geometry in VT cortex. Rather, we contend that representations of these object features in VT are closely coupled with the actions they afford in natural contexts, and that the actions play an unexpectedly large role that cannot be reduced to the objects alone. The dominance of the transitivity model over other models, particularly the object model, suggests that VT may encode properties of objects as they relate to the actions in which these objects play a meaningful role in everyday life. Action understanding for natural vision and behavior The foregoing discussion raises some questions we believe are deserving of reflection. Why have we historically underestimated the role of action meaning in shaping neural representational geometry? Why have we underestimated the importance of action representation in cortical areas like VT? With regard to the first question, one possible explanation is that we simply have not used a diverse or meaningful enough sample of stimuli to afford rich behavioral distinctions85,88,101,152. This limitation is likely a symptom of deeper biases in our approach, however. The vast majority of experiments in the action observation literature deliberately design stimuli and tasks to isolate particular features of actions chosen by the experimenter; for example, brief videos of an isolated hand acting on an object in two different ways62,151. The literature surrounding object representation in VT is, by and large, derived from contrasts between static images of isolated objects varying in particular ways4,36,115. On the other hand, many experiments aiming to localize neural sensitivity to biological motion or agency deliberately strip away information about the form of agents and objects47,52,94. Experiments following this “divide and conquer” strategy153 are useful in their own right, but they cannot give us a complete picture. We are trying to understand a system that is highly flexible and interconnected, that is exquisitely sensitive to context, and that is tuned to the statistical structure of the natural world65,154. The “confounds” we often strive to rule out in experimental manipulations are nearly ubiquitous in real-world contexts, and the brain appears to take advantage of these statistical dependencies155. This creates a dilemma: each individual experimental paradigm is designed to isolate and describe a particular piece of the puzzle—but this turns out to make the puzzle pieces very difficult to reassemble. For example, if we take an exquisitely face-selective puzzle piece156,157 and try to fit it back into the “big picture” of dynamic, natural vision, this seemingly well-behaved puzzle piece seems to change shape49,69,120,158,159. We encounter a similar dilemma in early visual cortex66,160. Experimentalists design paradigms to minimize unwanted variance and manufacture a certain kind of data, discarding the statistical dependencies that may cut across puzzle pieces. When it comes to the brain, if we try to isolate a particular puzzle piece, it may change shape. If we zoom in on a particular puzzle piece, it becomes difficult to judge its relative size; we run the risk of overestimating its importance relative to other puzzle pieces. Naturalistic paradigms offer a way to reassemble these puzzle pieces: they force us to (re)examine each piece in its surrounding context, where natural statistical dependencies cut across pieces, and allow us to better evaluate the relative contributions of different puzzle pieces to the larger picture64,65. The current paradigm serves as a compromise between structured experiments and unconstrained videos or theatrical films74,76,161–163. Our use of arrangement tasks allows us to solicit relatively rich, continuous behavioral judgments without (a) imposing our preconceptions via experimental design, and without (b) relying on predetermined annotations or questionnaires—for example, binary questions like “is this action directed at an object or set of objects?”80 or “is a social interaction present?”76, or Likert-scale ratings of questions like “are the people acting independently or jointly?”77. Our findings suggest that these behavioral arrangements better capture neural representational geometry than annotation-based semantic vectors74. The current work also has several limitations. First, our paradigm was not designed to examine individual differences in behavioral or neural representational geometry164,165. Second, our passive viewing task with intermittent probe verbs during fMRI acquisition cannot speak to prior work showing that neural representational geometry flexibly reshapes to subserve particular task goals68,78,120,166. Third, we do not test any artificial neural network models of visual processing among our selection of models76,77. Our paradigm yielded a large amount of explainable variance in neural representational geometry, particularly in LO and VT, with a maximum of intersubject Spearman r = 0.75, (R2 ≈ 0.56), compared to the similarly computed noise ceiling of Kendall’s τ = 0.26 for object images in VT148. The combination of best performing models—transitivity and sociality—yielded a maximum r ≈ 0.47 (R2 = 0.22), corresponding to roughly 40% of the meaningful variance in neural representational geometry. This means that more than half of the meaningful variance in neural representational geometry remains unexplained. We hope that neural network models for natural vision will ultimately unravel the neural computations giving rise to action understanding. That said, we are not overly optimistic that current neural network models will capture a large portion of this variance. Jiahui, Feilong and colleagues167, for instance, have shown that state-of-the-art face recognition networks capture a relatively small fraction—3% at best—of the representational geometry in neural responses to dynamic, naturalistic videos of faces (and account for only 27% of variance in behavioral judgments of face similarity). Our findings suggest that the simple categorization or recognition tasks and static image stimuli used to train typical visual neural networks may not suffice to learn the dynamic, behaviorally relevant features of human action representation. However, as neural networks advance to pursue more interactive, more social, more “human” objectives, they may very well close the gap. Methods Participants Twenty-three right-handed adults (12 females; mean age ± SD: 27.3 ± 2.4 years) participated in the fMRI experiment. Each participant completed two 1-hour scanning sessions probing action representation, an additional 1-hour movie-viewing session outside the scanner (first half of the movie) immediately followed by a 1.5 h movie-viewing session in the scanner (second half of the movie), and two 1 h behavioral sessions. A subset of 17 of these participants then completed three follow-up behavioral sessions, each lasting approximately 1 h. This amounts to roughly 3.5 h of scanning time (including structural scans) and 8 h of data collection in total per participant. A separate sample of 19 adults (11 females; mean age ± SD: 19.7 ± 2 years) participated in a 1 h eye-tracking session. Two of these participants were excluded due to incomplete data collection. All participants gave written, informed consent prior to participating in the study. The study was approved by the Institutional Review Board of Dartmouth College. Participants were compensated for their time. Data are available on OpenNeuro: https://openneuro.org/datasets/ds007942168. Code used for subsequent analyses is available on GitHub: https://github.com/HaxbyLab/action-geometry169. Stimuli and design Stimuli were selected to span the space of everyday human action as comprehensively as possible across a broad spectrum of perceptual and semantic features65,74,134,170–172. We initially developed a set of 18 action categories spanning social and nonsocial actions, then sought out at least five completely different videos per category, yielding 90 distinct video stimuli total. The videos were sampled from different sources on YouTube with the intention of maximizing variance in terms of actors, scenes, filming styles, and so on, while maintaining a focus on real-world, everyday depictions of human actions. Two research assistants (authors R.P. and M.K.T.) selected and curated the stimuli under the supervision of authors S.A.N., M.I.G., and J.V.H. A total of 181 different video clips were initially compiled. These video clips were annotated manually by both research assistants for the number of people depicted (including separate counts of men and women), as well descriptive words from the following categories: people-related words, body part-related words, object-related words, scene-related words, social/affective words, and verbs. The top five most relevant descriptive words were retained for each video clip. From this initial sample of 181 videos, the best 90 videos were selected to span the 18 social and nonsocial action categories outlined in the design. Each video was then manually trimmed (by M.K.T. and S.T.) using video editing software to avoid extraneous camera cuts and isolate the most representative 2.5 s segment. The selected stimuli bridge the gap between the curated narratives of commercial cinema and the authenticity of real-world scenes. The stimuli were categorized into 18 groups, delineated by social and nonsocial actions, with 5 exemplar clips per category. These action categories span a variety of different everyday activities quantified in the American Time Use Survey (ATUS); see Table S2 for the corresponding activities and their frequency in everyday American life. Social actions encompassed conversation, intimacy (e.g., hugging), teaching, and assembly-line work, while nonsocial actions included cooking, gardening, arts and crafts, and musical performance (e.g., playing an instrument). Additionally, five actions were classified under both social and nonsocial contexts: eating, dancing, exercising, cosmetics and grooming (e.g., hair styling, tooth brushing), and manual tool use (e.g., operating a power drill, using a saw). This categorization aimed to guarantee a diverse selection of content without participant awareness of the categorization or engagement in a categorization task. Despite the binary social/nonsocial categorization, some nonsocial clips contained social elements (e.g., bystanders). The selection deliberately varied in visual properties, the number of actors, and other semantic aspects such as actor gender and ethnicity, setting (indoor/outdoor), among others. We developed a condition-rich, rapid event-related design, treating each of the 90 stimuli as a distinct experimental condition11. Each trial consisted of a 2.5 s video clip presentation followed by a jittered interstimulus interval featuring a fixation cross, averaging 2.5 s (Fig. 1). Random variation in the jittered interstimulus fixation intervals was constrained such that no fixation interval was briefer than 2 s. The stimulus onset times were jittered using AFNI’s make_random_timings.py utility following an exponential decay curve173 (pp. 84–86). For each participant, 1000 random onset sequences were generated, evaluated for efficiency174, and the most efficient design was selected. To avoid overrepresented transitions between conditions by chance, type 1 index 1 serially balanced sequences were used to ensure that each trial type precedes and follows every other trial type175. However, because a type 1 index 1 sequence is overly long for 90 conditions, we counterbalanced the presentation order of the 18 action categories. In addition to the 18 categories, we included null fixation trials and probe trials (described below) for 20 total trial types (amounting to ~5% fixation trials). A type 1 index 1 sequence for n = 20 trial types comprises n2 = 400 trials, with 360 trials of interest (excluding null fixation trials and probe trials). For a single participant, two unique type 1 index 1 trial orders were constructed and used for two scanning sessions on separate days (800 total trials, 720 trials of interest, 8 trials per stimulus, 4 trials per stimulus per session). For each trial order used, we first generated 1000 sequences and selected the sequence with the highest efficiency. Trial sequences and timing were unique to each participant and session. Experimental stimuli were presented using PsychoPy176. For a single scanning session, 400 trials were divided into four scanning runs of 100 trials each, totaling eight runs over two sessions. Runs were organized into blocks of 20 trials, where each of the 18 action categories, alongside a null fixation trial and a probe trial, was featured once in a randomized order. Each time an action category occurred, we randomly sampled without replacement from the five video clip exemplars for that category such that each exemplar occurred once per run. For each scanner run, the last three trials of the previous run (approximately 15 s of stimulation) were prepended to the beginning of the run to reinstate the temporal context of the sequence. The volumes acquired during these prepended trials were discarded prior to analysis. The first run of a session was prepended with the last three trials of the final run for that session. These three preparatory trials at the beginning of each run were sampled from a separate set of 18 clips (one for each category) not otherwise used in the stimulus set to ensure that no exemplar repetitions occurred in any run. An additional 5 s of fixation and 15 s of fixation were appended to the beginning and end of each run, respectively, making each run 535 s in duration, or approximately 9 min. Participants were instructed to pay attention to the clips and to keep their eyes on the fixation cross between clips. To ensure participants remained vigilant, we included probe trials using a two-alternative forced-choice semantic task. Prior to the experiment and before each run, participants were informed that they would occasionally be presented with two verbs and asked to answer the following question: “Which of these two verbs is most closely related to the action depicted in the preceding clip?” (see Table S1 for a full list of probe verbs). Probe trials occurred five times per run (once for each block of the type 1 index 1 sequence). The locations (i.e., left or right) of the particular verbs in a given pair were determined randomly per probe trial. This semantic probe was presented for 2.5 s, and participants were instructed to respond during this period using either the left or right buttons of a single response box (held in the right hand). Participants were familiarized with the task and the format of the probe question prior to scanning. The probe verbs were sampled from the WordNet lexical database177,178(Table S1). For each implicit action category (as listed above, e.g., “conversation”), we retrieved four related verbs from WordNet (e.g., “argue,” “chat,” “converse,” and “discuss”). The related verbs were typically troponyms (i.e., subordinate verb categories) of some superordinate verb, e.g., “talk.” Troponyms were preferentially selected to minimize participant exposure to overarching categories. This means that most probe verbs were typically not a perfect description of the action depicted in the preceding clip, making the task non-trivial. The 80 probe verbs had a median depth of 3 (mean = 3.063, SD = 1.184) in the WordNet hierarchy, indicating that they were fairly specific subordinate categories. Note that verb hierarchies are generally shallower than noun hierarchies, rarely exceeding four levels177. The structure of the type 1 index 1 sequences ensured that each action category was followed by a semantic probe trial once per scanning session. Because the five action categories depicted in social and nonsocial had related sets of verbs, we ensured that probe verbs from the social or nonsocial version of an action category were never paired (e.g., a verb from the social eating category was never paired with a verb from the nonsocial eating category). For trials where a semantic probe trial immediately followed a fixation trial or another probe trial (each necessarily occurring once per session due to the type 1 index 1 sequence), we replaced the trailing probe trial with a fixation trial. Behavioral responses (i.e., button presses) were monitored online during the scanning session to ensure task compliance. However, the log of recorded button presses was incomplete due to a technical error, and therefore in-scanner behavioral responses were not further analyzed. For the movie session, the film Raiders of the Lost Ark was split into 8 roughly 15 min segments. The segments were 840, 860, 860, 815, 850, 860, 860, and 850 s in duration. The first four movie segments were viewed outside the scanner immediately before the scanning session. Participants then freely viewed the latter four segments of the movie in the scanner. MRI acquisition All functional and structural images were acquired using a 3 T Siemens Magnetom Prisma MRI scanner (Siemens, Erlangen, Germany) with a 32-channel phased-array head coil. Functional, blood-oxygenation-level-dependent (BOLD) images were acquired in an interleaved fashion using gradient-echo echo-planar imaging with pre-scan normalization, fat suppression, a multiband (i.e., simultaneous multi-slice; SMS) acceleration factor of 4 (using blipped CAIPIRINHA): TR/TE = 1000/33 ms, flip angle = 59°, bandwidth = 2368 Hz/Px, resolution = 2.5 mm3 isotropic voxels, matrix size = 96 × 96, FoV = 240 × 240 mm, 48 axial slices, anterior–posterior phase encoding. At the beginning of each run, three dummy scans were acquired to allow for signal stabilization. For each participant, eight runs were collected in two separate scanning sessions, each consisting of 535 volumes totaling 535 s (~9 min). At the beginning of each scanning session, a T1-weighted structural scan was acquired using a high-resolution single-shot MPRAGE sequence with an in-plane acceleration factor of 2 using GRAPPA: TR/TE/TI = 2300/2.32/933 ms, flip angle = 8°, resolution = 0.9375 × 0.9375 × 0.9 mm voxels, matrix size = 256 × 256, FoV = 240 × 240 × 172.8 mm, 192 sagittal slices. For the movie session, both functional and structural images were acquired using the parameters specified above. For each participant, four functional runs were collected. The eight runs are each roughly 15 min long, comprising 850, 860, 860, and 850 volumes (3420 volumes in total). Preprocessing All MRI data were preprocessed using fMRIPrep179. Anatomical images were skull-stripped using ANTs180. Tissue-based segmentation isolating gray matter, white matter, and cerebrospinal fluid was implemented using FSL’s FAST181. For each participant, anatomical images were registered across sessions using FreeSurfer182. Cortical surfaces were reconstructed from T1- and T2-weighted anatomical scans using FreeSurfer, spatially normalized based on sulcal curvature183. For the main experiment, a functional reference image was created for each participant using the median image after correction for head motion. Head motion parameters were estimated using FSL’s MCFLIRT184. Correction for slice-timing was performed using AFNI’s 3dTshift185. Functional images were then aligned to the gray–white matter boundary of the T1-weighted anatomical image (estimated by FreeSurfer) in a single interpolation step using FreeSurfer’s 9-parameter affine boundary-based registration algorithm186. Functional data were projected onto the surface by averaging, at each surface vertex, values at six intervals sampled along a normal spanning the white matter and pial boundaries. Surface data were normalized to the fsaverage6 template with 40,962 vertices per hemisphere (81,924 vertices total). General linear model Voxelwise general linear models were used to estimate response patterns for each of the 90 conditions. Stimulus-evoked response patterns for each event were modeled using a hemodynamic response function adjusted for a 2.5 s stimulus duration. The following nuisance variables were included in the model: six regressors accounting for head motion, 4th-order Legendre polynomial trends modeling slow baseline fluctuations, a regressor capturing framewise displacement187, and the first five principal components estimated from tissue-segmented cerebrospinal fluid (ventricle) time series (aCompCor188). The first three preparatory trials, as well as the probe verb trials, were modeled with two additional nuisance regressors. Voxelwise models were estimated using AFNI’s 3dREMLfit, which accounts for temporal autocorrelation using an autoregressive–moving-average ARMA(1,1) model. Regression coefficients for each of the 90 conditions of interest were estimated across the four runs in each of the two scanning sessions; four trials contributed to each coefficient. This resulted in two sets of coefficients, one for each session. We z-scored response profiles across the 90 conditions for each voxel prior to further analysis. For the movie session, voxelwise general linear models were used to regress out confounding variables. As above, nuisance variables included six regressors for head motion, a regressor for framewise displacement, 2nd-order Legendre polynomial trends, and the first five principal components from the ventricle time series. Bandpass filtering was also performed to remove temporal frequencies lower than 1/150 Hz and higher than 0.1 Hz. The regression model for the movie session was estimated using AFNI’s 3dTproject. The movie response time series at each voxel were z-scored prior to further analysis. Hyperalignment We used hyperalignment to align individual-specific cortical-functional topographies into a common response space171. Specifically, we used a hybrid hyperalignment algorithm that estimates alignment parameters from a combination of both functional response time series and functional connectivity103. We used a surface-based searchlight algorithm (20 mm searchlights) to construct a single whole-brain transformation (computed separately for each cerebral hemisphere) comprising locally-constrained transformation matrices189. Searchlight hyperalignment has previously been shown to improve the consistency of searchlight representational geometries across individual participants68. Hybrid-hyperalignment parameters were estimated from data acquired in an independent movie scanning session where participants watched the second half of Raiders of the Lost Ark (~1 h in duration and 3420 volumes of movie-watching in total). These participant-specific transformations were then applied to the whole-brain response maps comprising the 90 coefficients estimated from the first-level regression model. All subsequent analyses were applied to response patterns hyperaligned into this common response space. Both hyperalignment and the subsequent multivariate pattern analyses were performed using PyMVPA190. Regions of interest We constructed nine regions of interest (ROIs) intended to inclusively span a visual hierarchy for action understanding, ranging from early visual cortex into higher-order visual areas, areas of the traditional action observation network, as well as areas associated with biological motion and social cognition. Each ROI was constructed from one or more parcels from the multimodal parcellation by Glasser and colleagues191 on the fsaverage6 template183 to best reflect regions commonly reported in the literature. Here we describe the parcel names and numerical labels for each ROI. Early visual (EV) cortex (right: 1037 vertices; left: 1047 vertices): Primary Visual Cortex (V1, 1). Ventral temporal (VT) cortex (left: 1851 vertices; right: 1785 vertices): Third Visual Area (V3, 5), Fourth Visual Area (V4, 6), Eighth Visual Area (V8, 7), Fusiform Face Complex (FFC, 18), ParaHippocampal Area 1 (PHA1, 126), ParaHippocampal Area 3 (PHA3, 127), Area TF (TF, 135), Area TE2 posterior (TE2p, 136), VentroMedial Visual Area 1 (VMV1, 153), VentroMedial Visual Area 3 (VMV3, 154), ParaHippocampal Area 2 (PHA2, 155), VentroMedial Visual Area 2 (VMV2, 160), Ventral Visual Complex (VVC, 163). Lateral occipitotemporal (LO) cortex (left: 690 vertices; right: 746 vertices): Middle Superior Temporal Area (MST, 2), Middle Temporal Area (MT, 23), Area TemporoParietoOccipital Junction 2 (TPOJ2, 140), Area TemporoParietoOccipital Junction 3 (TPOJ3, 141), Area V4t (V4t, 156), Area FST (FST, 157), Area Lateral Occipital 3 (LO3, 159). Posterior superior temporal sulcus (pSTS; left: 765 vertices; right: 947 vertices): Superior Temporal Visual Area (STV, 28), Area TemporoParietoOccipital Junction 1 (TPOJ1, 139), Area PGi (PGi, 150). Temporoparietal junction (TPJ; left: 1128 vertices; right: 1078 vertices): Area PFm Complex (PFm, 149), Area PGi (PGi, 150), Area PGs (PGs, 151). Posterior parietal cortex (PPC; left: 1348 vertices; right: 1220 vertices): IntraParietal Sulcus Area 1 (IPS1, 17), Lateral Area 7A (7AL, 42), Lateral Area 7P (7Pl, 46), Area 7PC (7PC, 47), Area Lateral IntraParietal ventral (LIPv, 48), Ventral IntraParietal Complex (VIP, 49), Medial IntraParietal Area (MIP, 50). Anterior intraparietal cortex (AIP; left: 1479 vertices; right: 1301 vertices): Area PFt (PFt, 116), Anterior IntraParietal Area (AIP, 117), Area PF opercular (PFop, 147), Area PF Complex (PF, 148). Ventral premotor cortex (VPM; left: 769 vertices; right: 828 vertices): Premotor Eye Field (PEF, 11), Ventral Area 6 (6 v, 56), Rostral Area 6 (6r, 78), Area IFJa (IFJa, 79), Area IFJp (IFJp, 80). Posterior medial cortex (PMC; left: 2010 vertices; 2126 vertices): Sixth Visual Area (V6, 3), Parieto-Occipital Sulcus Area 2 (POS2, 15), PreCuneus Visual Area (PCV, 27), Medial Area 7P (7P, 29), Area 7m (7m, 30), Parieto-Occipital Sulcus Area 1 (POS1, 31), Area ventral 23 a + b (v23ab, 33), Area 31p ventral (31pv, 35), Medial Area 7A (7Am, 45), Dorsal Transitional Visual Area (DVT, 142), Area V6A (V6A, 152), Area 31pd (31pd, 161). Models of representational geometry Motion energy To capture the dynamic, low-level visual features of each stimulus, we submitted each clip to a neurally-inspired motion-energy model99,105,106,192 implemented using pymoten193. Pymoten uses a pyramid of spatiotemporal Gabor filters, which have multiple spatial and temporal frequencies, directions of motion, positions, and sizes, designed to mimic motion-sensitive receptive fields in primate visual cortex. The motion-energy model yields a 2530-dimensional vector at each frame of a clip. We then averaged these vectors across the 2.5 s duration of a clip. Note, however, that unlike Nishimoto and colleagues99, we did not demand that participants maintain central fixation for the sake of a more naturalistic viewing experience. This may reduce the overall effectiveness of the motion-energy model, as it does not account for saccades and foveation. Gaze trajectories An independent sample of N = 17 subjects participated in the eye-tracking experiment, where they were presented with the 90 video clips from the main scanning experiment. Each clip was presented for 2.5 s while eye movements were recorded using an SR Research EyeLink 1000 Plus. Clips were presented in random order in four blocks, with each stimulus occurring once per block (4 repetitions total). Participants were instructed to monitor for repetitions of the same clip stimulus, and 10 repetitions occurred per block. We directly analyzed the measured gaze trajectory (x and y coordinates over time). The trajectory was median filtered using a rolling window with a width of 84 ms and linear interpolation. Eye blinks indicated by the EyeLink software were censored and interpolated using the median filter194. Trials where the eyes were closed for the entire duration of the stimulus were excluded, and blocks missing several or more trials (due to measurement error or participant compliance) were not further analyzed. Gaze trajectories were then decimated from 2500 samples at a sampling rate of 1000 Hz to 60 samples over 2.5 s (one sample per frame at a frame rate of 24 Hz). For each block, the preprocessed gaze trajectories were used to construct an RDM capturing differences in spatiotemporal eye movement. To quantify the similarity of two gaze trajectories, we computed the Euclidean distance between the location of gaze (in two-dimensional screen coordinates) at each sample, and summed these distances across the stimulus presentation. To quantify the reliability of gaze trajectories, we first computed the Pearson correlation of gaze RDMs across blocks within each participant. In the sample of 17 participants, the mean inter-block pairwise correlation averaged across participants was 0.186 (SD = 0.137). To create a clean gaze RDM for further analysis, we excluded gaze RDMs from participants for which the inter-block (intra-participant) correlation of gaze RDMs was less than Pearson r = 0.1. Gaze trajectories for the remaining nine participants were used for all further analyses. The mean inter-block pairwise correlations for this subset of participants were 0.280 (averaged across participants, SD = 0.093). This suggests that there is modest consistency in each participant’s gaze allocation across blocks. To quantify inter-participant consistency of gaze trajectories, we averaged the gaze RDMs across blocks within each participant, then computed the Pearson correlation between these averaged gaze RDMs for each pair of participants. The mean inter-participant pairwise correlation of gaze RDMs was 0.390 (SD = 0.116 across pairs of participants), indicating that gaze RDMs are fairly consistent across individuals. The gaze RDMs for the remaining nine participants were averaged across blocks and across participants to construct a single gaze RDM. This average RDM was used in further analyses as a model of gaze allocation for the clip stimuli. Word embeddings Two annotators manually assigned semantic labels to the 90 clip stimuli (as in Huth et al.74). Annotators were instructed to consider several factors when labeling the clips: person-related features such as gender, ethnicity, appearance, and body parts; object-related features such as tools used; scene-related features, such as indoor and outdoor contexts; and verbs describing the actions depicted41,126. Annotators were then instructed to select from this exhaustive set of labels the five most descriptive or salient labels for each clip. To quantify the semantic relationships among clips according to their assigned labels, we extracted 300-dimensional word embeddings from word2vec107. We used pretrained semantic vectors based on the ~100 billion words Google News corpus. In this model, semantic structure is derived from co-occurrence statistics in large corpora of online text. Semantic relationships are encoded in the geometric relationships between word embeddings in a high dimensional vector space where more similar words are located nearer to each other. To assess inter-annotator agreement, we assigned word embeddings to each of the top five labels selected by the two annotators and averaged these five vectors per stimulus, resulting in 90 semantic vectors for each of the two annotators. For each stimulus, we then computed the Pearson correlation between vectors for the two annotators. The average correlation between the two annotators across clips was 0.702 (SD = 0.123, range: 0.364–0.942). This indicates substantial agreement between the two annotators. We then combined the two annotations and split the aggregated labels into verbs and nonverbs (person-, object-, and scene-related nouns and adjectives). The verb annotation comprised, on average, 3.311 words per stimulus (SD = 0.755, median = 3, range: 2–5 words), and the nonverb annotation comprised on average 4.722 words (SD = 0.633, median = 5, range: 3–6 words). For each stimulus, we separately averaged the verb embeddings and the nonverb embeddings. To construct verb and nonverb RDMs, we computed the cosine distance between the associated vectors for each pair of stimuli. Behavioral arrangements From the original sample of fMRI participants (N = 23), 17 participants returned for five ~1 h behavioral tasks after completing the scanning sessions. To acquire behavioral judgments of stimulus similarity, we used a multiple-item arrangement paradigm82,83. Participants were presented with sets of stimuli positioned outside a large circle (or “arena”), and were instructed to organize these stimuli within the circle (see Fig. 1B for an example of the starting positions and a final arrangement). Although in typical arrangement tasks participants are simply asked to judge the “similarity” between stimuli (where the features of “similarity” are left to the participant’s interpretation88), we designed several different arrangement tasks to orient participants toward theoretically motivated features of the stimuli81. Participants first performed two tasks in the two separate sessions: a “sociality” arrangement task and a “transitivity” arrangement task63, both designed to assess behaviorally relevant features of the dynamic (inter)actions depicted in the video stimuli. The task/session order was counterbalanced across participants. Although these tasks were inspired by Wurm and colleagues63, we aimed to elicit more rich, meaningful judgments of sociality and transitivity. In Wurm et al.63, sociality and transitivity are experimentally defined as binary factors: object-related (transitive) versus object-unrelated (intransitive), and social versus nonsocial. With respect to transitivity, this binary definition does not fully capture the many different ways objects are meaningfully incorporated into purposeful actions in everyday life. In the same vein, humans engage in a large variety of different social interactions. For this reason, we designed the transitivity task instructions to more inclusively prompt participants to consider the role of objects and action goals (transitivity) and the different kinds of social interactions (sociality) depicted in the stimuli. In the sociality arrangement task, participants were instructed to organize the stimuli according to the social interaction depicted (if any). They received the following instructions: “Move the images into the circle and organize them so that clips depicting similar types of social interaction are nearest to each other. The more similar the two clips are in terms of social interaction, the closer the images should be.” In the transitivity arrangement task, participants were instructed to organize the stimuli according to the role of objects (if any) and the goal of the action. They received the following instructions: “Move the images into the circle and organize them so that clips in which similar objects play similar roles, and in which the actions have similar goals, are nearest to each other. The more similar the two clips are in terms of objects and goals, the closer the images should be.” A reminder of the task (“social interaction” or “object/goal”) was present in the upper left corner for the duration of the experiment. The task was self-paced, and participants were verbally informed that they could ignore any task-irrelevant stimulus features. Participants later performed three additional behavioral arrangement tasks in three separate sessions: a “person” arrangement task, an “object” arrangement task, and a “scene” arrangement task, all related to the visual features depicted in static-image frames from each original stimulus. The task/session order for the static-image arrangement tasks was also counterbalanced across participants. In the person task, participants received the following instructions: “Move the images into the circle and organize them so that images depicting similar people are nearest to each other. The more similar the people, the closer the images should be.” In the object task, they received the following instructions: “Move the images into the circle and organize them so that images depicting similar objects are nearest to each other. The more similar the objects, the closer the images should be.” In the scene task, they received following instructions: “Move the images into the circle and organize them so that images depicting similar scenes or places are nearest to each other. The more similar the scenery, the closer the images should be.” For all arrangement tasks, participants were required to arrange 13 sets of stimuli. The first set included all 90 stimuli, while the subsequent 12 subsets each included 30 stimuli. We pseudo-randomly assigned stimuli to each of the 12 subsets82. We generated random subsets by repeatedly permuting the list of stimuli and selecting subsets of 30 stimuli while recording the number of unique stimulus pairs occurring across each set of 12 subsets. We repeated this procedure 1000 times and selected the set of 12 subsets with the greatest number of unique pairs. The stimuli were initially positioned at uniform intervals outside the arena, and the starting positions for each set of stimuli were determined randomly per participant. The final positions of the stimuli in each set were recorded after participants finalized their arrangement. Presenting all 90 stimuli in the first set served a dual purpose: first, participants were able re-familiarize themselves with the stimuli (both behavioral sessions occurred after the scanning sessions) and appreciate the scope of the stimulus set; second, this ensured that we acquired at least one measurement of the distance between every pair of stimuli. This first arrangement is inherently two-dimensional. However, aggregating distances from additional random subsets of the stimuli has been shown to afford psychological spaces exceeding two dimensions82. Each stimulus was represented by the first frame of the clip, cropped to a square and presented at 72 × 72 pixels. Stimulus images were resized so as to accommodate all 90 stimuli on the screen in the first set. Participants could use the mouse to either right-click and drag a stimulus to the desired location, or could left-click to select a stimulus and then click elsewhere to make the stimulus jump to the desired location. Finally, for the two video arrangement tasks (sociality and transitivity), participants could middle-click on a stimulus to inflate the stimulus to 256 × 144 pixels (the original aspect ratio) and play the 2.5 s video clip. In the static image arrangement tasks, the middle-clicked simply enlarged the still image. The experimental interface was created using PsychoPy176. For each set of stimuli arranged by a participant, we computed the pairwise Euclidean distances between all stimuli in screen coordinates. For the first stimulus set containing all 90 stimuli, this yielded a full dissimilarity matrix. We then averaged the sparse pairwise distances computed from each subsequent random subset of stimuli with this dissimilarity matrix. Finally, we averaged the resulting RDMs across participants for subsequent analyses. Representational similarity analysis We used representational similarity analysis (RSA) to evaluate different models of the neural representational spaces supporting naturalistic action understanding. As described in the previous section, we focused on nine models of representational geometry: (1) motion energy, (2) gaze, (3) nonverb and (4) verb semantic embeddings, behavioral arrangements of static images based on (5) scene, (6) person, and (7) object content, and behavioral arrangements of dynamic videos based on (8) sociality and (9) transitivity. Each of these representational models is summarized into a 90 × 90 representational dissimilarity matrix (RDM) and embodies a hypothesized representational space. We first used an exploratory surface-based searchlight analysis (10 mm radius) to map out the distribution of representational geometries throughout cortex195,196. To construct neural RDMs, we computed the pairwise Pearson correlations between response patterns (hyperaligned regression coefficients) for each of the 90 action stimuli across the two scanning sessions (split-data RDMs197,198), yielding a 90 × 90 neural RDM at each searchlight/cortical region. We first computed the intersubject correlation of searchlight representational geometries to demarcate an upper bound (or “noise ceiling”) of reliable variance in neural representational geometry199,200. For each searchlight, we computed the Spearman correlation between each participant’s RDM and the average RDM of the other participants and then averaged these correlation values (Fig. S1). To evaluate different representational models, for each searchlight/region, we computed the Spearman correlation between each participant’s neural RDM and each of the nine model RDMs. These correlation values were then submitted to a group-level statistical analysis. To quantify how well multiple RDMs jointly predict neural RDMs, we performed a multiple regression analysis: we ranked and standardized all model RDMs and used ordinary least-squares regression to predict the neural RDM. We quantified joint model performance using the coefficient of determination R2. Variance partitioning To quantify the unique variance explained by a given model, accounting for variance explained by all other models, we performed a variance partitioning analysis. Using a hierarchical regression procedure, we first estimated a full model comprising all 12 model RDMs: (1) motion energy, (2) gaze, (3) nonverb and (4) verb embeddings, (5) scene, (6) person, and (7) object RDMs derived from behavioral arrangements of static images based on visual content, (8) sociality and (9) transitivity RDMs based on behavioral arrangements of dynamic clips, as well as additional control models for the (10) body parts, (11) number of people, and (12) action categories. We then estimated a nested model containing all models except for the model(s) of interest. We quantified the unique variance explained by the model(s) of interest as the difference between the fit of the full model and the fit of the nested model excluding the model(s) of interest: unique R2 = full R2 – nested R2. This analysis accounts for collinearity between model RDMs (Fig. S12). Statistical evaluation To evaluate the statistical significance of Spearman correlations between searchlight neural RDMs and model RDMs, we performed a permutation test: the signs of participant-level correlation values were randomly flipped at each permutation, Fisher-transformed, and averaged (10,000 permutations). Significance was assessed by quantifying the proportion of permuted statistical values that exceed the actual test statistic201. For multiple regression analyses, the values are positively biased above zero, resulting in overly permissive permutation distributions (or t-values). To more fairly assess the R2 values, we performed a bootstrap hypothesis test: we randomly sampled participant-level R2 values with replacement and recomputed the mean at each iteration (10,000 iterations), resulting in a bootstrap distribution around the test statistic; we then subtracted the actual test statistic from the bootstrap distribution, effectively re-centering it around zero in order to assess significance202. Searchlight statistical maps were corrected for multiple tests by controlling the false discovery rate (FDR) at 0.001203. For the ROI analysis, we computed null distributions by permuting the condition labels prior to constructing RDMs (i.e., shuffling the rows and columns of the RDM). We computed two kinds of bootstrap distributions: (1) we resampled participants for typical population inference, and (2) we resampled both participants and stimuli to assess generalizability across stimuli200,204,205. When comparing performance between two models, we performed Wilcoxon signed-rank tests200 and controlled FDR across tests. All correlations were Fisher-transformed prior to averaging or statistical analysis, then converted back into correlations for visualization and numerical reporting. Representational structure in ventral temporal cortex Given the unexpectedly strong representation of action meaning in VT cortex, we sought to better understand the structure of representational geometries in VT searchlights. We first aimed to summarize how many distinct representational geometries are housed across the searchlights in VT. We first averaged all searchlight neural RDMs in VT across participants. We used principal component analysis (PCA) to reduce the number of searchlight RDMs from 1851 and 1785 searchlights in left and right VT, respectively, to a number of orthogonal PC RDMs. In this procedure, each PC corresponds to a vectorized RDM of 4005 elements corresponding to the 90 × 90 RDMs used throughout the manuscript. The first PC RDM (PC 1) corresponds to the dominant direction of variance across all searchlight RDMs in VT, while subsequent PCs correspond to orthogonal components of variance across searchlights in VT. We computed the cumulative proportion of variance explained by these PC RDMs to roughly quantify the number of distinct RDMs across VT searchlights. For each PC RDM, we also computed the semipartial Spearman correlation between that PC RDM and each of the nine model RDMs, controlling for all other model RDMs. We next aimed to determine whether VT could be characterized by clusters of searchlight RDMs. For left and right VT, we used k-means clustering with varying numbers of clusters ranging from k = 2 to k = 10. To effectively cluster VT searchlights (1851 in left hemisphere, 1785 in right hemisphere), we first reduced the dimensionality of the 4005-dimensional RDM features. We again used PCA (this time on the transpose of the VT searchlight RDMs matrix described in the previous paragraph) to reduce the RDM dimensionality from 4005 to 30 PCs, capturing roughly 95% of the variance in each hemisphere. In this case, each PC is a vector of length corresponding to the number of searchlights in a given hemisphere; we applied clustering to the searchlights in this 30-dimensional space. We used the k-means clustering algorithm in scikit-learn206 with k = 2–10, random initialization, and 100 initializations to stabilize the solutions, with default settings otherwise. For each clustering solution, we computed and visualized the silhouette coefficients207 across all samples. The silhouette score effectively measures how similar members of a given cluster are to other members of the same cluster as opposed to members of other clusters. In both hemispheres, silhouette scores increased from k = 2 and 3 up to k = 4, declined again for k = 5 and 6, then began increasing for larger numbers of clusters. Given that the silhouette scores reached a local maximum at k = 4, we focused on this particular cluster solution. We inspected cluster solutions at higher values of k, but as the number of clusters increases, the overall solution becomes more difficult to interpret. The clusters of searchlights we found at k = 4 were remarkably well-matched across left and right hemispheres, and were spatially contiguous across searchlights (despite the clustering algorithm receiving no explicit spatial or anatomical information). This spatial contiguity is likely due to the fact that searchlights are highly correlated with their neighbors due to their high overlap. Using the cluster labels, we next computed the centroid RDMs for each cluster: the mean RDM at the original 4005 length across all searchlights assigned to a given cluster. We next computed semipartial Spearman correlations between each cluster RDM and each of the nine model RDMs, controlling for all other model RDMs. To statistically evaluate these correlations, we used a bootstrapping procedure: we resampled participants with replacement, averaged VT searchlight RDMs across bootstrapped participants, recomputed the cluster centroid RDMs (based on the cluster labels derived from the original sample) for each cluster, and then recomputed the semipartial Spearman correlations. Finally, we computed 95% confidence intervals on each correlation based on their corresponding bootstrap distributions. Reporting summary Further information on research design is available in the Nature Portfolio Reporting Summary linked to this article. Supplementary information Supplementary information Reporting summary Transparent peer review file Supplementary information Reporting summary Transparent peer review file Source data Source Data Source Data