InterventionLens: A Multi-Agent Framework for Detecting ASD Intervention Strategies in Parent-Child Shared Reading

Xiao Wang, Lu Dong, Ifeoma Nwogu, Srirangaraj Setlur, Venu GovindarajuState University of New York at Buffalo, Buffalo, NY 14260, USA {xwang277, ludong, inwogu, setlur, govind}@buffalo.edu · 2026

Read InterventionLens: A Multi-Agent Framework for Detecting ASD Intervention Strategies in Parent-Child Shared Reading by Xiao Wang, Lu Dong, Ifeoma Nwogu, Srirangaraj Setlur, Venu GovindarajuState University of New York at Buffalo, Buffalo, NY 14260, USA {xwang277, ludong, inwogu, setlur, govind}@buffalo.edu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-13710.txt.

Introduction

Xiao Wang and co-authors present InterventionLens, a multi-agent framework for detecting ASD intervention strategies in parent-child shared reading. Home-based shared reading supports children with autism spectrum disorder at low cost. Expert annotation of caregiver strategies in home videos is costly and hard to scale, though. InterventionLens detects and temporally segments those strategies with a collaborative multi-agent architecture. It needs no task-specific training or fine-tuning and integrates multimodal interaction content. On the ASD-HI dataset it reached an F1 score near 79 percent.

Catalog record

CREATORXiao Wang, Lu Dong, Ifeoma Nwogu, Srirangaraj Setlur, Venu GovindarajuState University of New York at Buffalo, Buffalo, NY 14260, USA {xwang277, ludong, inwogu, setlur, govind}@buffalo.edu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE32278
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.13710

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

InterventionLens: A Multi-Agent Framework for Detecting ASD Intervention Strategies in Parent-Child Shared Reading Xiao Wang, Lu Dong, Ifeoma Nwogu, Srirangaraj Setlur, Venu GovindarajuState University of New York at Buffalo, Buffalo, NY 14260, USA {xwang277, ludong, inwogu, setlur, govind}@buffalo.edu Abstract Home-based interventions like parent-child shared reading provide a cost-effective approach for supporting children with autism spectrum disorder (ASD). However, analyzing caregiver intervention strategies in naturalistic home interactions typically relies on expert annotation, which is costly, time-intensive, and difficult to scale. To address this challenge, we propose InterventionLens, an end-to-end multi-agent system for automatically detecting and temporally segmenting caregiver intervention strategies from shared reading videos. Without task-specific model training or fine-tuning, InterventionLens uses a collaborative multi-agent architecture to integrate multimodal interaction content and perform fine-grained strategy analysis. Experiments on the ASD-HI dataset show that InterventionLens achieves an overall F1 score of 79.44%, outperforming the baseline by 19.72%. These results suggest that InterventionLens is a promising system for analyzing caregiver intervention strategies in home-based ASD shared reading settings. Additional resources will be released on the project page. 1 Introduction Autism Spectrum Disorder (ASD) is a prevalent neurodevelopmental disorder that substantially impairs social communication and daily functioning 4 ; 11 . While early intervention can improve core symptoms ( 10 ) , many families lack consistent access to effective services due to long-term costs and uneven resource distribution ( 20 ) . This gap has motivated scalable, caregiver-deliverable home-based interventions 21 ; 25 ; 31 ; 30 , among which shared book reading (SBR) is a particularly effective and accessible routine ( 6 ; 29 ) . In practice, families often record SBR sessions at home and submit the videos to speech-language pathologists (SLPs), who manually conduct video-by-video review to identify caregiver intervention strategies and provide guidance 1 . However, such expert-driven annotation is labor-intensive, costly, and difficult to scale. Therefore, a key technical need is to automatically detect intervention strategies from home-based videos. In this work, we focus on three caregiver intervention strategies used in parent-implemented ASD interventions: Modeling , where the caregiver demonstrates the target word or phrase for the child; Mand-Model , where the caregiver prompts the child to respond and then provides a model when needed; and Time Delay , where the caregiver deliberately pauses to create an opportunity for the child to initiate or complete the response 23 ; 22 ; 24 . These strategies originate from the Parent-Implemented Communication Strategies (PiCS) framework and were later adapted to shared reading and telepractice-based interventions for children with ASD 1 ; 2 ; 3 . Achieving this goal is inherently difficult because shared book reading (SBR) is highly dynamic and interactive, typically involving a tightly coupled, triadic interaction among a caregiver, a child, and a book. Caregivers drive this process through continuous verbal scaffolding while simultaneously coordinating multi-modal social cues —such as gaze, facial expressions, and gestures —to regulate child engagement 19 . Fine-grained recognition and reasoning of these interaction segments become even more challenging in unstructured, in-the-wild home settings. In these environments, non-fixed camera viewpoints, ambient noise, and variable interaction rhythms are compounded by the inherent acoustic differences between adult and child voices ( 18 ) . These factors drastically degrade the performance of mainstream Automatic Speech Recognition (ASR) systems like Whisper 5 , severely undermining the reliability of subsequent interaction modeling. Finally, these technical hurdles are further magnified by a severe scarcity of high-quality annotated data. Although the ASD-HI dataset 19 partially addresses the gap in home-based SBR, its limited scale cannot support the massive data demands of modern data-intensive model training. Consequently, developing a robust and generalizable automated analysis system under these compounding constraints remains a core open challenge in the field. To address these limitations, we propose InterventionLens, a hierarchical multi-agent framework for caregiver intervention detection without parameter updating. Our contributions are three-fold: • We propose a two-layer multi-agent architecture that decouples multimodal perception from intervention detection and segmentation, combining structured transcript and book-context modeling with coarse-to-fine expert-based temporal localization. • We introduce a progressive knowledge refinement mechanism that uses a small amount of labeled data to iteratively update decision rules and refine the agent’s structured guidance. By translating clinical standard operating procedures (SOPs) into explicit agent roles and decision constraints, our framework enables structured strategy analysis without data-intensive fine-tuning. • We demonstrate strong empirical performance on the ASD-HI dataset, where InterventionLens substantially outperforms existing baselines under strict evaluation criteria, improving the overall F1 score from 59.72% to 79.44% and Precision from 50.21% to 82.36%. 2 Related Work LLM-based Multi-Agent Systems: Large language model (LLM)-based multi-agent systems (MAS) have emerged as an effective paradigm for complex reasoning by decomposing problems into modular subtasks handled by specialized agents 14 ; 33 ; 26 ; 9 ; 8 ; 7 . Compared with single-agent systems, which often suffer from context dilution and error accumulation over long interactions, MAS can better isolate subproblems, reduce interference across reasoning stages, and enable targeted optimization for heterogeneous tasks 34 ; 12 ; 27 ; 15 ; 17 ; 28 . While prior MAS research has mainly focused on general planning, coding, and task-solving settings, the use of MAS for ASD intervention strategy detection in shared book reading interactions remains underexplored. Challenges in Multimodal Perception. Fine-grained analysis of parent-child shared book reading requires accurate speech recognition and reliable temporal grounding. However, child speech remains challenging for traditional ASR systems because of its substantial acoustic variability and pronunciation instability 13 ; 16 . In parallel, recent multimodal large language models have shown promising video understanding ability, but they still struggle to provide precise temporal localization in long-form videos 32 ; 35 , especially when second-level timestamps are required for segment-level analysis. These limitations make robust multimodal perception a key bottleneck for automated intervention detection in naturalistic home environments. ASD Intervention Detection: In the domain of ASD language intervention assessment, the ASD-HI dataset 19 provides the main benchmark for caregiver intervention strategy detection in home-based shared book reading videos.The strongest publicly available baseline uses Whisper for speech transcription and a single LLM agent guided by prompts designed by ASD intervention experts for intervention identification, followed by greedy sequential scanning for temporal segmentation. This benchmark establishes an important foundation for the task; however, its overall detection performance still leaves substantial room for improvement because it relies on error-prone transcripts and rigid heuristic boundary search. 3 Problem Formulation We formulate the task as a strategy detection problem over parent–child shared book reading videos. Given an input video, the objective is to temporally localize intervention strategies and classify each detected segment into its corresponding strategy category. Let V\in\mathcal{V} denote a parent-child shared book reading video, and let \mathcal{S}=\{\textit{Modeling},\textit{Mand\mbox{-}Model},\textit{Time Delay}\} be the intervention strategy space. The goal is to predict a set of temporally localized intervention strategy segments: \hat{\mathcal{Y}}=\{(\hat{t}_{m}^{s},\hat{t}_{m}^{e},\hat{s}_{m})\}_{m=1}^{M},\quad\hat{s}_{m}\in\mathcal{S}, (1) where M is the total number of predicted interventions. For the m -th predicted segment, \hat{t}_{m}^{s} and \hat{t}_{m}^{e} denote its predicted start and end timestamps, respectively, and \hat{s}_{m} represents its assigned strategy label. InterventionLens first utilizes the OPA module to process the raw video V into grounded interaction transcript features \mathcal{X} . Subsequently, the BMA module is applied to extract the book context representation \mathcal{B} aligned with the current interaction:

Read the complete text