Generating Dialogues from Egocentric Instructional Videos for Task Assistance: Dataset, Method and Benchmark
Lavisha Aggarwal, Vikas Bahirwani, Lin Li, Andrea Colaco · 2025
Read Generating Dialogues from Egocentric Instructional Videos for Task Assistance: Dataset, Method and Benchmark by Lavisha Aggarwal, Vikas Bahirwani, Lin Li, Andrea Colaco (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-11192.txt.
Introduction
Lavisha Aggarwal and co-authors turn in a 2025 study solo how-to videos into two-person teaching dialogues for hands-free task help. A helper who watches through your camera and talks you through a repair needs dialogue data tied to video steps, which is costly to record. Their automatic method uses large language models to script the exchange.
The result, HowToDIV, holds five hundred seven conversations, over sixty-six hundred question-answer pairs, and twenty-four hours of clips spanning cooking, mechanics, and planting. A Gemma-3 baseline sets the first scores on the new benchmark. Cheap synthetic dialogues may bootstrap real task assistants.
Catalog record
| CREATOR | Lavisha Aggarwal, Vikas Bahirwani, Lin Li, Andrea Colaco |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-11192
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Generating Dialogues from Egocentric Instructional Videos for Task Assistance: Dataset, Method and Benchmark Lavisha Aggarwal, Vikas Bahirwani, Lin Li, Andrea Colaco Abstract Many everyday tasks ranging from fixing appliances, cooking recipes to car maintenance require expert knowledge, especially when tasks are complex and multi-step. Despite growing interest in AI agents, there is a scarcity of dialogue-video datasets grounded for real world task assistance. In this paper, we propose a simple yet effective approach that transforms single-person instructional videos into task-guidance two-person dialogues, aligned with fine grained steps and video-clips. Our fully automatic approach, powered by large language models, offers an efficient alternative to the substantial cost and effort required for manual data collection. Using this technique, we build HowToDIV, a large-scale dataset containing 507 conversations, 6636 question-answer pairs and 24 hours of videoclips across diverse tasks in cooking, mechanics, and planting. Each session includes multi-turn conversation where an expert teaches a novice user how to perform a task step by step, while observing user’s surrounding through a camera and microphone equipped wearable device. We establish the baseline benchmark performance on HowToDIV dataset through Gemma-3 model for future research on this new task of dialogues for procedural-task assistance. Our dataset and code are publicly available at our project page: https://github.com/google/howtodiv . 1 Introduction Figure 1 : Overview of our approach, an automatic monologue instructional video to user-expert dialogue generation method leveraging pre-trained large language models. Starting from expert subtitles and video, we produce instructions for the given task, followed by two-person dialogues and video clips corresponding to each user turn. There has been a recent emergence of general-purpose AI agents assisting people and transforming every aspect of life. One such aspect involves the need for people to learn and perform new procedural tasks 1 1 1 Procedural task is defined as a task consisting of multiple well-defined steps to achieve an end goal. While steps can be performed in a few valid orders, some deviations are considered as user errors or mistakes. , such as assembling IKEA furniture, cooking new recipes, or performing mechanical tasks like repairing home appliances, or car or bike parts. These tasks consist of multiple complex steps, and are difficult to memorize and recall precisely, owing to various nuances and details involved. The advent of smart wearable devices embedded with AI agents along with sensors such as microphones, speaker, and cameras presents us a unique opportunity to build language vision models and agents for such procedural task assistance. However the lack of datasets, catering to the domain of step by step task assistance, makes it difficult to evaluate the performance of existing models and to further improve them. Datasets mimicking a real expert teaching a novice user how to initiate and complete a custom task can significantly aid boosting research in this direction. The user is expected to be equipped with a wearable device streaming real-time audio and video. This is fed to a vision-language model to respond to the user’s questions, guide them through the different steps and help finish the task. The AI agent is expected not only to know and describe task instructions one step at a time, but also to answer intermediate questions, clarify requested details, provide feedback and be vigilant enough to check for any user errors and correct them proactively. HoloAssist holoassist is a recent egocentric dataset consisting of a continuous video stream and dialogues for collaborative object manipulation tasks built via HoloLens between two users. However, HoloAssist is restricted to 20 tasks, limiting its scope along with requiring an enormous data collection undertaking with hundreds of hours of human oversight, annotation, making it both resource-intensive and difficult to scale for diverse use cases. Despite scanty dialogue datasets, we observed that a large number of instructional video datasets, such as NIV niv , EgoPER egoper , YouCookII youcook2 , Assembly101 assembly101 , CrossTask crosstask , EpicKITCHEN epickitchen , COIN coin , are readily available. Such video datasets consist of video recordings showing an expert demonstrating how to perform a specific task while narrating the instructions for different steps. These datasets, often extracted through Youtube videos, do not contain multi-person conversations or dialogues, rather a single person delivering a monologue and are meant for post consumption instead of real-time step-by-step guidance. They are sometimes annotated with fine grained steps for the given procedure. In order to build a large-scale task assistance dataset, spanning tasks from various spheres of life, human-facilitated data collection is infeasible due to high cost incurred as well as the large number of human hours needed for in-field collection, supervision & annotation. Therefore, it is of high research and production interest to explore cheaper alternatives leveraging existing models, datasets and knowledge pools. To this end, we propose a simple yet effective prompting-based method, for generating dialogues and video-clips by fusing the language interpolation and manipulation skills of a large language model (LLM) with domain-knowledge captured in instructional videos. Our algorithm consists of three main components: instruction formation, dialogue generation and video localization, as shown in Fig 1 . We utilize this algorithm to generate a large-scale task-assistance dataset, HowToDIV, built on top of two popular instructional video datasets: Narrated Instruction Videos (NIV) niv and Egocentric Procedural Error (EgoPER) egoper . It consists of real human-like conversations between an expert and a novice user, along with corresponding user’s point-of-view video-clips for every user-dialogue turn, and instruction steps for every conversation. The text generation is controllable with prompts and step annotations accompanying each dataset. Given a single person video with subtitles, our approach can generate user-expert Dialogues, Instructions and corresponding user Videoclips, without any manual intervention. We benchmark the HowToDIV dataset on the Gemma3 gemma3 LLM, evaluating the metrics: BLEU score, ROUGE score and LLM as a Judge score. Gemma3 responses yield an average score of 2.87 for LLM as a Judge score (on a scale of 1 - 5), 0.32 BLEU score, 0.270 ROUGE-L score. In summary, our contributions are following: • We propose an instructional monologue video to two-person dialogue generation technique, fusing a large language model’s inherent knowledge with domain information from procedural task video datasets. • We contribute HowToDIV, a large-scale multi-turn dataset consisting of 507 conversations, and 6636 user-expert dialogue turns and videos covering a diverse set of tasks across cooking, mechanical fixing and planting domains. The sessions consist of variations for user speech-styles (regular and concise), action-types (users accurately following steps vs making errors), along with the same task being performed with different step orderings. • We establish the benchmark performance of Gemma3, 4B model on HowToDIV dataset, achieving a BLEU score of 0.32, ROUGE-2 score of 0.125 and LLM-as-a-Judge score of 2.97 (on a 1 - 5 scale) as the baseline for the task of step-by-step procedural-task assistance dialogues. 2 Related Works Instructional Video Datasets. Instructional videos are a rich source for learning complex multi-step tasks, ranging from cooking recipes to car maintenance and appliance repair. Large-scale efforts like HowTo100M howto100m have significantly advanced the field by collecting 136 million video clips from 1.22 million narrated instructional web videos, covering over 23,000 distinct visual tasks. This dataset leverages automatically transcribed narrations as a form of weak supervision for learning text-video embeddings, demonstrating a scalable