DynamicTHOR: A Scalable Dataset of Human-Centric Dynamic Scenes for Embodied AI Authors: Chenxu Wang, Dunzheng Wang, Dong Wang, Xinghang Li, Yongchang Li, Huaping Liu Published: 2026 ABSTRACT AbstractAlthough many existing works have studied probabilistic or dynamic environments where the objects used in daily life may be moved due to human activities, the scale of datasets is usually limited due to the cost of human annotation or manual configuration. This paper introduces a framework that simulates human activities and corresponding object dynamics using Large Language Models (LLMs) and applies the simulated human residents to embodied scenes to generate dynamic scenes. Using this framework, we craft a dataset named DynamicTHOR with 50 characters and 100 dynamic scenes, which can be easily extended in scale. A user study comparing our generated scene dynamics with a baseline approach and human annotations validates that our framework successfully produces believable, diversified data of a quality comparable to human annotations. The novel framework and dataset can facilitate the study of embodied intelligence, such as the navigation task in dynamic scenarios.Subject terms: Electrical and electronic engineering, Computer science Background & Summary While static environments have been widely adopted in the study of Embodied AI, dynamic scenes, where daily objects may be moved according to human behaviors, have also attracted a lot of attention. The probabilistic nature of dynamic scenes introduces additional challenges, including constructing object-based memory systems1, proactive robot assistance2,3, and object-goal navigation with probabilistic object configurations4,5. However, most existing studies on dynamic scenes conduct evaluations either in human-annotated environments with only a few scenes or in simplified 2D environments, limiting the generalizability of the results. On the other hand, although embodied simulators have achieved fruitful success6–10, and large-scale scenes can be generated in various ways11–13, these approaches still generate static scenes, where neither human activity influences nor object dynamics are considered. To support future embodied AI research in dynamic scenes and address the lack of large-scale dynamic scene datasets, we introduce a framework for generating dynamic scenes by simulating human behaviors and configuring corresponding object relocations with time in static scenes to obtain dynamic scenes. Object positions in dynamic scenes will evolve over time according to human activities, as illustrated in Fig. 1. An illustration of the dynamic scene, where the daily objects may be moved over time according to human activities. The relations between human activities and the objects are highlighted with the red dashed lines, and the yellow marks denote the target place of object relocations. In this context, analyzing the human activity information and inferring the object placement distribution for robot navigation becomes an emerging challenge. To automatically and scalably simulate human indoor activities, we leverage Large Language Models (LLMs), which have been widely used for human simulation14–16 and have demonstrated strong content generation capabilities17–19. Our framework first hierarchically generates human activity schedules based on a customized persona, and then generates a set of probabilities for object relocations caused by these activities according to the persona. The simulated human avatars are then assigned to specific static scenes, and the object relocations are subsequently applied to obtain dynamic scenes. To trade off between scalability and simulation quality, we keep the human simulation agnostic to specific scene layouts, so that the simulated behaviors are applicable to all scenes in the same domain, e.g., the 10,000 scenes in the ProcTHOR dataset11. Leveraging this framework, we introduce DynamicTHOR, a comprehensive dataset that includes both simulated character activity data and dynamic scene data. Specifically, DynamicTHOR contains 50 characters, each with a 5-day activity schedule generated within the ProcTHOR environment context. These simulated activities are then used, together with the static scene data, to instantiate 100 dynamic scenes that capture temporal evolution driven by human behaviors. An example is presented in Fig. 2. To validate our framework and the generated dataset, we conduct a user study to compare our generated dynamic scenes with two baselines: the HOMER dataset and human annotations. The results show that the quality of our generated data is comparable to human annotations, and even better in comprehensiveness and diversity. An example of the dynamic scene evolution with first-person view at several viewpoints, where the daily object placement is changed over time according to the simulated human activities. Our framework and dataset address the lack of large-scale dynamic scene datasets for robotics research. They can facilitate a broad range of robotics research and real-world applications, such as object-goal navigation and robotic assistants. To demonstrate its applicability, we showcase a usage scenario involving object-goal navigation in dynamic scenes with baselines. Methods Dynamic Scene Definition In this work, we aim to generate dynamic scenes where the object placements change over time according to the influence of human activities. Specifically in the indoor domain, a dynamic scene S can be defined as a tuple: where H stands for the static house layout, L is the set of rooms, R is the set of immovable receptacles, O is the set of movable objects, P:O×[ts,te]↦R is the configuration of object positions over time, ts is the start time of the dynamic scene, and te is the end time. We further denote the dynamic scene at time t as St=(H,L,R,O,Pt), where Pt(⋅)=P(⋅,t). Embodied tasks can be set at any St within the constraint ts≤t≤te. Recall that the primary cause of scene dynamic is the influence of human activities, thus the evolution of the scene is defined by events, which can be represented as e=(ts(e),te(e),a(e),P(e)), where ts(e) and te(e) stand for the starting and ending time instant of the event, a(e) is a natural language description for the event, and P(e) is the object relocations caused by the event, defined as a set of object-receptacle pairs. Each dynamic scene is associated with a set of events, denoted as E. Given a static scene with H,L,R,O and a given time interval (ts, te), we aim to generate a reasonable event set E and subsequently generate the object configuration P. Scene Generation Framework As illustrated in Fig. 1, our framework takes pre-defined character information and domain-specific scene prior information as input, followed by three modules: generation of the human activity schedules; establishment of the activity database that models the environment evolution caused by each activity; and a scene configuration module that applies the object relocations to specific scenes to get the event set E. Both the generation of the schedule and the activity database are powered by LLMs, whereas the scene configuration module is implemented with a set of rules associated with the environment and the simulator, to ensure the final object placements are feasible and realistic. For each step of the generation, we assemble the information into a natural language prompt using a designed template, which also specifies the response format for the LLM. The natural language response is then parsed with a set of rules to get structured data. The external input data can be in pure textual form, where the characters are defined by 8 fields, divided into basic information and characteristic information as shown in Fig. 3(a). Character personalities can be customized by manual design or automatically generated as well. Learning from the success of SOTOPIA20, we include 50 characters that are initialized by GPT-4 with manual filtering and adjustment to avoid duplication and enrich the diversity. The scene prior information offers a profile of the target simulation domain, including the possible rooms, receptacles, objects, and possible relations. An overview of the dynamic scene generation framework. Given the information of characters and the scene prior, our framework first generates the activity schedules and the corresponding activity database to simulate a resident avatar. Static scenes can be extended to dynamic ones by sampling residents and configuring the object relocations caused by their activities. Notably, our framework is modularly designed, with activity schedules and the activity database functioning independently of specific scene layouts, enabling characters to be reused across different scenes. The amount of distinct characters determines the diversity and comprehensiveness of our dataset, while the data can be grounded in infinite scenes. Human Activity Schedule Generation Aiming to provide believable and personalized activity schedules, we draw on the idea of hierarchical planning from generative agents14 and implement a top-down framework for human schedule generation, which first generates a general plan and then decomposes the indoor parts into detailed activities, as illustrated in Fig. 3(b). Based on the provided character information, a general plan is first generated at the hour level with a personalized sketchy description, such as doing research or preparing for a party. The general plan is intended to reflect the characteristics of the persona and frame the general activity trajectory of the character. Since we ultimately aim at generating indoor dynamic scenes, the general plan is subsequently filtered by substituting the outside part with leave home to force the LLM to focus on the indoor activities. We then prompt the LLM to generate a fine-grained activity schedule at the minute level by decomposing the filtered general plan items into detailed activities, such as getting dressed and reading the newspaper. To guide the LLM, we additionally provide an activity list as examples and a reference for granularity, which is acquired from the HOMER dataset2 with manual adjustment. After each round of schedule generation, we prompt the LLM to recognize synonyms and extend the activity list continuously by merging new activities. Activity Database Generation After acquiring the human activity schedules, the natural subsequent step is to analyze how the activities impose effects on the environment. To this end, we build an activity database that hierarchically models the probabilities of object relocation caused by each activity as illustrated in Fig. 3(c), considering the characteristics of the persona. We assume an activity will be performed at only one location. Given an activity a, we denote the possibility of it being performed at location l as p(l|a) ∈ [0, 1], with the constraint ∑l∈Lp(l∣a)=1 for any activity a. The probabilities are under comprehensive consideration of the nature of activities, characteristics of the persona, and the available options of locations with possible receptacles that reflect the functionality of the locations. Once activities are paired with specific locations, we inspect the probabilities of objects being used or involved in each event, and which objects are considered to be moved due to the effect of the activities. For example, tableware may be used for having lunch at kitchen and subsequently placed on the dining table. Then, after tidying up the dining table at kitchen, they may be placed back to the countertop. In this regard, we use the probability p(o|a, l) ∈ [0, 1] to characterize the possibility of the object o being involved in activity a at location l. For some object o, if the probability p(o|a, l) > 0, i.e., it may be involved in the activity, then we use p(r|o, a, l) to denote the probability of the human moving an object with type o onto a receptacle r∈R. We assume that a used object will be placed onto exactly one receptacle. This results in ∑r∈Rp(r∣o,a,l)=1. Across the generation process, scene prior information including possible room types, room description, object types, and receptacle types, are injected into the prompts to guide the data generation. The scene prior information will also be used in an automatic rule-based data verification process, which removes invalid content (such as a nonexistent location) from the LLM outputs. Scene Configuration As shown in Fig. 3(d), the final step of our framework is to ground the human activities into specific scenes to instantiate the scene evolution by forming the event set E and calculate the object position configuration P. In this step, the object-receptacle relations that are output by the LLM are used to determine specific 3D coordinates for the objects in the environment. The configuration process is tightly coupled with the scene layout. In this paper, we build dynamic scenes based on the initial scenes provided by the ProcTHOR dataset11, including four types of rooms: bedroom, bathroom, living room, and kitchen, while the number of rooms may vary in different scenes. We start by assuming each person lives in a unique bedroom and first sample residents according to the number of bedrooms in the scene. With a given time interval, the event set E can be formed by incorporating all indoor activities of all sampled residents. For each event in E, we sample the effect following the hierarchical probabilistic model described earlier: first sampling the location of the activities, followed by the objects involved and the target receptacles for the used objects. In consideration of realism, residents are assigned specific bedrooms and bathrooms, i.e., they will always choose their assigned room whenever they want to go to a bedroom or a bathroom. Since the activity database is generated with consideration of all possible receptacles, the relocation probabilities will first be normalized by the existence of receptacles in the given scene. If there are multiple instances of the target receptacles in the sampled room, we randomly pick one. We then sample a position for placing the target object onto the receptacle. Implementation We use gpt-4-1106-preview in the dynamic scene generation process. For generating more diversified data, the model temperatures for schedule and activity probability generation are set from 0.2 to 0.5; whereas the synonym merging uses a temperature of 0 for stable behavior. Following the generation framework illustrated above, we first generate a character activity dataset that includes 50 distinct personas; each persona has a 5-day schedule generated from Wednesday to Sunday. The dataset includes 564 activities, of which 448 are considered primary after merging synonyms within each persona’s schedule; synonymous activities share the same environmental effects. For dynamic scene generation, we sample 100 scene layouts from the ProcTHOR dataset11, where each layout is guaranteed to include at least one room of each type (Kitchen, LivingRoom, Bathroom, and Bedroom). We then sample residents from the character activity dataset to build a dynamic scene dataset of 100 dynamic scenes. All code for reproducing the generation process, including the prompt templates, LLM inference parameters, and the data parsing mechanism, is included in our dataset21. Data Record The DynamicTHOR dataset is publicly available at Figshare21; it consists of two parts: the character activity dataset and the instantiated dynamic scenes. The record also contains two additional codebases, DynamicSceneGeneration and DynamicTHOR, for reproducing the data generation process and using the data, respectively. The general data record structure is illustrated in Fig. 4. The data file structure. Character Activity Dataset The character activity dataset is placed in the characters folder under the root path, and each subfolder corresponds to a character persona, where the subfolder name denotes the character’s name. There are four files under each character subfolder: persona.json, a JSON file that defines the persona; schedule.pkl, a pickle file storing the activity schedules; status.pkl, a pickle file storing auxiliary information, including the primary activity set and alias data; and event_base.pkl, a pickle file storing all activities and the probabilities associated with them. Dynamic Scenes The dynamic scene dataset is placed in the dynamic_scene folder under the root path, and each subfolder corresponds to an instantiated dynamic scene. Each dynamic scene folder contains five files: character.json, a JSON file that contains the character information in the dynamic scene; house_info.json, a JSON file that defines the house layout, directly imported from the ProcTHOR dataset; object_poses_dict.pkl, a pickle file storing the object distribution at all time points to facilitate fast scene evolution; schedules.pkl, the generated and applied schedules of the sampled characters; and tasks.pkl, the sampled navigation task examples. The main data file to load the dynamic scene is the object_poses_dict.pkl, which records the positions and rotations of objects at each timestamp. An example JSON snippet of object_poses_dict.pkl is presented in Fig. 5. An example JSON snippet demonstrating the data structure of the object_poses_dict.pkl. Repetitive items and extensive timestamps are omitted for brevity. Technical Validation To validate the technical quality of our dataset, we conduct a user study to assess the realism, comprehensiveness, and diversity of the generated human activity schedules and the corresponding object relocations, ensuring that the synthetic dynamics are suitable for downstream embodied AI research in dynamic scenes. User Study Design Since our dataset is largely based on synthetic data, it naturally requires an assessment of the quality of the human simulations to ensure that the dataset can support further research on dynamic goal navigation. To this end, we evaluate the generated human activity schedules and corresponding object relocations configured in dynamic scenes within the following dimensions: Believability, an idea borrowed from prior works on language agents14,22, indicating whether the generated content is reasonable and true to humans. Believability, an idea borrowed from prior works on language agents14,22, indicating whether the generated content is reasonable and true to humans. Comprehensiveness, measuring whether the content is detailed enough to depict human behaviors without omitting important activities or object usage. Comprehensiveness, measuring whether the content is detailed enough to depict human behaviors without omitting important activities or object usage. Diversity. The generated content is expected to be diversified and personalized, instead of monotonously repeating the same routine. Diversity. The generated content is expected to be diversified and personalized, instead of monotonously repeating the same routine. Both believability and comprehensiveness are measured for both the schedules and the object relocations at the persona level, whereas diversity is measured at the method level, across multiple personas. The exact survey questions are presented in Table 1. Participants are asked to give a score from 0 to 10 for each question, higher scores indicate better quality. Table 1Questions used in the questionnaires.MetricQuestionSchedule BelievabilityHow do you evaluate the believability of the schedules? (Do they look like schedules of real and specific humans?)Schedule ComprehensivenessHow do you evaluate the comprehensiveness of the schedules? (Do they cover sufficient types of activities and are detailed enough to profile humans’ indoor activities?)Relocation BelievabilityHow do you evaluate the believability of the object movements? (Are they reasonable? Will an ordinary person move the objects in that way?)Relocation ComprehensivenessHow do you evaluate the comprehensiveness of the object movements? (Is there any important object that is in the house and should be used but omitted? You should give a high score if no object is omitted.)Population DiversityHow do you evaluate the diversity of the schedules obtained by this method across different personas? (Generally, different people may have different schedules due to their characteristics. Do the schedules obtained by this method reflect that diversity?) Questions used in the questionnaires. To verify the effectiveness of our framework, we compare it with the following baselines: HOMER dataset2. Although it captures delicate human data in a vivid simulation, the dataset is not directly applicable to the THOR simulators. We extract the human schedules and the corresponding object relocations from the environment variation for comparison at the text level. HOMER dataset2. Although it captures delicate human data in a vivid simulation, the dataset is not directly applicable to the THOR simulators. We extract the human schedules and the corresponding object relocations from the environment variation for comparison at the text level. Human annotation. Following the conventional process of human annotation as adopted in prior work14, we recruit 10 college students (4 female, 6 male) to manually design a two-day schedule and corresponding object relocations for 15 distinct personas within 5 sampled scenes (3 personas per scene), with the same information as the input of our framework. The schedules are reviewed by the authors to ensure the quality. Human annotation. Following the conventional process of human annotation as adopted in prior work14, we recruit 10 college students (4 female, 6 male) to manually design a two-day schedule and corresponding object relocations for 15 distinct personas within 5 sampled scenes (3 personas per scene), with the same information as the input of our framework. The schedules are reviewed by the authors to ensure the quality. In this user study, 200 questionnaires are distributed through an international survey platform in five batches, each batch has 40 identical questionnaire instances that include the human behaviors acquired by all 3 methods in a single scene with 3 personas. We display the scene layouts, including rooms, receptacles, and movable objects, along with the characteristics of personas, the generated human activity schedules, and the corresponding object relocations. For data from the HOMER dataset, we explicitly state that the activities are in a different scene and the persona is not specified. The user study is conducted in a blind manner; the method name that corresponds to each set of human behaviors is invisible to the participants. The user study was conducted with informed consent and did not involve the collection or publication of any personally identifiable information. Only aggregate results are reported. All participants’ ability to read and understand English is verified through an embedded reading comprehension test. After filtering out those who failed to pass the reading comprehension test or failed to meet the minimum answer time requirement, a total of 166 questionnaires are collected for analysis. Validation results As the quantitative results presented in Fig. 6, the simulated human behaviors and the corresponding object relocations achieve the highest overall assessment. To assess the statistical robustness, we fitted a linear mixed-effects model (LMM) with score as the dependent variable, method and metric as fixed effects, and respondent identity as a random effect to account for inter-participant variability. There is no significant difference between the scores achieved by our method and the human annotation in Schedule Believability. In all other cases, our method outperforms other alternatives with a statistical significance level of p < 0.01. Our generated schedules show a comparable level of believability to human annotations, while significantly outperforming the HOMER, probably due to the limited choice of activity in rule-based schedule synthesis methods. Thanks to the hierarchical architecture, our framework can generate more detailed activities than the baselines. Surprisingly, our method also outperforms human annotation. We conjecture that humans may not be motivated enough to improve the level of detail of annotations. Not surprisingly, our framework exhibits better quality of object relocations than humans, since it might be hard to recognize all reasonable relocations among dozens of objects for human annotators. Although the HOMER dataset models human behaviors more delicately, users tend to give higher appraisal to our framework, possibly being influenced by the better quality of schedules. Benefiting from the generation capabilities of LLMs, our framework can produce more diversified schedules based on multifarious personas, which are not considered by the HOMER dataset. Average scores in the user study, where the error bar reflects the standard error. Though our dataset shows comparable and even better quality than the data collected from human annotation, the assessment is rather subjective and the individual difference is high. To quantify this, we additionally computed inter-annotator agreement using Kendall’s coefficient of concordance (W) for all the five indicators, resulting in 0.055 for Schedule Believability, 0.198 for Schedule Comprehensiveness, 0.074 for Relocation Believability, 0.196 for Relocation Comprehensiveness, and 0.137 for Population Diversity. This result implies that when the data quality is comparable, respondents may have different preferences and it may require a large sample size to show the statistical significance. Overall, the results verify the quality concerning the believability, comprehensiveness, and diversity of our dataset, which is comparable to or above average human level, further supporting the appropriateness of our dataset as a benchmark for studying dynamic semantic goal navigation. Usage Notes Dynamic scenes can be regarded as static scenes with one additional time dimension. Within the given time period t ∈ (ts, te), the scene at any time instant St can be loaded as an embodied environment with the code provided with our dataset21. Please note that the DynamicTHOR dataset is designed to be used together with the ProcTHOR environment, therefore the ProcTHOR codebase11 is necessary for running our code. We implement the DynamicTHOR by rearranging the object placement in the scene. Loaded dynamic scenes also support standard navigation experiments, we provide an example usage in our codebase with Embodied-Codebook as a baseline23. Please also note that the LLM-generated human behavior data is designed to be used for dynamic scenes for embodied AI research, and the technical validation is conducted at the scene dynamics level. Therefore, its quality and diversity are not guaranteed for other general purposes and may not be a perfect substitution of real-world data. Acknowledgements This work is supported by the National Natural Science Fund (Grant 62025304, 62120106005), and the Fundamental and Interdisciplinary Disciplines Breakthrough Plan of the Ministry of Education of China (grant JYB2025XDXM109). Author contributions Huaping Liu, Chenxu Wang, and Dunzheng Wang conceived the project. Chenxu Wang and Dunzheng Wang designed the method, generated the data, and conducted the technical validation. Dong Wang, Chenxu Wang, Xinghang Li, Dunzheng Wang, and Yongchang Li provided the usage notes. Chenxu Wang, Dong Wang, Dunzheng Wang, Xinghang Li, and Huaping Liu wrote the paper. All the authors read and approved the final manuscript. Data availability The dynamic scene dataset, including the character dataset and the dynamic scene dataset, is openly accessible at Figshare21. All data and code files described in the Data Record section can be accessed and downloaded there. Code availability Code for reproducing the dynamic scene generation is available at Huggingface (https://huggingface.co/spaces/DynamicScene/DynamicGeneration/tree/main) with an online demo. Code for loading and using the scenes is available at GitHub (https://github.com/wcx21/DynamicTHOR). Both codebases can also be found in the dataset record21. Competing interests The authors declare no competing interests. Footnotes Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. References Associated Data Data Availability Statement The dynamic scene dataset, including the character dataset and the dynamic scene dataset, is openly accessible at Figshare21. All data and code files described in the Data Record section can be accessed and downloaded there. Code for reproducing the dynamic scene generation is available at Huggingface (https://huggingface.co/spaces/DynamicScene/DynamicGeneration/tree/main) with an online demo. Code for loading and using the scenes is available at GitHub (https://github.com/wcx21/DynamicTHOR). Both codebases can also be found in the dataset record21.