DSADF: Thinking Fast and Slow for Decision Making
Alex Zhihao Dou, Dongfei Cui, Jun Yan, Weida Wang, Benteng Chen, Haoming Wang, Zeke Xie, Shufei Zhang · 2025
Read DSADF: Thinking Fast and Slow for Decision Making by Alex Zhihao Dou, Dongfei Cui, Jun Yan, Weida Wang, Benteng Chen, Haoming Wang, Zeke Xie, Shufei Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-08189.txt.
Introduction
Zhihao Dou and seven co-authors published a paper in 2025 on decision-making agents that combine fast habit with slow thought. Learning agents trained by trial and error react quickly but stumble in new settings. Big vision-language models reason well but respond slowly and expensively. The authors build a two-part system modeled on the fast and slow thinking described in psychology.
One part is a quick agent backed by memory for routine choices. The other is a vision-language model called in for deep analysis when the situation is unfamiliar. Tests in two game worlds, a crafting game and a household task game, showed clear gains over single-system rivals. The design aims at agents that stay nimble without giving up judgment.
Catalog record
| CREATOR | Alex Zhihao Dou, Dongfei Cui, Jun Yan, Weida Wang, Benteng Chen, Haoming Wang, Zeke Xie, Shufei Zhang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 86 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-08189
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
DSADF: Thinking Fast and Slow for Decision Making Alex Zhihao Dou, Dongfei Cui, Jun Yan, Weida Wang, Benteng Chen, Haoming Wang, Zeke Xie, Shufei Zhang Abstract Although Reinforcement Learning (RL) agents are effective in well-defined environments, they often struggle to generalize their learned policies to dynamic settings due to their reliance on trial-and-error interactions. Recent work has explored applying Large Language Models (LLMs) or Vision Language Models (VLMs) to boost the generalization of RL agents through policy optimization guidance or prior knowledge. However, these approaches often lack seamless coordination between the RL agent and the foundation model, leading to unreasonable decision-making in unfamiliar environments and efficiency bottlenecks. Making full use of the inferential capabilities of foundation models and the rapid response capabilities of RL agents and enhancing the interaction between the two to form a dual system is still a lingering scientific question. To address this problem, we draw inspiration from Kahneman’s theory of fast thinking (System 1) and slow thinking (System 2), demonstrating that balancing intuition and deep reasoning can achieve nimble decision-making in a complex world. In this study, we propose a Dual-System Adaptive Decision Framework (DSADF), integrating two complementary modules: System 1, comprising an RL agent and a memory space for fast and intuitive decision making, and System 2, driven by a VLM for deep and analytical reasoning. DSADF facilitates efficient and adaptive decision-making by combining the strengths of both systems. The empirical study in the video game environment: Crafter and Housekeep demonstrates the effectiveness of our proposed method, showing significant improvements in decision abilities for both unseen and known tasks. keywords reinforcement learning, vision language model, System 1 and System 2, decision-making † † equal-contributors: These authors contributed equally to this work. † † equal-contributors: These authors contributed equally to this work. † † equal-contributors: These authors contributed equally to this work. 1 Introduction The reinforcement learning (RL) framework, which is centered on maximizing cumulative rewards, involves an agent iteratively interacting with an environment to explore and refine its decision-making process, ultimately working towards a definite goal. RL has demonstrated remarkable success in a range of applications, including robotics Ibarz et al. 2021 ; Peng et al. 2018 , video games Mnih et al. 2013 , dialogue systems Du et al. 2019 , and autonomous vehicles Chen et al. 2019 . The critical challenge of RL agents is their generalizability to new unseen settings Di Langosco et al. 2022 ; Yang et al. 2019 . This limitation arises because agents rely on experiences specific to the training environment. More concerning is that even minor environmental modifications can substantially affect the agent’s policy generalization ability Ao et al. 2023 ; Rank et al. 2024 . The open environments to which agents must adapt are usually dynamically changing, making the learning process more complex. Agents must adapt to environmental changes, which requires the proposal of advanced RL methods. In addition, the RL process remains consistently challenging Cao et al. 2024 ; Dalal et al. 2024 for the long-horizon tasks. One critical issue is the sparse rewards caused by the Credit Assignment Problem (CAP) Cao et al. 2024 , where the agent struggles to correctly attribute the final reward to key decisions in the sequence, leading to reduced learning efficiency and difficulty optimizing its policy. Simultaneously, RL agents should learn high-level reasoning and low-level control Dalal et al. 2024 . This dual learning mechanism increases the complexity of training, as high-level reasoning requires the agent to plan long-term strategies, while low-level control involves fine-grained, immediate decisions. Recent research has explored leveraging the prior knowledge and natural language understanding capabilities of foundation models to improve the generalization of RL agents. These foundation models have been used for the extraction of multimodal features, integrating information from various modalities such as vision and language, enabling RL agents to understand complex environments Stooke et al. 2021 ; Laskin et al. 2020 ; Schwarzer et al. 2020 . In addition, Large Language Models (LLMs) and Vision Language Models (VLMs) can serve as decision-making assistants, using their reasoning and planning capabilities to support RL agents in formulating actions and strategies Chakraborty et al. 2023 ; Majumdar et al. 2020 . Through mechanisms like the Chain of Thought (CoT), LLMs and VLMs facilitate complex reasoning in RL agents, allowing them to decompose difficult tasks and adopt step-by-step problem solving approaches Du et al. 2023 ; Yan et al. 2023 ; Li et al. 2024 . In some cases, the foundation models are independent agents, executing entire task processes autonomously Inoue and Ohashi 2022 ; Li et al. 2022 . Nevertheless, despite the valuable insights of these works, they suffer from an unideal collaboration between RL agents and foundation models, leading to under-utilization of foundation models’ reasoning capabilities, inefficient execution and interaction. Moreover, some of these efforts do not address the challenges CAP poses to RL, causing difficulty in optimizing its policies Cao et al. 2024 . The key challenge of this problem is to design a synergic interaction mechanism that integrates the knowledge of the RL agent in navigating specific environments (exceptional in short-horizon tasks) with the knowledge and reasoning skills of the foundation models (delivering rational responses even for unknown cases). Positioning LLMs and VLMs as rational long-term planning systems and RL agents as intuitive short-term reactive systems, while facilitating interaction between the two, is a research assumption that has not been adequately explored. To this end, we draw inspiration from Kahneman’s cognitive theory Daniel 2017 , particularly the dual-process framework of “System 1” and “System 2,” which enhances both the efficiency of the decision and generalization. Fig. 1 illustrates such an insight that the planning ability of System 2 can boost the decision-making pipeline of System 1. Figure 1 : Fast thinking and slow thinking Essentially, two modes of human thinking are distinguished: System 1, a fast, intuitive process driven by emotions, memory, and experience, and System 2, a slower, more deliberate approach requiring focused attention for analytical problem-solving Wei et al. 2022 ; Jin et al. 2024 ; Zhou et al. 2024 . The dual-system approach has been preliminarily implemented in areas such as Visual Question Answering (VQA) Sun et al. 2024 and Continual Learning (CL) Qi et al. 2024 , demonstrating improvements in both efficiency and generalization. However, to the best of our knowledge, no research has explored the application of dual-system concepts in the more challenging decision-making of RL agents. Building upon such a motivation, we propose the D ual- S ystem A daptive D ecision F ramework (DSADF), an innovative approach with an efficient collaboration mechanism integrating System 1 and System 2 to achieve the generalized decision-making. In this framework, System 1 comprises an RL agent and a memory module, whereas System 2 consists of a VLM. System 2 harnesses the powerful reasoning capabilities of the foundation models to extract clues from instruction hints, breaking down long-horizon initial tasks into manageable single-step tasks. It enables the RL agent to concentrate on solving each single-step task individually, significantly boosting its learning and execution efficiency and maximizing its strengths in single-step tasks. Meanwhile, the VLM continuously updates the memory module based on historical steps, assessing the proficiency of RL agents with different