Leveraging Foundation Models for Causal Generative Modeling DOI: XXXXXXX.XXXXXXX ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Computing methodologies Causal reasoning and diagnostics
Aneesh Komanduri, Xintao Wu · 2026
Read Leveraging Foundation Models for Causal Generative Modeling DOI: XXXXXXX.XXXXXXX ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Computing methodologies Causal reasoning and diagnostics by Aneesh Komanduri, Xintao Wu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-23861.txt.
Introduction
Aneesh Komanduri and Xintao Wu build FM-CGM, a modular route to visual causal reasoning from pretrained foundation models. It needs no retraining for causality. Three tools form the pipeline: one pulls out concepts, one changes them, one draws the imagined result. A large reasoning model handles causal inference, while a diffusion model renders the alternatives. Their Causal Semantic Guidance steers edits through cross attention, so changes reach later concepts while untouched regions stay fixed, enabling faithful counterfactual images.
Catalog record
| CREATOR | Aneesh Komanduri, Xintao Wu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 33733 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.23861
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Leveraging Foundation Models for Causal Generative Modeling DOI: XXXXXXX.XXXXXXX ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Computing methodologies Causal reasoning and diagnostics Aneesh Komanduri, Xintao Wu Abstract . Causal generative modeling is essential for developing reliable and transparent AI systems capable of counterfactual reasoning. While existing approaches focus on integrating causal constraints during the training of generative models, they often lack a unified framework to leverage the zero-shot reasoning capabilities of pretrained foundation models. We introduce FM-CGM, a modular framework for end-to-end visual causal reasoning using pretrained foundation models. FM-CGM formalizes the causal pipeline through three core components: a concept extractor, a concept manipulator, and a counterfactual generator. By leveraging a large reasoning model for causal inference and a text-to-image diffusion model for generation, our approach enables zero-shot causal discovery, intervention, and counterfactual generation. We then develop Causal Semantic Guidance (CSG), a cross-attention-based mechanism that ensures semantic interventions propagate to descendant concepts while preserving invariant regions. We empirically show that our approach can identify plausible causal structures and is suitable for faithful counterfactual image generation. 1. Introduction Causal generative modeling ( Komanduri et al. 2024b ) has been studied both in the context of discovering causal concepts and their relationships from high-dimensional data ( Scholkopf et al. 2021 ) and reasoning about counterfactual scenarios ( Ribeiro et al. 2025 ) . This direction has contributed to improving the reliability and transparency of AI systems. Existing work in causal generative modeling primarily focuses on integrating causality directly into the training process of generative models such as GANs ( Kocaoglu et al. 2018 ) , VAEs ( Pawlowski et al. 2020 ; Komanduri et al. 2024a ) , and Diffusion models ( Komanduri et al. 2024c ) by modeling relationships among generative causal factors. Several works have also studied counterfactual image generation using state-of-the-art diffusion models ( Sanchez and Tsaftaris 2022 ; Spyrou et al. 2025 ) . Recently, due to the inference capabilities of large-scale generative models such as large language model (LLMs) and large reasoning models, there has been ongoing research utilizing the power of foundation models as domain experts to improve causal discovery and inference ( Vashishtha et al. 2023 ) , and causal agent frameworks to automate tasks in scientific discovery ( Verma et al. 2025 ) . However, there is no unified framework that leverages the capabilities of pretrained foundation models to discover generative factors, reason about interventions, and generate counterfactual scenarios, the complete pipeline of causal generative modeling. In this work, we propose a formalism for visual causal reasoning from the perspective of pretrained foundation models. We develop Foundation Model Powered Causal Generative Model (FM-CGM) , a high-level abstraction for causal generative modeling using pretrained foundation model components. We formalize the key components of a visual causal generative model in this setting as (1) concept extractor , which infers a set of concepts and their causal relationships from a given image, (2) concept manipulator , a mechanism to perform interventions on concept variables and propagate causal effects, and (3) a counterfactual generator , a generation mechanism to map the concepts under intervention to a counterfactual image. We represent each component using a powerful large-scale foundation model. Specifically, we implement a practical method consisting of a reasoning VLM (Qwen3-VL) as the concept extractor and manipulator and text-to-image generative model (Stable Diffusion XL) as the counterfactual generator. To translate semantic interventions inferred by the concept manipulator into pixel-space counterfactual edits, we propose causal semantic guidance (CSG) a procedure that operates through the text-to-image diffusion model’s text-conditioning mechanism. Concretely, we leverage cross-attention attribution maps to localize the regions associated with each concept and its descendants and construct an inference-time guidance mechanism that (a) amplifies edits for intervened concepts, (b) encourages consistent changes in descendant concepts, and (c) suppresses unintended changes to non-descendant (invariant) concepts. This yields counterfactual images that are both visually plausible and aligned with the intended causal semantics. Contributions. We summarize our contributions as follows (1) We propose a general and modular framework for end-to-end causal generative modeling with pretrained foundation model components including a concept extractor , concept manipulator , and counterfactual generator , each represented as a foundation model. (2) We develop causal semantic guidance (CSG) , a cross-attention-based guidance method that compositionally enforces the concept under intervention and its descendants, inferred by a large reasoning model, while preserving non-descendant concepts, enabling faithful counterfactual generation. (3) We demonstrate empirically that our method can perform minimal and faithful counterfactual edits. 2. Related Work Diffusion Models for Controllable Generation . Diffusion models ( Ho et al. 2020 ; Nichol and Dhariwal 2021 ; Song et al. 2021 ) have quickly become one of the most effective tools for controllable image generation with models such as DALL-E ( Ramesh et al. 2021 ) , StableDiffusion ( Rombach et al. 2022 ) , T2I-Adapter ( Mou et al. 2024 ) , ControlNet ( Zhang et al. 2023 ) , etc. Several different techniques have been proposed to achieve controllable generation, such as fine-tuning, adaptation, and post-hoc optimization. Recent work in diffusion probabilistic models has focused on interpretable properties to enable generative control. ( Wu et al. 2023 ) show that pretrained text-to-image diffusion models, such as Stable Diffusion, already have somewhat of a content-style disentanglement capability, and propose a lightweight optimization to learn a soft mixing of content and style-infused prompts for better disentanglement. ( Yang et al. 2023 ) study disentanglement properties of diffusion probabilistic models by learning separate gradient fields. ( Brack et al. 2024 ) propose LEDITS++, an image editing technique using cross-attention attribution maps capable of composing multiple concept edits. Causality-based Diffusion Models . ( Sanchez and Tsaftaris 2022 ) provide a causal interpretation of classifier guidance in diffusion models and propose a method to guide the generation of image counterfactuals using interventions on labeled data. ( Komanduri et al. 2024c ) explore counterfactual generation in diffusion models through guidance from disentangled causal representations. ( Ribeiro et al. 2025 ) study counterfactual identifiability in high-dimensions leveraging flow-matching. ( Spyrou et al. 2025 ) develop a framework for counterfactual video editing in pretrained text-to-video generative models by utilizing VLMs to refine counterfactual prompts given a fixed causal graph. ( Wang et al. 2023 ) formalize the notion of causal separability in text-to-image generative models through a counterfactual lens and show that concepts that are independently manipulable lie in orthogonal subspaces. Based on this intuition, they propose a simple projection mechanism to manipulate style components of generated images while preserving content. Large Vision-Language Models . Extending LLMs to incorporate visual inputs, large vision-language models (LVLMs) have shown impressive performance in tasks such as recognition and visual question answering (VQA). Previous work has shown that LVLMs can sometimes struggle to perform formal causal reasoning ( Komanduri et al. 2025 ) . Recently, there has been extensive development in improving the reasoning abilities of LLMs and LVLMs through reinforcement learning techniques with verifiable rewards. Recently, there has been a large improvement in visual reasoning capabilities in vision language models, including QwenVL ( Bai et al. 2025 ) , OpenAI o1, and Gemini-Thinking. These models have shown remarkable performance on math, coding, and visual reasoning tasks. 3. Preliminaries Structural Causal Model . ( Pearl