Linear-LLM-SCM: Benchmarking LLMs for Coefficient Elicitation in Linear-Gaussian Causal Models Kanta Yamaoka, Sumantrak Mukherjee, Thomas Gärtner, David Selby, Stefan Konigorski, Eyke Hüllermeier, Viktor Bengs, Sebastian Vollmer Abstract Large language models (LLMs) have shown potential in identifying qualitative causal relations, but their ability to perform quantitative causal reasoning---estimating effect sizes that parametrize functional relationships---remains underexplored in continuous domains. We introduce Linear-LLM-SCM, a plug-and-play benchmarking framework for evaluating LLMs on linear Gaussian structural causal model (SCM) parametrization when the DAG is given. The framework decomposes a DAG into local parent-child sets and prompts an LLM to produce a regression-style structural equation per node, which is aggregated and compared against available ground-truth parameters. Our experiments show several challenges in such benchmarking tasks, namely, strong stochasticity in the results in some of the models and susceptibility to DAG misspecification via spurious edges in the continuous domains. Across models, we observe substantial variability in coefficient estimates for some settings and sensitivity to structural and semantic perturbations, highlighting current limitations of LLMs as quantitative causal parameterizers. We also open-sourced 1 1 1 https://github.com/datasciapps/parameterize-dag-with-llm the benchmarking framework so that researchers can utilize their DAGs and any off-the-shelf LLMs plug-and-play for evaluation in their domains effortlessly. 1 Introduction Robust intelligence requires an agent to have an internal “world model” ( 10 ) , an internal causal mechanism to be able to infer causal structures and their effect relationships ( 27 ) . Recent work ( 28 ) suggests that any agent capable of solving complex decision tasks must effectively learn a causal model of its data-generating process. Given the recent advancement of large language models (LLMs), they exhibit the ability to encode a vast corpus of human knowledge, such as clinical knowledge ( 31 ) , one may also speculate whether LLMs are encoding literature with causal information and therefore constructing causal models. Per LLMs’ Causal Hierarchy , recently classified by 40 , today’s LLMs have shown promise in Type 1 tasks (identifying causal relationships using domain knowledge), but they struggle with Type 2 (discovering new knowledge from data) and Type 3 (quantitative estimation of consequences) tasks. LLMs can often (but not always) reconstruct the qualitative structure of a directed acyclic graph (DAG) as Type 1. However, Type 3, their ability to perform quantitative parameterization, estimating the specific effect sizes within those structures, remains under-explored in continuous variables, necessitating theoretical developments and empirical results. In this paper, we introduce Linear-LLM-SCM, a benchmarking framework designed to evaluate the quantitative causal capabilities of LLMs. Our framework programmatically decomposes DAGs into local parent-child structures, tasking LLMs with eliciting regression coefficients for linear Gaussian SCMs. Our framework itself is agnostic to LLMs and DAGs; in our open-sourced testbed, researchers can use their own DAGs and any off-the-shelf LLMs plug-and-play for evaluation in their domains with minimal effort. We investigate the following research questions (RQs): • RQ1: Given a pre-specified DAG, can LLMs elicit plausible regression coefficients in comparison to the real-world ground truths? • RQ2: How robust is this parameterization when facing adversarial conditions, such as DAG misspecification (spurious edges) or changes in variable units? • RQ3: What are the common failure modes encountered when using LLMs for parameterization? One may wonder why we assess DAG misspecification when assuming pre-specified structures. Real-world graphs are often uncertain: experts may omit or include edges, and data-driven discovery can yield multiple plausible DAGs in the face of uncertainty ( 25 ) . Today’s causal discovery largely focuses on point estimates ( 35 ) , while uncertainty-aware Bayesian structure learning remains challenging due to combinatoric complexity. Even in the future LLM-based causal discovery becomes more feasible, it may produce misspecified DAGs due to limited uncertainty quantification ( 26 ) . The downstream causal effect estimation, which comes after possibly misspecified DAGs and which we aim to investigate in our framework, also needs robustness in this regard. Employing such adversarial perturbations also help mitigate potential data leakage ( 38 ) in LLMs for evaluation in causality. Our key contributions can be summarized as follows: • We introduce Linear-LLM-SCM, a framework that allows us to evaluate LLMs’ ability to estimate linear-Gaussian SCM coefficients when the DAG structure is given, by decomposing the graph into local parent–child structures. • We open-source an evaluation pipeline that supports plug-and-play DAGs, variable metadata (units/ranges), as well as LLMs, and reports coefficient-distance and ordering metrics against ground truth. 2 2 2 https://github.com/datasciapps/parameterize-dag-with-llm • Broadly, our work provides a practical benchmarking framework of causal effect estimation given an assumed DAG, which helps further research of reasoning and generalizability of LLMs as motivated by 27 and 28 . 2 Background and Related Work Structural causal models (SCMs) formalize causal systems as a pair (\mathcal{G},\mathbf{F}) , where \mathcal{G} is a directed acyclic graph (DAG) encoding causal structure and \mathbf{F}=\{f_{i}\} is a set of structural equations, with each endogenous variable generated as X_{i}=f_{i}(\mathrm{Pa}(X_{i}),E_{i}) . Recent work has explored large language models (LLMs) for causal reasoning, primarily focusing on causal discovery and qualitative inference ( 22 ; 21 ; 15 ) . However, several studies raise concerns about LLMs’ causal capabilities, arguing that they mainly succeed at identifying causal relationships using prior knowledge (Type 1) , while struggling with discovering causal structure from data (Type 2) and quantitatively estimating intervention effects (Type 3) ( 40 ; 39 ; 38 ; 14 ) . Most existing work on LLM-based causal effect estimation focuses on discrete domains or requires numerical observational data and specialized architectures ( 4 ; 6 ; 41 ) . Closest to our setting, 3 combined LLMs with pre-specified causal graphs but estimate effects via sampling-based conditional distributions rather than directly eliciting structural parameters. In contrast, we study whether LLMs can directly estimate continuous linear causal effect parameters for pre-specified SCMs using only causal structure and variable semantics, without observational data. A detailed review of related work is provided in Appendix A.1 . 3 Linear-LLM-SCM Benchmarking Framework We provide pre-specified DAG structures that decompose real-world phenomena into parent-child interactions. The system iterates through the DAG, calling LLMs via prompt templates to elicit functional mappings for each parent-child set, then aggregating results to obtain full effect parameters. This assumes LLMs learned quantitative or qualitative information from their large training corpus. For quantitative aspects, LLMs may have encountered scientific literature with effects formalized as SCMs, Bayesian Networks, or linear regressions/ODEs/PDEs. For qualitative aspects, LLMs encountered natural language causal statements (e.g., “Coffee consumption positively regulates alertness after 1 hour”) in scientific and social media texts. 3.1 Overview of the benchmarking framework The Algorithm 1 describes the steps of our benchmarking framework. The framework requires the following input: DAG structure (nodes and directed edges), node descriptions with units (e.g., “GC”: “Glucose (micromolar, µM)”), and variable constraints (lower and upper bounds). The framework programmatically iterates through all parent-child sets. For each set, the program crafts a prompt with domain expert persona (“You are an expert in XYZ…”), the target variable, its direct parents, and their constraints. The prompt visibility includes only the current node and direct parents, plus shared phenomenon information. For example, for a DAG about a phenomenon, Cachexia, the prompt includes the following texts: “You are going to identify internal dynamics of a phenomenon, Cachexia. Cachexia is a complicated metabolic syndrome related to underlying illness and characterized by muscle mass loss with or without fat mass loss that is often associated with anorexia, an inflammatory process, insulin resistance, and increased protein turnover.” For details of each prompt to call to LLMs, we will discuss in Section 3.2 . Algorithm 1 Linear-LLM-SCM Benchmarking Framework 1: DAG structure \mathcal{G}=(\mathcal{V},\mathcal{E}) , Variable descriptions \mathcal{D} , Value ranges \mathcal{R} , Phenomenon overview \mathcal{P} , LLM \mathcal{M} 2: Aggregated set of linear coefficients \bm{\hat{\beta}} 3: Initialize \bm{\hat{\beta}}\leftarrow\emptyset 4: for each target variable X_{j}\in\mathcal{V} in topological order do 5: 1. Identify direct parents Pa(X_{j})\subset\mathcal{V} using edges \mathcal{E} \triangleright Decompose into local structures 6: 2. Construct prompt S_{j} incorporating: 7: • Domain expert persona and Phenomenon overview \mathcal{P} 8: • Short descriptions \mathcal{D} and Units for X_{j} and Pa(X_{j}) 9: • Hard constraints/Value ranges \mathcal{R} 10: 3. Call LLM: Response\leftarrow\mathcal{M}(S_{j}) requesting JSON format 11: 4. Parse Response : Extract numerical \hat{\beta} for Pa(X_{j}) and intercept \beta_{0} \triangleright This part has an iterative feedback mechanism, see Algorithm 2 12: 5. \bm{\hat{\beta}}\leftarrow\bm{\hat{\beta}}\cup\{\hat{\beta}_{i,j}\} 13: end for 14: return Aggregated parameterized SCM \bm{\hat{\beta}} for an entire graph structure \mathcal{G} The units of variables are specified in descriptions of direct parents and target variables, sourced from original literature or author judgment. We currently do not elicit confidence intervals for coefficients; residual variance is included as the last term in the functional specification. While Linear SCMs typically include uncertainty quantification, we have not yet implemented this in our prompt. We obtain the functional mapping of direct parents and target variables as text expressions within JSON objects returned by LLMs. While LLMs provide plausibility descriptions for debugging, we typically parse only the parameterized equation from the proposed_lin_str_eq field. The system programmatically extracts beta coefficients from the LLM structural equation format. 3.2 Prompt for Node Level Function Parametrization Designing the interaction between LLMs and DAGs is critical. Our current approach traverses the DAG node-by-node; for each variable, the LLM receives descriptions of only its direct parents to determine functional parameterization. While one could pass the entire graph via text representations (e.g., DAGitty by 33 or Mermaid by 32 ) or feed entire DAGs into visual language models, we chose the parent-child template based on Occam’s Razor. This focus on local structures also aligns with recent benchmarks ( 3 ; 24 ) . Our prompt includes the names and short descriptions of the target variable and its direct parents, alongside formatting instructions. Figure 1 shows an example of a prompt for a parent-child local structure. The prompt is consisting of three parts: First, expert persona, a summary of the phenomenon of interest, and variable units are introduced to the LLM. Next, the parameterization task, linear equation template, and variable ranges appear. Finally, the LLM is informed of the output format, which starts with the thought process and ends with the parameterization result as a string. This process corresponds to line 4 of Algorithm 1 . Figure 1: An example of a prompt for a local parent-child structure in a DAG. 3.3 Iterative Feedback When eliciting functional mappings per target variable, the LLM-SCM framework includes an iterative feedback refinement mechanism (Algorithm 2 ). This process requires pre-specified hard constraints for each variable. This matters because parameterization happens at the direct parent-child level and coefficients are used downstream; therefore the refinement enforces such constraints. For each variable, evaluate its possible value range based on parent constraints (C1) and the proposed parameterization, then compare it with the node’s constraints (C2) to check whether C2 includes C1. If not, re-parameterize using the last proposal, repeating up to the loop budget (e.g., 5). Algorithm 2 Iterative Feedback for Refinement with Hard Constraints 1: Target variable X_{j} , Parent hard constraints R_{Pa(X_{j})} , Node hard constraints C_{2} , Loop budget n 2: Accepted linear coefficients \hat{\beta} 3: for \textit{iteration}=1 to n do 4: Call LLM \mathcal{M} with prompt S_{j} to get proposal P \triangleright Includes parameterization \hat{\beta} 5: Calculate possible value range C_{1} of X_{j} based on R_{Pa(X_{j})} and P 6: if C_{2}\text{ includes }C_{1} then \triangleright Validation check 7: return \hat{\beta} (Accept proposal) 8: else 9: Reject proposal 10: Update prompt S_{j} with previous proposal and validation results 11: end if 12: end for 13: return Last available \hat{\beta} \triangleright Budget n reached 3.4 Metrics for Evaluating Parameterization Our framework performs comparisons between ground-truth effect parameters and the effect parameterization from LLMs. For DAGs with learned parameter ground truth (explained in the next section), we calculate metrics (M1)–(M4). For (M1)–(M3), we compute the L2 norm over all linear coefficients across nodes; the contribution aggregation differs by metric. These metrics capture distances between LLM-elicited parameter vectors per node \bm{\beta}_{LLM,j} and ground truth vectors per node \bm{\beta}_{GT,j} . For each vector, we denote j ’s each direct parents’ edges effect size (scaler) using index i : \beta_{LLM,j,i} and \beta_{GT,j,i} . (M4) captures relative effect-size ordering per node. Concretely, if a node j has parents a and b and the effect sizes are {\beta}_{LLM,j,a}=-0.8<{\beta}_{LLM,j,b}=0.5 while {\beta}_{GT,j,a}=-2<{\beta}_{GT,j,b}=3 , for this node we increment the sum by one before moving to the next node, finally obtaining (M4). The metrics are summarized as follows: • (M1) L2 Norm distance between LLM-elicited vs GT: M_{1}=\sqrt{\sum_{j\in V}\sum_{i\in Pa(j)}(\beta_{LLM,j,i}-\beta_{GT,j,i})^{2}} (1) • (M2) L2 Norm distance with node-wise effect normalization: M_{2}=\sqrt{\sum_{j\in V}\sum_{i\in Pa(j)}\left(\frac{\beta_{LLM,j,i}}{\|\bm{\beta}_{LLM,j}\|_{2}}-\frac{\beta_{GT,j,i}}{\|\bm{\beta}_{GT,j}\|_{2}}\right)^{2}} (2) • (M3) L2 Norm distance excluding edges with single parents: M_{3}=\sqrt{\sum_{j\in\{V:|Pa(X_{j})|>1\}}\sum_{i\in Pa(X_{j})}\left(\frac{\beta_{LLM,j,i}}{\|\bm{\beta}_{LLM,j}\|_{2}}-\frac{\beta_{GT,j,i}}{\|\bm{\beta}_{GT,j}\|_{2}}\right)^{2}} (3) • (M4) Effect size relative ordering matches per target variable: M_{4}=\sum_{j\in\{V:|Pa(X_{j})|>1\}}\mathbb{I}\left(\text{ordering of }\{\beta_{\text{LLM},i,j}\}_{i\in Pa(X_{j})}=\text{ordering of }\{\beta_{\text{GT},i,j}\}_{i\in Pa(X_{j})}\right) (4) 4 Experimental Setup We assume linear functional elicitation for pre-specified DAGs using LLMs given their nodes and directed edges. In other words, we will leave more complicated non-linear functional elicitation to our future work. This is due to verifiability. In Bayesian Network literature, if we consider the linear assumption, it is possible to obtain linear settings including learned parameters as well as DAG structures ( 18 ) for real-world settings; If we consider non-linear parameterization with non-synthetic example, obtaining the ground truths would be difficult. In our functional elicitation, we assume direct parameterization in a symbolic fashion using LLMs, but we do not sample many effect examples from LLMs unlike the most similar work to our knowledge ( 3 ) . For DAGs and their parameterization, we assume continuous variables but not discrete ones. 4.1 LLM Model Selection For our experiment, general purpose pre-trained LLMs (Gemini 2.5 Flash and Llama 3 family) as in Table 1 . We tried models with different model sizes and architectures either mixture-of-experts (MoE), where an input is routed into different experts, obtaining output in an ensemble fashion ( 13 ) , vs non-MoE ones. For Gemini 2.5 Flash, the model consists of sparse mixture-of-experts (MoE) ( 7 ) . Representative models from the open-weights community, Llama 3.1 8B and Llama 3.3 70B have dense transformer, where all parameters are used for inference ( 20 ) . Table 1: Benchmark model specifications for our study. MoE stands for mixture of experts. Dense means dense-transformer and this also indicates the model is non-MoE-based. The tick ✓indicates yes, and the cross \times indicates no. MODEL NAME MODEL SIZE ARCHITECTURE OPEN WEIGHTS Gemini 2.5 Flash Unknown MoE \times Llama 3.1 8B 8B Dense ✓ Llama 3.3 70B 70B Dense ✓ 4.2 DAGs With Ground Truth Effect Parameters In our experiment, we use real-world DAGs from a Bayesian Network repository, BnRep ( 18 ) . In the Appendix, Figure 4 describes how we selected DAGs and effect ground-truths from a large DAG repository. First, we kept the ones with a continuous Linear Gaussian setting which corresponds to the linear SCM setting. For convenience, we limit ourselves on DAGs with no more than 15 nodes. Moreover, we excluded DAGs where variable names consist of letters and numbered suffixes, for example, “X1” and “X2.” Table 2 lists the 7 resulting DAGs that we use in our experiments. Table 2: DAGs with ground-truths included for our study from BnRep repository. Literature indicates sources introducing either the DAG structure or DAG effect parameterization. VR. (Value Ranges) indicates whether value ranges are available in the original literature. NAME LITERATURE VR. NODES DOMAIN cachexia1 8 ; 5 Y 6 Genetics expenditure 34 ; 9 N (Crafted) 12 Economics foodsecurity 17 ; 1 Y 4 Social Sciences algal2 12 Y 9 Env. Science lexical 2 Y 8 Social Sciences liquefaction 11 Y 10 Earth Sciences stocks 29 Y 13 Economics 4.3 Adversarial Conditions Recall that our RQ2 was: How robust is this parameterization when facing adversarial conditions, such as DAG misspecification (spurious edges) or changes in variable units? In this section, we perform robustness checks with two additional and adversarial conditions, (I) changes in the unit values or (II) DAG misspecification. 4.3.1 (I) Tweaking Units to Check Robustness For the first type of adversarial conditions, namely changes in units, we use Cachexia1 DAG from Table 2 because the DAG comes with units and the DAG structure is relatively simpler among others. We captured aggregated trends on each model with temperature set to zero to make the behavior as deterministic as possible. We used 25 runs for each condition. In our experiments regarding (i), we employed the following two conditions: (A) the Cachexia DAG with real bounds and units, and (B) the Cachexia DAG with real bounds and tweaked units from micromolar to nanomolar. 4.3.2 (II) Simulated DAG Misspecification to Check Robustness We created multiple adversarial mutated examples. For each sampler, we added a spurious edge between two variables with no actual connections in the ground truths. Based on this, we performed parameterization to check how robust each LLM is. Based on the original Expenditure DAG (available in the Appendix, Figure 3 ), we created the following four DAG exemplars while keeping their acyclic nature. There are four mutated examples originating from the original DAG, namely adding the following spurious edges: (S1) Owner \rightarrow Expenditure , (S2) Majorcards \rightarrow Dependents , (S3) Owner \rightarrow Share , and (S4) Majorcards \rightarrow Selfemp . 5 Results and Discussion 5.1 RQ1: Direct Estimation Results Table 3 presents direct parameter estimation results across three LLM models on all 7 DAGs from Table 2 . Throughout, values after \pm indicate 95% CIs. For M1 metrics, which are simple L2 norm values, there is high variability among different DAG variants in the same model family. The M1 variability across models is also observed. This is because M1 is not scale invariant with respect to the variable ranges. Given this, we focus on metrics M2 (normalized), and M3 (focus on effect distance from multiple parents of a node), M4 (scale invariant, counting metrics). In M2, for each DAG, the best model varies. From M3, in 6/7 DAGs Gemini 2.5 Flash outperforms others. From M4, in 6/7 DAGs, Gemini 2.5 Flash is the best. In comparisons of Llama 3.1 8B and Llama 3.3 70B, in general, Llama 3.3 70B is in most cases better than the other, indicating larger parameter size can result in improvement in quantitative causal parameterization. Note that Llama 3.1 8B failed to generate parsable equations for DAG algal2 . We did not optimize for this case to avoid harming other model-DAG conditions. We also used temperature set to zero, which in theory should make models deterministic. Nevertheless, larger models (Gemini 2.5 Flash, Llama 3.3 70B) exhibit high stochasticity across metrics, with non-Gaussian distributions at n=25 samples, possibly due to hardware and software factors beyond our control via external APIs. This variability is concerning for safety-critical domains, e.g., medicine, where incorrect parameterization can propagate through downstream effects and hinder adoption, corroborating research on LLM non-determinism ( 16 ) . Table 3: Direct estimation results (Averaged n=25 , Temp 0). Values after \pm indicate 95% CIs. M1 : L2 distance; M2 : normalized L2; M3 : normalized L2 excluding single-parent edges; M4 : Effect relative order count. For M1 – M3 , lower is better ( \downarrow ); for M4 , higher is better ( \uparrow ). See Table 2 for DAG descriptions. MODEL DAG M1 \downarrow M2 \downarrow M3 \downarrow M4 \uparrow Gemini 2.5 Flash cachexia1 \textbf{13.783}\pm 1.826 2.449\pm 0.240 \textbf{1.074}\pm 0.114 \textbf{1.000}\pm 0.000 Llama 3.1 8B cachexia1 16.527\pm 0.000 \textbf{1.036}\pm 0.000 1.036\pm 0.000 0.000\pm 0.000 Llama 3.3 70B cachexia1 13.843\pm 0.938 1.994\pm 0.213 1.352\pm 0.071 0.440\pm 0.199 Gemini 2.5 Flash expenditure 148084.559\pm 22359.281 \textbf{0.998}\pm 0.226 \textbf{0.998}\pm 0.226 \textbf{7.520}\pm 0.200 Llama 3.1 8B expenditure \textbf{2463.377}\pm 0.000 2.053\pm 0.000 2.053\pm 0.000 7.000\pm 0.000 Llama 3.3 70B expenditure 27137.540\pm 12574.354 1.548\pm 0.234 1.548\pm 0.234 6.560\pm 0.199 Gemini 2.5 Flash foodsecurity 22.801\pm 0.038 2.017\pm 0.006 \textbf{0.236}\pm 0.050 \textbf{0.500}\pm 0.214 Llama 3.1 8B foodsecurity \textbf{22.727}\pm 0.006 \textbf{0.447}\pm 0.000 0.447\pm 0.000 0.000\pm 0.000 Llama 3.3 70B foodsecurity 22.914\pm 0.009 2.039\pm 0.001 0.395\pm 0.004 0.000\pm 0.000 Gemini 2.5 Flash algal2 \textbf{4.094}\pm 0.256 \textbf{0.514}\pm 0.098 \textbf{0.514}\pm 0.098 \textbf{2.000}\pm 0.000 Llama 3.1 8B algal2 Model output equations not parsable by the program. Llama 3.3 70B algal2 4.657\pm 0.004 0.559\pm 0.054 0.559\pm 0.054 \textbf{2.000}\pm 0.000 Gemini 2.5 Flash lexical 42.246\pm 0.650 \textbf{2.036}\pm 0.053 \textbf{2.036}\pm 0.053 \textbf{2.040}\pm 0.265 Llama 3.1 8B lexical 35.854\pm 0.000 3.252\pm 0.000 2.565\pm 0.000 1.000\pm 0.000 Llama 3.3 70B lexical \textbf{35.847}\pm 0.001 2.274\pm 0.037 2.274\pm 0.037 0.778\pm 0.374 Gemini 2.5 Flash liquefaction 12.294\pm 5.556 \textbf{0.844}\pm 0.034 \textbf{0.844}\pm 0.034 2.800\pm 0.160 Llama 3.1 8B liquefaction \textbf{9.999}\pm 0.000 2.012\pm 0.002 2.012\pm 0.002 \textbf{3.000}\pm 0.000 Llama 3.3 70B liquefaction 12.093\pm 0.221 1.271\pm 0.026 1.271\pm 0.026 \textbf{3.000}\pm 0.000 Gemini 2.5 Flash stocks \textbf{0.829}\pm 0.048 1.059\pm 0.122 1.006\pm 0.063 \textbf{3.478}\pm 0.242 Llama 3.1 8B stocks 0.893\pm 0.005 1.589\pm 0.003 1.589\pm 0.003 2.880\pm 0.130 Llama 3.3 70B stocks 1.060\pm 0.026 \textbf{0.934}\pm 0.059 \textbf{0.934}\pm 0.059 2.600\pm 0.226 5.2 RQ2: Robustness Results To assess the robustness of LLM-based parameter estimation, we examine how model predictions respond to adversarial conditions. Values after \pm indicate 95% CIs. Table 4 and Table 5 quantify performance degradation when facing unit value changes and DAG misspecification respectively. 5.2.1 (i) Unit Tweaking Robustness Table 4 evaluates model robustness when unit values for the Cachexia1 domain are tweaked from micromolar to nanomolar scale. This robustness check tests whether models maintain consistent causal effect estimates when facing realistic variations in measurement units, which is critical for practical deployment where domain experts may choose different units. Similar to the previous direct estimation result, focusing on M2-M4, sometimes, counter-intuitively tweaked units resulted in better parameterization. Admittedly, to conclude this, this would require more empirical coverage and we will extend this adversarial example of unit tweaking and we try to avoid overspeculation. Table 4: Robustness under unit tweak (Averaged n=25 , Temp 0). Values after \pm indicate 95% CIs. M1 : L2 distance; M2 : normalized L2; M3 : normalized L2 excluding single-parent edges; M4 : Effect relative order count. For M1 – M3 , lower is better ( \downarrow ); for M4 , higher is better ( \uparrow ). Units: (L)= \mu M, (T)=nM. MODEL CND. UNITS M1 \downarrow M2 \downarrow M3 \downarrow M4 \uparrow Gemini 2.5 Flash A L 13.783\pm 1.826 2.449\pm 0.240 1.074\pm 0.114 1.000\pm 0.000 B T \textbf{12.399}\pm 1.774 \textbf{1.769}\pm 0.372 \textbf{0.917}\pm 0.136 \textbf{1.120}\pm 0.206 Llama 3.1 8B A L \textbf{16.527}\pm 0.000 1.036\pm 0.000 1.036\pm 0.000 0.000\pm 0.000 B T 17.198\pm 0.000 \textbf{0.905}\pm 0.002 \textbf{0.905}\pm 0.002 0.000\pm 0.000 Llama 3.3 70B A L 13.843\pm 0.938 \textbf{1.994}\pm 0.213 1.352\pm 0.071 0.440\pm 0.199 B T \textbf{13.441}\pm 0.199 2.439\pm 0.160 \textbf{1.107}\pm 0.115 \textbf{0.640}\pm 0.250 5.2.2 (ii) DAG Misspecification Robustness Table 5 examines model performance when the Expenditure DAG is intentionally misspecified by adding spurious edges, where O indicates the original DAG while S1-S4 indicate mutated variants. As we discussed earlier, our analysis is based on the more reliable metrics, M2-M4. First, M2, which is normalized by node (vector at the node level is normalized), is the lowest with original DAG conditions compared to other mutated conditions. This is the case among all the models. Next, if we compare each value of M2 and M3 of the same model and the same DAG, they have the same values. Recall that contributions of M3 are filtered to keep the parent effects for a variable only when there are multiple parents. M2 and M3 values being the same makes sense due to the original expenditure DAG as in Figure 3 in the Appendix. All the variables that have at least one parent in the DAG have multiple parents already, so the filtering mechanism in M3 does not exclude any contributions from the parent effects. In this particular DAG, M2 and M3 are the same, but in other DAGs generally this does not happen. In M4, overall in each model, the relative effect order counts are the best for the original DAGs in Gemini 2.5 Flash and Llama 3.1 8B, except Llama 3.3 70B, where the original DAG is the second best per M4, while S4 is the best. However, there is overlap of 95% intervals among the two conditions. To summarize, generally, adversarial conditions yield lower M4 values, quantifying performance degradation given DAG misspecification with spurious edges. In comparisons of the models, Llama 3.1 8B has almost no variability within each DAG conditions. This is already mentioned in previous direct estimation results, and also the case in S1-S4 too. This indicates in a deterministic model, adding spurious edges do not introduce stochasticity in the coefficients. This indicates the stochasticity of the models may not come from confusion, but could be attributed to the model itself. Table 5: Robustness under DAG misspecification for the expenditure DAG (Averaged n=25 , Temp 0). Values after \pm indicate 95% CIs. M1 : L2 distance; M2 : normalized L2; M3 : normalized L2 excluding single-parent edges; M4 : Effect relative order count. For M1 – M3 , lower is better ( \downarrow ); for M4 , higher is better ( \uparrow ). Misspec: (O)=Original DAG, (S1)-(S4)=Spurious edges added. MODEL MISSPEC. M1 \downarrow M2 \downarrow M3 \downarrow M4 \uparrow Gemini 2.5 Flash O 148084.559\pm 22359.281 \textbf{0.998}\pm 0.226 \textbf{0.998}\pm 0.226 \textbf{7.520}\pm 0.200 S1 \textbf{81587.682}\pm 20626.822 1.372\pm 0.282 1.372\pm 0.282 6.200\pm 0.299 S2 138770.541\pm 21002.249 1.235\pm 0.267 1.235\pm 0.267 6.400\pm 0.253 S3 120240.669\pm 20548.802 1.355\pm 0.253 1.355\pm 0.253 6.560\pm 0.255 S4 106012.716\pm 23004.372 1.988\pm 0.189 1.988\pm 0.189 7.320\pm 0.271 Llama 3.1 8B O \textbf{2463.377}\pm 0.000 \textbf{2.053}\pm 0.000 \textbf{2.053}\pm 0.000 \textbf{7.000}\pm 0.000 S1 20151.134\pm 0.000 2.483\pm 0.000 2.483\pm 0.000 6.000\pm 0.000 S2 \textbf{2463.377}\pm 0.000 2.063\pm 0.001 2.063\pm 0.001 6.000\pm 0.000 S3 \textbf{2463.377}\pm 0.000 2.106\pm 0.000 2.106\pm 0.000 6.000\pm 0.000 S4 \textbf{2463.377}\pm 0.000 2.413\pm 0.000 2.413\pm 0.000 \textbf{7.000}\pm 0.000 Llama 3.3 70B O 27137.540\pm 12574.354 \textbf{1.548}\pm 0.234 \textbf{1.548}\pm 0.234 6.560\pm 0.199 S1 25843.000\pm 13086.141 1.703\pm 0.217 1.703\pm 0.217 5.520\pm 0.230 S2 29202.638\pm 13299.788 1.918\pm 0.186 1.918\pm 0.186 5.920\pm 0.251 S3 24257.613\pm 11979.590 2.038\pm 0.118 2.038\pm 0.118 5.360\pm 0.192 S4 \textbf{12524.332}\pm 7698.439 2.222\pm 0.138 2.222\pm 0.138 \textbf{6.680}\pm 0.271 Limitations and Future Work The parameterization degradation in the lens of M2-M4 with spurious edge addition suggests that LLMs are prone to being distracted by irrelevant context ( 30 ) . Our research scope was to provide a benchmarking framework; we do not address how to mitigate this LLMs’ susceptibility to perturbations, which we leave as future work. Another limitation is our focus on linear assumption in SCMs. We also admit the empirical coverage for the unit tweaking is limited and will plan to scale into different unit specifications in different DAGs. 6 Conclusion The Linear-LLM-SCM framework benchmarks the quantitative causal reasoning of LLMs by decomposing DAGs into parent-child structures for coefficient elicitation. Among Gemini 2.5 Flash, Llama 3.1 8B, Llama 3.3 70B, results indicate that Gemini 2.5 Flash achieves the highest performance across the tested DAGs when evaluated using normalized distance (M3) and relative effect size ordering (M4). These findings demonstrate the necessity of scale-invariant metrics, e.g., (M4), as L2 distance (M1) is influenced by variable ranges. A challenge identified is the stochasticity in Gemini 2.5 Flash and Llama 3.3 70B, which produced inconsistent results at a temperature of zero. This behavior contrasts with Llama 3.1 8B, which shows consistency but lower accuracy. Such variance presents risks for deployment in fields like healthcare or public policy, where decision-making requires deterministic parameterization since such inaccuracies might propagate through the downstream causal structure’s and affect the validity of other downstream nodes’ effects. Robustness testing shows susceptibility to DAG misspecification, in terms of adding spurious edges, resulting in performance degradation and lower effect ordering accuracy. Models struggle to assign zero or near-zero coefficients to these edges, indicating a lack of robustness to DAG structural uncertainty. The framework currently assesses linear-Gaussian models and is limited by its linear assumption. Future research should extend these benchmarks to non-linear functional forms and investigate methods to mitigate the impact of structural noise on performance. Overall, we plan to increase empirical coverage in the adversarial examples to further understand when the current LLMs performs well, poorly, how to mitigate, or what kind of architectural changes are required toward robust intelligence capable of quantitative causal reasoning. We hope our open-source framework will support further empirical investigations of the quantitative causal reasoning of LLMs in the community. 7