Plausible but Wrong: A case study on Agentic Failures in Astrophysical Workflows
Shivam Rawat and Lucie Flek · 2026
Read Plausible but Wrong: A case study on Agentic Failures in Astrophysical Workflows by Shivam Rawat and Lucie Flek (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-25345.txt.
Introduction
Rawat and Flek test CMBAgent on eighteen space tasks in two modes. In One-Shot mode, rich domain context lifts scores by about 6 times, but the main flaw is quiet wrong math: code that looks right yet gives wrong results. In Deep Research mode, the system also fails in silence, giving posteriors that clash. Scores are strong on crisply set tasks but fall on tasks that push reasoning bounds. The worst flaw is sure-but-wrong results. They share the frame for AI trust checks.
Catalog record
| CREATOR | Shivam Rawat and Lucie Flek |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 48597 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.25345
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Plausible but Wrong: A case study on Agentic Failures in Astrophysical Workflows Shivam Rawat and Lucie Flek Abstract Agentic AI systems are increasingly being integrated into scientific workflows, yet their behavior under realistic conditions remains insufficiently understood. We evaluate CMBAgent across two workflow paradigms and eighteen astrophysical tasks. In the One-Shot setting, access to domain-specific context yields an approximately ∼ 6 × {\sim}6\times performance improvement ( 0.85 0.85 vs. ≈ 0 \approx\!0 without context), with the primary failure mode being silent incorrect computation—syntactically valid code that produces plausible but inaccurate results. In the Deep Research setting, the system frequently exhibits silent failures across stress tests, producing physically inconsistent posteriors without self-diagnosis. Overall, performance is strong on well-specified tasks but degrades on problems designed to probe reasoning limits, often without visible error signals. These findings highlight that the most concerning failure mode in agentic scientific workflows is not overt failure, but confident generation of incorrect results. We release our evaluation framework to facilitate systematic reliability analysis of scientific AI agents. I Introduction Recent advances in large language models (LLMs) [ 5 , 24 , 1 ] and tool-augmented reasoning systems [ 26 , 20 , 18 ] have enabled the emergence of agentic AI systems capable of multi-step reasoning, external tool use, and autonomous decision-making. These systems are increasingly proposed as assistants for scientific workflows, where they can perform data analysis, hypothesis generation, and pipeline orchestration with limited human intervention. Despite promising demonstrations, existing work largely evaluates such systems through task-level success or synthetic benchmarks, offering limited insight into their reliability in realistic scientific settings. Scientific workflows differ fundamentally from standard AI benchmarks: they involve structured pipelines, domain-specific constraints, noisy data, and multi-stage reasoning, where early errors can propagate throughout the pipeline. As a result, aggregate success metrics alone are insufficient to characterize system behavior in these environments. Importantly, our focus is not on proposing or critiquing agentic architectures themselves, but on evaluating their scientific reliability and failure modes when deployed in realistic workflows. A central challenge is that agentic systems inherit known limitations of LLMs, including hallucination, brittle reasoning, and misalignment with task objectives. When embedded in multi-step workflows with external tool interaction, these issues can lead not only to incorrect intermediate steps but also to silent numerical errors and physically inconsistent outputs. Critically, such failures may not be detectable from final outputs alone, making systematic evaluation of reliability and error modes essential for scientific deployment. In this work, we present a structured empirical evaluation of an existing agentic system applied to astrophysical inference tasks. Rather than introducing a new model or framework, our goal is to characterize performance boundaries, assess robustness and calibration, and identify systematic failure modes in realistic scientific workflows. Astrophysics provides a particularly suitable testbed due to its well-defined physical models, established numerical tools, and availability of reference solutions from both simulation and observation. This allows controlled evaluation of correctness while maintaining real-world scientific relevance. We introduce a structured evaluation framework for agentic scientific workflows that integrates execution success, parameter accuracy, and numerical fidelity. Using this framework, we conduct a systematic analysis of reliability across both single-step and multi-step reasoning settings, identifying key failure modes such as silent numerical errors and physically inconsistent inference, and quantifying their prevalence under different task conditions. II Related Work LLM Agents and Tool Use. The ability of LLMs to interact with external tools has been a key driver in the transition from static text generation to dynamic agentic behavior. ReAct [ 26 ] established the paradigm of interleaving reasoning traces with action execution, enabling models to plan and adapt in response to environmental feedback. Toolformer [ 20 ] extended this by training models to invoke APIs autonomously through self-supervision, while WebGPT [ 18 ] demonstrated that grounding language models in live retrieval substantially improves factual reliability. More recent work has formalised tool-use evaluation, with benchmarks such as ToolBench [ 19 ] and TRAJECT-Bench [ 9 ] , revealing systematic failure modes including similar-tool confusion, parameter-blind selection, and performance degradation with trajectory length. Together, these works establish that structured tool access, rather than parametric knowledge alone, is what enables LLMs to operate effectively in multi-step workflows — a finding our results reinforce in the context of scientific computation. LLM Agents in Scientific Workflows. A growing number of works demonstrate that agentic systems can execute substantial portions of real scientific workflows, spanning hypothesis generation, data analysis, and manuscript production [ 8 , 21 , 23 , 16 , 10 ] . While these works establish the feasibility of agentic scientific automation, evaluation is typically limited to an end-to-end task success or qualitative expert assessment. Systematic analysis of reliability, failure modes, and error propagation in multi-step scientific pipelines remains largely unexplored. Reliability and Evaluation of LLMs. A substantial body of work examines the limitations of LLMs, including hallucination [ 11 , 2 ] , factual inconsistency [ 14 ] , and misalignment with user intent [ 3 ] . Critically, recent surveys show that hallucinations in agentic settings differ qualitatively from single-step LLM failures: they emerge through multi-step sequential interactions involving tool use, memory, and inter-agent communication, making them substantially harder to detect from final outputs alone [ 15 ] . More recent work has begun to formalize agent evaluation, distinguishing between tool-use capability, trajectory-level reasoning, and domain-specific performance [ 27 , 6 ] . However, systematic evaluation of reliability under realistic multi-step scientific workflows remains limited. Positioning of This Work. In contrast to prior work, we focus explicitly on reliability and failure characterization rather than task feasibility. Rather than proposing a new model or architecture, we provide a structured empirical evaluation of an existing deployed system under controlled conditions such as under-constrained inference, and compositional reasoning pipelines. This enables us to identify not only whether agentic systems succeed, but how and why they fail, and crucially, whether those failures are visible in the final output or silent. III Agentic System Under Consideration For our evaluation, we utilize CMBAgent , an open-source agentic framework for autonomous scientific discovery [ 25 ] . CMBAgent serves as the research backend in the Denario project [ 23 ] , generating research ideas, methodologies, and results through coordinated multi-agent planning. The outputs—including analyses, visualizations, and literature references—are automatically compiled into publication-ready manuscripts. While the Denario project demonstrates the feasibility of end-to-end autonomous scientific workflows and evaluates overall task success through expert assessment, a systematic investigation of reliability, robustness, error propagation, and failure modes remains limited. Such analysis is essential for assessing the suitability of agentic systems in scientific deployment settings where reproducibility and numerical correctness are critical. Here we complement existing feasibility studies by conducting a structured empirical evaluation focused on robustness, calibration, and failure characterization, thereby contributing toward a more rigorous evaluation framework for autonomous scientific agents. The framework provides multiple operational modes and workflows, which differ primarily in the number of active agents, session length, context persistence across tasks, and the extent of human intervention required for conducting scientific analyses. These are as follows: 1. One Shot. Executes a task in a single reasoning and execution pass without iterative planning or review. We instantiate two variants: one with a context-aware domain agent ( cambagent )