A Benchmark on LLM-Based Power Flow Computation: Do More Structured Prompts Help?

Tingwei Chen, Kaiyang Huang, Student Member, IEEE , and Kai Sun, Fellow, IEEE · 2026

Read A Benchmark on LLM-Based Power Flow Computation: Do More Structured Prompts Help? by Tingwei Chen, Kaiyang Huang, Student Member, IEEE , and Kai Sun, Fellow, IEEE (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-18642.txt.

Introduction

Tingwei Chen and two co-authors benchmark, in a 2026 paper, three language models on Gauss-Seidel AC power flow computation for a three-bus system across four prompt formats. Gemini 2 point 5 Pro with the simplest narrative prompt achieves the lowest error, zero point two five seven MW/MVar mean absolute error. Its JSON-structured prompt raises error about threefold. GPT-3 point 5 Turbo fails on at least 90 percent of cases under all formats. A 100-case replication confirms the ordering, yet no configuration proves reliable enough as a direct numerical solver.

Catalog record

CREATORTingwei Chen, Kaiyang Huang, Student Member, IEEE , and Kai Sun, Fellow, IEEE
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE26991
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.18642

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A Benchmark on LLM-Based Power Flow Computation: Do More Structured Prompts Help? Tingwei Chen, Kaiyang Huang, Student Member, IEEE , and Kai Sun, Fellow, IEEE Abstract We present a controlled benchmark evaluating three LLMs—Claude Sonnet 4.5, Gemini 2.5 Pro, and GPT-3.5 Turbo—across four prompt formats (from concise narrative to structured JSON with explicit iteration trace) on Gauss–Seidel AC power flow computation for a three-bus system. Against 50 test cases with reference solutions computed numerically, Gemini 2.5 Pro with the simplest narrative prompt achieves the lowest mean absolute error (MAE = 0.257 MW/MVar, 54% of cases within 5% relative error), while the same model with a JSON-structured prompt raises MAE to 0.789—a 3.1 × \times increase. Adding a worked example degrades accuracy for Gemini but provides a marginal gain for Claude. GPT-3.5 Turbo fails on at least 90% of cases under all prompt formats. An independent 100-case replication with related prompt-format families confirms the qualitative ordering (Gemini ≫ \gg Claude > > GPT-3.5): the best 100-case configuration (Gemini with explicit iteration trace) achieves MAE = 0.402 and 53% within 5%, while Claude Sonnet 4.5’s near-flat accuracy profile ( ≈ \approx 38% within 5% across formats) and GPT-3.5’s near-total ineffectiveness (92–97% above 20% error) both replicate. In neither evaluation does any configuration achieve sufficient reliability for use as a direct numerical solver. These findings offer a diagnostic baseline for practitioners and researchers evaluating LLMs for smart-grid decision-support assistance. I Introduction Power flow computation is the computational foundation of grid planning, operation, and stability analysis. As smart-grid deployments grow increasingly complex, operators and engineers rely on decision-support tools that can perform or explain numerical computations rapidly. Large language models (LLMs) are attractive candidates for such copilot roles—from explaining solver output to assisting with engineering feasibility checks. The Gauss–Seidel method, introduced decades ago and still taught as the canonical iterative solver in power engineering curricula [ 8 ] , requires careful execution of complex arithmetic across multiple convergence iterations. As LLMs attract growing interest for power system tasks [ 7 ] , a natural question arises: can these models reliably execute a classical iterative numerical algorithm—and does the way we ask matter? Prior LLM-for-power-systems work has focused on high-level tasks such as fault diagnosis, maintenance Q&A, and security analysis [ 7 ] . These tasks rely on text understanding and knowledge retrieval rather than precise arithmetic. Separately, the prompt engineering literature has shown that chain-of-thought decomposition and few-shot examples can improve LLM performance on mathematical reasoning [ 9 , 1 ] , but these benchmarks typically involve closed-form problems. Gauss–Seidel AC power flow requires multi-pass convergence, complex number arithmetic, and careful tracking of intermediate voltage estimates across iterations—a qualitatively different challenge. No prior study has systematically varied prompt representation for a classical iterative AC power flow solver in a controlled setting. We close this gap with a benchmark that systematically varies prompt format while holding the algorithm, test network, and evaluation protocol constant, enabling a diagnostic comparison of four prompt-package designs. Our paper makes three contributions: 1. We design four prompt formats spanning narrative, example-augmented, procedural-text, and JSON-structured representations for LLM-based Gauss–Seidel power flow. 2. We show that Gemini 2.5 Pro’s overall MAE varies by a factor of 3.1 \times across prompt formats—with the concise narrative format yielding the lowest error and the most complex format (structured JSON input with required iteration trace output) performing worst. 3. We show that GPT-3.5 Turbo places \geq 90% of cases above 20% relative error regardless of prompt design, and that no configuration achieves reliable accuracy sufficient for direct use as a numerical solver. The best result in our benchmark—Gemini 2.5 Pro with a narrative prompt—achieves overall MAE = 0.257 MW/MVar, with 54% of cases within 5% relative error (Fig. 1 ; darker green = lower MAE, each cell reports the fraction of cases with relative error < 5%, bold border marks the best entry). The same model with a JSON-structured prompt yields MAE = 0.789. This result challenges a common assumption in prompt engineering—that providing more explicit numerical instructions always improves accuracy on arithmetic tasks. The remainder of this paper is organized as follows. Section II reviews related work. Section III defines the problem and experimental setup. Section IV presents results and analysis. Section V concludes. Fig. 1: MAE heatmap for 12 LLM–prompt configurations on 50 three-bus Gauss–Seidel power flow test cases. II Related Work II-A LLMs for power system tasks. The application of LLMs to power engineering is an active and recent research area. Ruan et al. [ 7 ] examine security implications of deploying LLMs in power system contexts and document how LLMs can be prompted to assist with grid-related tasks—but treat the LLM as a text interface, not a numerical solver. More broadly, proposals for LLM-assisted fault diagnosis, load forecasting explanation, and operator Q&A all rely on retrieval or classification rather than arithmetic computation. To the best of our knowledge, no prior work has benchmarked LLM performance on direct execution of a classical iterative power flow algorithm under controlled prompt variation. II-B Prompt engineering for mathematical reasoning. Chain-of-thought prompting [ 9 ] showed that asking LLMs to produce explicit intermediate reasoning steps substantially improves accuracy on arithmetic and commonsense benchmarks. The zero-shot variant [ 5 ] demonstrated that the instruction “Let’s think step by step” transfers across tasks without labeled examples. Few-shot prompting [ 1 ] established that providing in-context worked examples helps LLMs generalise to structurally similar problems. However, these benchmarks use problems with closed-form or single-step solutions. Gauss–Seidel iteration requires the model to maintain a convergence trajectory across multiple passes, updating complex voltage phasors at each step. Whether chain-of-thought-style procedural instructions—or worked examples—transfer to multi-pass iterative algorithms is the question our benchmark directly addresses. II-C Benchmarking LLMs on scientific and engineering tasks. Lewkowycz et al. [ 6 ] (Minerva) tested LLMs on quantitative science problems and found that larger models improved substantially on multi-step mathematical derivations. Chen et al. [ 2 ] evaluated LLMs on code generation (HumanEval), showing that models can write executable numerical code. More recently, program-aided approaches [ 4 , 3 ] offload arithmetic to an interpreter, achieving stronger numerical accuracy than direct text-based computation. These approaches contrast with our setup, in which LLMs must carry out multi-pass arithmetic without tool assistance—the failure mode our benchmark is designed to expose. No benchmark to date targets direct LLM execution of power flow iteration, making our three-model, four-format study the first controlled experiment of this kind. III Problem Formulation and Experimental Setup III-A Three-Bus Power System and Reference Solver We use a three-bus system with one slack bus (bus 1) and two PQ load buses (buses 2 and 3), fully connected by three transmission lines. Bus 1 is held at V_{1}=1.05\angle 0^{\circ} pu on a 100-MVA base. All shunt susceptances are neglected. Across all 50 test cases, the resistance and reactance of each branch are drawn independently and uniformly: R\in[0.010,\,0.015] pu and X\in[0.050,\,0.070] pu (per-unit, 100-MVA base), as summarised in Table I . TABLE I: Branch parameter sampling ranges for the three-bus benchmark system (per-unit, 100-MVA base). Each branch is sampled independently and uniformly within the stated interval. Line (from–to) R (pu) X (pu) Bus 1–Bus 2 [0.010,\ 0.015] [0.050,\ 0.070]

Read the complete text