Benchmarking System Dynamics AI Assistants: Cloud Versus Local Large Language Models on CLD Extraction and Discussion

Terry Leitch · 2026

Read Benchmarking System Dynamics AI Assistants: Cloud Versus Local Large Language Models on CLD Extraction and Discussion by Terry Leitch (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-18566.txt.

Introduction

Leitch evaluates model families, both cloud APIs and local open-source models, on two benchmarks for System Dynamics AI assistance: causal loop diagram extraction and model discussion. On extraction, cloud models achieve 77 to 89 percent pass rates, while the best local model reaches 77 percent. On discussion, the best local models reach 50 to 100 percent on model building, but only 0 to 50 percent on error fixing, where long prompts expose memory limits. The central analysis compares reasoning versus instruction-tuned architectures, backends, and quantization levels, finding backend choice matters more than quantization.

Catalog record

CREATORTerry Leitch
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE63168
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.18566

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Benchmarking System Dynamics AI Assistants: Cloud Versus Local Large Language Models on CLD Extraction and Discussion Terry Leitch Abstract We present a systematic evaluation of large language model families—spanning both proprietary cloud APIs and locally-hosted open-source models—on two purpose-built benchmarks for System Dynamics AI assistance: the CLD Leaderboard (53 tests, structured causal loop diagram extraction) and the Discussion Leaderboard (interactive model discussion, feedback explanation, and model building coaching). On CLD extraction, cloud models achieve 77–89% overall pass rates; the best local model reaches 77% (Kimi K2.5 GGUF Q3, zero-shot engine), matching mid-tier cloud performance. On Discussion, the best local models achieve 50–100% on model building steps and 47–75% on feedback explanation, but only 0–50% on error fixing—a category dominated by long-context prompts that expose memory limits in local deployments. A central contribution of this paper is a systematic analysis of model type effects on performance: we compare reasoning vs. instruction-tuned architectures, GGUF (llama.cpp) vs. MLX (mlx_lm) backends, and quantization levels (Q3 / Q4_K_M / MLX-3bit / MLX-4bit / MLX-6bit) across the same underlying model families. We find that backend choice has larger practical impact than quantization level: mlx_lm does not enforce JSON schema constraints, requiring explicit prompt-level JSON instructions, while llama.cpp grammar-constrained sampling handles JSON reliably but causes indefinite generation on long-context prompts for dense models. We document the full parameter sweep ( t t , p p , k k ) for all local models, cleaned timing data (stuck requests excluded), and a practitioner guide for running 671B–123B parameter models on Apple Silicon. 1 Introduction Causal loop diagrams (CLDs) are a foundational tool of the System Dynamics methodology, encoding feedback structure, polarity, and causal pathways among model variables [ 1 ] . Automating CLD extraction from natural-language text—and enabling AI-assisted model discussion and critique—could substantially accelerate System Dynamics model construction and the teaching of the method. Large language models (LLMs) are natural candidates: they must parse causal language, identify polarities, track loop semantics, and produce structured graph output. Yet structured extraction under strict schema constraints is among the more demanding LLM tasks [ 2 ] , and iterative model refinement adds further complexity. The Discussion task additionally requires models to reason about model dynamics, identify feedback-driven behaviour, and diagnose formulation errors—tasks that require genuine domain understanding rather than pattern-matching. This paper evaluates a broad set of LLMs across both task families and contributes: 1. Ranked leaderboards for CLD extraction and Discussion tasks, comparing cloud and local models. 2. A systematic analysis of model type effects : reasoning vs. instruction-tuned architectures, GGUF vs. MLX backends, and quantization levels. 3. A full parameter sweep ( t , p , k ) for all local models, with analysis of optimal configurations per model class. 4. Cleaned timing analysis (stuck requests excluded) by task category for the top local models. 5. A practitioner guide documenting deployment challenges—JSON enforcement, context limits, template bugs, and crash recovery— encountered when running frontier open-source models on Apple Silicon. 2 Benchmarks 2.1 CLD Leaderboard (53 tests) Each test presents a text passage and asks the model to extract a structured CLD in JSON format, encoding variables, directed links, polarities, and optionally loop labels. Tests are evaluated by exact structured match against a ground-truth schema: a response is scored pass if and only if the extracted JSON matches the ground-truth on all required fields (variable names, edge directions, polarities, and any specified cardinality constraints), after normalisation of variable name casing and whitespace. Reasoning traces or chain-of-thought content prior to the final JSON are ignored in scoring; only the final structured output is evaluated. Full reproduction instructions are given in Section 14 . Model profiles and run configurations are archived at https://github.com/tleitch/sd-ai ( evals/model-profiles/ , evals/run-configs/ ); the upstream inference framework is at https://github.com/UB-IAD/sd-ai . Conformance (18 tests). Schema compliance: cardinality constraints, required vs. optional fields, correct polarity encoding. Models must not hallucinate extra variables or omit required links. Qualitative Causal Reasoning (3 tests). Second-order reasoning: identify mediated causation, distinguish direct from feedback-driven effects, handle ambiguous causal language. Iterative Model Building (8 tests). The model receives an existing CLD plus a passage describing additions, and must return the updated CLD with all pre-existing relationships preserved exactly. Qualitative Translation (24 tests). Extract a full CLD from natural-language system descriptions spanning ecology, economics, public health, and engineering domains. 2.2 Discussion Leaderboard The Discussion benchmark evaluates models acting as a System Dynamics mentor/critic via the Seldon engine. Tests are drawn from three categories: Model Building Steps. Given a modelling question, provide structured coaching toward correct model construction steps. Tests range from simple (single-step guidance) to medium (multi-step modelling sequences). Feedback Explanation. Explain the feedback dynamics driving model behaviour. Simple tests involve single-loop explanations; medium tests require multi-loop causal tracing. Error Fixing Suggestions. Identify and explain formulation errors in a provided model. These tests carry the longest prompts (80–146k tokens including full model context) and are the most demanding in terms of context window requirements. 2.3 Prompt Engines For CLD extraction, tests run under two prompt strategies, referred to consistently throughout this paper as: • few-shot ( qualitative engine): chain-of-thought system prompt with structured extraction examples included in context. • zero-shot ( qualitative-zero engine): task instructions only, no in-context examples. Both engine names appear in result table footnotes where space requires abbreviation ( qual. / qual-zero ). For Discussion, all tests run under the Seldon mentor engine. 2.4 Experimental Setup All runs use seed 4242. Local models are served via LM Studio (v1.x) or mlx_lm.server on an Apple Mac Studio (M3 Ultra, 512 GB unified memory, 2025) and queried through an OpenAI-compatible REST API [ 9 ] . Cloud APIs are accessed via their respective SDKs. Concurrency is set to 1 and tests run sequentially to avoid memory contention. 1 1 1 The 512 GB M3 Ultra configuration was available as a custom-order option and is no longer listed in Apple’s current Mac Studio lineup. Specifications are documented in [ 9 ] . 3 Models Evaluated 3.1 Cloud API Models Table 1 : Cloud API models evaluated. Model Family Type Gemini 2.5 Flash Google Gemini Instruction-tuned Gemini 2.5 Pro Google Gemini Instruction-tuned Gemini 3 Pro Preview Google Gemini Instruction-tuned Gemini 3.1 Pro Preview Google Gemini Instruction-tuned Gemini 3 Flash Preview Google Gemini Instruction-tuned GPT-5.1 OpenAI Instruction-tuned GPT-5.2 OpenAI Instruction-tuned Claude Sonnet 4.5 Anthropic Instruction-tuned Claude Opus 4.5 Anthropic Instruction-tuned o4-mini OpenAI Reasoning 3.2 Local Open-Source Models Table 2 : Local models evaluated. Backend: L=llama.cpp, M=mlx_lm. Type: R=Reasoning, I=Instruction-tuned. Model Quant Be. Ty. Params Ctx Storage Kimi K2.5 GGUF Q3 L R 671B 16–197k \sim 250 GB Kimi K2.5 MLX-3bit M R 671B 164k \sim 200 GB DeepSeek V3.2 Q4_K_M L I \sim 671B 64–164k \sim 400 GB DeepSeek V3.2 MLX-4bit M I \sim 671B 164k \sim 320 GB Qwen 3.5 397B Q4_K_M L I 397B 64k \sim 230 GB Qwen 3.5 397B MLX-6bit M I 397B 64k 301 GB GLM-5 MLX-4bit M R \sim 9B 64–197k \sim 6 GB Llama 4 Maverick Q4_K_S L I 17B \times 128E 64k \sim 100 GB Mistral Large 2411 Q6_K L I 123B 64k \sim 92 GB DeepSeek R1-0528 IQ4NL L R \sim 671B 33k \sim 360 GB Mistral Large and DeepSeek R1 were excluded from the main

Read the complete text