AgenticSimLaw: A Juvenile Courtroom Multi-Agent Debate Simulation for Explainable High-Stakes Tabular Decision Making

Jon Chun, Kathrine Elkins, Yong Suk Lee · 2026

Read AgenticSimLaw: A Juvenile Courtroom Multi-Agent Debate Simulation for Explainable High-Stakes Tabular Decision Making by Jon Chun, Kathrine Elkins, Yong Suk Lee (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-21936.txt.

Introduction

Jon Chun, Kathrine Elkins, and Yong Suk Lee build a courtroom inside a computer to test high-stakes decisions. Their 2026 preprint introduces AgenticSimLaw, a multi-agent debate framework aimed at tabular decision making, where choices carry real consequences and must be explained.

Unlike black-box systems, the framework assigns clear roles: a prosecutor, a defense, and a judge, modeled on a juvenile courtroom. The agents follow a structured debate over several turns, and each uses private reasoning strategies. Because every step is explicit, the whole decision process can be audited afterward. Transparent test-time reasoning of this kind could help institutions check automated decisions before trusting them with people's lives and livelihoods.

Catalog record

CREATORJon Chun, Kathrine Elkins, Yong Suk Lee
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE56 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-21936

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

AgenticSimLaw: A Juvenile Courtroom Multi-Agent Debate Simulation for Explainable High-Stakes Tabular Decision Making Jon Chun, Kathrine Elkins, Yong Suk Lee Abstract We introduce AgenticSimLaw, a role-structured, multi-agent debate framework that provides transparent and controllable test-time reasoning for high-stakes tabular decision-making tasks. Unlike black-box approaches, our courtroom-style orchestration explicitly defines agent roles (prosecutor, defense, judge), interaction protocols (7-turn structured debate), and private reasoning strategies, creating a fully auditable decision-making process. We benchmark this framework on young adult recidivism prediction using the NLSY97 dataset, comparing it against traditional chain-of-thought (CoT) prompting across almost 90 unique combinations of models and strategies. Our results demonstrate that structured multi-agent debate provides more stable and generalizable performance compared to single-agent reasoning, with stronger correlation between accuracy and F1-score metrics. Beyond performance improvements, AgenticSimLaw offers fine-grained control over reasoning steps, generates complete interaction transcripts for explainability, and enables systematic profiling of agent behaviors. While we instantiate this framework in the criminal justice domain to stress-test reasoning under ethical complexity, the approach generalizes to any deliberative, high-stakes decision task requiring transparency and human oversight. This work addresses key LLM-based multi-agent system challenges: organization through structured roles, observability through logged interactions, and responsibility through explicit non-deployment constraints for sensitive domains. Data, results, and code will be available on github.com under the MIT license. Introduction The introduction of the Transformer architecture in 2017 launched an era of rapid progress in AI, initially driven by the scaling of compute and data ( 14 ) . This led to a saturation of performance gains as training costs soared to over $100M USD for state-of-the-art (SOTA) models ( 3 ) . To use compute more efficiently, new training methodologies like LoRA and architectural innovations like Mixture of Experts (MoE) emerged ( 10 ; 14 ) . As models became more capable, traditional benchmarks were ”solved,” and a new focus on complex reasoning tasks in fields such as mathematics, physics, and chemistry took hold ( 11 ) . This shift has led to greater emphasis on reasoning during inference, or ”test-time compute.” A wide range of prompt engineering techniques have been developed to guide models towards better reasoning. From simple ”let’s think step by step” chain of thought (CoT) to more complex methods like Tree of Thought (ToT), self-consistency (SC), and ReAct, these strategies have formalized multi-turn reasoning and become a frontier for performance gains ( 22 ) . However, as LLM-based multi-agent systems (LaMAS) have emerged as powerful tools for complex problem-solving, critical questions arise about their organization, observability, and responsible deployment in high-stakes domains. In parallel, LLMs are increasingly applied to tasks traditionally handled by specialized statistical models, such as analyzing structured tabular data ( 6 ) . However, LLMs struggle with these tasks because their attention mechanisms are optimized for linear text and struggle to capture complex, non-sequential dependencies in tabular data ( 21 ) . This paper addresses the challenge of making LLM-based multi-agent reasoning transparent, controllable, and auditable through a structured interaction framework, instantiated on a particularly demanding use case: predicting criminal recidivism from tabular data. We selected recidivism prediction for three reasons: (1) it provides a difficult tabular reasoning benchmark where unstructured data confounds LLM processing; (2) the real-world NLSY97 dataset contains intricate feature relationships and noise that challenge both traditional ML and linguistic approaches; and (3) most critically for LaMAS research, this high-stakes, ethically sensitive domain demands transparent, accountable, and human-supervised decision-making—precisely the responsible AI properties that structured multi-agent systems should provide. We emphasize that our framework is designed for research, benchmarking, and transparency analysis, not for operational deployment in sentencing or parole decisions. The courtroom structure serves as an evaluation harness that can generalize to other deliberative contexts (medical diagnosis, credit assessment, policy analysis) where procedural transparency and role accountability are valued. We make the following contributions: • A reproducible multi-agent evaluation harness for high-stakes tabular prediction tasks, demonstrated on NLSY97 recidivism data, that provides explicit control over agent organization, interaction protocols, and reasoning transparency. • AgenticSimLaw, a courtroom-style multi-agent debate (MAD) framework with structured roles (prosecutor, defense, judge), a 7-turn interaction protocol, private strategy formulation, and complete logging of all utterances and belief updates for auditability. • A comprehensive benchmark comparing linguistic reasoning approaches—traditional statistical methods, single-agent CoT prompting, and structured multi-agent debate—across almost 90 unique combinations of open-source and commercial LLMs. • Empirical demonstration that structured multi-agent debate improves reasoning stability and generalizability (measured by stronger accuracy-F1 correlation) compared to traditional single-agent prompting, while providing richer audit trails and fine-grained control over test-time reasoning. • Analysis of the compute-transparency tradeoff in multi-agent systems, showing that AgenticSimLaw’s 9,100 token per-run cost delivers enhanced observability and metric stability that single-shot prompting cannot provide. Background and Related Work High-Stakes AI Decision Systems Artificial Intelligence has augmented human decision-making in numerous high-stakes environments, most notably in the US judicial system. The Correctional Offender Management Profiling for Alternative Sanctions (COMPAS) is a well-known proprietary algorithm used to make parole recommendations. A prominent critique by ProPublica highlighted significant racial disparities in its predictions, finding that ”blacks are almost twice as likely as whites to be labeled a higher risk but not actually reoffend” ( 1 ) . Subsequent research found that COMPAS was no more accurate or fair than a simple linear model using only two features, and its performance was comparable to that of laypeople ( 5 ) . Recent research shows COMPAS simultaneously reduced overall confinement while exacerbating racial disparities ( 28 ) , and that LLMs exhibit amplified cognitive biases—particularly omission bias—that could systematically distort legal decisions ( 26 ) . These findings underscore the critical need for transparency, fairness, and auditability in AI-driven decision systems—properties that LaMAS research aims to provide through structured agent interactions. Our dataset derives from the National Longitudinal Survey of Youth 1997 (NLSY97), a nationally representative survey of individuals born between 1980 and 1984. This longitudinal dataset provides rich, multi-dimensional data on demographic, social, and economic factors, allowing for more nuanced analysis than datasets based on static justice-related variables like those used by COMPAS. The NLSY97 data is rigorously anonymized to protect respondent confidentiality in compliance with Federal law and the Office of Management and Budget (OMB). We frame recidivism prediction as binary classification: predicting whether a young adult offender will be rearrested within three years based on tabular features. Traditional and Specialized Tabular Approaches Historically, traditional statistical machine learning models like XGBoost have been SOTA for tabular prediction tasks. More recently, specialized Tabular LLMs like TabPFN have shown promise in small data regimes, offering potential for accurate statistical analysis through natural language interfaces ( 16 ) . Our work evaluates LLM-based multi-agent reasoning against these traditional methods to establish a comprehensive benchmark that reveals where structured linguistic debate offers advantages beyond pure statistical modeling. Multi-Agent Debate and LLM Orchestration A new approach to improving reasoning is through Multi-Agent Debate (MAD), where multiple agents engage in argumentative processes to arrive at conclusions. Building on work showing significant performance gains by incentivizing extensive test-time reasoning ( 19 ; 4 ) , MAD systems have proven effective in improving knowledge representation and performance on various benchmarks ( 7 ; 27 ) . The dominant test-time compute approach uses single large models with internal chain-of-thought

Read the complete text