Audit Trails for Accountability in Large Language Models CCS: Human-centered computing Empirical studies in HCI CCS: Human-centered computing HCI theory, concepts and models
Victor Ojewale, Harini Suresh, Suresh Venkatasubramanian · 2026
Read Audit Trails for Accountability in Large Language Models CCS: Human-centered computing Empirical studies in HCI CCS: Human-centered computing HCI theory, concepts and models by Victor Ojewale, Harini Suresh, Suresh Venkatasubramanian (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-20727.txt.
Introduction
Victor Ojewale, Harini Suresh, and Suresh Venkatasubramanian ask how large language models can be held to account. Their 2026 paper notes that these models now sit inside big decisions. They help shape choices in health care, finance, jobs, and public services. Yet when a choice goes wrong, there is often no clear record of how it was made.
The authors propose LLM audit trails as a fix. An audit trail is a durable record of the process that others can review later. They frame it as a social and technical mechanism, not just a log file. It is meant to support continuous accountability over time. The paper gives designers and regulators a concrete way to demand a paper trail from AI systems.
Catalog record
| CREATOR | Victor Ojewale, Harini Suresh, Suresh Venkatasubramanian |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 50 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-20727
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Audit Trails for Accountability in Large Language Models CCS: Human-centered computing Empirical studies in HCI CCS: Human-centered computing HCI theory, concepts and models Victor Ojewale, Harini Suresh, Suresh Venkatasubramanian Abstract . Large language models (LLMs) are increasingly embedded in consequential decisions across healthcare, finance, employment, and public services. Yet accountability remains fragile because process transparency is rarely recorded in a durable and reviewable form. We propose LLM audit trails as a sociotechnical mechanism for continuous accountability. An audit trail is a chronological, tamper-evident, context-rich ledger of lifecycle events and decisions that links technical provenance (models, data, training and evaluation runs, deployments, monitoring) with governance records (approvals, waivers, and attestations), so organizations can reconstruct what changed, when, and who authorized it. This paper contributes: (1) a lifecycle framework that specifies event types, required metadata, and governance rationales; (2) a reference architecture with lightweight emitters, append only audit stores, and an auditor interface supporting cross organizational traceability; and (3) a reusable, open-source Python implementation that instantiates this audit layer in LLM workflows with minimal integration effort. We conclude by discussing limitations and directions for adoption. 1. Introduction As Large language models have moved from research prototypes to infrastructure that shapes decisions in finance, healthcare, law, and public services ( 31 ; 26 ; 25 ) it has become increasingly crucial that we have accountability and transparency mechanisms. When an error or harm occurs, organizations often cannot reconstruct which version was live, which data influenced it, who approved its release, or why particular changes were made. External audits and investigative reporting continue to surface harms ( 10 ; 16 ; 32 ) , yet the internal records needed to trace decisions are often fragmented across tools, stored in ad hoc ways, or never captured in the first place. The result is a persistent accountability gap. Internal auditing frameworks argue for end to end oversight that is built into everyday practice ( 36 ) and traceability has been proposed as a way to make accountability concrete in computing systems ( 24 ) . Documentation artifacts such as model cards, datasheets, and FactSheets improve disclosure about models and datasets ( 29 ; 19 ; 4 ) , but they remain snapshots at a point in time. MLOps practices improve technical provenance ( 23 ; 43 ) yet often omit governance details such as who approved a change, under what conditions, and for what reasons. At the same time, AI development increasingly resembles a supply chain with many hands and limited visibility across organizational boundaries ( 41 ; 45 ; 13 ; 9 ; 21 ) , which makes accountability hard to assign without a shared, time-stamped record of changes and approvals across the development lifecycle. We address this gap by proposing LLM audit trails as a way for organizations to embed accountability into the development and operation of LLM based systems. An audit trail is a chronological, tamper evident, and context rich ledger of events and decisions across selection, data use, fine tuning, evaluation, deployment, monitoring, and incident response. Rather than customizing logging for each pipeline, we conceptualize audit trails as a thin, system agnostic layer between heterogeneous LLM workflows and governance regimes. Audit trails turn transparency by design from principle into operational practice ( 17 ) and align with regulatory record keeping ( 33 , Article 12) and risk management guidance ( 40 ) . Our contributions are: (1) a lifecycle framework that specifies what to log and why across these stages, including the governance rationale for each stage; (2) a reference architecture for capturing, storing, and using logs that emphasizes tamper evidence, privacy, and cross organizational linkage; (3) a lightweight Python library that instruments common LLM development workflows to show that such trails are feasible with modest overhead. We motivate the need for audit trails through concrete scenarios, situate our proposal within work on accountability and documentation, present the framework and system design, demonstrate a proof of concept implementation, and revisit the scenarios to illustrate how audit trails can support investigation and governance in practice. 2. Why do we need audit trails? This section illustrates why audit trails are needed in practice by walking through two LLM deployment scenarios in finance, and healthcare. For each case we describe how the system is used in context, what the development and deployment choices look like in practice, what goes wrong, and what kinds of records investigators would need in order to understand and govern the system. Financial advice chatbot incident. Retail banks are actively piloting generative AI for customer-facing assistance and internal analyst workflows, with early deployments focusing on chat-based support, document summarization, and other productivity tasks ( 14 ; 22 ) Scenario. A retail bank’s digital channels team deploys BankGPT, an LLM based assistant to answer mortgage questions for retail customers, while human loan officers retain formal authority over lending decisions. The assistant is positioned as a first line tool that can explain products, estimate affordability, and route customers to appropriate staff. Operationalization. The bank’s ML team selects a proprietary base LLM from a provider and fine tunes it on historical customer service transcripts for mortgage support and bank mortgage policies, after a model risk committee approves the use of that base model and the fine tuning dataset. Engineers configure prompt templates, set the generation settings that control how long and how varied the assistant’s answers are, connect the model to internal product catalogs so it can look up current rates and eligibility rules, and define guardrail policies to block clearly inappropriate content. They then deploy the fine tuned model behind a feature flag to a subset of web and mobile users, with updates rolled out through the bank’s continuous integration and continuous deployment (CI/CD) pipeline. Key event. Months later, a branch loan officer escalates a complaint from a customer who followed the assistant’s guidance and was later told by underwriting that the recommended mortgage product was unsuitable given their income and risk profile. The bank’s risk and compliance teams open an internal review to understand whether the LLM behaved within its intended scope and whether deployment controls were followed. Governance needs. To investigate, the bank needs records that show which fine tuned model version was serving in the relevant channel at the time of the interaction, which prompt templates and decoding settings were active, and how rollout controls such as feature flags were configured. They also need to see what approvals governed that deployment, whether any emergency hotfixes were applied, and how those changes were authorized. In many current LLM deployments, such information is scattered across experiment trackers, CI logs, configuration files, and email threads, making it difficult to reconstruct a coherent timeline for internal review or external scrutiny. Clinical documentation and follow up gaps. Clinical documentation remains a major source of administrative burden, with time-motion and log-based studies showing substantial clinician time devoted to EHR work and after-hours documentation ( 39 ; 3 ) . In response, vendors and health systems are increasingly experimenting with “ambient” or note-drafting assistants that generate visit summaries and follow-up plans ( 38 ) . Scenario. An integrated health system deploys NoteAssist, an LLM based assistant that drafts visit summaries and suggested follow up plans for primary