The Format Tax
Ivan Yee Lee, Loris D’Antoni, Taylor Berg-Kirkpatrick · 2026
Read The Format Tax by Ivan Yee Lee, Loris D’Antoni, Taylor Berg-Kirkpatrick (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-03616.txt.
Introduction
Ivan Lee, Loris D'Antoni, and Taylor Berg-Kirkpatrick found that asking a language model to answer in JSON, XML, LaTeX, or Markdown often hurts its reasoning and writing. Most of the damage came from the format request itself, not from the decoding limits that force the shape. Letting the model think freely first and reformat afterward recovered most of the lost accuracy across six open models and four task types.
Catalog record
| CREATOR | Ivan Yee Lee, Loris D’Antoni, Taylor Berg-Kirkpatrick |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 60407 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.03616
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
The Format Tax Ivan Yee Lee, Loris D’Antoni, Taylor Berg-Kirkpatrick Abstract Asking a large language model to respond in JSON should be a formatting choice, not a capability tax. Yet we find that structured output requirements—JSON, XML, LaTeX, Markdown—substantially degrade reasoning and writing performance across open-weight models. The research response has focused on constrained decoding, but sampling bias accounts for only a fraction of the degradation. The dominant cost enters at the prompt: format-requesting instructions alone cause most of the accuracy loss, before any decoder constraint is applied. This diagnosis points to a simple principle: decouple reasoning from formatting. Whether by generating freeform first and reformatting in a second pass, or by enabling extended thinking within a single generation, separating the two concerns substantially recovers lost accuracy. Across six open-weight models, four API models, four formats, and tasks spanning math, science, logic, and writing, decoupling recovers most lost accuracy. Notably, most recent closed-weight models show little to no format tax, suggesting the problem is not inherent to structured generation but a gap that current open-weight models have yet to close. Code is available at https://github.com/ivnle/the-format-tax . 1 Introduction Structured output has become the default interface between language models and software systems. Function calling, tool use, and agentic frameworks all route model generations through JSON schemas or equivalent format constraints. Grammar-constrained decoding guarantees this by masking invalid tokens during generation, ensuring every output parses. The implicit contract: syntax enforcement should be invisible to reasoning. The model thinks the same thoughts; they simply emerge in a valid format. We show this contract is broken. Requiring structured output degrades reasoning and writing performance across open-weight models—though recent closed-weight models appear to have closed this gap. The degradation we observe is not new. Tam et al. 2024 documented accuracy drops when models generate structured output, and practitioners have long suspected that format requirements carry hidden costs. What remains unclear is why . The research response has targeted the decoder. Grammar-constrained decoding restricts sampling to tokens that satisfy the grammar, but this restriction distorts the model’s distribution—biasing it away from what the model would produce unconstrained. Grammar-aligned decoding methods ( Park et al. 2024 ) attempt to correct this. This line of work is principled—but it rests on an assumption: that the decoder is the primary source of degradation. What if sampling bias accounts for only a fraction of the loss? To answer this, we distinguish two sources of degradation by when they occur. The format-requesting prompt introduces one cost before generation begins: the model must now produce reasoning and valid structure in the same output. Constrained decoding introduces a second cost during generation by masking tokens. If the prompt-level cost dominates, no amount of decoder engineering can close the gap. We test this across MATH-500 (competition math), GPQA (graduate science), ZebraLogic (constraint satisfaction), and WritingBench (open-ended writing)—four tasks, four output formats, and ten models spanning open-weight and commercial APIs. The results confirm that the prompt-level cost dominates. Across most open-weight model–task–format combinations, structured output reduces accuracy compared to freeform generation, whether or not constrained decoding is applied. Standard prompt-level mitigations—few-shot examples and explicit schema descriptions—leave the majority of the degradation intact. What does work is decoupling reasoning from formatting. Letting the model reason freely before producing structured output—either via a second reformatting pass or extended thinking—recovers most of the lost accuracy. Recent closed-weight models show near-zero degradation, suggesting that the tax is not inherent to structured generation. The format tax, it turns out, is paid at the prompt—not the decoder. In summary, we (1) systematically measure the format tax across ten models, four formats, and tasks spanning reasoning and writing; (2) decompose it into prompt-level degradation (dominant) and decoder-level sampling bias (minor); (3) show that prompt-level mitigations—schema descriptions, few-shot examples—do not close the gap; (4) demonstrate that decoupling reasoning from formatting, via either two-pass generation or extended thinking, recovers most lost accuracy; and (5) show that recent closed-weight models resist the format tax, suggesting the problem is solvable. 2 Background A formal grammar \mathcal{G}=(V,\Sigma,R,S) consists of non-terminal symbols V , terminal symbols \Sigma , production rules R , and a start symbol S . It defines a language \mathcal{L}(\mathcal{G}) : the set of all strings derivable from S by applying rules in R . Any string in \mathcal{L}(\mathcal{G}) is well-formed by construction. Grammar-constrained decoding (GCD) masks tokens that would violate a target grammar, guaranteeing syntactically valid output ( Geng et al. 2024 ; Willard & Louf 2023 ; Park et al. 2025 ) . At each step, the model’s distribution is restricted to valid continuations: p_{\text{GCD}}(x_{t}\mid x_{<t},\mathcal{G})\propto p(x_{t}\mid x_{<t})\cdot\mathbf{1}[x_{t}\text{ valid under }\mathcal{G}] (1) This guarantee has driven rapid adoption: APIs must return parseable JSON, function-calling agents need well-formed arguments, and downstream systems cannot recover from malformed output. Major frameworks now offer built-in support, including OpenAI’s structured outputs, vLLM, Outlines, and Guidance. Grammar-constrained decoding operates myopically: at each step, it masks tokens that would immediately violate the grammar, without considering whether the remaining valid tokens lead to feasible completions. Grammar-aligned decoding (GAD; Park et al. 2024 ) methods address this by targeting the true intersection of the language model and the grammar: p_{\text{GAD}}(x)\propto p(x)\cdot\mathbf{1}[x\in\mathcal{L}(\mathcal{G})] (2) Sampling from this distribution exactly is intractable; practical methods approximate it via adaptive sampling ( Park et al. 2024 ) , MCMC ( Parys et al. 2025 ; Gonzalez et al. 2025 ) , or Sequential Monte Carlo ( Lipkin et al. 2025 ; Loula et al. 2025 ) . These methods share a common premise: that the myopia of token-level masking is the primary source of degradation, and that better approximations should recover lost reasoning quality. Rather than benchmark individual GAD implementations, we test this shared premise directly by measuring how much degradation is attributable to the decoder in the first place. Definition: Format Tax. Format constraints specify presentation (JSON, XML, etc.), not content—a pure transduction that should leave task performance unchanged. The format tax measures the gap between this expectation and reality: \text{Format Tax}=\text{Perf}_{\text{freeform}}-\text{Perf}_{\text{format-constrained}} A positive value indicates that enforcing format degrades performance. We operationalize performance as task accuracy for reasoning benchmarks and LLM-judged quality for writing tasks. 3 Experimental Setup Freeform ··T· x\sim p(x\mid\texttt{\textperiodcentered\textperiodcentered T}) 1-Turn 1-Turn + Think GET· x\sim p(x\mid\texttt{GET}) GET· r\sim p(r\mid\texttt{GET}) ; x\sim p(x\mid r,\texttt{GET}) G·T· x\sim p(x\mid\texttt{G\textperiodcentered T}) G·T· r\sim p(r\mid\texttt{G\textperiodcentered T}) ; x\sim p(x\mid r,\texttt{G\textperiodcentered T}) ·ET· x\sim p(x\mid\texttt{\textperiodcentered ET}) ·ET· r\sim p(r\mid\texttt{\textperiodcentered ET}) ; x\sim p(x\mid r,\texttt{\textperiodcentered ET}) GETC x\sim p_{\text{GCD}}(x\mid\texttt{GET}) GETC r\sim p(r\mid\texttt{GET}) ; x\sim p_{\text{GCD}}(x\mid r,\texttt{GET}) G·TC x\sim p_{\text{GCD}}(x\mid\texttt{G\textperiodcentered T}) G·TC r\sim p(r\mid\texttt{G\textperiodcentered T}) ; x\sim p_{\text{GCD}}(x\mid r,\texttt{G\textperiodcentered T}) ·ETC x\sim p_{\text{GCD}}(x\mid\texttt{\textperiodcentered ET}) ·ETC r\sim p(r\mid\texttt{\textperiodcentered ET}) ; x\sim p_{\text{GCD}}(x\mid r,\texttt{\textperiodcentered ET}) 2-Turn 2-Turn + Think GET· a\sim p(a\mid\texttt{\textperiodcentered\textperiodcentered T}) ; x\sim p(x\mid a,\texttt{GET}) GET· r,a\sim p(r,a\mid\texttt{\textperiodcentered\textperiodcentered T}) ; x\sim p(x\mid a,\texttt{GET}) G·T· a\sim p(a\mid\texttt{\textperiodcentered\textperiodcentered T}) ; x\sim p(x\mid a,\texttt{G\textperiodcentered T}) G·T· r,a\sim p(r,a\mid\texttt{\textperiodcentered\textperiodcentered T}) ; x\sim p(x\mid a,\texttt{G\textperiodcentered T}) ·ET· a\sim p(a\mid\texttt{\textperiodcentered\textperiodcentered T}) ; x\sim p(x\mid a,\texttt{\textperiodcentered ET}) ·ET· r,a\sim p(r,a\mid\texttt{\textperiodcentered\textperiodcentered T}) ; x\sim p(x\mid a,\texttt{\textperiodcentered ET}) GETC a\sim p(a\mid\texttt{\textperiodcentered\textperiodcentered T}) ; x\sim p_{\text{GCD}}(x\mid a,\texttt{GET}) GETC r,a\sim p(r,a\mid\texttt{\textperiodcentered\textperiodcentered T}) ; x\sim p_{\text{GCD}}(x\mid a,\texttt{GET}) G·TC a\sim p(a\mid\texttt{\textperiodcentered\textperiodcentered T}) ; x\sim p_{\text{GCD}}(x\mid a,\texttt{G\textperiodcentered T}) G·TC r,a\sim p(r,a\mid\texttt{\textperiodcentered\textperiodcentered T}) ; x\sim p_{\text{GCD}}(x\mid a,\texttt{G\textperiodcentered T}) ·ETC a\sim p(a\mid\texttt{\textperiodcentered\textperiodcentered T}) ; x\sim