SLOT : Structuring the Output of Large Language Models

Darren Yow-Bang Wang, Zhengyuan Shen, Soumya Smruti Mishra · 2025

Read SLOT : Structuring the Output of Large Language Models by Darren Yow-Bang Wang, Zhengyuan Shen, Soumya Smruti Mishra (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-04016.txt.

Introduction

A team of researchers presents a post-processing layer that forces language model output into set formats in a paper from 2025. Structured output matters for agents and data extraction, yet models often drift off schema. Their system, SLOT, uses a small fine-tuned model to rewrite raw output into the required shape. It works with any large model and any schema. The catalog names three authors while the head byline lists six, so team framing is used here.

A tuned mid-size model hit near-perfect schema accuracy and high content fidelity. It beat a leading commercial model by wide margins. Even a one-billion-parameter model matched big proprietary systems once equipped. The authors target reliable generation on modest hardware.

Catalog record

CREATORDarren Yow-Bang Wang, Zhengyuan Shen, Soumya Smruti Mishra
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE58 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-04016

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SLOT : Structuring the Output of Large Language Models Darren Yow-Bang Wang, Zhengyuan Shen, Soumya Smruti Mishra Abstract Structured outputs are essential for large language models (LLMs) in critical applications like agents and information extraction. Despite their capabilities, LLMs often generate outputs that deviate from predefined schemas, significantly hampering reliable application development. We present SLOT ( S tructured L LM O utput T ransformer), a model-agnostic approach that transforms unstructured LLM outputs into precise structured formats. While existing solutions predominantly rely on constrained decoding techniques or are tightly coupled with specific models, SLOT employs a fine-tuned lightweight language model as a post-processing layer, achieving flexibility across various LLMs and schema specifications. We introduce a systematic pipeline for data curation and synthesis alongside a formal evaluation methodology that quantifies both schema accuracy and content fidelity. Our results demonstrate that fine-tuned Mistral-7B model with constrained decoding achieves near-perfect schema accuracy (99.5%) and content similarity (94.0%), outperforming Claude-3.5-Sonnet by substantial margins (+25 and +20 percentage points, respectively). Notably, even compact models like Llama-3.2-1B can match or exceed the structured output capabilities of much larger proprietary models when equipped with SLOT , enabling reliable structured generation in resource-constrained environments. 1 Introduction The emergence of large language models (LLMs) has led to a wide range of applications that leverage their natural language understanding capabilities. In such applications, developers are often required to carefully craft prompts to elicit specific and reliable responses from LLMs, followed by post-processing of the generated outputs to derive structured and precise results. This process becomes particularly critical when LLMs are integrated as components of within more complex systems, such as for function calling or multi-agent collaboration, where inaccuracies in earlier stages can propagate through the workflow, adversely affecting overall system performance. An emerging requirement for LLM-based applications is to support structured output. Although certain proprietary models ( Hurst et al., 2024 , e.g., GPT-4o) inherently support structured output through specialized training and constrained decoding (CD) mechanisms, replicating this capability for other LLMs presents significant challenges. In particular, for platforms that serve a diverse set LLMs, including open-weight and proprietary models, post-training each model for structured output without undermining general purpose performance is neither feasible nor scalable. This challenge is further exacerbated by the continual introduction of new models, making per-model adaptation impractical in the dynamic deployment environment. Figure 1: SLOT converts a textual LLM response into structured JSON with a pre-defined schema. To tackle these challenges, we introduce SLOT for converting unstructured text into structured output. Different from existing approaches that relies on LLM post-training and constrained decoding, SLOT post-processes LLM’s unstructured output by leveraging a lightweight language model, therefore being task and model-agnostic. Specifically, we fine-tune a lightweight language model to map unstructured text to the target schema, without modifying the underlying LLM. SLOT ensures broad compatibility with both current and emerging models, regardless of their task specialization or output constraints, effectively bridging the gap between the generative flexibility of LLMs and the rigorous structural requirement for downstream integration in modern software systems. Our main contributions are as follows: • We introduce SLOT for structured output conversion applicable to any textual LLM outputs. • We develop a synthetic data pipeline for diverse training data creation. • We present evaluation metrics for SLOT covering schema accuracy and content similarity. • We demonstrate lightweight models including Llama-3.2 (1B/3B) and Mistral-7B outperform existing solutions across evaluation dimensions through supervised fine-tuning. 2 Background and Problem Formulation Structured Output and JSON Schema. In the context of LLM, the term structured output means that model-generated content conforms to a pre-defined, machine readable format rather than free-form natural language Dong et al. (2024) ; Jiang et al. (2023b) ; OpenAI (2024) . Generating structured data from unstructured inputs enables the ability of LLM-based applications to answer questions via function calling, extract structured data and build multi-step agent workflows that allow LLMs to take actions. One of the most widely-used structure format is JSON Schema , which is a declarative language to define structure and constraints for JSON data. JSON schema maintains a consistent pattern, making it easier to ensure data validity and exchange structured data between applications. In this paper, we focus specifically on JSON schema due to its wide adoption. Existing Approaches. To enable the model generation to conform to a target JSON schema, existing works mainly opted for three approaches, which have their strengths and weaknesses. Direct prompting, although straightforward, often yields lower performance Tam et al. (2024) . Constrained decoding methods, exemplified by works like Dong et al. (2024) ; OpenAI (2024) , does not rely on dedicated post-training, but only supports limited types of JSON schemas and may underperform training-based methods. Post-training generally improves performance compared to constrained decoding, but also requires access to model weights. The training itself adds complexity, and is not scalable for adaptation to new LLMs. 1 1 1 We should note that training-based methods can be combined with constrained decoding methods, which leads to improved performance as we observed in experiments ( § 6 ). Problem Formulation. We focus on the post-processing setting ( Fig. 1 ), where we assume access to an LLM’s unstructured output. We define our task as a text-to-structure problem, i.e., to transform an existing LLM’s free-form text output into a structured format according to a specified JSON schema . A more detailed comparison between SLOT and existing mechanisms is presented in Appx ( Fig. 4 ). Given x as an input text and f as the formatting specification, let M_{\theta} be a generative model parameterized by \theta : M_{\theta}(x,f)\rightarrow y^{\prime} , and we seek to optimize the probability distribution P\big(y|x,f;\theta) where y is the groundtruth structured output and y^{\prime} the model’s structured output. Note that the input text x typically represents a response from an LLM rather than a user query. Our post-processing based method is flexible and does not require access to LLM model weights, making it suitable for the case of serving a diverse array of LLMs in a platform. 3 Evaluation Framework The evaluation of structured output is inherently multi-faceted, as structure-wise, the output needs to adhere to the target JSON schema; while also not derailing from the unstructured input’s semantic meaning. Therefore, we design our evaluation framework with the focus on (1) schema accuracy and (2) content similarity . Schema Accuracy. We define schema accuracy as A_{s}(y^{\prime}\mid f) : whether the response JSON string y^{\prime} exactly matches the user-demanded schema f in terms of key strings and value types. To accurately assess the LLM’s formatting capability, we directly evaluate response y^{\prime} . The response y^{\prime} must be a valid JSON string by itself to be considered correct. Content Similarity. The desired structured output should not derail from the unstructured input in terms of semantic meaning. Assume that the groundtruth JSON structured output y is provided, we can compute the content similarity between groundtruth output y and model’s output y^{\prime} as \textit{sim}_{C}(y,y^{\prime}) . In practice, we evaluate the semantic similarity between y and y^{\prime} . For each value in the prediction y^{\prime} , we compute its semantic similarity using a pre-trained Sentence-BERT model Reimers and Gurevych (2019) ; Zhang et al. (2020) against the corresponding value (with matching key; including

Read the complete text