Breaking the Autoregressive Chain: Hyper-Parallel Decoding for Efficient LLM-Based Attribute Value Extraction
Theodore Glavas, Nikhita Vedula, Dushyanta Dhyani, Yilun Zhu, Shervin Malmasi · 2026
Read Breaking the Autoregressive Chain: Hyper-Parallel Decoding for Efficient LLM-Based Attribute Value Extraction by Theodore Glavas, Nikhita Vedula, Dushyanta Dhyani, Yilun Zhu, Shervin Malmasi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-26209.txt.
Introduction
Glavas and four colleagues devised Hyper-Parallel Decoding, a way to pull many attribute values from one document at once instead of one token after another. By reordering token positions and stacking documents inside a single prompt, the method decodes up to 96 tokens together. It works with any large language model and cut inference cost and time sharply on extraction tasks without losing accuracy.
Catalog record
| CREATOR | Theodore Glavas, Nikhita Vedula, Dushyanta Dhyani, Yilun Zhu, Shervin Malmasi |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 53469 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.26209
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Breaking the Autoregressive Chain: Hyper-Parallel Decoding for Efficient LLM-Based Attribute Value Extraction Theodore Glavas, Nikhita Vedula, Dushyanta Dhyani, Yilun Zhu, Shervin Malmasi Abstract Some text generation tasks, such as Attribute Value Extraction (AVE), require decoding multiple independent sequences from the same document context. While standard autoregressive decoding is slow due to its sequential nature, the independence between output sequences offers an opportunity for parallelism. We present Hyper-Parallel Decoding, a novel decoding algorithm that accelerates offline decoding by leveraging both shared memory and computation across batches. HPD enables out-of-order token generation through position ID manipulation, significantly improving efficiency. Experiments on AVE show that attribute-value pairs are conditionally independent, enabling us to parallelize value generation within each prompt. By further stacking multiple documents within a single prompt, we can decode in parallel up to 96 tokens per prompt. HPD works with all LLMs, and reduces both inference costs and total inference time by up to 13.8X without compromising output quality, potentially saving hundreds of thousands of dollars on industry AVE tasks. Although designed for attribute extraction, HPD makes no assumptions unique to the AVE domain and can in theory be applied to other scenarios with independent output structures. Figure 1: AVE on the e-commerce domain: Given a set of attributes for the category Television , values for each attribute are autoregressively generated using an LLM. Our method, Hyper-Parallel Decoding (HPD) parallelizes the extraction of values within the prompt to dramatically increase throughput and decrease cost. 1 Introduction Information Extraction (IE) is a broad subcategory of Natural Language Processing (NLP) tasks that extract structured information from unstructured data. Generative IE has emerged from applying LLMs to IE in domains like e-commerce, medicine and real estate ( 7 ; 1 ; 17 ) , and recent work has shown that LLM-based IE is highly effective ( 26 ) . However, the computational cost of generative LLMs limits their adoption in real-world scenarios ( 41 ) . Prior work has proposed several techniques such as quantization, pruning and distillation ( 42 ) to effectively accelerate LLM inference, but inference costs can still be prohibitive on large-scale data. Moreover, these approaches are fundamentally limited by the autoregressive decoding process, which inherently limits the potential for parallelization on efficient hardware such as GPUs. This motivates our central research question: How can we overcome the autoregressive bottleneck to dramatically increase the speed of LLM-based IE and reduce cost, while maintaining output quality? We focus on Attribute Value Extraction (AVE), a sub-task of IE. AVE requires extracting values for multiple predefined attributes from a single document (see Figure 1 ). Since AVE often needs to be performed on massive sets of documents, it can greatly benefit from offline batched inference. Specifically, e-commerce is a popular domain for AVE as product listings often contain noisy, unstructured text. Crucial applications such as search ( 21 ; 34 ) , recommendation ( 16 ; 31 ) and question answering ( 39 ; 27 ) can greatly benefit from structured information in the form of attribute-value pairs. Autoregressive decoding is slow due to its sequential nature ( 29 ) . However, for tasks like AVE, outputs are independent and can be decoded in parallel. Prior literature has explored the idea of decomposing LLM generation into parallel components to accelerate decoding speed. For example, 23 show that independent reasoning chains can be decoded in parallel for problem solving. Although single-batch latency is reduced, such methods do not translate to the offline setting where batched inference is used, and where throughput and cost are the target metrics. Approaches such as Medusa ( 8 ) support batched inference and can increase throughput by proposing a chain of multiple consecutive tokens in parallel. However, as these tokens are dependent on each other, throughput is limited by the maximum length of the proposals and the verification step required to ensure quality. To address the above challenges, we introduce Hyper-Parallel Decoding (HPD) , a novel inference method that can increase the parallelization of generative AVE, unlocking extreme throughput gains up to 13X. Our key insight is that different generative tasks (e.g. attribute values) can be treated as conditionally independent given a common document context. HPD leverages this inherent structure of AVE to break the autoregressive dependency during generation, and simultaneously generate multiple attribute-value pairs. We further introduce an additional parallelization dimension by also extracting attributes of multiple documents within a shared prompt in parallel, analogous to how CPU hyper-threading can simultaneously complete multiple instructions. Our findings demonstrate that generating non-consecutive, independent tokens in parallel is far more efficient for increasing throughput than prior work. Although we evaluate our approach on e-commerce datasets, HPD makes no assumption exclusive to this setting and can in theory be applied to any domain where the conditional independence of the LLM output holds. We make our code available. 1 1 1 https://github.com/networkslab/HPD To summarize, our main contributions are: [1] We introduce Hyper-Parallel Decoding, a novel, high speed inference method that can increase the parallelization of attribute-value pair extraction to generate up to 96 tokens per prompt. [2] We demonstrate compatibility with a variety of other cost-reduction techniques including quantization, knowledge distillation and batched inference, enabling strong performance in the cost-constrained offline inference setting. [3] We demonstrate the effectiveness of HPD on three e-commerce AVE benchmark datasets, resulting in no quality drop and achieving upwards of 13X execution time reduction and 13X monetary cost reduction for select LLMs. 2 Related Work AVE Early work in AVE formulated the problem as a sequence labeling task, identifying spans of the input document as attribute values. Tagging was performed using specialized domain rules ( 38 ; 32 ) and later generalized using BiLSTM-CRF ( 15 ) and BERT-based ( 36 ) architectures. AVE was later formulated as a question answering (QA) task, with AVEQA ( 33 ) and MAVEQA ( 37 ) using BERT ( 11 ) and ETC ( 2 ) encoders. Both tagging and QA-based AVE rely on the existence of a comprehensive labeled training set, and struggle to generalize to unseen/implicit attributes and values. The generative IE paradigm ( 5 ; 26 ) takes advantage of LLMs to offer superior performance in AVE tasks with few labels or high data sparsity, which is common in the e-commerce domain. Recently, ExtractGPT ( 7 ) has shown that generating attribute-value pairs with LLMs for product AVE is highly effective ( 6 ; 30 ; 28 ) . Parallel LLM Inference We limit our review to works which increase LLM inference parallelism, as other efficient LLM methods are largely orthogonal to our work. Research in LLM inference parallelization often aims to reduce latency in an online setting, where requests are streamed one at a time. Skeleton-of-Thought ( 23 ) breaks down complex reasoning problems into a set of parallel sub-tasks, which can be simultaneously decoded via API calls to an LLM service. This takes advantage of the unused batch dimension in online inference, but does not translate to the offline setting where we assume GPU VRAM to already be saturated by batching independent prompts. APAR ( 20 ) partially addresses this issue using paged-attention to share part of the common key-value cache across batch elements. However, the attention