Budget-Aware Routing for Long Clinical Text

Khizar Qureshi, Geoffrey Martin, Yifan Peng · 2026

Read Budget-Aware Routing for Long Clinical Text by Khizar Qureshi, Geoffrey Martin, Yifan Peng (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-00336.txt.

Introduction

Khizar Qureshi and 2 co-authors study how to feed long clinical texts to language models under a strict token budget. Clinical notes are long, mixed, and often repetitive, while the questions asked of them are short and high stakes. Their problem is to choose which pieces of a document to keep so a standard generator can answer within fixed cost and delay limits.

They frame the choice as a knapsack problem and propose an objective that balances relevance, coverage, and diversity. They compare ways of cutting documents into units, from sentences to clusters, plus a routing rule that adapts to the budget. On discharge notes, research abstracts, and a long-text benchmark, no single strategy won everywhere. Selector choice mattered more than unit choice, and one popular quality score saturated while another kept separating good summaries from bad.

Catalog record

CREATORKhizar Qureshi, Geoffrey Martin, Yifan Peng
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.00336

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Budget-Aware Routing for Long Clinical Text Khizar Qureshi, Geoffrey Martin, Yifan Peng Abstract A key challenge for large language models is token cost per query and overall deployment cost. Clinical inputs are long, heterogeneous, and often redundant, while downstream tasks are short and high stakes. We study budgeted context selection, where a subset of document units is chosen under a strict token budget so an off-the-shelf generator can meet fixed cost and latency constraints. We cast this as a knapsack-constrained subset selection problem with two design choices, unitization that defines document segmentation and selection that determines which units are kept. We propose RCD , a monotone submodular objective that balances relevance, coverage, and diversity. We compare sentence, section, window, and cluster-based unitization, and introduce a routing heuristic that adapts to the budget regime. Experiments on MIMIC discharge notes, Cochrane abstracts, and L-Eval show that optimal strategies depend on the evaluation setting. Positional heuristics perform best at low budgets in extractive tasks, while diversity-aware methods such as MMR improve LLM generation. Selector choice matters more than unitization, with cluster-based grouping reducing performance and other schemes behaving similarly. ROUGE saturates for LLM summaries, while BERTScore better reflects quality differences. We release our code at https://github.com/stone-technologies/ACL_budget_paper . 1 Introduction Long-context capability is now a headline feature of large language models, yet clinical deployment is constrained by simple arithmetic. Each additional input token increases inference cost and latency, and clinical systems invoke models repeatedly across high-volume workflows. Discharge notes, radiology reports, and evidence syntheses are long because they are templated, redundant, and stitched from multiple sources. For example, MIMIC-IV discharge notes have a median length in the low thousands of tokens ( Johnson et al. 2023 ) , and long-document benchmarks such as L-Eval routinely exceed ten thousand tokens per instance ( An et al. 2024 ; Bai et al. 2023 ) . Cost scales linearly with the token budget and with operational volume. The American Hospital Association reports that community hospitals in the United States account for tens of millions of admissions annually ( American Hospital Association 2025 ) . Even a single large hospital generates on the order of ten thousand long notes per year ( Rosenbloom et al. 2010 ) . Let p_{\mathrm{in}} and p_{\mathrm{out}} denote the price per million input and output tokens, respectively. Processing a document with input budget B and output length S incurs a fee. \mathrm{Cost}(B,S)=\frac{p_{\mathrm{in}}B+p_{\mathrm{out}}S}{10^{6}}, (1) with current prices reported by API providers ( OpenAI 2025 ) . A reduction of even a few hundred tokens per call can translate into substantial annual savings at a hospital scale and can reduce end-to-end latency for time-sensitive decision support. In practice, such systems support tasks ranging from automated discharge summary generation to real-time clinical decision support, where both cost and response time directly affect adoption. These constraints motivate a system perspective where we treat context construction as a budget-constrained optimization problem that lies between raw text and an off-the-shelf generator. Classical submodular maximization under knapsack constraints provides a principled mathematical foundation ( Nemhauser et al. 1978 ; Sviridenko 2004 ; Krause and Golovin 2014 ) , but our focus differs from the canonical setting in three ways. First, the constraint is measured in tokens, which are coupled directly to monetary and latency budgets. Second, the objective must behave predictably across budgets, since practitioners often operate under heterogeneous constraints across users, devices, and clinical services. Third, the system must choose among multiple selectors and representations, rather than commit to a single surrogate objective. This motivates the routing layer studied in this paper. Our central finding is that the optimal selection strategy depends on the evaluation paradigm. For extractive evaluation, positional heuristics such as Lead dominate at low budgets because clinical documents front-load important content and ROUGE rewards lexical overlap. For LLM-based generation, diversity-aware selection (MMR) consistently outperforms positional methods because the LLM benefits from non-redundant input rather than verbatim coverage. A budget-aware routing heuristic captures these regime-dependent differences at near-oracle performance. 1.1 Our contributions We adopt a system-first view in which the context builder is treated as an explicit module with a measurable input budget, compute footprint, and an auditable output context. The module receives a long document D and a budget B , and emits a shorter context C_{B} that is passed to a downstream generator. This decoupling facilitates model-agnostic analysis of unitization and selection and permits comparisons across budget regimes. Our contributions are as follows. (i) We formalize budgeted context construction with explicit unitization and selection stages, and we evaluate sentence-, window-, section-, and cluster-based unitization strategies. (ii) We introduce RCD (Relevance-Coverage-Diversity), a monotone submodular objective that combines relevance, facility-location coverage ( Lin and Bilmes 2011 ) , and log-determinant diversity ( Kulesza and Taskar 2012 ) under a token knapsack constraint. (iii) We implement a unified selector suite including lead baselines ( See et al. 2017 ) , shuffled controls, sliding-window selection, hierarchical expansion, graph-based semantic clustering, maximal marginal relevance (MMR) ( Carbonell and Goldstein 1998 ) , and RCD. (iv) We propose a lightweight budget-aware router that selects an algorithm based on budget and document statistics, and we evaluate it against oracle upper and lower bounds. (v) We report an experimental study across MIMIC discharge notes, Cochrane abstracts, and L-Eval long-document summarization tasks, including both extractive evaluation and end-to-end LLM generation. 2 Related Work A recent wave of benchmarks evaluates model behavior as input length grows. L-Eval collects long documents with human-written questions and summarization tasks ( An et al. 2024 ) . LongBench evaluates long context understanding across multiple tasks ( Bai et al. 2023 ) . RULER and needle-in-a-haystack evaluations isolate retrieval behavior ( Hsieh et al. 2024 ; Kamradt 2023 ) . Our work is orthogonal to these efforts: we change the context that reaches the model rather than the model itself. A pragmatic response to a long context is to compress prompts or select salient content. LLMLingua compresses prompts while preserving answer quality ( Jiang et al. 2023 ) . Retrieval-augmented generation selects passages from external corpora ( Lewis et al. 2020 ) . Our setting differs in that we select from a single long document under a strict token budget rather than retrieving from a large corpus. Submodular functions formalize diminishing returns and have a long history in summarization ( Lin and Bilmes 2011 ) . Maximal marginal relevance trades relevance against redundancy using a similarity kernel ( Carbonell and Goldstein 1998 ) . Determinantal point processes provide diversity through log-determinant objectives ( Kulesza and Taskar 2012 ) . We build directly on these ideas, adding explicit token budgets and a routing layer that adapts to the budget regime. Neural extractive summarizers such as BertSumExt ( Liu and Lapata 2019 ) learn sentence-level selection end-to-end. Our work is complementary as we study lightweight, training-free selectors under explicit token budgets, isolating the effect of selection policy from learned extraction. 3 Methodology 3.1 Framework overview Given a long document D and a token budget B , our pipeline has four stages. (1) Unitization : We segment D into sentence-level candidates and estimate costs c_{i} . (2) Representation : We encode each unit with lexical or semantic features and compute relevance signals. (3)

Read the complete text