Optimizing Diversity and Quality through Base-Aligned Model Collaboration

Yichen Wang, Chenghao Yang, Tenghao Huang, Muhao Chen, Jonathan May, Mina Lee · 2025

Read Optimizing Diversity and Quality through Base-Aligned Model Collaboration by Yichen Wang, Chenghao Yang, Tenghao Huang, Muhao Chen, Jonathan May, Mina Lee (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-05650.txt.

Introduction

Yichen Wang and co-authors address a trade-off in aligned language models: their answers are better, but also more alike. Fine-tuning for quality tends to flatten variety, above all in open-ended tasks like story writing. Their framework, BACo, mixes a base model with its aligned twin while generating, token by token.

Routing rules use uncertainty and content signals to pick which model supplies each token. The mix runs in a single pass, with no extra training or costly decoding. Across three open-ended tasks and many quality and diversity metrics, BACo beat prior inference-time methods. The best router lifted diversity and quality together by about a fifth, and human raters agreed.

Catalog record

CREATORYichen Wang, Chenghao Yang, Tenghao Huang, Muhao Chen, Jonathan May, Mina Lee
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE78 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-05650

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Optimizing Diversity and Quality through Base-Aligned Model Collaboration Yichen Wang, Chenghao Yang, Tenghao Huang, Muhao Chen, Jonathan May, Mina Lee Abstract Alignment has greatly improved large language models (LLMs)’ output quality at the cost of diversity , yielding highly similar outputs across generations. We propose Base-Aligned Model Collaboration ( BACo ), an inference-time token-level model collaboration framework that dynamically combines a base LLM with its aligned counterpart to optimize diversity and quality. Inspired by prior work ( 16 ) , BACo employs routing strategies that determine, at each token, from which model to decode based on next-token prediction uncertainty and predicted contents’ semantic role. Prior diversity-promoting methods, such as retraining, prompt engineering, and multi-sampling methods, improve diversity but often degrade quality or require costly decoding or post-training. In contrast, BACo achieves both high diversity and quality post hoc within a single pass, while offering strong controllability. We explore a family of routing strategies, across three open-ended generation tasks and 13 metrics covering diversity and quality, BACo consistently surpasses state-of-the-art inference-time baselines. With our best router, BACo achieves a 21.3% joint improvement in diversity and quality. Human evaluations also mirror these improvements. The results suggest that collaboration between base and aligned models can optimize and control diversity and quality. Code: github.com/YichenZW/base-align-collab Data: huggingface.co/datasets/ZachW/base-align-collab Reading list: github.com/YichenZW/awesome-llm-diversity 1 Introduction While alignment greatly improved large language models (LLMs)’ output quality in terms of instruction following and downstream task performance ( 53 ) , it results in a stark reduction in output diversity ( 29 ; 89 ; 78 ; 69 ) . Across repeated sampling, a model after alignment (i.e., aligned model) tends to produce highly similar outputs, whereas a model before alignment (i.e., base model) yields diverse outputs. For example, when prompted with “suggest a summer trip destination in the US,” a base model may produce diverse destinations across generations, while the aligned model often converges on a single dominant one ( Figure 1 A). This diversity-quality trade-off undermines open-ended use cases (e.g., creative writing and dialogue) by encouraging formulaic language use ( 5 ; 88 ) , diminishing creativity ( 78 ) , and suppressing ideation in human-AI interaction ( 54 ; 47 ; 1 ) . These findings motivate methods to improve diversity in aligned LLMs. \begin{overpic}[figures/Pic1-v4.1.pdf] \put(13.0,-3.7){{(A) Generated outputs example.}} \par\put(56.0,-3.7){{(B) The diversity-quality trade-off{} space.}} \end{overpic} Figure 1: BACo is an inference-time token-level model collaboration framework that combines a base model’s diversity with its aligned counterpart’s quality. (A) A comparison of generated outputs. The aligned model produces high-quality but low-diversity outputs, while the base model produces high-diversity but low-quality outputs. BACo optimizes both diversity and quality by dynamically routing between them. The probabilities of token(s) are in grey next to text boxes. (B) Illustration of the diversity-quality trade-off space. Single models face a steep trade-off, where improving diversity by adjusting configuration (e.g., by increasing temperature) degrades quality. BACo achieves a better Pareto curve and allows for easy traversal across this frontier by adjusting the router’s threshold. The examples in this figure are modified for simplicity. Prior diversity-promoting methods attempt to address the diversity-quality trade-off at both the training and the inference stages ( Section 7 ). The former ( 33 ; 9 ; 38 ) incorporate explicit diversity objectives into preference optimization during reinforcement learning (e.g., by measuring deviation among winning responses or comparing alternative response groups). While effective at improving diversity, such methods require modifying the model’s output distribution, which can compromise desirable alignment properties such as safety and helpfulness ( 58 ) . The latter consists of decoding-based techniques, such as adjusting temperature and beam search ( 74 ) , as well as prompt-based techniques, including in-context learning ( 49 ) , prompt paraphrasing ( 89 ) , and multilingual back-translation ( 77 ) . However, these inference-time techniques typically require multiple decoding passes or long-horizon planning to improve diversity, and may still disproportionately degrade generation quality ( 56 ) . This reveals a fundamental limitation of forcing a single model to excel at both diversity and quality. To overcome this limitation, we introduce Base-Aligned Model Collaboration ( BACo ), an inference-time token-level model collaboration framework ( Section 3 ). Different from prior works, BACo combines a base model’s diversity with its aligned counterpart’s quality. BACo operates via a lightweight, token-level routing strategy that dynamically switches between the two models in a single decoding pass, requiring no fine-tuning or specialized prompting ( Figure 1 A). Recently, 16 demonstrated that base and aligned models largely agree on next-token predictions, a phenomenon known as the superficial alignment ( 40 ) , suggesting that collaboration between them is feasible. Their method, nudging , employs a small aligned model to guide a larger base model during decoding, effectively improving its quality but without considering diversity. In contrast, our goal is to jointly optimize diversity and quality ( Figure 1 B) by collaboratively employing a base and aligned model. However, the trade-off cannot be entirely resolved. Different tasks naturally favor different points along the diversity-quality spectrum. For example, argumentative writing tasks often demand high-quality (i.e., accurate and factual) outputs with less emphasis on diversity (e.g., multiple valid expressions), whereas brainstorming tasks benefit more from high-diversity (i.e., broader exploration and novelty), even if some outputs are lower quality (e.g., less polished in writing). Moreover, different people may prefer different trade-offs depending on their preferences or intent. Therefore, a practical framework should enable controllability , the ability to adjust along the diversity-quality spectrum according to task or preference. BACo supports such controllability by an adjustable routing threshold ( Figure 1 ) that tunes the attribution of base and aligned model. Moreover, as a single routing strategy does not guarantee diversity across all aspects, we further curate a family of routing strategies within a proposed design space. This design space is primarily organized by the type of information used as prior when making routing decisions. Specifically, the model’s next-token prediction uncertainty (logit-based strategies), predicted contents’ semantic role (content-based strategies), and their combinations. The design space provides a principled foundation for designing, analyzing, and comparing collaborative performance. We validate our approach across three open-ended generation tasks: instruction-following, dialogue, and creative writing ( Section 4 ). We apply 11 automatic metrics for diversity and 2 for quality, together forming 11 \times 2 bi-dimensional evaluation spaces. Additionally, we extend our evaluation beyond instruction-following tasks, which involve relatively short outputs, to long-form generation, where we measure discourse-level diversity in plot structure and emotional flow ( 72 ) . Finally, we complement automatic evaluations with human evaluations of diversity and quality, providing human-centered validation and comparison with automatic results. Through extensive experiments ( Section 5 ), we demonstrate that BACo achieves a new state-of-the-art in optimizing diversity-quality trade-off. Even a simple router that routes randomly yields a 19.0% joint improvement in diversity and quality across tasks and metrics, underscoring the effectiveness of the framework. Our best router achieves a 21.3% gain compared with all baselines (e.g., base or aligned model alone with varying temperature), with even larger improvements on semantic diversity metrics. These results are further supported by human evaluations. Furthermore, we analyze model collaboration patterns under BACo and human-perceived creativity of its outputs in Section 6 . View the

Read the complete text