Generative Retrieval via Diffusion Transformer with Metric-Ordered Sequence Training and Hybrid-Policy Preference Optimization

Chenghao Liu, Yu Zhang, Zhongtao Jiang, Kun Xu, Zhenwei An, Renzhi Wang, Zhao Wang, Jiachen Zhang, Yuxiao Zhang, Kun Xu, Songfang Huang Peking University · 2026

Read Generative Retrieval via Diffusion Transformer with Metric-Ordered Sequence Training and Hybrid-Policy Preference Optimization by Chenghao Liu, Yu Zhang, Zhongtao Jiang, Kun Xu, Zhenwei An, Renzhi Wang, Zhao Wang, Jiachen Zhang, Yuxiao Zhang, Kun Xu, Songfang Huang Peking University (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-26899.txt.

Introduction

Chenghao Liu and nine colleagues tackle pattern-preserving attribute retrieval, a search task where both likeness and a target trait matter. A user supplies seed items showing a fine pattern. They want more items that keep that pattern but score higher on some attribute. Averaging the seeds keeps the pattern but stays low on the attribute. Chasing the attribute alone drifts to alien patterns.

Their generative approach creates query vectors rather than ranking by fixed similarity. A diffusion transformer generates the queries, trained in stages. Metric-ordered pretraining teaches it sequences ranked by measured closeness. A preference optimisation stage with hybrid policies sharpens the trade-off. Fine-tuning on domain tail centroids, the centres of the high-attribute tail, completes the recipe.

Catalog record

CREATORChenghao Liu, Yu Zhang, Zhongtao Jiang, Kun Xu, Zhenwei An, Renzhi Wang, Zhao Wang, Jiachen Zhang, Yuxiao Zhang, Kun Xu, Songfang Huang Peking University
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE88 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.26899

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Generative Retrieval via Diffusion Transformer with Metric-Ordered Sequence Training and Hybrid-Policy Preference Optimization Chenghao Liu, Yu Zhang, Zhongtao Jiang, Kun Xu, Zhenwei An, Renzhi Wang, Zhao Wang, Jiachen Zhang, Yuxiao Zhang, Kun Xu, Songfang Huang Peking University Abstract Embedding-based retrieval ranks items by their similarity to a query in a shared vector space and usually aims to return the highest-scoring items. In many production settings this is not what is wanted: given a seed set that expresses a fine-grained pattern, one needs more items that both satisfy a target attribute and stay within that pattern. We formalize this as pattern-preserving attribute retrieval . The two goals pull against each other: averaging the seeds preserves the pattern but stays in a low-attribute region, while global attribute retrieval drifts to unrelated patterns. We approach the task with continuous generative retrieval, where a model reads a sequence of item embeddings and generates one or more query embeddings for nearest-neighbor search. We propose MO-DiT+HPPO (Metric-Ordered Diffusion Transformer with Hybrid-Policy Preference Optimization, HPPO), a staged framework with large-scale raw-sequence pretraining, multi-domain metric-ordered continuation pretraining, domain-specific tail-centroid supervised fine-tuning, and HPPO. The metric-ordered stages convert sparse online retrieval labels into in-pattern trajectories ordered from low to high predicted attribute density, so a single continuation-pretrained model learns the metric-improvement direction across all domains. HPPO then aligns the final query distribution with the true online objective: in each round it forms a hybrid-policy candidate pool—deterministic high-metric constructions together with the current policy’s own guidance-scale fan—labels every candidate with the real online intersection metric Joint@K , and applies iterated, reference-anchored preference optimization with held-out-validation early stopping; a simple tradeoff-aware Pareto pair filter then keeps only winner pairs that do not lower same-pattern purity, so updates raise the attribute metric without sacrificing the pattern. Across four large-scale attribute domains under strict item- and pattern-holdout protocols, metric-ordered DiT training improves the primary intersection metric over a strong pretrained generative retriever, and HPPO improves it further, with gains that are large and paired-bootstrap significant on seven of the eight domain-split cells and a marginal tie on the hardest domain’s pattern split. The Pareto filter is the key ingredient: on most domain–split cells it raises Joint@K while keeping the same-pattern share higher than an unconstrained preference policy, pushing the attribute–pattern frontier outward rather than merely sliding along it. Metric-predictor validation, matched order ablations, CPT/SFT comparisons, and a candidate-policy ablation show where the gains come from and that static and policy-generated candidates are complementary. † † footnotetext: ∗ Equal contribution. † Corresponding authors.Emails: [email protected], [email protected], [email protected], [email protected], Emails: [email protected], [email protected], [email protected], [email protected], Emails: [email protected], [email protected], [email protected]. 1 Introduction Figure 1: Pattern-preserving attribute retrieval and the MO-DiT+HPPO pipeline. Given a seed set from one fine-grained pattern, the objective is to retrieve top- K neighbors that are both attribute-positive and of the seed pattern. Average pooling preserves the pattern but remains in a low-attribute-density region, whereas attribute-only retrieval drifts toward unrelated patterns. The generative retriever instead synthesizes a continuous query embedding near a high-density region of the same pattern. MO-DiT+HPPO follows a staged pipeline (bottom): large-scale raw-sequence pretraining; metric-ordered sequence construction; shared multi-domain continuation pretraining and domain-specific tail-centroid SFT, both trained on the constructed metric-ordered sequences; and online-reward Hybrid-Policy Preference Optimization. The item thumbnails are randomly sampled from a public dataset ( 18 ) and shown for illustration only. Large retrieval systems increasingly serve attribute-seeking objectives. A platform may need to find items that satisfy a quality, safety, policy, originality, or integrity criterion. In many such tasks, the desired result is not an arbitrary high-scoring item. Retrieved items should also match the fine-grained pattern represented by a seed set: the same topic, visual style, failure mode, or retrieval intent. A typical operational case illustrates the tension: reviewers discover a small batch of items that express one new variant of a policy violation—a specific format, template, or tactic—and need to retrieve many more items of exactly that variant for measurement and treatment. A global attribute classifier returns positives of every variant and floods the review queue with unrelated patterns, while nearest-neighbor expansion of the seeds returns the same pattern at mostly ordinary, low-positive-rate items. The task therefore sits between two standard tools. Averaging the seed embeddings stays close to the seed pattern but retrieves mostly ordinary examples; optimizing only for the target attribute retrieves many positives from the wrong patterns. The desired query moves from the seed pattern toward a nearby high-density region of the same pattern. We study this problem in continuous generative retrieval. Each item has a frozen multimodal embedding. A generative retriever receives a sequence of item embeddings and produces a query embedding; the query then retrieves the top- K items whose embeddings are most similar to it, by approximate nearest-neighbor (ANN) search over a large vector index. Continuous generation is attractive because the output query is not restricted to an existing item or to a fixed average of inputs. However, it raises a central supervision question: what target teaches the model to move toward higher-density examples while preserving the pattern? We propose MO-DiT+HPPO (Metric-Ordered Diffusion Transformer with Hybrid-Policy Preference Optimization, HPPO), a staged training framework for attribute-seeking and pattern-preserving generative retrieval; Figure 1 illustrates the task and the staged pipeline. The first stage is large-scale raw-sequence pretraining, which gives the diffusion transformer a general continuous retrieval prior. The second stage is metric-ordered continuation pretraining, shared across domains. We use online recall density as an accessible proxy metric: for an anchor embedding, attribute density is the fraction of top- K retrieved neighbors that satisfy the target attribute. Because dense online labeling is expensive, we label a sparse subset online, train a lightweight metric predictor over frozen embeddings, and use the predictor to order items inside each latent pattern cluster. The resulting training examples are sequences that move from lower-metric to higher-metric items within the same pattern; mixing the ordered sequences of all domains lets one continuation-pretrained model internalize the metric-improvement direction as a general capability rather than a per-domain memorization. The third stage is domain-specific tail-centroid supervised fine-tuning. It maps the input prefix of a metric-ordered sequence to the centroid of its high-metric tail. The centroid target aggregates multiple high-metric examples, reducing sensitivity to a single noisy top item. The fourth stage is HPPO, which labels candidate query embeddings with the final online intersection metric and applies iterated reference-anchored preference optimization. Its key ingredient is a tradeoff-aware Pareto pair filter that prevents preference updates from buying attribute gains at the cost of pattern drift, so alignment improves the primary metric without sacrificing pattern purity. This mirrors the role of post-training alignment: CPT and SFT provide a stable pattern-preserving generator, while HPPO aligns the final query distribution with the true online objective. Our contributions are: • We formalize pattern-preserving attribute retrieval for continuous generative retrieval, and define an intersection metric ( Joint@K ) that penalizes both attribute failure and pattern drift. • We introduce metric-ordered sequence construction , which turns sparse online retrieval labels into large-scale in-pattern training trajectories that move from low to high attribute density within a single pattern. • We show that a shared multi-domain metric-ordered continuation pretraining, followed by domain-specific tail-centroid supervised fine-tuning, improves same-pattern attribute retrieval across four large-scale domains.

Read the complete text