Polaris: Scaling Up Instruction-Guided Image Generation Towards Millions of Personalized Style Needs

Zhi-Kai Chen, Jun-Peng Jiang, Jun-Jie Tao, De-Chuan Zhan, Han-Jia Ye · 2026

Read Polaris: Scaling Up Instruction-Guided Image Generation Towards Millions of Personalized Style Needs by Zhi-Kai Chen, Jun-Peng Jiang, Jun-Jie Tao, De-Chuan Zhan, Han-Jia Ye (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-01858.txt.

Introduction

Chen and co-authors build a system for personal image styles at huge scale. Users want image models to match diverse personal tastes. Fine-tuning per style is costly and hard to scale. A large pool of tuned modules now exists. Each module serves a narrow set of style needs.

Their Polaris system picks and blends modules per request. It reads a user's instruction to guide the search. The hard part is finding the right few modules. Thousands of mixed candidates fill the pool. The picked parts must also align for the task. Tests are reported on generation and editing jobs. The design aims to reuse the pool, not retrain models.

Catalog record

CREATORZhi-Kai Chen, Jun-Peng Jiang, Jun-Jie Tao, De-Chuan Zhan, Han-Jia Ye
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.01858

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Polaris: Scaling Up Instruction-Guided Image Generation Towards Millions of Personalized Style Needs Zhi-Kai Chen, Jun-Peng Jiang, Jun-Jie Tao, De-Chuan Zhan, Han-Jia Ye Abstract Users increasingly expect image generation models to quickly adapt to highly diverse and personalized requirements, such as producing images with distinctive styles or characteristics. Traditional approaches rely on fine-tuning, which is costly and difficult to scale. To cope with these limitations, the community has accumulated a growing library of fine-tuned modules and adapters, where each component targets specific generation needs and collectively serves as a foundation for handling new demands. This naturally raises a question: instead of repeatedly training new models, can we systematically exploit this expanding ecosystem to better fulfill user instructions? To this end, we present Polaris, an intelligent retrieval framework that automatically selects and integrates suitable models from the model library based on a user’s instructions. The key insight is that harnessing such a massive and heterogeneous pool requires not only finding the most relevant modules among thousands of candidates, but also aligning them effectively for instruction-driven generation and editing. Polaris addresses this challenge by indexing over 6,500 checkpoints and 75,000 adapters, and retrieving the most relevant components given a user’s input and instruction. In doing so, it delivers scalable, controllable, and well-aligned generation—without any additional training. Figure 1 : Although large models fine-tuned on trillions of tokens better understand diverse user instructions, such as style generation, they still exhibit significant blind spots. We introduce Polaris, a retrieval framework over the Stable Diffusion model library that automatically identifies and invokes the most relevant models, addressing these blind spots and enabling diverse, personalized image generation. 1 Introduction The field of image generation has witnessed remarkable progress ( Zhang et al. 2023 ; Wei et al. 2025 ; Singh 2023 ) , particularly with the emergence of diffusion models that demonstrate strong capabilities in producing high-quality, diverse images ( Song et al. 2020 ; Rombach et al. 2022 ; Podell et al. 2023 ; Yang et al. 2023 ) . These models have been successfully applied to a wide range of practical downstream tasks, including super-resolution ( Gao et al. 2023 ) , image completion ( Saharia et al. 2022 ) , style transfer ( Qi et al. 2024 ) , image segmentation ( Tian et al. 2024 ) , and image editing ( Brooks et al. 2023a ) . Such advances have made generative models increasingly accessible to end users. Users’ needs are often highly diverse ( Michelessa et al. 2025 ) and deeply personalized with real-time feedback. For example, they may wish to generate images in distinctive artistic styles, replicate niche visual domains, or perform complex editing operations on existing content. Yet, due to issues such as data imbalance ( Johnson & Khoshgoftaar 2019 ) and catastrophic forgetting ( Goodfellow et al. 2015 ; Loke et al. 2025 ; Lopez-Paz & Ranzato 2022 ) , no single model can reliably master this broad spectrum of styles and tasks. Even large-scale and heavily fine-tuned foundation models still fail to handle many of these long-tail requirements, despite being trained on billions of examples. For precise user needs, small-scale finetuning or generating task-specific adapters such as LoRA ( Hu et al. 2022 ) often provides far more accurate customization than relying solely on a single foundation model. Yet each new requirement still necessitates training a new adapter or checkpoint, which incurs non-negligible costs in data collection and tuning. Building on this context, it is important to recognize that many users have already adopted more lightweight strategies in practice. Instead of training models from scratch, which is costly and time-consuming, they often start from a base model and leverage community-released checkpoints or adapters that capture distinctive styles, domains, or editing capabilities. Such components are openly shared through platforms including Civitai, PixAI, and Tensor.art, which collectively host tens of thousands of publicly available resources. This ecosystem substantially reduces the cost of customization and has enabled the rapid prototyping of diverse generative workflows. Motivated by these observations, we ask whether these community-contributed modules can be systematically harnessed to extend and promote the capabilities of our model. If feasible, this paradigm would provide a lightweight and scalable alternative to traditional personalization methods, while simultaneously raising important challenges in component selection and integration for downstream tasks such as instruction-driven generation. However, systematically exploiting this rich and heterogeneous ecosystem is far from trivial. Two core challenges arise. First, user queries for personalized image generation are often semantically complex and multimodal. A system must not only interpret open-ended instructions and visual references, but also accurately ground them to the underlying style or domain that best reflects the user’s intent. Second, as the pool of checkpoints and adapters grows to tens of thousands, retrieval must be efficient and scalable; naïve search quickly becomes impractical for interactive generation. To address these challenges, we introduce Polaris, a unified framework for instruction-driven style and capability selection. Polaris formulates this process as a multimodal retrieval task: it maps user intent to appropriate community models across style, domain, and concept dimensions. It incorporates an efficient search-and-rerank pipeline that ensures both scalability and responsiveness at interactive latency. With these designs, Polaris systematically harnesses community-contributed checkpoints and adapters to deliver instruction-driven image generation without requiring any additional training. It can directly parse user instructions, recommend the most suitable models, and produce high-quality outputs. Compared with approaches that optimize solely for instructional input, Polaris achieves substantial gains by leveraging its large-scale model library. As shown in Fig. 1 , when large pretrained models struggle with rare or specialized concepts, Polaris offers an alternative generation route that covers these blind spots through targeted community adapters, while maintaining clear advantages in inference efficiency. Together, these results establish Polaris as a scalable and effective paradigm for controllable image generation and editing. Our contributions can be summarized as follows: • We emphasize the need to address highly diverse, user-specific requirements while maintaining efficiency, and demonstrate that leveraging a large model library offers a practical and flexible solution. • Our Polaris retrieves models using combined text and image queries and incorporates an efficient reranking strategy, enabling fast and scalable selection from tens of thousands of candidates. • By harnessing the model library, Polaris delivers high-quality outputs that adapt to diverse user instructions with real-time performance, enabling scalable and customizable image generation. Remark. Polaris focuses on a style-aware instruction setting, where the central requirement is to interpret and realize stylistic intent. This differs from the conventional image-editing setting, which centers on localized content modifications, whereas Polaris enables style-driven transformations and style-consistent generation, including capabilities beyond standard editing. 2 Related Works 2.1 Diffusion-Based Text-to-Image Generation. Diffusion models generate images by progressively denoising random noise under the guidance of a learned score function ( Ho et al. 2020 ; Song et al. 2021 ) , which enables high-quality and diverse synthesis. Stable Diffusion ( Rombach et al. 2022 ) further introduced the latent diffusion framework, operating in a compressed latent space to achieve an efficient tradeoff between computational cost and image fidelity. Since its initial release, multiple versions (v1, v2, SDXL ( Podell et al. 2023 ) , and the recent SD3/3.5 ( Esser et al. 2024 ) ) have been

Read the complete text