FAPE-IR: Frequency-Aware Planning and Execution Framework for All-in-One Image Restoration
Jingren Liu, Shuning Xu, Qirui Yang, Yun Wang, Xiangyu Chen, Zhong Ji · 2025
Read FAPE-IR: Frequency-Aware Planning and Execution Framework for All-in-One Image Restoration by Jingren Liu, Shuning Xu, Qirui Yang, Yun Wang, Xiangyu Chen, Zhong Ji (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-14099.txt.
Introduction
Jingren Liu, Shuning Xu, Qirui Yang, and three co-authors build one image restoration system that handles many kinds of damage. Photos may be blurred, noisy, dark, or compressed, and older unified methods rely on rigid task designs that adapt poorly to mixed real-world damage. Their framework, FAPE-IR, splits the job into planning and execution along frequency lines.
A frozen multimodal language model inspects each damaged image and writes a short frequency-aware restoration plan. That plan steers a diffusion executor whose mixture-of-experts module picks high or low frequency specialists. Adversarial training and a frequency loss cut artifacts. Tests across seven restoration tasks reached state-of-the-art results, with strong zero-shot handling of mixed damage.
Catalog record
| CREATOR | Jingren Liu, Shuning Xu, Qirui Yang, Yun Wang, Xiangyu Chen, Zhong Ji |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 110 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-14099
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
FAPE-IR: Frequency-Aware Planning and Execution Framework for All-in-One Image Restoration Jingren Liu, Shuning Xu, Qirui Yang, Yun Wang, Xiangyu Chen, Zhong Ji Abstract All-in-One Image Restoration (AIO-IR) aims to develop a unified model that can handle multiple degradations under complex conditions. However, existing methods often rely on task-specific designs or latent routing strategies, making it hard to adapt to real-world scenarios with various degradations. We propose FAPE-IR, a Frequency-Aware Planning and Execution framework for image restoration. It uses a frozen Multimodal Large Language Model (MLLM) as a planner to analyze degraded images and generate concise, frequency-aware restoration plans. These plans guide a LoRA-based Mixture-of-Experts (LoRA-MoE) module within a diffusion-based executor, which dynamically selects high- or low-frequency experts, complemented by frequency features of the input image. To further improve restoration quality and reduce artifacts, we introduce adversarial training and a frequency regularization loss. By coupling semantic planning with frequency-based restoration, FAPE-IR offers a unified and interpretable solution for all-in-one image restoration. Extensive experiments show that FAPE-IR achieves state-of-the-art performance across seven restoration tasks and exhibits strong zero-shot generalization under mixed degradations. 1 Introduction Figure 1 : Comparison of AIO-IR methods: (a) multi-branch mappings with task-level priors; (b) task-specific routing/clustering; (c) our FAPE-IR, unifying understanding and restoration. Image restoration seeks to recover clean images from degradations such as rain, snow, fog, blur, low-light, and noise, and is fundamental to safety- and quality-critical applications in medical imaging 7 ; 85 , consumer photography 101 , and autonomous driving 87 ; 25 ; 42 . Traditional pipelines typically employ specialized architectures or optimization strategies for each type of degradation 63 ; 107 ; 112 ; 28 . Although recent single-degradation models report strong results on curated benchmarks 100 ; 84 ; 115 ; 31 ; 20 , real-world images rarely present a single known corruption; degradations are often unknown, co-occurring, and subject to distribution shift, which undermines stability and controllability. This gap motivates All-in-One Image Restoration (AIO-IR), which aims to handle multiple degradations within a single model 8 ; 49 ; 78 ; 68 ; 106 ; 11 ; 59 ; 10 . Yet most current efforts still assume task separability 49 ; 1 or tether restoration to reconstruction pipelines driven by task-level priors 86 ; 96 , preventing models from autonomously perceiving and disentangling degradation attributes. As a result, model generalization to real-world scenarios remains inadequate. Within current AIO-IR methods, a prevailing strategy is to employ multi-branch mappings (see Figure 1 -(a)), injecting task-level conditions via explicit priors (e.g., prompts, embeddings, or specialized encoders) and learning the restoration paths accordingly 57 ; 39 ; 51 ; 50 ; 8 ; 114 . This design is prone to severe cross-task interference. In a unified model with shared parameters, gradients from different degradations conflict, making it difficult to reach task-specific optima simultaneously. Moreover, both training and inference depend on prior labels or textual inputs, incurring substantial annotation and prompt-engineering costs. A complementary strategy clusters features or employs specialized routing mechanisms within the model (see Figure 1 -(b)) 16 ; 78 ; 109 ; 113 ; 90 ; 13 . Because tasks are strongly separated in latent space, their learning processes become nearly independent, which hampers the discovery of shared structures and compositionality across degradations. Consequently, robustness to compound degradations is limited, and generalization suffers. More critically, both lines lack semantic understanding and rely on opaque degradation pipelines, hindering content-aware adaptation and interpretability. These limitations call for a unified, semantics-aware AIO-IR framework in which degradation understanding and restoration are coupled, and task knowledge is both disentangled and shared in an adaptive way. To instantiate such a framework, we adopt diffusion-based image restoration, which has recently become a popular choice for AIO-IR due to its strong generative priors 57 ; 8 ; 114 ; 16 ; 113 ; 90 ; 13 . This formulation naturally models diverse degradations, yet prior work typically treats diffusion as a task-conditioned generator driven by labels or hand-crafted priors, leaving degradation modeling opaque and under-exploiting cross-degradation interactions. We instead regard diffusion as the execution engine in a unified multimodal understanding–generation paradigm. In this paradigm, degradations are first parsed and organized, and then restored. Inspired by recent unified models 99 ; 40 ; 95 ; 17 ; 9 ; 102 , we instantiate FAPE-IR, a multimodal large language model (MLLM) + diffusion framework for low-level vision (see Figure 1 -(c)). An MLLM explicitly parses image content and degradation semantics and outputs understanding and planning tokens that condition a diffusion executor to perform content-adaptive restoration in the frequency domain, grouping tasks into high- and low-frequency regimes, thereby encouraging sharing within same-frequency tasks and isolating conflicting ones. Concretely, FAPE-IR couples a multimodal planner with a diffusion-based executor to mitigate cross-task gradient conflict and isolation from a frequency perspective. In the planning stage, we employ a label-free procedure to extract as many degradation-related low-level features as possible, and introduce several well-designed instructions that constrain the model to output: the degradation types present in the image, the primary frequency bands requiring restoration, the underlying causes, and the proposed restoration pipeline. This design yields effective, per-image understanding that more accurately guides the restoration process. In the execution stage, tokens produced by the planner, augmented with high-/low-frequency visual features from SigLIP-v2 79 and a VAE 30 , guide the restoration of the entire image. Furthermore, we propose a Frequency-Aware LoRA-MoE architecture, where text tokens from the planner and the high-/low-frequency signals extracted from intermediate executor’s features drive the MoE gating to select among high-/low-frequency experts dynamically and interpretably. For the overall training recipe, we provide theoretical and empirical evidence that adversarial training on diffusion-pretrained weights can yield higher fidelity and fewer artifacts than other fine-tuning methods under our setting. Building on this, we further introduce an energy-based frequency regularization loss so that each frequency expert is optimized in its most suitable band. Our main contributions are summarized as follows: (i) We introduce FAPE-IR, a unified understanding–generation AIO-IR framework that couples a multimodal planner with a diffusion-based executor, enabling semantics-driven, content-adaptive restoration across diverse degradations. (ii) We develop a frequency-aware modeling strategy that dynamically routes information between high- and low-frequency experts, mitigating cross-task interference while promoting shared representations among related tasks. (iii) We conduct comprehensive experiments on a wide range of single-degradation and mixed-degradation benchmarks, demonstrating that FAPE-IR achieves state-of-the-art or competitive performance and exhibits strong zero-shot generalization to unseen compound degradations. 2 Related Works 2.1 Diffusion-based AIO-IR In recent years, diffusion-based AIO-IR methods have advanced rapidly. Existing approaches fall into two families: (i) multi-branch mappings and (ii) clustering or routing. The former relies on injecting task-level priors into a shared backbone via task encoders or textual/latent prompts. PromptIR 58 , Prompt-in-Prompt 39 , and InstructIR 14 unify degradations through prompts or instructions. ProRes 51 and DA-CLIP 50 supply content/degradation embeddings via visual prompts or CLIP 61 controllers and fuse them with cross-attention. UniRestore 8 injects task cues into diffusion, whereas UniRes 114 composes task specialists at sampling time. The latter adapts internal representations by clustering features or routing to experts. AdaIR 16 and DFPIR 78 mine frequency cues and reshape separable, degradation-aware features. AMIRNet 110 performs label-free, layer-wise clustering with domain alignment. Within diffusion, DiffUIR 113 disentangles shared/private factors