AURORA : A daptive U nified R epresentation f O r R obust U ltrasound A nalysis
Ufaq Khan 1 , L D M S Sai Teja 2 , Ayuba Shakiru 3 , Mai A. Shaaban 1 Yutong Xie 1 , Muhammad Bilal 3 , Muhammad Haris Khan 1 · 2026
Read AURORA : A daptive U nified R epresentation f O r R obust U ltrasound A nalysis by Ufaq Khan 1 , L D M S Sai Teja 2 , Ayuba Shakiru 3 , Mai A. Shaaban 1 Yutong Xie 1 , Muhammad Bilal 3 , Muhammad Haris Khan 1 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-19364.txt.
Introduction
Khan and co-authors build one model that handles many ultrasound tasks across organs and scanners. Ultrasound images shift a lot with device, operator, and body target. Their AURORA uses a transformer encoder from the Qwen3-VL family. Token features are fused across scales, then split to small heads for each task. Task-aware sampling and loss balance keep the mixed training fair. Scores rise from 67 to 85 percent on validation data. The test score averages 81.84 percent across all tasks.
Catalog record
| CREATOR | Ufaq Khan 1 , L D M S Sai Teja 2 , Ayuba Shakiru 3 , Mai A. Shaaban 1 Yutong Xie 1 , Muhammad Bilal 3 , Muhammad Haris Khan 1 |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 24765 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.19364
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
AURORA : A daptive U nified R epresentation f O r R obust U ltrasound A nalysis Ufaq Khan 1 , L D M S Sai Teja 2 , Ayuba Shakiru 3 , Mai A. Shaaban 1 Yutong Xie 1 , Muhammad Bilal 3 , Muhammad Haris Khan 1 Abstract Ultrasound images can vary widely across scanners, operators, and anatomical targets, so models trained in one setting often generalize poorly to new hospitals and clinical conditions. The Foundation Model Challenge for Ultrasound Image Analysis (FMC-UIA) reflects this scenario by requiring a single model to handle multiple tasks like segmentation, detection, classification, and landmark regression across diverse organs and datasets. We propose a unified multi-task framework based on a transformer visual encoder from the Qwen3-VL family. Intermediate token features are projected into spatial feature maps and fused using a lightweight multi-scale feature pyramid, enabling both pixel-level predictions and global reasoning within a shared representation. Each task is addressed with a small task-specific prediction head, while training uses task-aware sampling and selective loss balancing to manage heterogeneous supervision and reduce task imbalance. Our method is designed to be straightforward to optimize and adaptable across a wide range of ultrasound analysis tasks. The score improved from 67% to 85% on validation and reached 81.84% average on the official test set across all tasks. The code is publicly available at Github . † † address: 1 MBZUAI, Abu Dhabi, UAE, 2 NIT Silchar, India, 3 Birmingham City University, UK. 1 Introduction Ultrasound is a first-line imaging modality due to its portability, real-time, and free ionizing radiation. At the same time, automated analysis is difficult: appearance changes with probe pose and operator technique; speckle statistics vary across vendors and settings; and clinically relevant targets are usually small, occluded, or weakly contrasted [ 1 ] . Such that models trained in one setting often degrade when deployed on new devices, institutions, or patient cohorts [ 2 ] . Recent work has moved from task-specific ultrasound models toward generalist representations inspired by foundation models in vision. Promptable segmentation frameworks such as ”Segment Anything” shows how large-scale pretraining can transfer across diverse settings [ 3 , 4 ] , and ”MedSAM” demonstrates the value of domain-tailored pretraining in medical imaging [ 5 ] . For ultrasound, modality-specific efforts including UltraSam [ 6 ] , UltraFedFM [ 7 ] , and large-scale self-supervised pretraining studies [ 8 ] further highlight that scale and diversity can improve robustness. However, most ultrasound foundation models remain single-task, limiting support for masks, labels, boxes, and landmarks in one model. The Foundation Model Challenge for Ultrasound Image Analysis (FMC-UIA 2026) [ 9 ] addresses this gap by requiring a single submission to perform segmentation, classification, detection [ 10 ] , and landmark regression across many organs, with evaluation on unseen domains to stress-test generalization [ 11 ] . In this paper, we present a unified multi-task baseline aligned with the challenge protocol and built on a pretrained transformer vision encoder. We use the vision encoder from Qwen3-VL [ 12 ] as backbone and bridge intermediate token representations to multi-scale spatial feature maps through a lightweight feature pyramid, enabling one shared representation to support both dense and global/coordinate objectives. Contributions. We propose (i) a unified multi-task ultrasound framework based on a pretrained Qwen3-VL vision encoder, enabling a single model to jointly address segmentation, detection, classification, and landmark regression through a shared representation; (ii) a token-to-map bridging mechanism with multi-scale (FPN-style) feature fusion that converts intermediate transformer token activations into spatial feature maps, supporting both dense prediction and global/coordinate reasoning; and (iii) an efficient task-routing and optimization strategy comprising lightweight task-specific heads with explicit routing, together with task-aware sampling and selective loss balancing to accommodate heterogeneous supervision, alleviate task imbalance, and improve memory and training efficiency. Figure 1: Overview of the proposed unified multi-task framework. Shared Qwen3-VL encoder extracts multi-depth tokens, reshaped and fused into a four-level pyramid. Dense heads use pyramid features; global heads use pooled multi-scale embeddings. 2 Method 2.1 Overview Given an ultrasound image X\in\mathbb{R}^{H\times W\times 3} and task identifier t , our model learns a task-conditional predictor f_{\theta}(X,t)\mapsto\hat{y}_{t} , where \hat{y}_{t} corresponds to segmentation, detection, classification, or landmark regression. The architecture follows a shared private design: a shared visual encoder produces a common representation, which is transformed into a multi-scale feature pyramid, and lightweight task-specific heads are attached. Routing is explicit via t , with dense tasks (segmentation/detection) operating on pyramid features and global tasks (classification/regression) using a pooled embedding derived from the pyramid. An overview is shown in Fig. 1 . 2.2 Visual encoder and intermediate representations We use the Qwen3-VL vision encoder as backbone and discard the language model to reduce memory. Inputs are resized to a fixed square resolution and preprocessed to normalized pixels with a grid specification; for still images the temporal grid is 1, yielding a spatial lattice of size h{\times}w (grayscale is replicated to 3 channels). To obtain multi-level features, we attach forward hooks to four transformer blocks roughly uniformly spaced in depth, producing token activations \mathbf{T}^{(\ell)}\!\in\!\mathbb{R}^{B\times N\times C} . Using N{=}hw , we reshape them into spatial maps \mathbf{F}^{(\ell)}\!\in\!\mathbb{R}^{B\times C\times h\times w} , ranging from low-level detail to high-level semantic representations. 2.3 Token pyramid and feature pyramid fusion The channel dimensionality of each hooked map \mathbf{F}^{(\ell)} depends on the backbone; therefore, we project all maps to a shared width D using a learnable 1{\times}1 convolution instantiated lazily on first use. We construct a four-level hierarchy by resizing features to relative scales \{1,\tfrac{1}{2},\tfrac{1}{4},\tfrac{1}{8}\} of the finest level and fuse them with a top-down FPN with lateral connections, yielding pyramid features \{\mathbf{P}_{2},\mathbf{P}_{3},\mathbf{P}_{4},\mathbf{P}_{5}\} at channel width D . Dense heads operate on the multi-scale pyramid, whereas global tasks use a pooled representation obtained by concatenating global average pooled vectors from each level, i.e., \mathbf{e}=[\mathrm{GAP}(\mathbf{P}_{2});\mathrm{GAP}(\mathbf{P}_{3});\mathrm{GAP}(\mathbf{P}_{4});\mathrm{GAP}(\mathbf{P}_{5})]\in\mathbb{R}^{4D} , which captures both fine-grained detail and coarse contextual information. 2.4 Task-specific heads and loss functions Segmentation head. Segmentation exploits the full multi-scale pyramid. Each level is refined with lightweight residual blocks and channel recalibration, and we apply an Atrous Spatial Pyramid Pooling (ASPP) module at the coarsest level to enlarge the effective receptive field. A top-down decoder then upsamples and merges features using gated skip connections. A final 1{\times}1 convolution predicts logits for the maximum number of classes across all segmentation tasks; for a given task, we slice the corresponding channels and upsample predictions to the input resolution. We train the head with a standard weighted cross-entropy combined with a soft Dice term (computed over foreground classes), using \mathcal{L}_{\mathrm{seg}}=\mathcal{L}_{\mathrm{ce}}+\lambda_{\mathrm{dice}}\mathcal{L}_{\mathrm{dice}} . Detection head. Detection is implemented as an anchor-free grid predictor operating on the finest pyramid level \mathbf{P}_{2} . To inject additional context, we resize \mathbf{P}_{3} and \mathbf{P}_{4} to the \mathbf{P}_{2} resolution, concatenate them with \mathbf{P}_{2} and normalized coordinate channels, and feed the resulting tensor to a lightweight convolutional tower. The head predicts, at each grid location, a bounding box (x_{1},y_{1},x_{2},y_{2}) and an objectness score. Following the challenge protocol, inference selects the box at the location with maximum objectness, avoiding other post-processing. For training, we treat locations whose grid center lies inside the ground-truth box as positives and optimize a focal-weighted binary cross-entropy for objectness together with box regression and an IoU-based penalty, i.e.,