Combining Microscopy Data and Metadata for Reconstruction of Cellular Traction Forces Using a Hybrid Vision Transformer–U-Net

Yunfei Huang, Elena Van der Vorst, Alexander Richard, Benedikt Sabass · 2026

Read Combining Microscopy Data and Metadata for Reconstruction of Cellular Traction Forces Using a Hybrid Vision Transformer–U-Net by Yunfei Huang, Elena Van der Vorst, Alexander Richard, Benedikt Sabass (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-13400.txt.

Introduction

Yunfei Huang and co-authors propose ViT+UNet to reconstruct cellular traction forces from microscopy data. Traction force microscopy measures the forces cells exert on the matrix around them. Deep models still struggle with inference across scales and with using context like cell type. Their hybrid joins a U-Net with a Vision Transformer in one architecture. It beat both standalone models at predicting traction force fields in their tests. It also generalized across scales and noise levels, and a set input format lets metadata like cell type boost accuracy.

Catalog record

CREATORYunfei Huang, Elena Van der Vorst, Alexander Richard, Benedikt Sabass
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47022
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.13400

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Combining Microscopy Data and Metadata for Reconstruction of Cellular Traction Forces Using a Hybrid Vision Transformer–U-Net Yunfei Huang, Elena Van der Vorst, Alexander Richard, Benedikt Sabass Abstract Traction force microscopy (TFM) is a widely used technique for quantifying the forces that cells exert on their surrounding extracellular matrix. Although deep learning methods have recently been applied to TFM data analysis, several challenges remain—particularly achieving reliable inference across multiple spatial scales and integrating additional contextual information such as cell type to improve accuracy. In this study, we propose ViT+UNet , a robust deep learning architecture that integrates a U-Net with a Vision Transformer. Our results demonstrate that this hybrid model outperforms both standalone U-Net and Vision Transformer architectures in predicting traction force fields. Furthermore, ViT+UNet exhibits superior generalization across diverse spatial scales and varying noise levels, enabling its application to TFM datasets obtained from different experimental setups and imaging systems. By appropriately structuring the input data, our approach also allows the inclusion of metadata, in our case cell-type information, to enhance prediction specificity and accuracy. I Introduction Cells generate mechanical forces on their surrounding extracellular matrix (ECM) and neighboring cells. These cellular traction forces regulate fundamental biological processes such as migration [ 42 , 7 , 21 ] , wound healing [ 9 , 1 ] , and immune responses [ 39 , 32 , 5 ] . Dysregulation of these forces has been implicated in various diseases ranging from cancer [ 27 , 2 , 3 , 17 ] to liver fibrosis [ 10 , 20 ] . Accurate quantification of these mechanical interactions is therefore essential for understanding cellular behavior under both physiological and pathological conditions. Traction force microscopy (TFM) is an effective and widely used technique for inferring cellular traction forces \mathbf{f} on a substrate from measured displacement fields \mathbf{u} based on the known mechanical properties of the substrate, as illustrated in Fig. ( 1 -A). The displacement field is typically obtained by tracking fluorescent marker beads embedded within the substrate using particle image velocimetry (PIV). The mechanical relationship between force and displacement can be numerically approximated using methods such as the boundary element method (BEM) [ 34 , 25 ] , finite element method (FEM) [ 47 , 51 ] , or Fourier-transform traction cytometry (FTTC) [ 34 , 23 ] . Determining unknown traction forces from measured displacements constitutes an inverse problem that is inherently ill-posed—small errors in measurement can lead to large instabilities in the reconstructed force fields. Classical approaches rely on regularization techniques such as L_{2} regularization [ 36 , 34 ] , elastic-net regularization [ 25 ] , or non-parametric Bayesian regularization to stabilize solutions [ 25 , 23 ] . Despite these advances, traditional physics-based methods remain computationally expensive, limited to single deterministic predictions, and sensitive to experimental noise. Recently, machine learning (ML) models have emerged as powerful tools for solving this inverse problem by learning direct mappings between displacement data and traction forces rather than relying solely on physical modeling with regularization techniques. ML-based approaches enable faster inference once trained while maintaining high predictive performance even under complex experimental conditions. Most existing deep-learning frameworks focus on two-dimensional (2D) traction or stress fields derived from displacement data [ 40 , 28 , 29 ] . Furthermore, a trained 2D machine-learning model can infer both displacement and traction force fields directly from sequential cell images [ 38 ] . In another approach, a deep-learning model was first trained to learn surface wrinkles and subsequently used to train another model that predicts the traction force field from images or surface wrinkle patterns [ 30 ] . All these trained models are based on measured displacement information as input to infer the traction force field. Because of the robust capability of neural networks to extract nonlinear patterns and hidden features that extend beyond direct or analytically tractable physical relationships, advanced architectures have also been proposed to directly infer traction forces from cellular protein images without explicit displacement information [ 35 ] . Furthermore, three-dimensional deep-learning–based frameworks have been developed to capture out-of-plane forces and complex 3D cell–matrix interactions [ 18 , 4 ] , enabling data-driven TFM analysis in three-dimensional collagen hydrogels. Although existing machine-learning models efficiently predict traction forces with high robustness across different datasets, they still face challenges arising from untested techniques, evolving algorithms, and newly developed datasets. U-Net [ 33 ] and Vision Transformers (ViT) [ 16 ] are two prominent neural network architectures widely applied in image classification [ 12 , 46 , 8 , 6 ] and segmentation tasks [ 22 , 41 , 37 , 49 ] . U-Net excels at identifying short-range dependencies associated with local spatial patterns, whereas ViT specializes in capturing long-range dependencies related to global contextual information. Consequently, several advanced hybrid architectures have been developed that combine both approaches—including Swin-UNet [ 11 ] , TransUNet [ 13 ] , SUNet [ 19 ] , META-UNet [ 43 ] , and After-UNet [ 45 ] —for various image analysis tasks; however, whether these hybrid networks improve traction force prediction remains unclear. Moreover, ViT provides a robust architecture for integrating both image and text metadata inputs across different tasks—for example, image generation [ 14 ] and image segmentation [ 31 ] . Nevertheless, the use of such multimodal (image–text) metadata paired inputs in TFM applications remains largely unexplored. Generalization—the ability of a model to perform well on unseen data—is an essential metric for evaluating machine-learning performance. It ensures applicability beyond the training dataset and is crucial for real-world biological imaging applications, where variability arises due to differences in experimental setups or imaging scales. In deep-learning-based TFM studies, generalization has been extensively discussed and investigated across different cell types and biochemical perturbations [ 35 ] ; however, achieving consistent performance across diverse spatial scales and varying noise levels remains challenging. In this work, we introduce a hybrid neural network architecture (ViT+UNet) for traction force microscopy that combines the classical U-Net for capturing short-range dependencies with a robust Vision Transformer (ViT) for modeling long-range contextual relationships. We compare the performance of ViT+UNet with that of the individual U-Net and ViT models, as well as with Bayesian Fourier-transform traction cytometry (BFTTC) [ 23 ] . Furthermore, we investigate the generalization capabilities of these models across diverse spatial scales and evaluate their predictive robustness under varying noise levels. In addition, we develop extended versions of ViT+UNet and ViT capable of inferring traction forces using metadata by incorporating both displacement fields and cell-type information. Fig. 1: Schematic diagram of traction force microscopy and the architectures of deep learning models used for traction force analysis. (A) Experimental setup: a single cell adheres to the surface of an elastic gel, and the traction force field is calculated from the measured displacement field. (B) Architecture of U-Net, consisting of downsampling, a middle block, and upsampling stages. The black arrows indicate 2D convolutions; red arrows represent downsampling through 2D convolutions; blue arrows denote 2D transposed convolutions used for upsampling; and purple arrows indicate skip connections (copy operations). The input and output of the model correspond to the displacement field and traction force field, respectively. (C) Architecture of Vision Transformer (ViT), which includes image pre-processing, a transformer

Read the complete text