Multimodal Diffusion to Mutually Enhance Polarized Light and Low Resolution EBSD Data
Harry Dong, Timofey Efimov, Megna Shah, Jeff Simmons, Sean Donegan, Marc De Graef, Yuejie Chi · 2026
Read Multimodal Diffusion to Mutually Enhance Polarized Light and Low Resolution EBSD Data by Harry Dong, Timofey Efimov, Megna Shah, Jeff Simmons, Sean Donegan, Marc De Graef, Yuejie Chi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-22212.txt.
Introduction
Dong and six colleagues use an open multi-input diffusion model to learn the dynamics between EBSD and polarized light data. The model tackles inverse problems for 3-D electron back-scattered diffraction. Trained solely on fake data once, the model carries over strongly to real data that are low-detail, noisy, damaged, and misaligned. With test-time scaling, it gains on grain edge guessing, upscaling, and cleanup. The model shows little gap from full-detail scores with only 25 percent of EBSD data and damaged polarized light data.
Catalog record
| CREATOR | Harry Dong, Timofey Efimov, Megna Shah, Jeff Simmons, Sean Donegan, Marc De Graef, Yuejie Chi |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 46822 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.22212
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Multimodal Diffusion to Mutually Enhance Polarized Light and Low Resolution EBSD Data Harry Dong, Timofey Efimov, Megna Shah, Jeff Simmons, Sean Donegan, Marc De Graef, Yuejie Chi Abstract In spite of the utility of 3-D electron back-scattered diffraction (EBSD) microscopy, the data collection process can be time-consuming with serial-sectioning. Hence, it is natural to look at other modalities, such as polarized light (PL) data, to accelerate EBSD data collection, supplemented with shared information. Complementarily, features in chaotic PL data could even be enriched with a handful of EBSD measurements. To inherently learn the complex dynamics between EBSD and PL to solve these inverse problems, we use an unconditional multimodal diffusion model, motivated by progress in diffusion models for inverse problems. Although trained solely on synthetic data once, our model has strong generalizable capabilities on real data which can be low-resolution, noisy, corrupted, and misregistered. With inference-time scaling, we show gains in performance on a variety of objectives including grain boundary prediction, super-resolution, and denoising. With our model, we demonstrate that there is little difference from full resolution performance with only 25% (1/4 the resolution) of EBSD data and corrupted PL data. 1 Introduction Electron BackScatter Diffraction (EBSD) is a workhorse modality for modern microstructure characterization because of the richness of the information it provides. Since each pixel in the output orientation map is a result of a machine indexed full diffraction pattern, this makes serial data acquisition with EBSD a potential bottleneck in 3-D serial sectioning experiments, and much attention has been dedicated to improving the throughput of the technique, both in the form of faster and more sensitive detectors [ 11 ] and better indexing algorithms [ 48 ] . By contrast, polarized light (PL) microscopy is very fast, since it acquires an entire image in parallel, but the technique suffers from the fact that it can only be applied to materials with a non-cubic crystal structure; in optically uniaxial materials like hexagonal close-packed Ti, only the orientation of the crystallographic c -axis can be determined, leaving the orientation in the plane normal to this axis as a degree of freedom. For optically bi-axial materials this limitation does not apply. Nevertheless, it is tempting to consider a data fusion approach between minimal data acquisition with EBSD and the balance with PL microscopy in order to improve the overall throughput of microstructure characterization. This work presents a novel algorithm using generative artificial intelligence (AI), which applies inverse solvers with diffusion to recover features of an EBSD map (i.e., a corresponding field of orientation values at every pixel), from an input polarized light image. An EBSD map can be useful both for the explicit orientations of each grain in a polycrystalline material and for the simpler problem of just finding the boundaries of each grain. Both of these issues can be addressed from the results of this work. Figure 1 : Super-resolution using diffusion models. The multimodal (EBSD and PL) model is more robust as the observed resolution diminishes. Best viewed zoomed in. Generally, we aim to use fewer EBSD measurements, which are relatively more expensive to collect, in conjunction with a greater quantity of cheaper PL measurements to accomplish downstream objectives that can benefit more from both modalities than either alone . As we shall see, this is deeply rooted as an inverse problem. More concretely, the aim of solving inverse problems is to recover some underlying ground truth \bm{\mathcal{X}}^{\star} from observation \bm{\mathcal{Y}} with the following relationship: \displaystyle\bm{\mathcal{Y}}=f(\bm{\mathcal{X}}^{\star})+\bm{\mathcal{E}}, (1) where f is known as the forward model and \bm{\mathcal{E}} is measurement noise. Relating EBSD data as \bm{\mathcal{X}}^{\star} and PL data as \bm{\mathcal{Y}} in this manner means f=f_{\text{E$\mapsto$P}} is a complicated nonlinear and discontinuous function. To make matters worse, the inverse f_{\text{E$\mapsto$P}}^{-1} is ill-posed since there is information in EBSD that cannot be captured by PL [ 28 , 29 ] . However, we hypothesize at the slice level, spatial and macroscopic features in PL data can contain emergent information that is missing in pointwise low quality EBSD observations. Diffusion models [ 25 , 47 ] have been established as a highly expressive generative model to capture complex information in the data distribution in recent years. Extensively tested on natural image generation [ 35 , 36 , 38 ] , diffusion models have also shown efficacy as priors to solve inverse problems for natural [ 10 , 33 , 58 ] , medical [ 34 , 26 , 56 ] , and microscopy images [ 18 , 42 , 19 ] . Since diffusion models are learned from the data of interest directly, they are capable of capturing characteristic that are difficult for pre-determined, hand-crafted priors such as sparsity [ 2 , 6 ] or total variation [ 41 ] . Diffusion models also generate images from randomly initialized noise, and repeated inference from different initializations with an inverse solver can lead to different output reconstructions of \bm{\mathcal{X}}^{\star} from the same input observation \bm{\mathcal{Y}} . Hence, inverse solvers with diffusion models have the advantage that instead of producing one reconstruction per observation, they can capture distributions of possible reconstructions via repeated inference . Akin to Monte Carlo sampling from the posterior distribution, a set of output reconstructions provides measures of uncertainty, which are richer representations than a single predicted reconstruction. Though very successful in the domain of natural images, several technical challenges arise when we think about diffusion in our setting. One major challenge is that the forward model f=f_{\text{E$\mapsto$P}} and its gradients are complex nonlinear functions, limiting the use of many existing diffusion-based inverse problem solvers. To correctly guide the reconstruction with multimodal information, we reformulate the original problem of reconstructing just one modality from another into joint multimodal reconstruction via a multimodal diffusion model, following Efimov et al. [ 18 ] . By jointly learning the probability distribution of all modalities with a multimodal diffusion model, the new forward model becomes just a linear masking operator, f_{\text{EP}} . Consequently, the new reconstruction or inverse problem also becomes linear, bypassing the need to explicitly have a forward model between the two modalities. To illustrate the difference between unimodal and multimodal diffusion, let us adapt ( 1 ) for each case. Define perfect EBSD measurements, observed EBSD measurements, and EBSD measurement noise to be \bm{\mathcal{X}}^{\star}_{\text{E}} , \bm{\mathcal{Y}}_{\text{E}} , and \bm{\mathcal{E}}_{\text{E}} respectively, and similarly for PL ( \bm{\mathcal{X}}^{\star}_{\text{P}} , \bm{\mathcal{Y}}_{\text{P}} , and \bm{\mathcal{E}}_{\text{P}} ). The forward model in the unimodal case is \displaystyle\bm{\mathcal{Y}}_{\text{P}} \displaystyle=f_{\text{E$\mapsto$P}}(\bm{\mathcal{X}}^{\star}_{\text{E}})+\bm{\mathcal{E}}_{\text{P}}. (2) The multimodal case concatenates the modalities into \bm{\mathcal{X}}^{\star}_{\text{EP}}=[\bm{\mathcal{X}}^{\star}_{\text{E}}\quad\bm{\mathcal{X}}^{\star}_{\text{P}}] for the target, \bm{\mathcal{Y}}_{\text{EP}}=[\bm{\mathcal{Y}}_{\text{E}}\quad\bm{\mathcal{Y}}_{\text{P}}] for the measurement, and \bm{\mathcal{E}}_{\text{EP}}=[\bm{\mathcal{E}}_{\text{E}}\quad\bm{\mathcal{E}}_{\text{P}}] for the noise. Hence, the new multimodal forward model becomes \displaystyle\bm{\mathcal{Y}}_{\text{EP}} \displaystyle=f_{\text{EP}}(\bm{\mathcal{X}}^{\star}_{\text{EP}})+\bm{\mathcal{E}}_{\text{EP}}, (3) where f_{\text{EP}} is a masking operation such that we only partially observe EBSD images but fully observe PL images. In practice, f_{\text{EP}} can involve intentional choices like subsampling EBSD data to be at a lower resolution for faster data collection. By learning an unconditional multimodal diffusion model over both modalities, we implicitly capture the relationship between EBSD and PL data. Moreover, by learning an unconditional multimodal diffusion model as a joint prior on both modalities, the synthesis of information between EBSD and PL data can capture more holistic representations of the material than either alone. The second