Applying Medical Imaging Tractography Techniques to Painterly Rendering of Images

Alberto Di Biase · 2025

Read Applying Medical Imaging Tractography Techniques to Painterly Rendering of Images by Alberto Di Biase (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-00702.txt.

Introduction

Alberto Di Biase borrows a medical imaging technique and turns it into a paintbrush. Doctors use diffusion tensor imaging and tractography to trace fibrous tissue by following local orientation through the body. Painterly rendering faces the same geometry problem: strokes should follow the forms inside an image, the way an artist's hand does. Di Biase maps one field onto the other, explicitly and in tractography's own language.

A tractography algorithm lays down brush strokes that mimic a painting process. For local direction, the structural tensor of the image replaces the gradient, giving richer orientation information. Portraits and general images demonstrate the method, and the paper discusses the parallels between fiber tracking and stroke placement at length. Code is public, inviting artists and researchers to experiment with the crossover.

Catalog record

CREATORAlberto Di Biase
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE20 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-00702

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Applying Medical Imaging Tractography Techniques to Painterly Rendering of Images Alberto Di Biase Abstract Doctors and researchers routinely use diffusion tensor imaging (DTI) and tractography to visualize the fibrous structure of tissues in the human body. This paper explores the connection of these techniques to the painterly rendering of images. Using a tractography algorithm the presented method can place brush strokes that mimic the painting process of human artists, analogously to how fibres are tracked in DTI. The analogue to the diffusion tensor for image orientation is the structural tensor, which can provide better local orientation information than the gradient alone. I demonstrate this technique in portraits and general images, and discuss the parallels between fibre tracking and brush stroke placement, and frame it in the language of tractography. This work presents an exploratory investigation into the cross-domain application of diffusion tensor imaging techniques to painterly rendering of images. All the code is available at https://github.com/tito21/st-python . Figure 1 : Example of stylization using tractography. 1 Introduction An invaluable tool for visualizing the fibrous structure of tissue in the human body is tractography. Tractography algorithms use the diffusion tensor to determine the local orientation of fibres in tissue, and then place streamlines along these orientations to create a 3D representation of the fibre pathways. This problem is surprisingly similar to the placing coherent brush strokes in painterly rendering of images. This paper explores this similarity in a cross-domain framing, applying tractography techniques to place brush strokes in images. I present the parallels between the two problems and introduce the computer graphics community to the mathematical framework of tractography highlighting the potential for cross-disciplinary collaboration. The strokes of a painter follow the contours and edges of the image subject. This motivates the need for the extraction of gradient based quantities. In my algorithm, this is done by the eigen decomposition of the structural tensor (analogue to the diffusion tensor). Next, the extracted vectors are followed with a simple tractography algorithm and then rendered with an artistic brush stroke. An example of an image stylized with the proposed method is shown in figure 1 . The article is divided as follows: first, an overview of the diffusion tensor imaging including a review of the tractography algorithm is given, followed by a brief discussion of existing painterly rendering techniques. Next, a full description of the proposed algorithm is presented. After that, results from the NPRportrait and NPRgeneral are presented followed by discussions and conclusions. 1.1 Primer on Diffusion Tensor Imaging and Tractography DTI is an image modality in Magnetic Resonance Imaging (MRI) that measures the diffusion of water across the body [ 15 , 2 ] . One of the main applications of this technique is to visually represent the nerve tracts of the brain (and other organs), helping doctors and researchers to study neurological disease [ 11 ] . In DTI the acquired images are later used to fit the diffusion tensor of water for that voxel (pixel). This is a 3\times 3 symmetric matrix that quantifies how much water can diffuse in each direction. Because water in biological tissue is restricted by cell membranes the shape of the tensor is elongated in areas were a predominant orientation is present such as in the fibre tracts. In this case the tensor is said to be anisotropic. In the case of free water, there is no preferred orientation and the tensor is isotropic. To measure the amount of anisotropy in the tensor several metrics exist. The most common being the fractional anisotropy (FA) which is a scalar value ranging from 0 (completely isotropic) to 1 (completely anisotropic). An eigenvalue decomposition is performed on the tensor to find the predominant orientation. Larger eigenvalues indicate more diffusion in the direction of the associated eigenvector. The eigensystem is generally sorted in descending order and the first eigenvector is followed to create streamlines that represent fibres in the tissue. This process is called tractography [ 15 ] . First introduced by Basser, et al. in 2000, [ 1 ] tractography consists of following the primary orientation of the diffusion tensor forming tracts that can be visualized in 3D. The most basic form of this is done by solving this differential equation: \frac{d\vec{r}(s)}{ds}=\vec{u_{1}}(\vec{r}(s)) With \vec{r}(s) being the tract and \vec{u_{1}} the primary eigenvector (in diffusion the one associated with the largest eigenvalue, and in structural tensor the smallest). Solving this equation can be done by the well established Runge–Kutta methods [ 8 ] (RK2(3) is used in this work). Solving the presented differential equation was the first method proposed for tractography but more advanced techniques exist. For example, probabilistic tractography methods that take into account uncertainty in the tensor estimation [ 3 ] . 1.2 Related Works in image stylization In the past other methods for image stylization have used the normal to the gradient to orient the brush strokes [ 13 , 9 ] . These methods also achieve a painterly effect by following the direction normal to the gradient. The algorithms to place the brush strokes are determined by a simple procedure that can be viewed as solving the tractography differential equation with Euler’s method. The structural tensor as a measure of orientation produces smoother brush strokes without the need to do additional post-processing. Additionally, it is more robust to noise and can provide more information about the image structure. Metrics such as the coherence [ 10 ] of the tensor can be used to control the brush stroke width and length. In animation the structural tensor can also be used in 3D to orient the brush strokes in a coherent manner [ 7 ] . To the best of my knowledge the structural tensor has not been used for image stylization in the literature. 2 Methods 2.1 Structural Tensor For non diffusion images an analogue to the diffusion tensor is the structural tensor [ 12 , 4 ] . This tensor captures the local changes in intensity of the image. Areas with flat texture will exhibit an isotropic tensor, while areas with edges or linear structures will exhibit an anisotropic tensor oriented along the structure. As with the diffusion tensor an eigenvalue decomposition is performed to extract the local orientations. The structural tensor is defined by the tensor that minimizes the changes in intensities ( V ) when moving in an arbitrary direction. The general solution is S=\sum\nabla V(\nabla V)^{T} , where \nabla V is the gradient of the image and the summation is over a neighbourhood of the image. In practice the structure tensor is calculated using the following procedure: 1. Calculate the derivatives I_{x} and I_{y} using a Sobel filter or a derivative of Gaussian filter. (in this work a derivative of Gaussian with standard deviation \sigma=1.0 is used). 2. Average the products of the gradients with a Gaussian filter with a standard deviation \rho (in this work \rho=1.0 is used): ( \overline{I_{ij}}=K_{\rho}*(I_{i}I_{j}) ). 3. Build the tensor as: S=\begin{bmatrix}\overline{I_{xx}}&\overline{I_{xy}}\\ \overline{I_{xy}}&\overline{I_{yy}}\end{bmatrix} Again, if there are areas of uniform intensities in one direction, the tensor will be oriented in that direction and is anisotropic. The other case, an isotropic tensor would be an area with uniform intensities in all directions. The orientation of the

Read the complete text