Spline refinement with differentiable rendering
Frans Zdyb, Albert Alonso, Julius B. Kirkegaard · 2025
Read Spline refinement with differentiable rendering by Frans Zdyb, Albert Alonso, Julius B. Kirkegaard (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-14525.txt.
Introduction
Frans Zdyb, Albert Alonso, and Julius Kirkegaard sharpen the tracing of thin worm shapes in microscope images in a 2025 paper. Thin, overlapping forms are hard to trace well. Coordinate-based tools find them with care but draw their curves less well than pixel tools. The team adds a clean-up step that draws each curve and checks the drawing against the real image, then nudges the curve to fit.
The step needs no training data and can swap in for a classic curve tool. It lifts curve quality and holds up when image style shifts from the lab norm. Tests used a small worm that is a workhorse of drug and health research. On those images, the method joined the strengths of both older styles. It also shrank the gap between fake test images and real ones.
Catalog record
| CREATOR | Frans Zdyb, Albert Alonso, Julius B. Kirkegaard |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 18 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-14525
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Spline refinement with differentiable rendering Frans Zdyb, Albert Alonso, Julius B. Kirkegaard Abstract Detecting slender, overlapping structures remains a challenge in computational microscopy. While recent coordinate-based approaches improve detection, they often produce less accurate splines than pixel-based methods. We introduce a training-free differentiable rendering approach to spline refinement, achieving both high reliability and sub-pixel accuracy. Our method improves spline quality, enhances robustness to distribution shifts, and shrinks the gap between synthetic and real-world data. Being fully unsupervised, the method is a drop-in replacement for the popular active contour model for spline refinement. Evaluated on C. elegans nematodes, a popular model organism for drug discovery and biomedical research, we demonstrate that our approach combines the strengths of both coordinate- and pixel-based methods. 1 Introduction Automated detection and tracking methods form the basis for studying physiological behavior, drug effects, and motility in model organisms, enabling quantification of assays in biomedical research [ 10 ] . Large animal models, such as mice and rats, are widely used for preclinical drug testing, allowing the quantification of complex physiological responses [ 2 , 3 ] . Zebrafish provide a versatile system for high-throughput screening, as their transparent larvae enable real-time observation of drug-induced behavioral changes [ 25 , 19 ] . Fruit flies with their rapid generation time, serve as powerful genetic models for studying neural function and the effects of drugs on behavior [ 30 , 24 ] . The detection and tracking of the behavior of these higher-order organisms are typically facilitated by landmark-based deep learning methods [ 21 , 18 , 20 ] , which enable precise monitoring of limb movements, wing beats, and overall locomotion patterns. At the other scale, small model organisms such as the nematode Caenorhabditis elegans offer a simple and cost-effective model for studying neuroactive compounds and locomotion-based toxicity [ 23 , 11 , 26 , 15 ] . Due to their visually homogeneous appearance, landmark-based deep learning methods [ 18 , 20 ] are not ideal for high-throughput analysis of C. elegans assays. Instead, pixel-based segmentation methods can often estimate nematode poses extremely accurately [ 27 , 13 , 22 , 5 ] . However, these methods struggle when the images are at low resolution and worms appear thin, or when worms are occluded either due to self-overlap or due to imaging at high densities. These challenges are overcome by spline-predicting methods [ 9 , 4 , 31 ] , which can track worms during self-coiling and at extreme densities. These methods operate under a trade-off, however: Their accuracy is not pixel-level-accurate and are highly adversely affected when applied for out-of-distribution inference. Here, we propose an unsupervised model that uses differentiable rendering [ 16 ] to refine spline predictions for pixel-level-accuracy. Our method optimizes a reconstructed microscopy image, adjusting the background, textures, colors, spline shapes, and widths via differentiable rendering. We compare this to the active contour model [ 14 ] , an unsupervised refinement approach, which in contrast relies solely on feedback from the spline centerline, and therefore requires careful image preprocessing to work. This model works per-spline and therefore does not generalize well to overlapping splines. In contrast, in our approach, all splines are fitted simultaneously to an image. We note that supervised deep learning refinement methods [ 17 , 29 ] are always faster and have greater detection accuracy. However, these rely on pixel-perfect labels, which often are unavailable. Such labels can be generated synthetically [ 4 ] , where ground truth is fully controlled. In that case, the challenge is that synthetic data introduces a gap between simulation and reality, as model assumptions and textures differ from real-world conditions. Thus supervised refinement is possible for synthetic data, but bridging the reality-gap introduces the need for unsupervised refinement methods, such as the active contour model or the approach taken here. 2 Method The input to our model is one or more (potentially overlapping) splines that need to be refined. These could, for instance, be the output of a deep learning detection model. Our refinement approach, illustrated in Fig. 1 , reconstructs the input image through differentiable rendering by optimizing parameters to minimize the difference between the input and reconstructed images. Our primary concern are the parameters that define the splines, but the optimization is performed jointly with the background colors and texture of the image. Further, as our approach is fully differentiable, it is simple to regularize, ensuring e.g. smoothly curved spline (curvature regularization) or minor changes to total spline length. Figure 1: Overview of the spline refinement method using differentiable rendering. The process starts with initial spline guesses, potentially from deep learning models. Through differentiable rasterization (orange arrows), an image is rendered from spline control points and, background and texture adjustments. The loss \mathcal{L} is computed by comparing the rendered and input images, and gradients (blue arrows) update trainable parameters (bottom). At convergence, the refined spline (bottom right) accurately reconstructs the input shape (top right), as well as provides auxiliary data e.g. spline thickness. We model the spline placement, spline fitting and rendering, along with background and texture generation, as an end-to-end differentiable program. A set of parameters define the location, shape, and width of each spline. Specifically, for each set of K knots \{\mathbf{x}\}^{K}=\{(x_{1},y_{1},w_{1}),...,(x_{K},y_{K},w_{K})\} we fit three natural cubic splines describing parametric curves of varying width, \boldsymbol{r}^{n}:[0,1]\rightarrow[0,\textsc{res}]^{2}\times[0,1]=(r_{x}(s),r_{y}(s),r_{w}(s)), (1) where (r_{x},r_{y}) are coordinates in the image. r_{w} scales a global width parameter W which is shared for all splines. Additional parameters define background properties such as brightness, contrast and color gradient, and the texture of the image (Fig. 1 ). We have developed the method towards identifying splines whose texture is approximately constant along the spline, and we therefore learn only one (shared) blob texture. 2.1 Differentiable rendering To render the reconstruction image, we sample a large set of interpolating (x,y,w) points uniformly along each spline. Each (x,y) point is rendered onto a separate image by differentiable resampling of a learned blob image, which is scaled by W\cdot w . The radially-symmetric blob image is produced by evaluating f(z) in a small grid \{(x_{ij},y_{ij})\} , where z=\exp(-\sqrt{x^{2}+y^{2}}) and f is a learned 1D spline function. Summation over these scaled single-blob images produces a rendered spline. The images generated for each spline are combined using summation and max operations, with a parameter controlling the interpolation between the two. This allows for matching how overlapping splines interact visually. A background is generated and added, and the result is convolved with a learned 3\times 3 convolutional filter, which allows for brightness/contrast matching and minor blurring. 2.2 Optimization Our loss function is the reconstruction error between the input image \mathbf{Y} and the generated image \hat{\mathbf{Y}} , \mathcal{L}=||\mathbf{Y}-\hat{\mathbf{Y}}||^{2}. (2) Akin to the active contour model [ 14 ] , we regularize the optimization with terms that exploit prior information (barred parameters): \displaystyle\mathcal{L}_{\text{reg}}=\ \lambda_{1}\sum_{i}^{N}(\ell_{i}-\bar{\ell}_{i})^{2}+\lambda_{2}\sum_{i}^{N}C_{i}+\lambda_{3}\sum_{n}(\min_{k}w_{k}-\bar{w})^{2}+\lambda_{4}(W-\bar{W})^{2}. (3) Respectively, these terms regularize the spline length \ell_{i} using the initial length \bar{\ell}_{i} from the input knots, the curvature C_{i} (smoothness) of the spline, and the minimum ( \min w_{k}W ) and maximum ( W ) point width. Figure 2: Robustness analysis of the proposed method on SOSB (64 frames). Average DTW after transformations: rotation (left), translation (middle), and scaling (right). The initial guess