The Role of Recurrency in Image Segmentation for Noisy and Limited Sample Settings

David Calhas, João Marques, Arlindo L. Oliveira · 2024

Read The Role of Recurrency in Image Segmentation for Noisy and Limited Sample Settings by David Calhas, João Marques, Arlindo L. Oliveira (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-15734.txt.

Introduction

David Calhas, João Marques, and Arlindo Oliveira published a paper in 2024 testing whether giving image-segmentation networks a memory loop helps when data are scarce and noisy. Brains are recurrent: they revisit and revise their first impressions. Most top vision models are one-shot, so the authors grafted three kinds of recurrence, self-organizing, relational, and memory-based, onto a standard segmentation model, each driven by minimizing an energy score.

Tests on artificial and medical images stressed the models with heavy noise and tiny training sets, exactly where revision should pay. It did not. The recurrent versions failed to beat their plain feed-forward base, and the authors report the negative result plainly. Recurrence alone, they conclude, is not the brain's secret; something more is doing the work.

Catalog record

CREATORDavid Calhas, João Marques, Arlindo L. Oliveira
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-15734

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

The Role of Recurrency in Image Segmentation for Noisy and Limited Sample Settings David Calhas, João Marques, Arlindo L. Oliveira Abstract The biological brain has inspired multiple advances in machine learning. However, most state-of-the-art models in computer vision do not operate like the human brain, simply because they are not capable of changing or improving their decisions/outputs based on a deeper analysis. The brain is recurrent, while these models are not. It is therefore relevant to explore what would be the impact of adding recurrent mechanisms to existing state-of-the-art architectures and to answer the question of whether recurrency can improve existing architectures. To this end, we build on a feed-forward segmentation model and explore multiple types of recurrency for image segmentation. We explore self-organizing, relational, and memory retrieval types of recurrency that minimize a specific energy function. In our experiments, we tested these models on artificial and medical imaging data, while analyzing the impact of high levels of noise and few-shot learning settings. Our results do not validate our initial hypothesis that recurrent models should perform better in these settings, suggesting that these recurrent architectures, by themselves, are not sufficient to surpass state-of-the-art feed-forward versions and that additional work needs to be done on the topic. 1 Introduction In semantic segmentation , given an input, the corresponding segmentation mask associates a class to each pixel of the image. The U-Net architecture is widely used in image segmentation [ 1 , 2 , 3 ] and can obtain good predictions in many different settings. Although the U-Net architecture has some similarities to the known structure of the visual cortex [ 4 ] , it is critically different in that it is unable to use feedback from higher levels to improve its prediction. This is also the case for many state-of-the-art models that do not have a feedback cycle in their computational graph [ 5 ] . It is believed that the brain can be best described as a dynamical system, containing an internal state that represents its belief about the world [ 6 ] . When the stimuli are different from what is expected given its internal state, it processes the input and corrects its expectation [ 7 ] . From to static dynamic \Rightarrow \Rightarrow \mathbf{r} \mathbf{r} t t 0 0 Figure 1 : This figure illustrates the fact that a system with feedback connections is able to change its decision with time. For that, at least one cycle is needed in the computational graph. In this work, we study the hypothesis that there is a fundamental limitation on current machine learning models because they are not able to process their decisions and use intermediate decisions to move towards a more correct/certain one. To this end, we apply recurrency to a state-of-the-art segmentation model and explore its application on artificial and medical images. Our contributions are the following: • we describe how three recurrent types of paradigms used in this work (self-organizing maps, conditional random fields, and modern Hopfield networks, described in section 3 ) can be added or appended at any level of the U-Net. Specifically, we use the EfficientUnet++ [ 8 ] backbone as the base model, which is based on the U-Net++ architecture, with an EfficientNet [ 9 ] backend; • we propose and make available a new dataset with irregular polygon shapes (see section 4.1.1 ) and define two experimental settings, the noisy and limited sample size cases, to analyze feed-forward and recurrent models (see sections 4.2 and 4.3 ); • using artificially generated data, we show that self-organizing recurrency is the best type of model for noisy settings (see section 5 ) and recurrency is preferred to feed forward in limited sample size settings (see sections 5 and 6 ); • we present experimental results showing that none of the models used has significant success in a real world medical imaging dataset (see sections 4.1.2 , 5 , and 6 ). 2 Problem description Let \mathbf{X}=\{\mathbf{x}_{1},\dots,\mathbf{x}_{N}\} be a set of images (two-dimensional representations) and \mathbf{Y}=\{\mathbf{y}_{1},\dots,\mathbf{y}_{N}\} the corresponding set of pixelwise labels, with \mathbf{x}_{i}\in\mathbb{R}^{H\times W\times C}\wedge\mathbf{y}_{i}\in\mathbb{R}^{H\times W\times L},\forall i\in\{1,\dots,N\} , where H and W define the dimension of the image, C the number of channels of the original representation ( C=3 for RGB), and L the number of classes. Our goal is to compute the parameters \theta of a function f , such that \theta=\mbox{argmin}_{\theta}\frac{1}{N}\sum_{i}^{N}\mathcal{L}\left(f(\mathbf{x}_{i};\theta),\mathbf{y}_{i}\right), (1) where \mathcal{L} is an objective function, e.g. cross-entropy. 3 Recurrent Models Gradient descent is a form of recurrency, where we updates are performed recurrently as illustrated in figure 2 . s \frac{\partial E}{\partial s} compute energy E(.) derive w.r.t. s Update with s-\alpha\frac{\partial E}{\partial s} Figure 2 : An illustration of recurrency, where the dynamics are governed by gradient descent, which implies the minimization of an energy function E w.r.t. a state s . Consider a state s that evolves in time according to the dynamics \tau\frac{ds}{dt}=-\frac{\partial E}{\partial s(t)}, (2) where the evolution is governed by an energy function E that is being minimized. Then we can update a state s with gradient descent as s(t+1)=s(t)-\alpha\frac{\partial E}{\partial s(t)}, (3) where \alpha is the learning rate. The goal of this study is to explore multiple forms of energy-based recurrency and bridge its dynamics with human neuronal processing, such as self-organizing maps [ 10 ] , conditional random fields [ 11 ] , and Hopfield networks [ 12 ] . 3.1 Self-organizing map Originally proposed by Kohonen [10] , a self-organizing map is a neural network architecture that learns the topology of a space and is inspired in the way some neuronal cells respond to orientation [ 13 ] . A self-organizing map is defined by a set of nodes, \mathcal{V}\in\mathbb{R}^{M\times M\times L} , that define a square lattice. The lattice can be rectangular if we define \mathcal{V}\in\mathbb{R}^{H\times W\times L}withH\neq W . Each node \mathbf{v}\in\mathbb{R}^{L} is a feature vector to be optimized according to the algorithm. A B C D E Figure 3 : A self-organizing map square lattice. Given a point \mathbf{x} , we first compute the distance to all nodes and select the closest node, \mathbf{v}^{*}=argmin_{\mathbf{v}\in\mathcal{V}}||\mathbf{x}-\mathbf{v}||^{2}_{2} . The algorithm proceeds to update the closest node as \mathbf{v}^{*}=\mathbf{v}^{*}+\alpha\left(\mathbf{x}-\mathbf{v}^{*}\right), (4) where \alpha is the learning rate. The same update, \mathbf{v}^{\prime}=\mathbf{v}^{\prime}+\alpha\left(\mathbf{x}-\mathbf{v}^{\prime}\right) , is done for any node \mathbf{v}^{\prime} that is a neighbour of \mathbf{v}^{*} . For a more compact notation, let \mathcal{N}:\mathbb{R}^{2}\to\mathbb{R}^{5\times L} be the neighbor function of a node \mathbf{v} , that given its lattice coordinates i , returns the 4 direct neighbors and \mathbf{v} itself. For the sake of simplicity, we interchange between coordinates giving i and the representation of the node \mathbf{v} as input to \mathcal{N} , such that the neighbors \mathbf{v} are \mathcal{N}(i)=\mathcal{N}(\mathbf{v}) . Then the update rule for the set of neighbors is \mathcal{N}(\mathbf{v}^{*})=\mathcal{N}(\mathbf{v}^{*})+\alpha\left(\mathbf{x}-\mathcal{N}(\mathbf{v}^{*})\right). (5) Note that this imposes a competitive dynamic for the nodes when iterating through the points of the space we are learning from. Space regions with more density are going to have more nodes nearby, while regions with less density are going to have fewer nodes. Regions with no density will not have nodes. Self-organizing map energy. The energy of a map \mathcal{V}\in\mathbb{R}^{M\times M\times C} given a set of instances \mathbf{Y} is defined as

Read the complete text