The FlEye camera: Sampling the joint distribution of natural scenes and motion

Charles J. Edelson, Paul Smith, Sima Setayeshgar, William Bialek, Rob R. de Ruyter van Steveninck · 2024

Read The FlEye camera: Sampling the joint distribution of natural scenes and motion by Charles J. Edelson, Paul Smith, Sima Setayeshgar, William Bialek, Rob R. de Ruyter van Steveninck (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-21081.txt.

Description

A 2024 arXiv research paper by Charles J. Edelson, Paul Smith, Sima Setayeshgar, William Bialek, Rob R. de Ruyter van Steveninck. To make efficient use of limited physical resources, the brain must match its coding and computational strategies to the statistical structure of input signals.

Catalog record

CREATORCharles J. Edelson, Paul Smith, Sima Setayeshgar, William Bialek, Rob R. de Ruyter van Steveninck
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE65 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-21081

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

The FlEye camera: Sampling the joint distribution of natural scenes and motion Charles J. Edelson, Paul Smith, Sima Setayeshgar, William Bialek, Rob R. de Ruyter van Steveninck Abstract To make efficient use of limited physical resources, the brain must match its coding and computational strategies to the statistical structure of input signals. An attractive testing ground for these principles is the problem of motion estimation in the fly visual system: we understand the optics of the compound eye, have a quantitative description of input signals and noise from the retina, and can record from output neurons that encode estimates of different velocity components. Furthermore, recent work provides a nearly complete wiring diagram of the intervening circuitry. What is missing is a characterization of the visual signals and motions that flies encounter in a natural context. We attack this directly with the development of a specialized camera that matches the high temporal resolution, optical properties, and spectral sensitivity of the fly’s eye; inertial motion sensors provide ground truth about rotations and translations through the world. We describe the design, construction, and performance characteristics of this FlEye camera. To illustrate the opportunities created by this instrument we use data on movies and motion to construct optimal local motion estimators that can be compared with the responses of the fly’s motion sensitive neurons. I Introduction It is an old idea that the algorithms for information processing in the brain should be adapted to the structure of physical signals in the world [ 1 ] . These ideas became more precise in the mid–twentieth century with the development of information theory [ 2 ] and theories for signal detection and estimation [ 3 , 4 ] . These approaches are probabilistic, with strong connections to statistical physics [ 5 ] . As Barlow emphasized, this means that neural coding and computation will be driven by the distribution of signals in the natural sensory environment [ 6 ] . The resulting ideas of efficient representation and inference continue to have impact on thinking about the brain and about the physics of life more generally [ 7 , 8 ] . The insect visual system provides an important testing ground for these theories [ 9 , 10 , 11 , 12 , 13 ] . Despite their small size and “basic” morphologies, flies and other flying insects are able to perform complex aerial acrobatics with a precision and ease that would make even the most skilled of human pilots jealous. This aerial prowess has led to an interest in understanding how flies both plan and execute their flights. One avenue to answering this question is determining how a fly senses the world around it and then uses its sensory signals to control its many gravity defying feats. This approach has served as a guiding principle for decades of research, revealing a rich collection of sensory signals used in a variety of clever ways to control flight [ 14 , 15 , 16 , 17 , 18 , 19 ] . The input-output relationships for a number of these systems have been studied in detail, and in some cases this has been explored down to the level of plausible neural control mechanisms for specific flight behaviors [ 20 , 21 ] . Of its many sensory systems, the fly’s visual system may be the most iconic. For the human observer, insect compound eyes are among the most foreign features of their morphology, and indeed the fly visual system operates in a regime quite different from our own. The optics of the compound eye provide much lower spatial resolution than the lens of the human eye, but fly photoreceptors have much wider temporal bandwidths, giving the insects greater time resolution and allowing faster behavioral responses [ 22 , 23 ] . The submillimeter focal length of the fly’s optics would be inconvenient for humans, but allows flies to see what is at the end of their legs, useful for animals that walk on their food [ 24 ] . The fly visual system also is attractive for its experimental accessibility. We understand the diffraction–limited optics of the compound eye [ 25 , 26 , 24 , 27 ] , and extensive quantitative measurements have been made on the signals and noise in photoreceptor cells and their immediate synaptic targets, the large monpoloar cells (LMCs). Some of the earliest recordings of neural responses to single photons were in insect photoreceptors [ 28 ] , and LMCs act as nearly ideal photon counters up to rates of \sim 10^{6}\,{\rm s}^{-1} before saturating [ 29 , 22 ] . Near the output of the fly’s visual system, in the lobula plate, is a dense collection of large identified neurons which respond to optical flow patterns and provide motion information that guides flight trajectory planning, pose correction, and spatial mapping [ 30 , 31 , 32 , 33 , 34 , 35 , 36 ] . Very recent work in fruit flies has mapped the connectivity in all the neurons leading from photoreceptors to the lobula plate, and more [ 37 ] . The responses of motion-sensitive neurons in the lobula plate adapt to changes in the statistical structure of simplified visual inputs, including mean light levels, the dynamic range and correlation times of light intensity variations, the mean and variance of movement velocities [ 12 , 38 , 39 , 40 ] . In some cases it has been possible to show that these adaptations serve to maximize information transfer [ 12 , 41 ] . Related to this, under some conditions the motion estimates encoded by single cells in the lobula plate reach a precision close to the physical limits imposed by diffraction blur in the compound eye and noise in the photoreceptors [ 42 , 43 ] . All of this encourages us to think that the physical principles of efficient representation and inference are relevant, at least in this small corner of the fly’s brain. What is missing from this body of work is a compelling characterization of the visual inputs and movements that flies encounter in nature. Building on earlier efforts, we have made a direct attack on the problem of characterizing the joint distribution of visual inputs and motion statistics by designing and constructing a camera that samples natural scenes in a manner similar to that of a fly’s eye [ 44 ] . The FlEye camera, as we have affectionately dubbed it, accomplishes this through a combination of careful optical and electronic design choices, enabling it to achieve a signal-to-noise ratio orders of magnitude better than the fly photoreceptor, thus giving us nearly the true light intensities that a fly would experience. The FlEye camera is equipped with an inertial motion unit (IMU) to allow for simultaneous sampling of motion trajectories. Thus, recording a stroll through the woods with the FlEye camera is equivalent to collecting samples from the joint distribution of fly-like visual inputs and motion trajectories. With the camera, we will be able to assemble a large database of movies and motion, sampled from a fly eye’s view, providing a resource with which to address a number of questions. In this article we

Read the complete text