Early jump-out corner detectors

Two corner detectors are presented, one of which works by testing similarity of image patches along the contour direction to detect curves in the image contour, and the other of which uses direct estimation image curvature along the contour direction. The operators are fast, robust to noise, and self-thresholding. An interpretation of the Kitchen-Rosenfeld corner operator is presented which shows that this operator can also be viewed as the second derivative of the image function along the edge direction.

Neighborhood coherence and edge based approaches to film scene extraction

In order to enable high-level semantics-based video annotation and interpretation, we tackle the problem of automatic decomposition of motion pictures into meaningful story units, namely scenes. Since a scene is a complicated and subjective concept, we first propose guidelines from film production to determine when a scene change occurs in film. We examine different rules and conventions followed as part of Film Grammar to guide and shape our algorithmic solution for determining a scene boundary. Two different techniques are proposed as new solutions in this paper. Our experimental results on 10 full-length movies show that our technique based on shot sequence coherence performs well and reasonably better than the color edges-based approach.

Local energy, the pre-envelope, and filter resolution

We examine the construction of new filters for computing local energy, and compare these filters with the Gabor filters and the three-point-filter of Venkatesh [l]. Further, we demonstrate that the effect of convolution with complex Gabor filters is to band-pass (with some differentiating effect) and compute the local energy of the result. The magnitude of the resulting local energy is then used to detect features [2], [3] (step features, texture etc.), and the phase is used to classify the detected features [l], [4] or provide disparity information for stereo [5] and motion work [6], [7]. Each of these types of information can be obtained at multiple resolutions, enabling the use of course to fine strategies for computing disparity, and allowing the discrimination of image textures on the basis of which parts of the Fourier domain they dominate [8], [9].

Multiple-object tracking in cluttered and crowded public spaces

This paper addresses the problem of tracking moving objects of variable appearance in challenging scenes rich with features and texture. Reliable tracking is of pivotal importance in surveillance applications. It is made particularly difficult by the nature of objects encountered in such scenes: these too change in appearance and scale, and are often articulated (e.g. humans). We propose a method which uses fast motion detection and segmentation as a constraint for both building appearance models and their robust propagation (matching) in time. The appearance model is based on sets of local appearances automatically clustered using spatio-kinetic similarity, and is updated with each new appearance seen. This integration of all seen appearances of a tracked object makes it extremely resilient to errors caused by occlusion and the lack of permanence of due to low data quality, appearance change or background clutter. These theoretical strengths of our algorithm are empirically demonstrated on two hour long video footage of a busy city marketplace.

An integrative framework for the appraisal of coloration in nature

The world in color presents a dazzling dimension of phenotypic variation. Biological interest in this variation has burgeoned, due to both increased means for quantifying spectral information and heightened appreciation for how animals view the world differently than humans. Effective study of color traits is challenged by how to best quantify visual perception in nonhuman species. This requires consideration of at least visual physiology but ultimately also the neural processes underlying perception. Our knowledge of color perception is founded largely on the principles gained from human psychophysics that have proven generalizable based on comparative studies in select animal models. Appreciation of these principles, their empirical foundation, and the reasonable limits to their applicability is crucial to reaching informed conclusions in color research. In this article, we seek a common intellectual basis for the study of color in nature. We first discuss the key perceptual principles, namely, retinal photoreception, sensory channels, opponent processing, color constancy, and receptor noise. We then draw on this basis to inform an analytical framework driven by the research question in relation to identifiable viewers and visual tasks of interest. Consideration of the limits to perceptual inference guides two primary decisions: first, whether a sensory-based approach is necessary and justified and, second, whether the visual task refers to perceptual distance or discriminability. We outline informed approaches in each situation and discuss key challenges for future progress, focusing particularly on how animals perceive color. Given that animal behavior serves as both the basic unit of psychophysics and the ultimate driver of color ecology/evolution, behavioral data are critical to reconciling knowledge across the schools of color research.

Artificial selection for food colour preferences

Colour is an important factor in food detection and acquisition by animals using visually based foraging. Colour can be used to identify the suitability of a food source or improve the efficiency of food detection, and can even be linked to mate choice. Food colour preferences are known to exist, but whether these preferences are heritable and how these preferences evolve is unknown. Using the freshwater fish Poecilia reticulata, we artificially selected for chase behaviour towards two different-coloured moving stimuli: red and blue spots. A response to selection was only seen for chase behaviours towards the red, with realized heritabilities ranging from 0.25 to 0.30. Despite intense selection, no significant chase response was recorded for the blue-selected lines. This lack of response may be due to the motion-detection mechanism in the guppy visual system and may have novel implications for the evolvability of responses to colour-related signals. The behavioural response to several colours after five generations of selection suggests that the colour opponency system of the fish may regulate the response to selection.

Reconstruction from image correspondences

A single picture provides a largely incomplete representation of the scene one is looking at. Usually it reproduces only a limited spatial portion of the scene according to the standpoint and the viewing angle, besides it contains only instantaneous information. Thus very little can be understood on the geometrical structure of the scene, the position and orientation of the observer with respect to it remaining also hard to guess. When multiple views, taken from different positions in space and time, observe the same scene, then a much deeper knowledge is potentially achievable. Understanding inter-views relations enables construction of a collective representation by fusing the information contained in every single image. Visual reconstruction methods confront with the formidable, and still unanswered, challenge of delivering a comprehensive representation of structure, motion and appearance of a scene from visual information. Multi-view visual reconstruction deals with the inference of relations among multiple views and the exploitation of revealed connections to attain the best possible representation. This thesis investigates novel methods and applications in the field of visual reconstruction from multiple views. Three main threads of research have been pursued: dense geometric reconstruction, camera pose reconstruction, sparse geometric reconstruction of deformable surfaces. Dense geometric reconstruction aims at delivering the appearance of a scene at every single point. The construction of a large panoramic image from a set of traditional pictures has been extensively studied in the context of image mosaicing techniques. An original algorithm for sequential registration suitable for real-time applications has been conceived. The integration of the algorithm into a visual surveillance system has lead to robust and efficient motion detection with Pan-Tilt-Zoom cameras. Moreover, an evaluation methodology for quantitatively assessing and comparing image mosaicing algorithms has been devised and made available to the community. Camera pose reconstruction deals with the recovery of the camera trajectory across an image sequence. A novel mosaic-based pose reconstruction algorithm has been conceived that exploit image-mosaics and traditional pose estimation algorithms to deliver more accurate estimates. An innovative markerless vision-based human-machine interface has also been proposed, so as to allow a user to interact with a gaming applications by moving a hand held consumer grade camera in unstructured environments. Finally, sparse geometric reconstruction refers to the computation of the coarse geometry of an object at few preset points. In this thesis, an innovative shape reconstruction algorithm for deformable objects has been designed. A cooperation with the Solar Impulse project allowed to deploy the algorithm in a very challenging real-world scenario, i.e. the accurate measurements of airplane wings deformations.

Methodologies for visual correspondence

Visual correspondence is a key computer vision task that aims at identifying projections of the same 3D point into images taken either from different viewpoints or at different time instances. This task has been the subject of intense research activities in the last years in scenarios such as object recognition, motion detection, stereo vision, pattern matching, image registration. The approaches proposed in literature typically aim at improving the state of the art by increasing the reliability, the accuracy or the computational efficiency of visual correspondence algorithms. The research work carried out during the Ph.D. course and presented in this dissertation deals with three specific visual correspondence problems: fast pattern matching, stereo correspondence and robust image matching. The dissertation presents original contributions to the theory of visual correspondence, as well as applications dealing with 3D reconstruction and multi-view video surveillance.

Computation of direction selectivity in retinal starburst amacrine cell dendrites – studied using electrophysiological recordings and two-photon imaging

Neuronal circuits in the retina analyze images according to qualitative aspects such as color or motion, before the information is transmitted to higher visual areas of the brain. One example, studied for over the last four decades, is the detection of motion direction in ‘direction selective’ neurons. Recently, the starburst amacrine cell, one type of retinal interneuron, has emerged as an essential player in the computation of direction selectivity. In this study the mechanisms underlying the computation of direction selective calcium signals in starburst cell dendrites were investigated using whole-cell electrical recordings and two-photon calcium imaging. Analysis of the somatic electrical responses to visual stimulation and pharmacological agents indicated that the directional signal (i) is not computed presynaptically to starburst cells or by inhibitory network interactions. It is thus computed via a cell-intrinsic mechanism, which (ii) depends upon the differential, i.e. direction selective, activation of voltage-gated channels. Optically measuring dendritic calcium signals as a function of somatic voltage suggests (iii) a difference in resting membrane potential between the starburst cell’s soma and its distal dendrites. In conclusion, it is proposed that the mechanism underlying direction selectivity in starburst cell dendrites relies on intrinsic properties of the cell, particularly on the interaction of spatio-temporally structured synaptic inputs with voltage-gated channels, and their differential activation due to a somato-dendritic difference in membrane potential.

Untersuchungen zum Sehen von bewegten Objekten beim Goldfisch (Carassius auratus)

In der vorliegenden Arbeit wurde das Objektbewegungssehen des Goldfischs betrachtet. Zuerst musste eine geeignete Methode gefunden werden, diese Form der Bewegungswahrnehmung untersuchen zu können, da bisherige Experimente zum Bewegungssehen beim Goldfisch ausschließlich mit Hilfe der optomotorischen Folgereaktion gemacht wurden. Anschließend sollte die Frage geklärt werden, ob das Objektbewegungssehen genau wie das Bewegungssehen einer Großfeldbewegung farbenblind ist und welcher Zapfentyp daran beteiligt ist. Die Verwendung eines Zufallpunktmusters zur Dressur auf ein bewegtes Objekt hat sich als äußert erfolgreich herausgestellt. Diese Methode hat den Vorteil, dass sich die Versuchstiere ausschließlich aufgrund der Bewegungsinformation orientieren können. In den Rot-Grün- und Blau-Grün-Transferversuchen zeigte sich, dass das Objektbewegungssehen beim Goldfisch farbenblind ist, aber erstaunlicherweise nicht vom L-Zapfen vermittelt wird, sondern wahrscheinlich vom M-Zapfen. Welchen Vorteil es haben könnte, dass für die verschiedenen Formen der Bewegungswahrnehmung verschiedene Eingänge benutzt werden, kann mit diesen Versuchen nicht geklärt werden. Farbenblindheit des Bewegungssehens scheint eine Eigenschaft visueller Systeme allgemein zu sein. Beim Menschen ist diese Frage im Moment noch nicht geklärt und wird weiterhin diskutiert, da es sowohl Experimente gibt, die zeigen, dass es farbenblind ist, als auch andere, die Hinweise darauf geben, dass es nicht farbenblind ist. Der Vorteil der Farbenblindheit eines bewegungsdetektierenden visuellen Systems zeigt sich auch in der Technik beim Maschinen Sehen. Hier wird ebenfalls auf Farbinformation verzichtet, was zum einen eine Datenreduktion mit sich bringt und zum anderen dazu führt, dass korrespondierende Bildpunkte leichter gefunden werden können. Diese werden benötigt, um Bewegungsvektoren zu bestimmen und letztlich Bewegung zu detektieren.

A new method to measure higher visual functions in an immersive environment

BACKGROUND: Higher visual functions can be defined as cognitive processes responsible for object recognition, color and shape perception, and motion detection. People with impaired higher visual functions after unilateral brain lesion are often tested with paper pencil tests, but such tests do not assess the degree of interaction between the healthy brain hemisphere and the impaired one. Hence, visual functions are not tested separately in the contralesional and ipsilesional visual hemifields. METHODS: A new measurement setup, that involves real-time comparisons of shape and size of objects, orientation of lines, speed and direction of moving patterns, in the right or left visual hemifield, has been developed. The setup was implemented in an immersive environment like a hemisphere to take into account the effects of peripheral and central vision, and eventual visual field losses. Due to the non-flat screen of the hemisphere, a distortion algorithm was needed to adapt the projected images to the surface. Several approaches were studied and, based on a comparison between projected images and original ones, the best one was used for the implementation of the test. Fifty-seven healthy volunteers were then tested in a pilot study. A Satisfaction Questionnaire was used to assess the usability of the new measurement setup. RESULTS: The results of the distortion algorithm showed a structural similarity between the warped images and the original ones higher than 97%. The results of the pilot study showed an accuracy in comparing images in the two visual hemifields of 0.18 visual degrees and 0.19 visual degrees for size and shape discrimination, respectively, 2.56° for line orientation, 0.33 visual degrees/s for speed perception and 7.41° for recognition of motion direction. The outcome of the Satisfaction Questionnaire showed a high acceptance of the battery by the participants. CONCLUSIONS: A new method to measure higher visual functions in an immersive environment was presented. The study focused on the usability of the developed battery rather than the performance at the visual tasks. A battery of five subtasks to study the perception of size, shape, orientation, speed and motion direction was developed. The test setup is now ready to be tested in neurological patients.

Simultaneous 3D object tracking and camera parameter estimation by Bayesian methods and transdimensional MCMC sampling

Multi-camera 3D tracking systems with overlapping cameras represent a powerful mean for scene analysis, as they potentially allow greater robustness than monocular systems and provide useful 3D information about object location and movement. However, their performance relies on accurately calibrated camera networks, which is not a realistic assumption in real surveillance environments. Here, we introduce a multi-camera system for tracking the 3D position of a varying number of objects and simultaneously refin-ing the calibration of the network of overlapping cameras. Therefore, we introduce a Bayesian framework that combines Particle Filtering for tracking with recursive Bayesian estimation methods by means of adapted transdimensional MCMC sampling. Addi-tionally, the system has been designed to work on simple motion detection masks, making it suitable for camera networks with low transmission capabilities. Tests show that our approach allows a successful performance even when starting from clearly inaccurate camera calibrations, which would ruin conventional approaches.

Magnocellular mediated visual-spatial attention and reading ability

This study explores the relationship between attentional processing mediated by visual magnocellular (MC) processing and reading ability. Reading ability in a group of primary school children was compared to performance on a visual cued coherent motion detection task. The results showed that a brief spatial cue was more effective in drawing attention either away or towards a visual target in the group of readers ranked in the upper 25% of the sample compared to lower ranked readers. Regression analysis showed a significant relationship between attentional processing and reading when the effects of age and intellectual ability were removed. Results suggested a stronger relationship between visual attentional and non-word reading compared to irregular word reading. (C) 2004 Lippincott Williams & Wilkins, Inc.

Sensory processing, reading, IQ, and attention

Detection thresholds for two visual- and two auditory-processing tasks were obtained for 73 children and young adults who varied broadly in reading ability. A reading-disabled subgroup had significantly higher thresholds than a normal-reading subgroup for the auditory tasks only. When analyzed across the whole group, the auditory tasks and one of the visual tasks, coherent motion detection, were significantly related to word reading. These effects were largely independent of ADHD ratings; however, none of these measures accounted for significant variance in word reading after controlling for full-scale IQ. In contrast, phoneme awareness, rapid naming, and nonword repetition each explained substantial, significant word reading variance after controlling for IQ, suggesting more specific roles for these oral language skills in the development of word reading. © 2004 Elsevier Inc. All rights reserved.

An automated method for the determination of electroreception thresholds in weakly electric fish

Weakly electric fish produce a dual function electric signal that makes them ideal models for the study of sensory computation and signal evolution. This signal, the electric organ discharge (EOD), is used for communication and navigation. In some families of gymnotiform electric fish, the EOD is a dynamic signal that increases in amplitude during social interactions. Amplitude increase could facilitate communication by increasing the likelihood of being sensed by others or by impressing prospective mates or rivals. Conversely, by increasing its signal amplitude a fish might increase its sensitivity to objects by lowering its electrolocation detection threshold. To determine how EOD modulations elicited in the social context affect electrolocation, I developed an automated and fast method for measuring electroreception thresholds using a classical conditioning paradigm. This method employs a moving shelter tube, which these fish occupy at rest during the day, paired with an electrical stimulus. A custom built and programmed robotic system presents the electrical stimulus to the fish, slides the shelter tube requiring them to follow, and records video of their movements. I trained the electric fish of the genus Sternopygus was trained to respond to a resistive stimulus on this apparatus in 2 days. The motion detection algorithm correctly identifies the responses 91% of the time, with a false positive rate of only 4%. This system allows for a large number of trials, decreasing the amount of time needed to determine behavioral electroreception thresholds. This novel method enables the evaluation the evolutionary interplay between two conflicting sensory forces, social communication and navigation.