Datasets for the evaluation of multi-sensor perception in natural environments with challenging conditions

This paper presents large, accurately calibrated and time-synchronised datasets, gathered outdoors in controlled environmental conditions, using an unmanned ground vehicle (UGV), equipped with a wide variety of sensors. It discusses how the data collection process was designed, the conditions in which these datasets have been gathered, and some possible outcomes of their exploitation, in particular for the evaluation of performance of sensors and perception algorithms for UGVs.

Combining multiple sensor modalities for a localisation robust to smoke

This paper proposes an approach to obtain a localisation that is robust to smoke by exploiting multiple sensing modalities: visual and infrared (IR) cameras. This localisation is based on a state-of-the-art visual SLAM algorithm. First, we show that a reasonably accurate localisation can be obtained in the presence of smoke by using only an IR camera, a sensor that is hardly affected by smoke, contrary to a visual camera (operating in the visible spectrum). Second, we demonstrate that improved results can be obtained by combining the information from the two sensor modalities (visual and IR cameras). Third, we show that by detecting the impact of smoke on the visual images using a data quality metric, we can anticipate and mitigate the degradation in performance of the localisation by discarding the most affected data. The experimental validation presents multiple trajectories estimated by the various methods considered, all thoroughly compared to an accurate dGPS/INS reference.

Wind speed perception and risk

Background How accurately do people perceive extreme wind speeds and how does that perception affect the perceived risk? Prior research on human–wind interaction has focused on comfort levels in urban settings or knock-down thresholds. No systematic experimental research has attempted to assess people's ability to estimate extreme wind speeds and perceptions of their associated risks. Method We exposed 76 people to 10, 20, 30, 40, 50, and 60 mph (4.5, 8.9, 13.4, 17.9, 22.3, and 26.8 m/s) winds in randomized orders and asked them to estimate wind speed and the corresponding risk they felt. Results Multilevel modeling showed that people were accurate at lower wind speeds but overestimated wind speeds at higher levels. Wind speed perceptions mediated the direct relationship between actual wind speeds and perceptions of risk (i.e., the greater the perceived wind speed, the greater the perceived risk). The number of tropical cyclones people had experienced moderated the strength of the actual–perceived wind speed relationship; consequently, mediation was stronger for people who had experienced fewer storms. Conclusion These findings provide a clearer understanding of wind and risk perception, which can aid development of public policy solutions toward communicating the severity and risks associated with natural disasters.

The role of active locomotion in space perception

It has been shown that active control of locomotion increases accuracy and precision of nonvisual space perception, but psychological mechanisms of this enhancement are poorly understood. The present study explored a hypothesis that active control of locomotion enhances space perception by facilitating crossmodal interaction between visual and nonvisual spatial information. In an experiment, blindfolded participants walked along a linear path under one of the following two conditions: (1) They walked by themselves following a guide rope; and (2) they were led by an experimenter. Subsequently, they indicated the walked distance by tossing a beanbag to the origin of locomotion. The former condition gave participants greater control of their locomotion, and thus represented a more active walking condition. In addition, before each trial, half the participants viewed the room in which they performed the distance perception task. The other half remained blindfolded throughout the experiment. Results showed that although the room was devoid of any particular cues for walked distances, visual knowledge of the surroundings improved the precision of nonvisual distance perception. Importantly, however, the benefit of preview was observed only when participants walked more actively. This indicates that active control of locomotion allowed participants to better utilize their visual memory of the environment for perceiving nonvisually encoded distance, suggesting that active control of locomotion served as a catalyst for integrating visual and nonvisual information to derive spatial representations of higher quality.

Cortical organization of environmental sounds by attribute

Semantic knowledge is supported by a widely distributed neuronal network, with differential patterns of activation depending upon experimental stimulus or task demands. Despite a wide body of knowledge on semantic object processing from the visual modality, the response of this semantic network to environmental sounds remains relatively unknown. Here, we used fMRI to investigate how access to different conceptual attributes from environmental sound input modulates this semantic network. Using a range of living and manmade sounds, we scanned participants whilst they carried out an object attribute verification task. Specifically, we tested visual perceptual, encyclopedic, and categorical attributes about living and manmade objects relative to a high-level auditory perceptual baseline to investigate the differential patterns of response to these contrasting types of object-related attributes, whilst keeping stimulus input constant across conditions. Within the bilateral distributed network engaged for processing environmental sounds across all conditions, we report here a highly significant dissociation within the left hemisphere between the processing of visual perceptual and encyclopedic attributes of objects.

The influence of colour and sound on neuronal activation during visual object naming

This paper investigates how neuronal activation for naming photographs of objects is influenced by the addition of appropriate colour or sound. Behaviourally, both colour and sound are known to facilitate object recognition from visual form. However, previous functional imaging studies have shown inconsistent effects. For example, the addition of appropriate colour has been shown to reduce antero-medial temporal activation whereas the addition of sound has been shown to increase posterior superior temporal activation. Here we compared the effect of adding colour or sound cues in the same experiment. We found that the addition of either the appropriate colour or sound increased activation for naming photographs of objects in bilateral occipital regions and the right anterior fusiform. Moreover, the addition of colour reduced left antero-medial temporal activation but this effect was not observed for the addition of object sound. We propose that activation in bilateral occipital and right fusiform areas precedes the integration of visual form with either its colour or associated sound. In contrast, left antero-medial temporal activation is reduced because object recognition is facilitated after colour and form have been integrated.

Close-quarters Quadrotor flying for a pole inspection with position based visual servoing and high-speed vision

This paper presents a 100 Hz monocular position based visual servoing system to control a quadrotor flying in close proximity to vertical structures approximating a narrow, locally linear shape. Assuming the object boundaries are represented by parallel vertical lines in the image, detection and tracking is achieved using Plücker line representation and a line tracker. The visual information is fused with IMU data in an EKF framework to provide fast and accurate state estimation. A nested control design provides position and velocity control with respect to the object. Our approach is aimed at high performance on-board control for applications allowing only small error margins and without a motion capture system, as required for real world infrastructure inspection. Simulated and ground-truthed experimental results are presented.

Facial motion engages predictive visual mechanisms

We employed a novel cuing paradigm to assess whether dynamically versus statically presented facial expressions differentially engaged predictive visual mechanisms. Participants were presented with a cueing stimulus that was either the static depiction of a low intensity expressed emotion; or a dynamic sequence evolving from a neutral expression to the low intensity expressed emotion. Following this cue and a backwards mask, participants were presented with a probe face that displayed either the same emotion (congruent) or a different emotion (incongruent) with respect to that displayed by the cue although expressed at a high intensity. The probe face had either the same or different identity from the cued face. The participants' task was to indicate whether or not the probe face showed the same emotion as the cue. Dynamic cues and same identity cues both led to a greater tendency towards congruent responding, although these factors did not interact. Facial motion also led to faster responding when the probe face was emotionally congruent to the cue. We interpret these results as indicating that dynamic facial displays preferentially invoke predictive visual mechanisms, and suggest that motoric simulation may provide an important basis for the generation of predictions in the visual system.

Potentiation of the early visual response to learned danger signals in adults and adolescents

The reinforcing effects of aversive outcomes on avoidance behaviour are well established. However, their influence on perceptual processes is less well explored, especially during the transition from adolescence to adulthood. Using electroencephalography, we examined whether learning to actively or passively avoid harm can modulate early visual responses in adolescents and adults. The task included two avoidance conditions, active and passive, where two different warning stimuli predicted the imminent, but avoidable, presentation of an aversive tone. To avoid the aversive outcome, participants had to learn to emit an action (active avoidance) for one of the warning stimuli and omit an action for the other (passive avoidance). Both adults and adolescents performed the task with a high degree of accuracy. For both adolescents and adults, increased N170 event-related potential amplitudes were found for both the active and the passive warning stimuli compared with control conditions. Moreover, the potentiation of the N170 to the warning stimuli was stable and long lasting. Developmental differences were also observed; adolescents showed greater potentiation of the N170 component to danger signals. These findings demonstrate, for the first time, that learned danger signals in an instrumental avoidance task can influence early visual sensory processes in both adults and adolescents.

An organization of visual and auditory fear conditioning in the lateral amygdala

Pavlovian fear conditioning is an evolutionary conserved and extensively studied form of associative learning and memory. In mammals, the lateral amygdala (LA) is an essential locus for Pavlovian fear learning and memory. Despite significant progress unraveling the cellular mechanisms responsible for fear conditioning, very little is known about the anatomical organization of neurons encoding fear conditioning in the LA. One key question is how fear conditioning to different sensory stimuli is organized in LA neuronal ensembles. Here we show that Pavlovian fear conditioning, formed through either the auditory or visual sensory modality, activates a similar density of LA neurons expressing a learning-induced phosphorylated extracellular signal-regulated kinase (p-ERK1/2). While the size of the neuron population specific to either memory was similar, the anatomical distribution differed. Several discrete sites in the LA contained a small but significant number of p-ERK1/2-expressing neurons specific to either sensory modality. The sites were anatomically localized to different levels of the longitudinal plane and were independent of both memory strength and the relative size of the activated neuronal population, suggesting some portion of the memory trace for auditory and visually cued fear conditioning is allocated differently in the LA. Presenting the visual stimulus by itself did not activate the same p-ERK1/2 neuron density or pattern, confirming the novelty of light alone cannot account for the specific pattern of activated neurons after visual fear conditioning. Together, these findings reveal an anatomical distribution of visual and auditory fear conditioning at the level of neuronal ensembles in the LA.

Visual predictive control of spiral motion

This paper deals with constrained image-based visual servoing of circular and conical spiral motion about an unknown object approximating a single image point feature. Effective visual control of such trajectories has many applications for small unmanned aerial vehicles, including surveillance and inspection, forced landing (homing), and collision avoidance. A spherical camera model is used to derive a novel visual-predictive controller (VPC) using stability-based design methods for general nonlinear model-predictive control. In particular, a quasi-infinite horizon visual-predictive control scheme is derived. A terminal region, which is used as a constraint in the controller structure, can be used to guide appropriate reference image features for spiral tracking with respect to nominal stability and feasibility. Robustness properties are also discussed with respect to parameter uncertainty and additive noise. A comparison with competing visual-predictive control schemes is made, and some experimental results using a small quad rotor platform are given.

Age-related changes in perception of movement in driving scenes

Purpose Age-related changes in motion sensitivity have been found to relate to reductions in various indices of driving performance and safety. The aim of this study was to investigate the basis of this relationship in terms of determining which aspects of motion perception are most relevant to driving. Methods Participants included 61 regular drivers (age range 22–87 years). Visual performance was measured binocularly. Measures included visual acuity, contrast sensitivity and motion sensitivity assessed using four different approaches: (1) threshold minimum drift rate for a drifting Gabor patch, (2) Dmin from a random dot display, (3) threshold coherence from a random dot display, and (4) threshold drift rate for a second-order (contrast modulated) sinusoidal grating. Participants then completed the Hazard Perception Test (HPT) in which they were required to identify moving hazards in videos of real driving scenes, and also a Direction of Heading task (DOH) in which they identified deviations from normal lane keeping in brief videos of driving filmed from the interior of a vehicle. Results In bivariate correlation analyses, all motion sensitivity measures significantly declined with age. Motion coherence thresholds, and minimum drift rate threshold for the first-order stimulus (Gabor patch) both significantly predicted HPT performance even after controlling for age, visual acuity and contrast sensitivity. Bootstrap mediation analysis showed that individual differences in DOH accuracy partly explained these relationships, where those individuals with poorer motion sensitivity on the coherence and Gabor tests showed decreased ability to perceive deviations in motion in the driving videos, which related in turn to their ability to detect the moving hazards. Conclusions The ability to detect subtle movements in the driving environment (as determined by the DOH task) may be an important contributor to effective hazard perception, and is associated with age, and an individuals' performance on tests of motion sensitivity. The locus of the processing deficits appears to lie in first-order, rather than second-order motion pathways.

Visual rehabilitation in low-moderate keratoconus : intracorneal ring segment implantation followed by same-day topography-guided photorefractive keratectomy and collagen cross linking

AIM: To present the results of same-day topography-guided photorefractive keratectomy (TG-PRK) and corneal collagen crosslinking (CXL) after previous intrastromal corneal ring segment (ISCR) implantation for keratoconus. METHODS: An experimental clinical study on twenty-one eyes of 19 patients aged, 27.1±6.6 years (range: 19 – 43 years), with low to moderate keratoconus who were selected to undergo customized TG-PRK immediately followed by same-day CXL, 9 months after ISCR implantation in a university ophthalmology clinic. Refraction, uncorrected (UDVA) and corrected distance visual acuities (CDVA), keratometry (K) values, central corneal thickness (CCT) and coma were assessed 3 months after TG/PRK and CXL. RESULTS: After TG-PRK/CXL: the mean UDVA (logMAR) improved significantly from 0.66±0.41 to 0.20±0.25 (P<0.05); K flat value decreased from: 48.44±3.66 D to 43.71±1.95 D; K steep value decreased from 45.61±2.40 D to 41.56±2.05D; K average also decreased from 42.42±2.07 D to 47.00±2.66 D (P<0.05 for all). The mean sphere and cylinder decreased significantly post-surgery from, -3.10±2.99 D to -0.11±0.93 D and from, -3.68±1.53 to -1.11±0.75D respectively, while the CDVA, CCT and coma showed no significant changes. Compared to post-ISCR, significant reductions (P ˂ 0.05 or all) in all K-values, sphere and cylinder were observed after TG-PRK/CXL. CONCLUSION: Same-day combined topography-guided PRK and corneal crosslinking following placement of ICRS is a safe and potentially effective option in treating low-moderate keratoconus. It significantly improved all visual acuity, reduced keratometry, sphere and astigmatism, but caused no change in central corneal thickness and coma.

Visual control for automated aircraft collision avoidance systems

This thesis presents a new vision-based decision and control strategy for automated aircraft collision avoidance that can be realistically applied to the See and Avoid problem. The effectiveness of the control strategy positions the research as a major contribution toward realising the simultaneous operation of manned and unmanned aircraft within civilian airspace. Key developments include novel classical and visual predictive control frameworks, and a performance evaluation technique aligned with existing aviation practise and applicable to autonomous systems. The overall approach is demonstrated through experimental results on a small multirotor unmanned aircraft, and through high fidelity probabilistic simulation studies.

Reactive reaching and grasping on a humanoid: Towards closing the action-perception loop on the iCub

We propose a system incorporating a tight integration between computer vision and robot control modules on a complex, high-DOF humanoid robot. Its functionality is showcased by having our iCub humanoid robot pick-up objects from a table in front of it. An important feature is that the system can avoid obstacles - other objects detected in the visual stream - while reaching for the intended target object. Our integration also allows for non-static environments, i.e. the reaching is adapted on-the-fly from the visual feedback received, e.g. when an obstacle is moved into the trajectory. Furthermore we show that this system can be used both in autonomous and tele-operation scenarios.