Shared autonomy for close-quarters navigation and control of a VTOL platform

This thesis presents an approach for a vertical infrastructure inspection using a vertical take-off and landing (VTOL) unmanned aerial vehicle and shared autonomy. Inspecting vertical structure such as light and power distribution poles is a difficult task. There are challenges involved with developing such an inspection system, such as flying in close proximity to a target while maintaining a fixed stand-off distance from it. The contributions of this thesis fall into three main areas. Firstly, an approach to vehicle dynamic modeling is evaluated in simulation and experiments. Secondly, EKF-based state estimators are demonstrated, as well as estimator-free approaches such as image based visual servoing (IBVS) validated with motion capture ground truth data. Thirdly, an integrated pole inspection system comprising a VTOL platform with human-in-the-loop control, (shared autonomy) is demonstrated. These contributions are comprehensively explained through a series of published papers.

Stepping accuracy and visuomotor control among older adults : effect of target contrast and refractive blur

Purpose: Older adults have increased visual impairment, including refractive blur from presbyopic multifocal spectacle corrections, and are less able to extract visual information from the environment to plan and execute appropriate stepping actions; these factors may collectively contribute to their higher risk of falls. The aim of this study was to examine the effect of refractive blur and target visibility on the stepping accuracy and visuomotor stepping strategies of older adults during a precision stepping task. Methods: Ten healthy, visually normal older adults (mean age 69.4 ± 5.2 years) walked up and down a 20 m indoor corridor stepping onto selected high and low-contrast targets while viewing under three visual conditions: best-corrected vision, +2.00 DS and +3.00 DS blur; the order of blur conditions was randomised between participants. Stepping accuracy and gaze behaviours were recorded using an eyetracker and a secondary hand-held camera. Results: Older adults made significantly more stepping errors with increasing levels of blur, particularly exhibiting under-stepping (stepping more posteriorly) onto the targets (p<0.05), while visuomotor stepping strategies did not significantly alter. Stepping errors were also significantly greater for the low compared to the high contrast targets and differences in visuomotor stepping strategies were found, including increased duration of gaze and increased interval between gaze onset and initiation of the leg swing when stepping onto the low contrast targets. Conclusions: These findings highlight that stepping accuracy is reduced for low visibility targets, and for high levels of refractive blur at levels typically present in multifocal spectacle corrections, despite significant changes in some of the visuomotor stepping strategies. These findings highlight the importance of maximising the contrast of objects in the environment, and may help explain why older adults wearing multifocal spectacle corrections exhibit an increased risk of falling.

An organization of visual and auditory fear conditioning in the lateral amygdala

Pavlovian fear conditioning is an evolutionary conserved and extensively studied form of associative learning and memory. In mammals, the lateral amygdala (LA) is an essential locus for Pavlovian fear learning and memory. Despite significant progress unraveling the cellular mechanisms responsible for fear conditioning, very little is known about the anatomical organization of neurons encoding fear conditioning in the LA. One key question is how fear conditioning to different sensory stimuli is organized in LA neuronal ensembles. Here we show that Pavlovian fear conditioning, formed through either the auditory or visual sensory modality, activates a similar density of LA neurons expressing a learning-induced phosphorylated extracellular signal-regulated kinase (p-ERK1/2). While the size of the neuron population specific to either memory was similar, the anatomical distribution differed. Several discrete sites in the LA contained a small but significant number of p-ERK1/2-expressing neurons specific to either sensory modality. The sites were anatomically localized to different levels of the longitudinal plane and were independent of both memory strength and the relative size of the activated neuronal population, suggesting some portion of the memory trace for auditory and visually cued fear conditioning is allocated differently in the LA. Presenting the visual stimulus by itself did not activate the same p-ERK1/2 neuron density or pattern, confirming the novelty of light alone cannot account for the specific pattern of activated neurons after visual fear conditioning. Together, these findings reveal an anatomical distribution of visual and auditory fear conditioning at the level of neuronal ensembles in the LA.

Vision-based control for mining automation

The mining industry is highly suitable for the application of robotics and automation technology since the work is both arduous and dangerous. Visual servoing is a means of integrating noncontact visual sensing with machine control to augment or replace operator based control. This article describes two of our current mining automation projects in order to demonstrate some, perhaps unusual, applications of visual servoing, and also to illustrate some very real problems with robust computer vision

Modeling and control of a robotic power line inspection vehicle

Power line inspection is a vital function for electricity supply companies but it involves labor-intensive and expensive procedures which are tedious and error-prone for humans to perform. A possible solution is to use an unmanned aerial vehicle (UAV) equipped with video surveillance equipment to perform the inspection. This paper considers how a small, electrically driven rotorcraft conceived for this application could be controlled by visually tracking the overhead supply lines. A dynamic model for a ducted-fan rotorcraft is presented and used to control the action of an Air Vehicle Simulator (AVS), consisting of a cable-array robot. Results show how visual data can be used to determine, and hence regulate in closed loop, the simulated vehicle’s position relative to the overhead lines.

Design and control of MIRA: A lightweight climbing robot for ship inspection

The inspection of marine vessels is currently performed manually. Inspectors use tools (e.g. cameras and devices for non-destructive testing) to detect damaged areas, cracks, and corrosion in large cargo holds, tanks, and other parts of a ship. Due to the size and complex geometry of most ships, ship inspection is time-consuming and expensive. The EU-funded project INCASS develops concepts for a marine inspection robotic assistant system to improve and automate ship inspections. In this paper, we introduce our magnetic wall–climbing robot: Marine Inspection Robotic Assistant (MIRA). This semiautonomous lightweight system is able to climb a vessels steel frame to deliver on-line visual inspection data. In addition, we describe the design of the robot and its building subsystems as well as its hardware and software components.

Visual openness in urban environments: Measuring visual openness and visibility to natural landscapes in a changing urban landscape in Gold Coast, Australia

The visual characteristics of urban environments have been changing dramatically with the growth of cities around the world. Protection and enhancement of landscape character in urban environments have been one of the challenges for policy makers in addressing sustainable urban growth. Visual openness and enclosure in urban environments are important attributes in perception of visual space which affect the human interaction with physical space and which can be often modified by new developments. Measuring visual openness in urban areas results in more accurate, reliable, and systematic approach to manage and control visual qualities in growing cities. Recent advances in techniques in geographic information systems (GIS) and survey systems make it feasible to measure and quantify this attribute with a high degree of realism and precision. Previous studies in this field do not take full advantage of these improvements. This paper proposes a method to measure the visual openness and enclosure in a changing urban landscape in Australia, on the Gold Coast, by using the improved functionality in GIS. Using this method, visual openness is calculated and described for all publicly accessible areas in the selected study area. A final map is produced which shows the areas with highest visual openness and visibility to natural landscape resources. The output of this research can be used by planners and decision-makers in managing and controlling views in complex urban landscapes. Also, depending on the availability of GIS data, this method can be applied to any region including non-urban landscapes to help planners and policy-makers manage views and visual qualities.

An anticipative reactive control strategy to deal with unforeseen obstacles during a multi-sensor-based navigation task

We address the problem of the rangefinder-based avoidance of unforeseen static obstacles during a visual navigation task. We extend previous strategies which are efficient in most cases but remain still hampered by some drawbacks (e.g., risks of collisions or of local minima in some particular cases, etc.). The key idea is to complete the control strategy by adding a controller providing the robot some anticipative skills to guarantee non collision and by defining more general transition conditions to deal with local minima. Simulation results show the proposed strategy efficiency.

Functional activation during the Rapid Visual Information Processing task in a middle aged cohort: An fMRI study

The Rapid Visual Information Processing (RVIP) task, a serial discrimination task where task performance believed to reflect sustained attention capabilities, is widely used in behavioural research and increasingly in neuroimaging studies. To date, functional neuroimaging research into the RVIP has been undertaken using block analyses, reflecting the sustained processing involved in the task, but not necessarily the transient processes associated with individual trial performance. Furthermore, this research has been limited to young cohorts. This study assessed the behavioural and functional magnetic resonance imaging (fMRI) outcomes of the RVIP task using both block and event-related analyses in a healthy middle aged cohort (mean age = 53.56 years, n = 16). The results show that the version of the RVIP used here is sensitive to changes in attentional demand processes with participants achieving a 43% accuracy hit rate in the experimental task compared with 96% accuracy in the control task. As shown by previous research, the block analysis revealed an increase in activation in a network of frontal, parietal, occipital and cerebellar regions. The event related analysis showed a similar network of activation, seemingly omitting regions involved in the processing of the task (as shown in the block analysis), such as occipital areas and the thalamus, providing an indication of a network of regions involved in correct trial performance. Frontal (superior and inferior frontal gryi), parietal (precuenus, inferior parietal lobe) and cerebellar regions were shown to be active in both the block and event-related analyses, suggesting their importance in sustained attention/vigilance. These networks and the differences between them are discussed in detail, as well as implications for future research in middle aged cohorts.

MPC controlled multirotor with suspended slung load: System architecture and visual load detection

There is an increased interest in the use of Unmanned Aerial Vehicles for load transportation from environmental remote sensing to construction and parcel delivery. One of the main challenges is accurate control of the load position and trajectory. This paper presents an assessment of real flight trials for the control of an autonomous multi-rotor with a suspended slung load using only visual feedback to determine the load position. This method uses an onboard camera to take advantage of a common visual marker detection algorithm to robustly detect the load location. The load position is calculated using an onboard processor, and transmitted over a wireless network to a ground station integrating MATLAB/SIMULINK and Robotic Operating System (ROS) and a Model Predictive Controller (MPC) to control both the load and the UAV. To evaluate the system performance, the position of the load determined by the visual detection system in real flight is compared with data received by a motion tracking system. The multi-rotor position tracking performance is also analyzed by conducting flight trials using perfect load position data and data obtained only from the visual system. Results show very accurate estimation of the load position (~5% Offset) using only the visual system and demonstrate that the need for an external motion tracking system is not needed for this task.

Towards vision-based deep reinforcement learning for robotic motion control

This paper introduces a machine learning based system for controlling a robotic manipulator with visual perception only. The capability to autonomously learn robot controllers solely from raw-pixel images and without any prior knowledge of configuration is shown for the first time. We build upon the success of recent deep reinforcement learning and develop a system for learning target reaching with a three-joint robot manipulator using external visual observation. A Deep Q Network (DQN) was demonstrated to perform target reaching after training in simulation. Transferring the network to real hardware and real observation in a naive approach failed, but experiments show that the network works when replacing camera images with synthetic images.

Evaluation of visual stress symptoms in Age-matched Dyslexic, Meares–Irlen syndrome (MIS) and normal adults

AIM To examine the prevalence of dyslexia and Meares–Irlen syndrome (MIS) among female students and determine their level of visual stress in comparison with normal subjects. METHODS A random sample of 450 female medical students of King Saud University Riyadh (age range, 18 - 30 years) responded to a wide range of questions designed to accomplish the aims of this study. The detailed questionnaire consisted of 54 questions with twelve questions enquiring on ocular history and demography of participants while 42 questions were on visual symptoms. Items were categorized into; critical and non-critical questions (CQ and NCQ) and were rated on four point Likert scale. Based on the responses obtained, the subjects were grouped into normal (control), dyslexic with or without MIS (Group 1) and subjects with MIS only (Group 2). Responses were analysed as averages and mean scores were calculated and compared between-groups using one way analysis of variance to evaluate total (TVSS = NCQ + CQ), critical and non-critical visual stress scores. The relationship between categorical variables such as age, handedness and condition were assessed with Chi- Square test. RESULTS The completion rate was 96.8% and majority of the respondents (92%) were normal readers, 2% dyslexic and 6% had MIS. They were age-matched. More than half of the participants had visited an eye care practitioner in the last 2yrs. About 13% were recommended eye exercises and one participant experienced pattern glare. Hand preference was not associated with any condition but Group 1 subjects (3/9, 33%) were significantly more likely to be diagnosed of lazy eye than Group 2 (2/27, 7%) and control (27/414, 5%) subjects. The mean ± SD of TVSS responses were 63 ± 14 but it was 44 ± 9 for CQ and 19 ± 5 for NCQ. Responses from all three variables were normally distributed but the CQ responses were on the average more positive (82%) in Group 2 and less positive (46%) in Group 1 than control. With NCQ, the responses were equally less positive in Group 1 and 2 than control. Group 2 subjects showed significantly higher TVSS (P = 0.002), NCQ (P = 0.006) and CQ (P = 0.008) visual stress scores than control but no difference between Group 1 and control subjects, was observed for all scores (P > 0.05, for all comparisons). CONCLUSION The prevalence of dyslexia and MIS among Saudi female students was 2 and 6%, respectively. Critical questions performed best for assessing visual stress symptoms in dyslexic and MIS subjects. Generally, students with MIS were more sensitive to visual stress than normal students but dyslexics were more likely to present with a lazy eye than MIS and normal readers.

A Probabilistic Approach to the Primary Visual Cortex

What can the statistical structure of natural images teach us about the human brain? Even though the visual cortex is one of the most studied parts of the brain, surprisingly little is known about how exactly images are processed to leave us with a coherent percept of the world around us, so we can recognize a friend or drive on a crowded street without any effort. By constructing probabilistic models of natural images, the goal of this thesis is to understand the structure of the stimulus that is the raison d etre for the visual system. Following the hypothesis that the optimal processing has to be matched to the structure of that stimulus, we attempt to derive computational principles, features that the visual system should compute, and properties that cells in the visual system should have. Starting from machine learning techniques such as principal component analysis and independent component analysis we construct a variety of sta- tistical models to discover structure in natural images that can be linked to receptive field properties of neurons in primary visual cortex such as simple and complex cells. We show that by representing images with phase invariant, complex cell-like units, a better statistical description of the vi- sual environment is obtained than with linear simple cell units, and that complex cell pooling can be learned by estimating both layers of a two-layer model of natural images. We investigate how a simplified model of the processing in the retina, where adaptation and contrast normalization take place, is connected to the nat- ural stimulus statistics. Analyzing the effect that retinal gain control has on later cortical processing, we propose a novel method to perform gain control in a data-driven way. Finally we show how models like those pre- sented here can be extended to capture whole visual scenes rather than just small image patches. By using a Markov random field approach we can model images of arbitrary size, while still being able to estimate the model parameters from the data.

Refractive surgery for iatrogenic and congenital anisometropia and mild visual impairment

In anisometropia, the two eyes have unequal refractive power. Anisometropia is a risk factor for amblyopia. The visual deficiencies are thought to be irreversible after the first decade of life. There is, however, accumulating evidence that neural plasticity exists also in adult brains. The aim of this study was to investigate functional outcome of excimer laser refractive surgery in adult anisometropic and visually impaired patients. Additional goal was to examine changes in the primary visual cortex (V1) using multifocal functional magnetic resonance imaging (mffMRI) after laser refractive surgery. Study I comprised of 57 anisometropic patients (anisometropia of ≥3.25 diopters) and 174 isometropic myopic subjects formed the control group. A significant improvement in best-spectacle-corrected visual acuity (BSCVA) among myopic control subjects was evident 3 months postoperatively. The improvement in BSCVA was significantly slower for anisometropic patients and the improvement appeared to persist to the end of the follow-up (24 months). In study II we found that refractive surgery may be also successfully used for iathrogenic anisometropia. In Study III we evaluated mildly visually impaired adult patients after refractive surgery. There was a statistically significant improvement in BSCVA among visually impaired patients and the difference in the mean BSCVA between visually impaired patients and isometropic myopic control subjects diminished during follow-up. Study IV was a prospective follow-up trial examining the changes in the primary visual cortex after refractive surgery. Two anisometropic patients and two isometropic myopic patients were examined with a 61-region mffMRI before refractive surgery and at three, six, nine and twelve months postoperatively. In this study, a dramatic decrease in the number of active voxels in the fovea was found among anisometropic patients. The results presented in this thesis revealed that refractive surgery may be successfully used for the treatment of anisometropic adults with both congenital and iatrogenic anisometropia and for mildly visually impaired adults. The findings in conclusion strengthen our hypothesis of plastic changes in the visual cortex of adult anisometropic and mildly visually impaired patients after refractive surgery.

Evaluation of Vestibular Function with Visual Feedback Posturography and Motorized Head Impulse Test.

Objectives: To evaluate the applicability of visual feedback posturography (VFP) for quantification of postural control, and to characterize the horizontal angular vestibulo-ocular reflex (AVOR) by use of a novel motorized head impulse test (MHIT). Methods: In VFP, subjects standing on a platform were instructed to move their center of gravity to symmetrically placed peripheral targets as fast and accurately as possible. The active postural control movements were measured in healthy subjects (n = 23), and in patients with vestibular schwannoma (VS) before surgery (n = 49), one month (n = 17), and three months (n = 36) after surgery. In MHIT we recorded head and eye position during motorized head impulses (mean velocity of 170º/s and acceleration of 1 550º/s²) in healthy subjects (n = 22), in patients with VS before surgery (n = 38) and about four months afterwards (n = 27). The gain, asymmetry and latency in MHIT were calculated. Results: The intraclass correlation coefficient for VFP parameters during repeated tests was significant (r = 0.78-0.96; p < 0.01), although two of four VFP parameters improved slightly during five test sessions in controls. At least one VFP parameter was abnormal pre- and postoperatively in almost half the patients, and these abnormal preoperative VFP results correlated significantly with abnormal postoperative results. The mean accuracy in postural control in patients was reduced pre- and postoperatively. A significant side difference with VFP was evident in 10% of patients. In the MHIT, the normal gain was close to unity, the asymmetry in gain was within 10%, and the latency was a mean ± standard deviation 3.4 ± 6.3 milliseconds. Ipsilateral gain or asymmetry in gain was preoperatively abnormal in 71% of patients, whereas it was abnormal in every patient after surgery. Preoperative gain (mean ± 95% confidence interval) was significantly lowered to 0.83 ± 0.08 on the ipsilateral side compared to 0.98 ± 0.06 on the contralateral side. The ipsilateral postoperative mean gain of 0.53 ± 0.05 was significantly different from preoperative gain. Conclusion: The VFP is a repeatable, quantitative method to assess active postural control within individual subjects. The mean postural control in patients with VS was disturbed before and after surgery, although not severely. Side difference in postural control in the VFP was rare. The horizontal AVOR results in healthy subjects and in patients with VS, measured with MHIT, were in agreement with published data achieved using other techniques with head impulse stimuli. The MHIT is a non-invasive method which allows reliable clinical assessment of the horizontal AVOR.