Computer vision onboard UAVs for civilian tasks

Computer vision is much more than a technique to sense and recover environmental information from an UAV. It should play a main role regarding UAVs’ functionality because of the big amount of information that can be extracted, its possible uses and applications, and its natural connection to human driven tasks, taking into account that vision is our main interface to world understanding. Our current research’s focus lays on the development of techniques that allow UAVs to maneuver in spaces using visual information as their main input source. This task involves the creation of techniques that allow an UAV to maneuver towards features of interest whenever a GPS signal is not reliable or sufficient, e.g. when signal dropouts occur (which usually happens in urban areas, when flying through terrestrial urban canyons or when operating on remote planetary bodies), or when tracking or inspecting visual targets—including moving ones—without knowing their exact UMT coordinates. This paper also investigates visual serving control techniques that use velocity and position of suitable image features to compute the references for flight control. This paper aims to give a global view of the main aspects related to the research field of computer vision for UAVs, clustered in four main active research lines: visual serving and control, stereo-based visual navigation, image processing algorithms for detection and tracking, and visual SLAM. Finally, the results of applying these techniques in several applications are presented and discussed: this study will encompass power line inspection, mobile target tracking, stereo distance estimation, mapping and positioning.

Design of automatic vision-based inspection system for solder joint segmentation

Purpose: Computer vision has been widely used in the inspection of electronic components. This paper proposes a computer vision system for the automatic detection, localisation, and segmentation of solder joints on Printed Circuit Boards (PCBs) under different illumination conditions. Design/methodology/approach: An illumination normalization approach is applied to an image, which can effectively and efficiently eliminate the effect of uneven illumination while keeping the properties of the processed image the same as in the corresponding image under normal lighting conditions. Consequently special lighting and instrumental setup can be reduced in order to detect solder joints. These normalised images are insensitive to illumination variations and are used for the subsequent solder joint detection stages. In the segmentation approach, the PCB image is transformed from an RGB color space to a YIQ color space for the effective detection of solder joints from the background. Findings: The segmentation results show that the proposed approach improves the performance significantly for images under varying illumination conditions. Research limitations/implications: This paper proposes a front-end system for the automatic detection, localisation, and segmentation of solder joint defects. Further research is required to complete the full system including the classification of solder joint defects. Practical implications: The methodology presented in this paper can be an effective method to reduce cost and improve quality in production of PCBs in the manufacturing industry. Originality/value: This research proposes the automatic location, identification and segmentation of solder joints under different illumination conditions.

Practitioner influence on contact lens prescribing in the UK

Contact lenses are mainly fitted by registered optometrists and contact lens opticians in the UK. Data we have gathered from annual contact lens fitting surveys over the past 12 years indicate that, on average, registered optometrists and contact lens opticians undertake 3.2 and 7.1 contact lens fits per week (p < 0.0001). More experienced practitioners tend to fit older patients. Practitioners fitting more lenses per year tend to fit a higher proportion of soft lenses. Contact lens opticians tend to fit a higher proportion of patients with planned replacement and daily disposable lenses compared with optometrists.

Eyelid pressure on the cornea

The eyelids play an important role in lubricating and protecting the surface of the eye. Each blink serves to spread fresh tears, remove debris and replenish the smooth optical surface of the eye. Yet little is known about how the eyelids contact the ocular surface and what pressure distribution exists between the eyelids and cornea. As the principal refractive component of the eye, the cornea is a major element of the eye’s optics. The optical properties of the cornea are known to be susceptible to the pressure exerted by the eyelids. Abnormal eyelids, due to disease, have altered pressure on the ocular surface due to changes in the shape, thickness or position of the eyelids. Normal eyelids also cause corneal distortions that are most often noticed when they are resting closer to the corneal centre (for example during reading). There were many reports of monocular diplopia after reading due to corneal distortion, but prior to videokeratoscopes these localised changes could not be measured. This thesis has measured the influence of eyelid pressure on the cornea after short-term near tasks and techniques were developed to quantify eyelid pressure and its distribution. The profile of the wave-like eyelid-induced corneal changes and the refractive effects of these distortions were investigated. Corneal topography changes due to both the upper and lower eyelids were measured for four tasks involving two angles of vertical downward gaze (20° and 40°) and two near work tasks (reading and steady fixation). After examining the depth and shape of the corneal changes, conclusions were reached regarding the magnitude and distribution of upper and lower eyelid pressure for these task conditions. The degree of downward gaze appears to alter the upper eyelid pressure on the cornea, with deeper changes occurring after greater angles of downward gaze. Although the lower eyelid was further from the corneal centre in large angles of downward gaze, its effect on the cornea was greater than that of the upper eyelid. Eyelid tilt, curvature, and position were found to be influential in the magnitude of eyelid-induced corneal changes. Refractively these corneal changes are clinically and optically significant with mean spherical and astigmatic changes of about 0.25 D after only 15 minutes of downward gaze (40° reading and steady fixation conditions). Due to the magnitude of these changes, eyelid pressure in downward gaze offers a possible explanation for some of the day-to-day variation observed in refraction. Considering the magnitude of these changes and previous work on their regression, it is recommended that sustained tasks performed in downward gaze should be avoided for at least 30 minutes before corneal and refractive assessment requiring high accuracy. Novel procedures were developed to use a thin (0.17 mm) tactile piezoresistive pressure sensor mounted on a rigid contact lens to measure eyelid pressure. A hydrostatic calibration system was constructed to convert raw digital output of the sensors to actual pressure units. Conditioning the sensor prior to use regulated the measurement response and sensor output was found to stabilise about 10 seconds after loading. The influences of various external factors on sensor output were studied. While the sensor output drifted slightly over several hours, it was not significant over the measurement time of 30 seconds used for eyelid pressure, as long as the length of the calibration and measurement recordings were matched. The error associated with calibrating at room temperature but measuring at ocular surface temperature led to a very small overestimation of pressure. To optimally position the sensor-contact lens combination under the eyelid margin, an in vivo measurement apparatus was constructed. Using this system, eyelid pressure increases were observed when the upper eyelid was placed on the sensor and a significant increase was apparent when the eyelid pressure was increased by pulling the upper eyelid tighter against the eye. For a group of young adult subjects, upper eyelid pressure was measured using this piezoresistive sensor system. Three models of contact between the eyelid and ocular surface were used to calibrate the pressure readings. The first model assumed contact between the eyelid and pressure sensor over more than the pressure cell width of 1.14 mm. Using thin pressure sensitive carbon paper placed under the eyelid, a contact imprint was measured and this width used for the second model of contact. Lastly as Marx’s line has been implicated as the region of contact with the ocular surface, its width was measured and used as the region of contact for the third model. The mean eyelid pressures calculated using these three models for the group of young subjects were 3.8 ± 0.7 mmHg (whole cell), 8.0 ± 3.4 mmHg (imprint width) and 55 ± 26 mmHg (Marx’s line). The carbon imprints using Pressurex-micro confirmed previous suggestions that a band of the eyelid margin has primary contact with the ocular surface and provided the best estimate of the contact region and hence eyelid pressure. Although it is difficult to directly compare the results with previous eyelid pressure measurement attempts, the eyelid pressure calculated using this model was slightly higher than previous manometer measurements but showed good agreement with the eyelid force estimated using an eyelid tensiometer. The work described in this thesis has shown that the eyelids have a significant influence on corneal shape, even after short-term tasks (15 minutes). Instrumentation was developed using piezoresistive sensors to measure eyelid pressure. Measurements for the upper eyelid combined with estimates of the contact region between the cornea and the eyelid enabled quantification of the upper eyelid pressure for a group of young adult subjects. These techniques will allow further investigation of the interaction between the eyelids and the surface of the eye.

The performance of a prolonged reading task by students with low vision

Purpose: Students often read for long periods and prolonged reading practice may be important for developing reading skills. For students with low vision, reading at a close working distance imposes high demands on their near visual functions, which might make it difficult to sustain prolonged reading. The aim of this study was to investigate the performance of a prolonged reading task by students with low vision. Method: Forty students with low vision, aged from eight to 20 years and without any intellectual, reading or learning disability, were recruited through the Paediatric Low Vision Clinic, Buranda, Queensland. Following a preliminary vision examination, reading performance measures—critical print size (CPS), maximum oral reading rates (MORR), near text visual acuity— were recorded using the Bailey-Lovie text reading charts before and after a 30-minute prolonged reading task. Results: The mean age of the participants was 13.03 ± 3 years. The distance and near visual acuities ranged between -0.1 to 1.24 logMAR and 0.0 to 1.60 logMAR, respectively. The mean working distance of the participants was 11.2 ± 5.8 cm. Most of the participants (65 per cent) in this study were able to complete the prolonged reading task. Overall, there was no significant change in CPS, MORR and near text visual acuity following the prolonged task (p > 0.05). MORR was significantly correlated to age and near text visual acuity (p < 0.05). Conclusions: In this study, students with low vision were able to maintain their reading performance over a 30-minute prolonged reading task. Overall, there was no significant increase or decrease in reading performance following a prolonged reading task performed at their habitual close working distances but there were wide individual variations within the group.

Can students with low vision maintain sustained reading rates?

Purpose: Students with low vision may be disadvantaged when compared with their normally sighted peers, as they frequently work at very short working distances and need to use low vision devices. The aim of this study was to examine the sustained reading rates of students with low vision and compare them with their peers with normal vision. The effects of visual acuity, acuity reserve and age on reading rate were also examined. Method: Fifty-six students (10 to 16 years of age), 26 with low vision and 30 with normal vision were required to read text continuously for 30 minutes. Their position in the text was recorded at two-minute intervals. Distance and near visual acuity, working distance, cause of low vision, reading rates and reading habits were recorded. Results: A total of 80.7 per cent of the students with low vision maintained a constant reading rate during the 30 minutes of reading, although they read at approximately half the rate (104 wpm) compared with their normally sighted peers (195 wpm). Only four of the low vision subjects could not complete the reading task. Reading rates increased significantly with acuity reserve and distance and near visual acuity but there was no significant relationship between age and sustained reading rate. Conclusions: The majority of students with low vision were able to maintain appropriate reading rates to cope in integrated educational settings. Surprisingly only relatively few subjects (16 per cent) used their prescribed low vision devices even though the average accommodative demand was 9 D and generally, they revealed a greater dislike of reading compared to students with normal vision.

Development of a method to measure upper eyelid pressure on the cornea

Purpose: The cornea is known to be susceptible to forces exerted by eyelids. There have been previous attempts to quantify eyelid pressure but the reliability of the results is unclear. The purpose of this study was to develop a technique using piezoresistive pressure sensors to measure upper eyelid pressure on the cornea. Methods: The technique was based on the use of thin (0.18 mm) tactile piezoresistive pressure sensors, which generate a signal related to the applied pressure. A range of factors that influence the response of this pressure sensor were investigated along with the optimal method of placing the sensor in the eye. Results: Curvature of the pressure sensor was found to impart force, so the sensor needed to remain flat during measurements. A large rigid contact lens was designed to have a flat region to which the sensor was attached. To stabilise the contact lens during measurement, an apparatus was designed to hold and position the sensor and contact lens combination on the eye. A calibration system was designed to apply even pressure to the sensor when attached to the contact lens, so the raw digital output could be converted to actual pressure units. Conclusions: Several novel procedures were developed to use tactile sensors to measure eyelid pressure. The quantification of eyelid pressure has a number of applications including eyelid reconstructive surgery and the design of soft and rigid contact lenses.

Influence of presbyopic corrections on driving-related eye and head movements

Purpose: To investigate whether wearing different presbyopic refractive corrections alters the pattern of eye and head movements when searching for dynamic targets in driving-related traffic scenes. Methods: Eye and head movements of 20 presbyopes (mean age = 56.2 ± 5.7 years), who had no experience of wearing presbyopic corrections or were unadapted wearers were recorded using the faceLABTM eye and head tracker, while wearing five different corrections: single vision lenses (SV), progressive addition lenses (PALs), bifocal spectacles (BIF), monovision and multifocal contact lenses (MTF CLs) in random order (within-subjects comparison). Recorded traffic scenes of suburban roads and expressways with edited targets were viewed as dynamic stimuli. Results: The magnitude of eye and head movements was significantly greater for SV, BIF and PALs than monovision and MTF CLs (p < 0.001). In addition, BIF wear led to more eye movements than PAL wear (p = 0.017), while PAL wear resulted in greater head movements than SV wear (p = 0.018). The ratio of eye to head movement was smaller for PALs than all other groups (p < 0.001). The number of saccades made to fixate a target was significantly higher for BIF and PALs than monovision or MTF CLs (p < 0.05). Conclusions: Different presbyopic corrections can alter eye and head movement patterns. Wearing spectacles such as BIF and PALs produced relatively greater eye and head movements and saccades when viewing dynamic targets. The impact of these changes in eye and head movement patterns may have implications for driving performance under real world driving conditions.

Effect of optical aberrations on the colour appearance of small lights

Purpose: Small red lights (one minute of arc or less) change colour appearance with positive defocus. We investigated the influence of longitudinal chromatic aberration and monochromatic aberrations on the colour appearance of small narrow band lights. Methods: Seven cyclopleged, trichromatic observers viewed a small light (one minute of arc, λmax = 510, 532, 550, 589, 620, 628 nm, approximately 19 per cent Weber contrast) centred within a black annulus (4.5 minutes of arc) and surrounded by a uniform white field (2,170 cd/m2). Pupil size was four millimetres. An optical trombone varied focus. Longitudinal chromatic aberration was controlled with a two component Powell achromatising lens that neutralises the eye’s chromatic aberration; a doublet that doubles and a triplet that reverses the eye’s chromatic aberration. Astigmatism and higher order monochromatic aberrations were corrected using adaptive optics. Results: Observers reported a change in appearance of the small red light (628 nm) without the Powell lens at +0.49 ± 0.21 D defocus and with the doublet at +0.62 ± 0.16 D. Appearance did not alter with the Powell lens, and five of seven observers reported the phenomenon with the triplet for negative defocus (-0.80 ± 0.47 D). Correction of aberrations did not significantly affect the magnitude at which the appearance of the red light changed (+0.44 ± 0.18 D without correction; +0.46 ± 0.16 D with correction). The change in colour appearance with defocus extended to other wavelengths (λmax = 510 to 620 nm), with directions of effects being reversed for short wavelengths relative to long wavelengths. Conclusions: Longitudinal chromatic aberrations but not monochromatic aberrations are involved in changing the appearance of small lights with defocus.

Simulated cataracts and their effect on speech intelligibility

Purpose: The classic study of Sumby and Pollack (1954, JASA, 26(2), 212-215) demonstrated that visual information aided speech intelligibility under noisy auditory conditions. Their work showed that visual information is especially useful under low signal-to-noise conditions where the auditory signal leaves greater margins for improvement. We investigated whether simulated cataracts interfered with the ability of participants to use visual cues to help disambiguate the auditory signal in the presence of auditory noise. Methods: Participants in the study were screened to ensure normal visual acuity (mean of 20/20) and normal hearing (auditory threshold ≤ 20 dB HL). Speech intelligibility was tested under an auditory only condition and two visual conditions: normal vision and simulated cataracts. The light scattering effects of cataracts were imitated using cataract-simulating filters. Participants wore blacked-out glasses in the auditory only condition and lens-free frames in the normal auditory-visual condition. Individual sentences were spoken by a live speaker in the presence of prerecorded four-person background babble set to a speech-to-noise ratio (SNR) of -16 dB. The SNR was determined in a preliminary experiment to support 50% correct identification of sentence under the auditory only conditions. The speaker was trained to match the rate, intensity and inflections of a prerecorded audio track of everyday speech sentences. The speaker was blind to the visual conditions of the participant to control for bias.Participants’ speech intelligibility was measured by comparing the accuracy of their written account of what they believed the speaker to have said to the actual spoken sentence. Results: Relative to the normal vision condition, speech intelligibility was significantly poorer when participants wore simulated catarcts. Conclusions: The results suggest that cataracts may interfere with the acquisition of visual cues to speech perception.

The effect of monocular and binocular cataract blur on postural stability among older adults

Purpose: First-eye cataract surgery can reduce the rate of falls among older adults, yet the effect of second-eye surgery on the rate of falling remains unclear. The present study investigated the effect of monocular and binocular simulated cataract blur on postural stability among older adults. Methods: Postural stability was assessed on 34 healthy older adults (mean 68.2 years, SD 3.5) with normal vision, using a portable force platform (BT4, HUR Labs, Finland) which collected data on centre of pressure (COP) displacement. Stability was assessed on firm and foam surfaces under four binocular viewing conditions using Vistech filters to simulate cataract blur: [1] best-corrected vision both eyes; [2] blur over non-dominant eye, [3] blur over dominant eye and [4] blur over both eyes. Binocular logMAR visual acuity, Pelli-Robson contrast sensitivity and stereoacuity were also measured under these viewing conditions and ocular dominance measured using the hole-in-card test. Generalized estimating equations with an exchangeable correlation structure examined the effect of the surface and vision conditions on postural stability. Results: Visual acuity and contrast sensitivity were significantly reduced under monocular and binocular cataract blur compared to normal viewing. All blur conditions resulted in loss of stereoacuity. Binocular cataract blur significantly reduced postural stability compared to normal vision on the firm (COP path length; p=0.013) and foam surface (anterior-posterior COP RMS, COP path length and COP area; p<0.01). However, no significant differences in postural stability were found between the monocular blur conditions compared to normal vision, or between the dominant and non-dominant monocular blur conditions on either the firm or foam surfaces. Conclusions: Findings indicate that binocular blur significantly impairs postural stability, and suggests that improvements in postural stability may justify first-eye cataract surgery, particularly during somatosensory disruption. Postural stability was not significantly impaired in the monocular cataract blur conditions compared to the normal vision condition, nor was there any effect of ocular dominance on postural stability in the presence of monocular cataract blur.

Vision-based topological mapping and localisation

Competent navigation in an environment is a major requirement for an autonomous mobile robot to accomplish its mission. Nowadays, many successful systems for navigating a mobile robot use an internal map which represents the environment in a detailed geometric manner. However, building, maintaining and using such environment maps for navigation is difficult because of perceptual aliasing and measurement noise. Moreover, geometric maps require the processing of huge amounts of data which is computationally expensive. This thesis addresses the problem of vision-based topological mapping and localisation for mobile robot navigation. Topological maps are concise and graphical representations of environments that are scalable and amenable to symbolic manipulation. Thus, they are well-suited for basic robot navigation applications, and also provide a representational basis for the procedural and semantic information needed for higher-level robotic tasks. In order to make vision-based topological navigation suitable for inexpensive mobile robots for the mass market we propose to characterise key places of the environment based on their visual appearance through colour histograms. The approach for representing places using visual appearance is based on the fact that colour histograms change slowly as the field of vision sweeps the scene when a robot moves through an environment. Hence, a place represents a region of the environment rather than a single position. We demonstrate in experiments using an indoor data set, that a topological map in which places are characterised using visual appearance augmented with metric clues provides sufficient information to perform continuous metric localisation which is robust to the kidnapped robot problem. Many topological mapping methods build a topological map by clustering visual observations to places. However, due to perceptual aliasing observations from different places may be mapped to the same place representative in the topological map. A main contribution of this thesis is a novel approach for dealing with the perceptual aliasing problem in topological mapping. We propose to incorporate neighbourhood relations for disambiguating places which otherwise are indistinguishable. We present a constraint based stochastic local search method which integrates the approach for place disambiguation in order to induce a topological map. Experiments show that the proposed method is capable of mapping environments with a high degree of perceptual aliasing, and that a small map is found quickly. Moreover, the method of using neighbourhood information for place disambiguation is integrated into a framework for topological off-line simultaneous localisation and mapping which does not require an initial categorisation of visual observations. Experiments on an indoor data set demonstrate the suitability of our method to reliably localise the robot while building a topological map.

Contributions of vision-proprioception interactions to the estimation of time-varying hand and target locations

We investigated the relative importance of vision and proprioception in estimating target and hand locations in a dynamic environment. Subjects performed a position estimation task in which a target moved horizontally on a screen at a constant velocity and then disappeared. They were asked to estimate the position of the invisible target under two conditions: passively observing and manually tracking. The tracking trials included three visual conditions with a cursor representing the hand position: always visible, disappearing simultaneously with target disappearance, and always invisible. The target’s invisible displacement was systematically underestimated during passive observation. In active conditions, tracking with the visible cursor significantly decreased the extent of underestimation. Tracking of the invisible target became much more accurate under this condition and was not affected by cursor disappearance. In a second experiment, subjects were asked to judge the position of their unseen hand instead of the target during tracking movements. Invisible hand displacements were also underestimated when compared with the actual displacement. Continuous or brief presentation of the cursor reduced the extent of underestimation. These results suggest that vision–proprioception interactions are critical for representing exact target–hand spatial relationships, and that such sensorimotor representation of hand kinematics serves a cognitive function in predicting target position. We propose a hypothesis that the central nervous system can utilize information derived from proprioception and/or efference copy for sensorimotor prediction of dynamic target and hand positions, but that effective use of this information for conscious estimation requires that it be presented in a form that corresponds to that used for the estimations.