Selective combination of visual and thermal imaging for resilient localization in adverse conditions : day and night, smoke and fire

Long-term autonomy in robotics requires perception systems that are resilient to unusual but realistic conditions that will eventually occur during extended missions. For example, unmanned ground vehicles (UGVs) need to be capable of operating safely in adverse and low-visibility conditions, such as at night or in the presence of smoke. The key to a resilient UGV perception system lies in the use of multiple sensor modalities, e.g., operating at different frequencies of the electromagnetic spectrum, to compensate for the limitations of a single sensor type. In this paper, visual and infrared imaging are combined in a Visual-SLAM algorithm to achieve localization. We propose to evaluate the quality of data provided by each sensor modality prior to data combination. This evaluation is used to discard low-quality data, i.e., data most likely to induce large localization errors. In this way, perceptual failures are anticipated and mitigated. An extensive experimental evaluation is conducted on data sets collected with a UGV in a range of environments and adverse conditions, including the presence of smoke (obstructing the visual camera), fire, extreme heat (saturating the infrared camera), low-light conditions (dusk), and at night with sudden variations of artificial light. A total of 240 trajectory estimates are obtained using five different variations of data sources and data combination strategies in the localization method. In particular, the proposed approach for selective data combination is compared to methods using a single sensor type or combining both modalities without preselection. We show that the proposed framework allows for camera-based localization resilient to a large range of low-visibility conditions.

Perception Quality Evaluation with Visual and Infrared Cameras in Challenging Environmental Conditions

This work aims to contribute to the reliability and integrity of perceptual systems of unmanned ground vehicles (UGV). A method is proposed to evaluate the quality of sensor data prior to its use in a perception system by utilising a quality metric applied to heterogeneous sensor data such as visual and infrared camera images. The concept is illustrated specifically with sensor data that is evaluated prior to the use of the data in a standard SIFT feature extraction and matching technique. The method is then evaluated using various experimental data sets that were collected from a UGV in challenging environmental conditions, represented by the presence of airborne dust and smoke. In the first series of experiments, a motionless vehicle is observing a ’reference’ scene, then the method is extended to the case of a moving vehicle by compensating for its motion. This paper shows that it is possible to anticipate degradation of a perception algorithm by evaluating the input data prior to any actual execution of the algorithm.

Practical visual odometry for car-like vehicles

A method for calculating visual odometry for ground vehicles with car-like kinematic motion constraints similar to Ackerman's steering model is presented. By taking advantage of this non-holonomic driving constraint we show a simple and practical solution to the odometry calculation by clever placement of a single camera. The method has been implemented successfully on a large industrial forklift and a Toyota Prado SUV. Results from our industrial test site is presented demonstrating the applicability of this method as a replacement for wheel encoder-based odometry for these vehicles.

Visual field coordinates of pupillary circular axis and optical axis

Purpose: We term the visual field position from which the pupil appears most nearly circular as the pupillary circular axis (PCAx). The aim was to determine and compare the horizontal and vertical co-ordinates of the PCAx and optical axis from pupil shape and refraction information for only the horizontal meridian of the visual field. Method: The PCAx was determined from the changes with visual field angle in the ellipticity and orientation of pupil images out to ±90° from fixation along the horizontal meridian for the right eyes of 30 people. This axis was compared with the optical axis determined from the changes in the astigmatic components of the refractions for field angles out to ±35° in the same meridian. Results: The mean estimated horizontal and vertical field coordinates of the PCAx were (‒5.3±1.9°, ‒3.2±1.5°) compared with (‒4.8±5.1°, ‒1.5±3.4°) for the optical axis. The vertical co-ordinates of the two axes were just significantly different (p =0.03) but there was no significant correlation between them. Only the horizontal coordinate of the PCAx was significantly related to the refraction in the group. Conclusion: On average, the PCAx is displaced from the line-of-sight by about the same angle as the optical axis but there is more inter-subject variation in the position of the optical axis. When modelling the optical performance of the eye, it appears reasonable to assume that the pupil is circular when viewed along the line-of-sight.

IMU aided 3D visual odometry for car-like vehicles

We present a method for calculating odome- try in three-dimensions for car-like ground ve- hicles with an Ackerman-like steering model. In our approach we use the information from a single camera to derive the odometry in the plane and fuse it with roll and pitch informa- tion derived from an on-board IMU to extend to three-dimensions, thus providing odometric altitude as well as traditional x and y transla- tion. We have mounted the odometry module on a standard Toyota Prado SUV and present results from a car-park environment as well as from an off-road track.

Complete-linkage clustering for voice activity detection in audio and visual speech

We propose a novel technique for conducting robust voice activity detection (VAD) in high-noise recordings. We use Gaussian mixture modeling (GMM) to train two generic models; speech and non-speech. We then score smaller segments of a given (unseen) recording against each of these GMMs to obtain two respective likelihood scores for each segment. These scores are used to compute a dissimilarity measure between pairs of segments and to carry out complete-linkage clustering of the segments into speech and non-speech clusters. We compare the accuracy of our method against state-of-the-art and standardised VAD techniques to demonstrate an absolute improvement of 15% in half-total error rate (HTER) over the best performing baseline system and across the QUT-NOISE-TIMIT database. We then apply our approach to the Audio-Visual Database of American English (AVDBAE) to demonstrate the performance of our algorithm in using visual, audio-visual or a proposed fusion of these features.

Visual openness in urban environments: Measuring visual openness and visibility to natural landscapes in a changing urban landscape in Gold Coast, Australia

The visual characteristics of urban environments have been changing dramatically with the growth of cities around the world. Protection and enhancement of landscape character in urban environments have been one of the challenges for policy makers in addressing sustainable urban growth. Visual openness and enclosure in urban environments are important attributes in perception of visual space which affect the human interaction with physical space and which can be often modified by new developments. Measuring visual openness in urban areas results in more accurate, reliable, and systematic approach to manage and control visual qualities in growing cities. Recent advances in techniques in geographic information systems (GIS) and survey systems make it feasible to measure and quantify this attribute with a high degree of realism and precision. Previous studies in this field do not take full advantage of these improvements. This paper proposes a method to measure the visual openness and enclosure in a changing urban landscape in Australia, on the Gold Coast, by using the improved functionality in GIS. Using this method, visual openness is calculated and described for all publicly accessible areas in the selected study area. A final map is produced which shows the areas with highest visual openness and visibility to natural landscape resources. The output of this research can be used by planners and decision-makers in managing and controlling views in complex urban landscapes. Also, depending on the availability of GIS data, this method can be applied to any region including non-urban landscapes to help planners and policy-makers manage views and visual qualities.

A method for estimating intrinsic noise in electroretinographic (ERG) signals

Purpose To develop a signal processing paradigm for extracting ERG responses to temporal sinusoidal modulation with contrasts ranging from below perceptual threshold to suprathreshold contrasts. To estimate the magnitude of intrinsic noise in ERG signals at different stimulus contrasts. Methods Photopic test stimuli were generated using a 4-primary Maxwellian view optical system. The 4-primary lights were sinusoidally temporally modulated in-phase (36 Hz; 2.5 - 50% Michelson). The stimuli were presented in 1 s epochs separated by a 1 ms blank interval and repeated 160 times (160.16 s duration) during the recording of the continuous flicker ERG from the right eye using DTL fiber electrodes. After artefact rejection, the ERG signal was extracted using Fourier methods in each of the 1 s epochs where a stimulus was presented. The signal processing allows for computation of the intrinsic noise distribution in addition to the signal to noise (SNR) ratio. Results We provide the initial report that the ERG intrinsic noise distribution is independent of stimulus contrast whereas SNR decreases linearly with decreasing contrast until the noise limit at ~2.5%. The 1ms blank intervals between epochs de-correlated the ERG signal at the line frequency (50 Hz) and thus increased the SNR of the averaged response. We confirm that response amplitude increases linearly with stimulus contrast. The phase response shows a shallow positive relationship with stimulus contrast. Conclusions This new technique will enable recording of intrinsic noise in ERG signals above and below perceptual visual threshold and is suitable for measurement of continuous rod and cone ERGs across a range of temporal frequencies, and post-receptoral processing in the primary retinogeniculate pathways at low stimulus contrasts. The intrinsic noise distribution may have application as a biomarker for detecting changes in disease progression or treatment efficacy.

Comparison of visual status of Iranian military and commercial drivers

Background There is no legal requirement for Iranian military truck drivers to undergo regular visual checkups as compared to commercial truck drivers. Objectives This study aimed to evaluate the impact of drivers’ visual checkups by comparing the visual function of Iranian military and commercial truck drivers. Patients and Methods In this comparative cross-sectional study, two hundred military and 200 commercial truck drivers were recruited and their Visual Acuity (VA), Visual Field (VF), color vision and Contrast Sensitivity (CS) were assessed and compared using the Snellen chart, confrontation screening method, D15 test and Pelli-Robson letter chart, respectively. A questionnaire regarding driving exposure and history of motor-vehicle crashes (MVCs) was also filled by drivers. Results were analyzed using an independent samples t-test, one-way ANOVA (assessing difference in number of MVCs across different age groups), chi-square test and Pearson correlation at statistical significance level of P < 0.05. Results Mean age was 41.6 ± 9.2 for the military truck drivers and 43.4 ± 10.9 for commercial truck drivers (P > 0.05). No significant difference between military and commercial drivers was found in terms of driving experience, number of MVCs, binocular VA, frequency of color vision defects and CS scores. In contrast, the last ocular examination was significantly earlier in military drivers than commercial drivers (P < 0.001). In addition, 4% of military drivers did not meet the national standards to drive as opposed to 2% of commercial drivers. There was a significant but weak correlation between binocular VA and age (r = 0.175, P < 0.001). However, CS showed a significantly moderate correlation with age (r = -0.488, P < 0.001). Conclusions The absence of legal requirement for regular eye examination in military drivers caused the incompetent drivers to be missed in contrast to commercial drivers. The need for scientific revision of VA standard for Iranian drivers is also discussed. The CS measurement in visual checkups of older drivers deserves to be investigated more thoroughly.

“I understand. I am a participant”: Navigating the ‘fuzzy’ boundaries of visual methods in qualitative longitudinal research

In this chapter we discuss how utilising the participatory visual methodology, photovoice, in an aged care context with its unique communal setting raised several ‘fuzzy boundary’ ethical dilemmas. To illustrate these challenges, we draw on immersive field notes from an ongoing qualitative longitudinal research (QLR) exploring the lived experience of aged care from the perspective of older residents, and focus on interactions with one participant, 81 year old Cassie. We explore how the camera, which is integral to the photovoice method, altered the researcher/participant ethical dynamics by becoming a continual ‘connector’ to the researcher. The camera took on a distinct agency, acting as a non-threatening ‘portal’ that lengthened contact, provided informal opportunities to alter the relationship dynamics and enabled unplanned participant revelation.

MPC controlled multirotor with suspended slung load: System architecture and visual load detection

There is an increased interest in the use of Unmanned Aerial Vehicles for load transportation from environmental remote sensing to construction and parcel delivery. One of the main challenges is accurate control of the load position and trajectory. This paper presents an assessment of real flight trials for the control of an autonomous multi-rotor with a suspended slung load using only visual feedback to determine the load position. This method uses an onboard camera to take advantage of a common visual marker detection algorithm to robustly detect the load location. The load position is calculated using an onboard processor, and transmitted over a wireless network to a ground station integrating MATLAB/SIMULINK and Robotic Operating System (ROS) and a Model Predictive Controller (MPC) to control both the load and the UAV. To evaluate the system performance, the position of the load determined by the visual detection system in real flight is compared with data received by a motion tracking system. The multi-rotor position tracking performance is also analyzed by conducting flight trials using perfect load position data and data obtained only from the visual system. Results show very accurate estimation of the load position (~5% Offset) using only the visual system and demonstrate that the need for an external motion tracking system is not needed for this task.

Augmenting process elicitation with visual priming: An empirical exploration of user behaviour and modelling outcomes

Business process models have become an effective way of examining business practices to identify areas for improvement. While common information gathering approaches are generally efficacious, they can be quite time consuming and have the risk of developing inaccuracies when information is forgotten or incorrectly interpreted by analysts. In this study, the potential of a role-playing approach to process elicitation and specification has been examined. This method allows stakeholders to enter a virtual world and role-play actions similarly to how they would in reality. As actions are completed, a model is automatically developed, removing the need for stakeholders to learn and understand a modelling grammar. An empirical investigation comparing both the modelling outputs and participant behaviour of this virtual world role-play elicitor with an S-BPM process modelling tool found that while the modelling approaches of the two groups varied greatly, the virtual world elicitor may not only improve both the number of individual process task steps remembered and the correctness of task ordering, but also provide a reduction in the time required for stakeholders to model a process view.

Extending visual perception with haptic exploration for improved scene understanding

Scene understanding has been investigated from a mainly visual information point of view. Recently depth has been provided an extra wealth of information, allowing more geometric knowledge to fuse into scene understanding. Yet to form a holistic view, especially in robotic applications, one can create even more data by interacting with the world. In fact humans, when growing up, seem to heavily investigate the world around them by haptic exploration. We show an application of haptic exploration on a humanoid robot in cooperation with a learning method for object segmentation. The actions performed consecutively improve the segmentation of objects in the scene.

Visual performance at passive level crossings with long sighting distances

The third edition of the Australian Standard AS1742 Manual of Uniform Traffic Control Devices Part 7 provides a method of calculating the sighting distance required to safely proceed at passive level crossings based on the physics of moving vehicles. This required distance becomes greater with higher line speeds and slower, heavier vehicles so that it may return quite a long sighting distance. However, at such distances, there are also concerns around whether drivers would be able to reliably identify a train in order to make an informed decision regarding whether it would be safe to proceed across the level crossing. In order to determine whether drivers are able to make reliable judgements to proceed in these circumstances, this study assessed the distance at which a train first becomes identifiable to a driver as well as their, ability to detect the movement of the train. A site was selected in Victoria, and 36 participants with good visual acuity observed 4 trains in the 100-140 km/h range. While most participants could detect the train from a very long distance (2.2 km on average), they could only detect that the train was moving at much shorter distances (1.3 km on average). Large variability was observed between participants, with 4 participants consistently detecting trains later than other participants. Participants tended to improve in their capacity to detect the presence of the train with practice, but a similar trend was not observed for detection of the movement of the train. Participants were consistently poor at accurately judging the approach speed of trains, with large underestimations at all investigated distances.

An approximate method for the solution of an influence function foil rolling model

Fleck and Johnson (Int. J. Mech. Sci. 29 (1987) 507) and Fleck et al. (Proc. Inst. Mech. Eng. 206 (1992) 119) have developed foil rolling models which allow for large deformations in the roll profile, including the possibility that the rolls flatten completely. However, these models require computationally expensive iterative solution techniques. A new approach to the approximate solution of the Fleck et al. (1992) Influence Function Model has been developed using both analytic and approximation techniques. The numerical difficulties arising from solving an integral equation in the flattened region have been reduced by applying an Inverse Hilbert Transform to get an analytic expression for the pressure. The method described in this paper is applicable to cases where there is or there is not a flat region.