Cascading appearance-based features for visual speaker verification

The cascading appearance-based (CAB) feature extraction technique has established itself as the state of the art in extracting dynamic visual speech features for speech recognition. In this paper, we will focus on investigating the effectiveness of this technique for the related speaker verification application. By investigating the speaker verification ability of each stage of the cascade we will demonstrate that the same steps taken to reduce static speaker and environmental information for the speech recognition application also provide similar improvements for speaker recognition. These results suggest that visual speaker recognition can improve considerable when conducted solely through a consideration of the dynamic speech information rather than the static appearance of the speaker's mouth region.

FAB-MAP + RatSLAM : appearance-based SLAM for multiple times of day

Appearance-based mapping and localisation is especially challenging when separate processes of mapping and localisation occur at different times of day. The problem is exacerbated in the outdoors where continuous change in sun angle can drastically affect the appearance of a scene. We confront this challenge by fusing the probabilistic local feature based data association method of FAB-MAP with the pose cell filtering and experience mapping of RatSLAM. We evaluate the effectiveness of our amalgamation of methods using five datasets captured throughout the day from a single camera driven through a network of suburban streets. We show further results when the streets are re-visited three weeks later, and draw conclusions on the value of the system for lifelong mapping.

Cascading appearance-based features for visual voice activity detection

The detection of voice activity is a challenging problem, especially when the level of acoustic noise is high. Most current approaches only utilise the audio signal, making them susceptible to acoustic noise. An obvious approach to overcome this is to use the visual modality. The current state-of-the-art visual feature extraction technique is one that uses a cascade of visual features (i.e. 2D-DCT, feature mean normalisation, interstep LDA). In this paper, we investigate the effectiveness of this technique for the task of visual voice activity detection (VAD), and analyse each stage of the cascade and quantify the relative improvement in performance gained by each successive stage. The experiments were conducted on the CUAVE database and our results highlight that the dynamics of the visual modality can be used to good effect to improve visual voice activity detection performance.

Continuous appearance-based trajectory SLAM

This paper describes a novel probabilistic approach to incorporating odometric information into appearance-based SLAM systems, without performing metric map construction or calculating relative feature geometry. The proposed system, dubbed Continuous Appearance-based Trajectory SLAM (CAT-SLAM), represents location as a probability distribution along a trajectory, and represents appearance continuously over the trajectory rather than at discrete locations. The distribution is evaluated using a Rao-Blackwellised particle filter, which weights particles based on local appearance and odometric similarity and explicitly models both the likelihood of revisiting previous locations and visiting new locations. A modified resampling scheme counters particle deprivation and allows loop closure updates to be performed in constant time regardless of map size. We compare the performance of CAT-SLAM to FAB-MAP (an appearance-only SLAM algorithm) in an outdoor environment, demonstrating a threefold increase in the number of correct loop closures detected by CAT-SLAM.

Loop closure detection on a suburban road network using a continuous appearance-based trajectory

This paper presents a novel technique for performing SLAM along a continuous trajectory of appearance. Derived from components of FastSLAM and FAB-MAP, the new system dubbed Continuous Appearance-based Trajectory SLAM (CAT-SLAM) augments appearancebased place recognition with particle-filter based ‘pose filtering’ within a probabilistic framework, without calculating global feature geometry or performing 3D map construction. For loop closure detection CAT-SLAM updates in constant time regardless of map size. We evaluate the effectiveness of CAT-SLAM on a 16km outdoor road network and determine its loop closure performance relative to FAB-MAP. CAT-SLAM recognizes 3 times the number of loop closures for the case where no false positives occur, demonstrating its potential use for robust loop closure detection in large environments.

Improving recall in appearance-based visual SLAM using visual expectation

In this paper, we present a new algorithm for boosting visual template recall performance through a process of visual expectation. Visual expectation dynamically modifies the recognition thresholds of learnt visual templates based on recently matched templates, improving the recall of sequences of familiar places while keeping precision high, without any feedback from a mapping backend. We demonstrate the performance benefits of visual expectation using two 17 kilometer datasets gathered in an outdoor environment at two times separated by three weeks. The visual expectation algorithm provides up to a 100% improvement in recall. We also combine the visual expectation algorithm with the RatSLAM SLAM system and show how the algorithm enables successful mapping

CAT-SLAM : probabilistic localisation and mapping using a continuous appearance-based trajectory

This paper describes a new system, dubbed Continuous Appearance-based Trajectory Simultaneous Localisation and Mapping (CAT-SLAM), which augments sequential appearance-based place recognition with local metric pose filtering to improve the frequency and reliability of appearance-based loop closure. As in other approaches to appearance-based mapping, loop closure is performed without calculating global feature geometry or performing 3D map construction. Loop-closure filtering uses a probabilistic distribution of possible loop closures along the robot’s previous trajectory, which is represented by a linked list of previously visited locations linked by odometric information. Sequential appearance-based place recognition and local metric pose filtering are evaluated simultaneously using a Rao–Blackwellised particle filter, which weights particles based on appearance matching over sequential frames and the similarity of robot motion along the trajectory. The particle filter explicitly models both the likelihood of revisiting previous locations and exploring new locations. A modified resampling scheme counters particle deprivation and allows loop-closure updates to be performed in constant time for a given environment. We compare the performance of CAT-SLAM with FAB-MAP (a state-of-the-art appearance-only SLAM algorithm) using multiple real-world datasets, demonstrating an increase in the number of correct loop closures detected by CAT-SLAM.

OpenFABMAP : an open source toolbox for appearance-based loop closure detection

Appearance-based loop closure techniques, which leverage the high information content of visual images and can be used independently of pose, are now widely used in robotic applications. The current state-of-the-art in the field is Fast Appearance-Based Mapping (FAB-MAP) having been demonstrated in several seminal robotic mapping experiments. In this paper, we describe OpenFABMAP, a fully open source implementation of the original FAB-MAP algorithm. Beyond the benefits of full user access to the source code, OpenFABMAP provides a number of configurable options including rapid codebook training and interest point feature tuning. We demonstrate the performance of OpenFABMAP on a number of published datasets and demonstrate the advantages of quick algorithm customisation. We present results from OpenFABMAP’s application in a highly varied range of robotics research scenarios.

Towards persistent localization and mapping with a continuous appearance-based topology

Appearance-based localization can provide loop closure detection at vast scales regardless of accumulated metric error. However, the computation time and memory requirements of current appearance-based methods scale not only with the size of the environment but also with the operation time of the platform. Additionally, repeated visits to locations will develop multiple competing representations, which will reduce recall performance over time. These properties impose severe restrictions on long-term autonomy for mobile robots, as loop closure performance will inevitably degrade with increased operation time. In this paper we present a graphical extension to CAT-SLAM, a particle filter-based algorithm for appearance-based localization and mapping, to provide constant computation and memory requirements over time and minimal degradation of recall performance during repeated visits to locations. We demonstrate loop closure detection in a large urban environment with capped computation time and memory requirements and performance exceeding previous appearance-based methods by a factor of 2. We discuss the limitations of the algorithm with respect to environment size, appearance change over time and applications in topological planning and navigation for long-term robot operation.

Capping computation time and storage requirements for appearance-based localization with CAT-SLAM

Appearance-based localization is increasingly used for loop closure detection in metric SLAM systems. Since it relies only upon the appearance-based similarity between images from two locations, it can perform loop closure regardless of accumulated metric error. However, the computation time and memory requirements of current appearance-based methods scale linearly not only with the size of the environment but also with the operation time of the platform. These properties impose severe restrictions on longterm autonomy for mobile robots, as loop closure performance will inevitably degrade with increased operation time. We present a set of improvements to the appearance-based SLAM algorithm CAT-SLAM to constrain computation scaling and memory usage with minimal degradation in performance over time. The appearance-based comparison stage is accelerated by exploiting properties of the particle observation update, and nodes in the continuous trajectory map are removed according to minimal information loss criteria. We demonstrate constant time and space loop closure detection in a large urban environment with recall performance exceeding FAB-MAP by a factor of 3 at 100% precision, and investigate the minimum computational and memory requirements for maintaining mapping performance.

Application of an appearance-based intervention to improve sun protection outcomes of outdoor workers in Queensland, Australia

Outdoor workers are exposed to high levels of ultraviolet radiation (UVR) and may thus be at greater risk to experience UVR-related health effects such as skin cancer, sun burn, and cataracts. A number of intervention trials (n=14) have aimed to improve outdoor workers’ work-related sun protection cognitions and behaviours. Only one study however has reported the use of UV-photography as part of a multi-component intervention. This study was performed in the USA and showed long-term (12 months) improvements in work-related sun protection behaviours. Intervention effects of the other studies have varied greatly, depending on the population studied, intervention applied, and measurement of effect. Previous studies have not assessed whether: - Interventions are similarly effective for workers in stringent and less stringent policy organisations; - Policy effect is translated into workers’ leisure time protection; - Implemented interventions are effective in the long-term; - The facial UV-photograph technique is effective in Australian male outdoor workers without a large additional intervention package, and; - Such interventions will also affect workers’ leisure time sun-related cognitions and behaviours. Therefore, the present Protection of Outdoor Workers from Environmental Radiation [POWER]-study aimed to fill these gaps and had the objectives of: a) assessing outdoor workers’ sun-related cognitions and behaviours at work and during leisure time in stringent and less stringent sun protection policy environments; b) assessing the effect of an appearance-based intervention on workers’ risk perceptions, intentions and behaviours over time; c) assessing whether the intervention was equally effective within the two policy settings; and d) assessing the immediate post-intervention effect. Effectiveness was described in terms of changes in sun-related risk perceptions and intentions (as these factors were shown to be main precursors of behaviour change in many health promotion theories) and behaviour. The study purposefully selected and recruited two organisations with a large outdoor worker contingent in Queensland, Australia within a 40 kilometre radius of Brisbane. The two organisations differed in the stringency of implementation and reinforcement of their organisational sun protection policy. Data were collected from 154 male predominantly Australian born outdoor workers with an average age of 37 years and predominantly medium to fair skin (83%). Sun-related cognitions and behaviours of workers were assessed using self-report questionnaires at baseline and six to twelve months later. Variation in follow-up time was due to a time difference in the recruitment of the two organisations. Participants within each organisation were assigned to an intervention or control group. The intervention group participants received a one-off personalised Skin Cancer Risk Assessment Tool [SCRAT]-letter and a facial UV-photograph with detailed verbal information. This was followed by an immediate post-intervention questionnaire within three months of the start of the study. The control group only received the baseline and follow-up questionnaire. Data were analysed using a variety of techniques including: descriptive analyses, parametric and non-parametric tests, and generalised estimating equations. A 15% proportional difference observed was deemed of clinical significance, with the addition of reported statistical significance (p<0.05) where applicable. Objective 1: Assess and compare the current sun-related risk perceptions, intentions, behaviours, and policy awareness of outdoor workers in stringent and less stringent sun protection policy settings. Workers within the two organisations (stringent n=89 and less stringent n=65) were similar in their knowledge about skin cancer, self efficacy, attitudes, and social norms regarding sun protection at work and during leisure time. Participants were predominantly in favour of sun protection. Results highlighted that compared to workers in a less stringent policy organisation working for an organisation with stringent sun protection policies and practices resulted in more desirable sun protection intentions (less willing to tan p=0.03) ; actual behaviours at work (sufficient use of upper and lower body protection, headgear, and sunglasses (p<0.001 for all comparisons), and greater policy awareness (awareness of repercussions if Personal Protective Equipment (PPE) was not used, p<0.001)). However the effect of the work-related sun protection policy was found not to extend to leisure time sun protection. Objective 2: Compare changes in sun-related risk perceptions, intentions, and behaviours between the intervention and control group. The effect of the intervention was minimal and mainly resulted in a clinically significant reduction in work-related self-perceived risk of developing skin cancer in the intervention compared to the control group (16% and 32% for intervention and control group, respectively estimated their risk higher compared to other outdoor workers: , p=0.11). No other clinical significant effects were observed at 12 months follow-up. Objective 3: Assess whether the intervention was equally effective in the stringent sun protection policy organisation and the less stringent sun protection policy organisation. The appearance-based intervention resulted in a clinically significant improvement in the stringent policy intervention group participants’ intention to protect from the sun at work (workplace*time interaction, p=0.01). In addition to a reduction in their willingness to tan both at work (will tan at baseline: 17% and 61%, p=0.06, at follow-up: 54% and 33%, p=0.07, stringent and less stringent policy intervention group respectively. The workplace*time interaction was significant p<0.001) and during leisure time (will tan at baseline: 42% and 78%, p=0.01, at follow-up: 50% and 63%, p=0.43, stringent and less stringent policy intervention group respectively. The workplace*time interaction was significant p=0.01) over the course of the study compared to the less stringent policy intervention group. However, no changes in actual sun protection behaviours were found. Objective 4: Examine the effect of the intervention on level of alarm and concern regarding the health of the skin as well as sun protection behaviours in both organisations. The immediate post-intervention results showed that the stringent policy organisation participants indicated to be less alarmed (p=0.04) and concerned (p<0.01) about the health of their skin and less likely to show the facial UV-photograph to others (family p=0.03) compared to the less stringent policy participants. A clinically significantly larger proportion of participants from the stringent policy organisation reported they worried more about skin cancer (65%) and skin freckling (43%) compared to those in the less stringent policy organisation (46%,and 23% respectively , after seeing the UV-photograph). In summary the results of this study suggest that the having a stringent work-related sun protection policy was significantly related to for work-time sun protection practices, but did not extend to leisure time sun protection. This could reflect the insufficient level of sun protection found in the general Australian population during leisure time. Alternatively, reactance caused by being restricted in personal decisions through work-time policy could have contributed to lower leisure time sun protection. Finally, other factors could have also contributed to the less than optimal leisure time sun protection behaviours reported, such as unmeasured personal or cultural barriers. All these factors combined may have lead to reduced willingness to take proper preventive action during leisure time exposure. The intervention did not result in any measurable difference between the intervention and control groups in sun protection behaviours in this population, potentially due to the long lag time between the implementation of the intervention and assessment at 12-months follow-up. In addition, high levels of sun protection behaviours were found at baseline (ceiling effect) which left little room for improvement. Further, this study did not assess sunscreen use, which was the predominant behaviour assessed in previous effective appearance-based interventions trials. Additionally, previous trials were mainly conducted in female populations, whilst the POWER-study’s population was all male. The observed immediate post-intervention result could be due to more emphasis being placed on sun protection and risks related to sun exposure in the stringent policy organisation. Therefore participants from the stringent policy organisation could have been more aware of harmful effects of UVR and hence, by knowing that they usually protect adequately, not be as alarmed or concerned as the participants from the less stringent policy organisation. In conclusion, a facial UV-photograph and SCRAT-letter information alone may not achieve large changes in sun-related cognitions and behaviour, especially of assessed 6-12 months after the intervention was implemented and in workers who are already quite well protected. Differences found between workers in the present study appear to be more attributable to organisational policy. However, against a background of organisational policy, this intervention may be a useful addition to sun-related workplace health and safety programs. The study findings have been interpreted while respecting a number of limitations. These have included non-random allocation of participants due to pre-organised allocation of participants to study group in one organisation and difficulty in separating participants from either study group. Due to the transient nature of the outdoor worker population, only 105 of 154 workers available at baseline could be reached for follow-up. (attrition rate=32%). In addition the discrepancy in the time to follow-up assessment between the two organisations was a limitation of the current study. Given the caveats of this research, the following recommendations were made for future research: - Consensus should be reached to define "outdoor worker" in terms of time spent outside at work as well as in the way sun protection behaviours are measured and reported. - Future studies should implement and assess the value of the facial UV-photographs in a wide range of outdoor worker organisations and countries. - More timely and frequent follow-up assessments should be implemented in intervention studies to determine the intervention effect and to identify the best timing of booster sessions to optimise results. - Future research should continue to aim to target outdoor workers’ leisure time cognitions and behaviours and improve these if possible. Overall, policy appears to be an important factor in workers’ compliance with work-time use of sun protection. Given the evidence generated by this research, organisations employing outdoor workers should consider stringent implementation and reinforcement of a sun protection policy. Finally, more research is needed to improve ways to generate desirable behaviour in this population during leisure time.

Towards persistent indoor appearance-based localization, mapping and navigation using CAT-Graph

The challenge of persistent appearance-based navigation and mapping is to develop an autonomous robotic vision system that can simultaneously localize, map and navigate over the lifetime of the robot. However, the computation time and memory requirements of current appearance-based methods typically scale not only with the size of the environment but also with the operation time of the platform; also, repeated revisits to locations will develop multiple competing representations which reduce recall performance. In this paper we present a solution to the persistent localization, mapping and global path planning problem in the context of a delivery robot in an office environment over a one-week period. Using a graphical appearance-based SLAM algorithm, CAT-Graph, we demonstrate constant time and memory loop closure detection with minimal degradation during repeated revisits to locations, along with topological path planning that improves over time without using a global metric representation. We compare the localization performance of CAT-Graph to openFABMAP, an appearance-only SLAM algorithm, and the path planning performance to occupancy-grid based metric SLAM. We discuss the limitations of the algorithm with regard to environment change over time and illustrate how the topological graph representation can be coupled with local movement behaviors for persistent autonomous robot navigation.

Image Region Salience for Improving Appearance-Based Place Recognition using a Supervised Classifier System

Many state of the art vision-based Simultaneous Localisation And Mapping (SLAM) and place recognition systems compute the salience of visual features in their environment. As computing salience can be problematic in radically changing environments new low resolution feature-less systems have been introduced, such as SeqSLAM, all of which consider the whole image. In this paper, we implement a supervised classifier system (UCS) to learn the salience of image regions for place recognition by feature-less systems. SeqSLAM only slightly benefits from the results of training, on the challenging real world Eynsham dataset, as it already appears to filter less useful regions of a panoramic image. However, when recognition is limited to specific image regions performance improves by more than an order of magnitude by utilising the learnt image region saliency. We then investigate whether the region salience generated from the Eynsham dataset generalizes to another car-based dataset using a perspective camera. The results suggest the general applicability of an image region salience mask for optimizing route-based navigation applications.

Continuous appearance-based localisation and mapping

This thesis presents a novel approach to mobile robot navigation using visual information towards the goal of long-term autonomy. A novel concept of a continuous appearance-based trajectory is proposed in order to solve the limitations of previous robot navigation systems, and two new algorithms for mobile robots, CAT-SLAM and CAT-Graph, are presented and evaluated. These algorithms yield performance exceeding state-of-the-art methods on public benchmark datasets and large-scale real-world environments, and will help enable widespread use of mobile robots in everyday applications.

Towards training-free appearance-based localization : probabilistic models for whole-image descriptors

Whole image descriptors have recently been shown to be remarkably robust to perceptual change especially compared to local features. However, whole-image-based localization systems typically rely on heuristic methods for determining appropriate matching thresholds in a particular environment. These environment-specific tuning requirements and the lack of a meaningful interpretation of these arbitrary thresholds limits the general applicability of these systems. In this paper we present a Bayesian model of probability for whole-image descriptors that can be seamlessly integrated into localization systems designed for probabilistic visual input. We demonstrate this method using CAT-Graph, an appearance-based visual localization system originally designed for a FAB-MAP-style probabilistic input. We show that using whole-image descriptors as visual input extends CAT-Graph’s functionality to environments that experience a greater amount of perceptual change. We also present a method of estimating whole-image probability models in an online manner, removing the need for a prior training phase. We show that this online, automated training method can perform comparably to pre-trained, manually tuned local descriptor methods.