Selective combination of visual and thermal imaging for resilient localization in adverse conditions : day and night, smoke and fire

Long-term autonomy in robotics requires perception systems that are resilient to unusual but realistic conditions that will eventually occur during extended missions. For example, unmanned ground vehicles (UGVs) need to be capable of operating safely in adverse and low-visibility conditions, such as at night or in the presence of smoke. The key to a resilient UGV perception system lies in the use of multiple sensor modalities, e.g., operating at different frequencies of the electromagnetic spectrum, to compensate for the limitations of a single sensor type. In this paper, visual and infrared imaging are combined in a Visual-SLAM algorithm to achieve localization. We propose to evaluate the quality of data provided by each sensor modality prior to data combination. This evaluation is used to discard low-quality data, i.e., data most likely to induce large localization errors. In this way, perceptual failures are anticipated and mitigated. An extensive experimental evaluation is conducted on data sets collected with a UGV in a range of environments and adverse conditions, including the presence of smoke (obstructing the visual camera), fire, extreme heat (saturating the infrared camera), low-light conditions (dusk), and at night with sudden variations of artificial light. A total of 240 trajectory estimates are obtained using five different variations of data sources and data combination strategies in the localization method. In particular, the proposed approach for selective data combination is compared to methods using a single sensor type or combining both modalities without preselection. We show that the proposed framework allows for camera-based localization resilient to a large range of low-visibility conditions.

Summarisation of short-term and long-term videos using texture and colour

We present a novel approach to video summarisation that makes use of a Bag-of-visual-Textures (BoT) approach. Two systems are proposed, one based solely on the BoT approach and another which exploits both colour information and BoT features. On 50 short-term videos from the Open Video Project we show that our BoT and fusion systems both achieve state-of-the-art performance, obtaining an average F-measure of 0.83 and 0.86 respectively, a relative improvement of 9% and 13% when compared to the previous state-of-the-art. When applied to a new underwater surveillance dataset containing 33 long-term videos, the proposed system reduces the amount of footage by a factor of 27, with only minor degradation in the information content. This order of magnitude reduction in video data represents significant savings in terms of time and potential labour cost when manually reviewing such footage.

An organization of visual and auditory fear conditioning in the lateral amygdala

Pavlovian fear conditioning is an evolutionary conserved and extensively studied form of associative learning and memory. In mammals, the lateral amygdala (LA) is an essential locus for Pavlovian fear learning and memory. Despite significant progress unraveling the cellular mechanisms responsible for fear conditioning, very little is known about the anatomical organization of neurons encoding fear conditioning in the LA. One key question is how fear conditioning to different sensory stimuli is organized in LA neuronal ensembles. Here we show that Pavlovian fear conditioning, formed through either the auditory or visual sensory modality, activates a similar density of LA neurons expressing a learning-induced phosphorylated extracellular signal-regulated kinase (p-ERK1/2). While the size of the neuron population specific to either memory was similar, the anatomical distribution differed. Several discrete sites in the LA contained a small but significant number of p-ERK1/2-expressing neurons specific to either sensory modality. The sites were anatomically localized to different levels of the longitudinal plane and were independent of both memory strength and the relative size of the activated neuronal population, suggesting some portion of the memory trace for auditory and visually cued fear conditioning is allocated differently in the LA. Presenting the visual stimulus by itself did not activate the same p-ERK1/2 neuron density or pattern, confirming the novelty of light alone cannot account for the specific pattern of activated neurons after visual fear conditioning. Together, these findings reveal an anatomical distribution of visual and auditory fear conditioning at the level of neuronal ensembles in the LA.

A neuronal topography of visual and acoustic fear conditioning within the amygdala

The lateral amygdala (LA) receives information from auditory and visual sensory modalities, and uses this information to encode lasting memories that predict threat. One unresolved question about the amygdala is how multiple memories, derived from different sensory modalities, are organized at the level of neuronal ensembles. We previously showed that fear conditioning using an auditory conditioned stimulus (CS) was spatially allocated to a stable topography of neurons within the dorsolateral amygdala (LAd) (Bergstrom et al, 2011). Here, we asked how fear conditioning using a visual CS is topographically organized within the amygdala. To induce a lasting fear memory trace we paired either an auditory (2 khz, 55 dB, 20 s) or visual (1 Hz, 0.5 s on/0.5 s off, 35 lux, 20 s) CS with a mild foot shock unconditioned stimulus (0.6 mA, 0.5 s). To detect learning-induced plasticity in amygdala neurons, we used immunohistochemistry with an antibody for phosphorylated mitogen-activated protein kinase (pMAPK). Using a principal components analysis-based approach to extract and visualize spatial patterns, we uncovered two unique spatial patterns of activated neurons in the LA that were associated with auditory and visual fear conditioning. The first spatial pattern was specific to auditory cued fear conditioning and consisted of activated neurons topographically organized throughout the LAd and ventrolateral nuclei (LAvl) of the LA. The second spatial pattern overlapped for auditory and visual fear conditioning and was comprised of activated neurons located mainly within the LAvl. Overall, the density of pMAPK labeled cells throughout the LA was greatest in the auditory CS group, even though freezing in response to the visual and auditory CS was equivalent. There were no differences detected in the number of pMAPK activated neurons within the basal amygdala nuclei. Together, these results provide the first basic knowledge about the organizational structure of two different fear engrams within the amygdala and suggest they are dissociable at the level of neuronal ensembles within the LA

The impact of simulated astigmatism on functional measures of visual performance

Uncorrected refractive error, including astigmatism, is a leading cause of reversible visual impairment. While the ability to perform vision-related daily activities is reduced when people are not optimally corrected, only limited research has investigated the impact of uncorrected astigmatism. Given the capacity to perform vision-related daily activities involves integration of a range of visual and cognitive cues, this research examined the impact of simulated astigmatism on visual tasks that also involved cognitive input. The research also examined whether the higher levels of complexity inherent in Chinese characters makes them more susceptible to the effects of astigmatism. The effects of different powers of astigmatism, as well as astigmatism at different axes were investigated in order to determine the minimum level of astigmatism that resulted in a decrement in visual performance.

Putting an "end" to the motor cortex representations of action words

Language processing is an example of implicit learning of multiple statistical cues that provide probabilistic information regarding word structure and use. Much of the current debate about language embodiment is devoted to how action words are represented in the brain, with motor cortex activity evoked by these words assumed to selectively reflect conceptual content and/or its simulation. We investigated whether motor cortex activity evoked by manual action words (e.g., caress) might reflect sensitivity to probabilistic orthographic-phonological cues to grammatical category embedded within individual words. We first review neuroimaging data demonstrating that nonwords evoke activity much more reliably than action words along the entire motor strip, encompassing regions proposed to be action category specific. Using fMRI, we found that disyllabic words denoting manual actions evoked increased motor cortex activity compared with non-body-part-related words (e.g., canyon), activity which overlaps that evoked by observing and executing hand movements. This result is typically interpreted in support of language embodiment. Crucially, we also found that disyllabic nonwords containing endings with probabilistic cues predictive of verb status (e.g., -eve) evoked increased activity compared with nonwords with endings predictive of noun status (e.g., -age) in the identical motor area. Thus, motor cortex responses to action words cannot be assumed to selectively reflect conceptual content and/or its simulation. Our results clearly demonstrate motor cortex activity reflects implicit processing of ortho-phonological statistical regularities that help to distinguish a word's grammatical class.

Comparison of visual inspection and structural-health monitoring as bridge condition assessment methods

This paper presents the results of a research project aimed at examining the capabilities and challenges of two distinct but not mutually exclusive approaches to in-service bridge assessment: visual inspection and installed monitoring systems. In this study, the intended functionality of both approaches was evaluated on its ability to identify potential structural damage and to provide decision-making support. Inspection and monitoring are compared in terms of their functional performance, cost, and barriers (real and perceived) to implementation. Both methods have strengths and weaknesses across the metrics analyzed, and it is likely that a hybrid evaluation technique that adopts both approaches will optimize efficiency of condition assessment and ultimately lead to better decision making.

Comparison of visual status of Iranian military and commercial drivers

Background There is no legal requirement for Iranian military truck drivers to undergo regular visual checkups as compared to commercial truck drivers. Objectives This study aimed to evaluate the impact of drivers’ visual checkups by comparing the visual function of Iranian military and commercial truck drivers. Patients and Methods In this comparative cross-sectional study, two hundred military and 200 commercial truck drivers were recruited and their Visual Acuity (VA), Visual Field (VF), color vision and Contrast Sensitivity (CS) were assessed and compared using the Snellen chart, confrontation screening method, D15 test and Pelli-Robson letter chart, respectively. A questionnaire regarding driving exposure and history of motor-vehicle crashes (MVCs) was also filled by drivers. Results were analyzed using an independent samples t-test, one-way ANOVA (assessing difference in number of MVCs across different age groups), chi-square test and Pearson correlation at statistical significance level of P < 0.05. Results Mean age was 41.6 ± 9.2 for the military truck drivers and 43.4 ± 10.9 for commercial truck drivers (P > 0.05). No significant difference between military and commercial drivers was found in terms of driving experience, number of MVCs, binocular VA, frequency of color vision defects and CS scores. In contrast, the last ocular examination was significantly earlier in military drivers than commercial drivers (P < 0.001). In addition, 4% of military drivers did not meet the national standards to drive as opposed to 2% of commercial drivers. There was a significant but weak correlation between binocular VA and age (r = 0.175, P < 0.001). However, CS showed a significantly moderate correlation with age (r = -0.488, P < 0.001). Conclusions The absence of legal requirement for regular eye examination in military drivers caused the incompetent drivers to be missed in contrast to commercial drivers. The need for scientific revision of VA standard for Iranian drivers is also discussed. The CS measurement in visual checkups of older drivers deserves to be investigated more thoroughly.

“I understand. I am a participant”: Navigating the ‘fuzzy’ boundaries of visual methods in qualitative longitudinal research

In this chapter we discuss how utilising the participatory visual methodology, photovoice, in an aged care context with its unique communal setting raised several ‘fuzzy boundary’ ethical dilemmas. To illustrate these challenges, we draw on immersive field notes from an ongoing qualitative longitudinal research (QLR) exploring the lived experience of aged care from the perspective of older residents, and focus on interactions with one participant, 81 year old Cassie. We explore how the camera, which is integral to the photovoice method, altered the researcher/participant ethical dynamics by becoming a continual ‘connector’ to the researcher. The camera took on a distinct agency, acting as a non-threatening ‘portal’ that lengthened contact, provided informal opportunities to alter the relationship dynamics and enabled unplanned participant revelation.

Relationships between user experience and intuitiveness of visual and physical interactions

This paper investigates the effects of experience on the intuitiveness of physical and visual interactions performed by airport security screeners. Using portable eye tracking glasses, 40 security screeners were observed in the field as they performed search, examination and interface interactions during airport security x-ray screening. Data from semi structured interviews was used to further explore the nature of visual and physical interactions. Results show there are positive relationships between experience and the intuitiveness of visual and physical interactions performed by security screeners. As experience is gained, security screeners are found to perform search, examination and interface interactions more intuitively. In addition to experience, results suggest that intuitiveness is affected by the nature and modality of activities performed. This inference was made based on the dominant processing styles associated with search and examination activities. The paper concludes by discussing the implications that this research has for the design of visual and physical interfaces. We recommend designing interfaces that build on users’ already established intuitive processes, and that reduce the cognitive load incurred during transitions between visual and physical interactions.

Evaluation of visual stress symptoms in Age-matched Dyslexic, Meares–Irlen syndrome (MIS) and normal adults

AIM To examine the prevalence of dyslexia and Meares–Irlen syndrome (MIS) among female students and determine their level of visual stress in comparison with normal subjects. METHODS A random sample of 450 female medical students of King Saud University Riyadh (age range, 18 - 30 years) responded to a wide range of questions designed to accomplish the aims of this study. The detailed questionnaire consisted of 54 questions with twelve questions enquiring on ocular history and demography of participants while 42 questions were on visual symptoms. Items were categorized into; critical and non-critical questions (CQ and NCQ) and were rated on four point Likert scale. Based on the responses obtained, the subjects were grouped into normal (control), dyslexic with or without MIS (Group 1) and subjects with MIS only (Group 2). Responses were analysed as averages and mean scores were calculated and compared between-groups using one way analysis of variance to evaluate total (TVSS = NCQ + CQ), critical and non-critical visual stress scores. The relationship between categorical variables such as age, handedness and condition were assessed with Chi- Square test. RESULTS The completion rate was 96.8% and majority of the respondents (92%) were normal readers, 2% dyslexic and 6% had MIS. They were age-matched. More than half of the participants had visited an eye care practitioner in the last 2yrs. About 13% were recommended eye exercises and one participant experienced pattern glare. Hand preference was not associated with any condition but Group 1 subjects (3/9, 33%) were significantly more likely to be diagnosed of lazy eye than Group 2 (2/27, 7%) and control (27/414, 5%) subjects. The mean ± SD of TVSS responses were 63 ± 14 but it was 44 ± 9 for CQ and 19 ± 5 for NCQ. Responses from all three variables were normally distributed but the CQ responses were on the average more positive (82%) in Group 2 and less positive (46%) in Group 1 than control. With NCQ, the responses were equally less positive in Group 1 and 2 than control. Group 2 subjects showed significantly higher TVSS (P = 0.002), NCQ (P = 0.006) and CQ (P = 0.008) visual stress scores than control but no difference between Group 1 and control subjects, was observed for all scores (P > 0.05, for all comparisons). CONCLUSION The prevalence of dyslexia and MIS among Saudi female students was 2 and 6%, respectively. Critical questions performed best for assessing visual stress symptoms in dyslexic and MIS subjects. Generally, students with MIS were more sensitive to visual stress than normal students but dyslexics were more likely to present with a lazy eye than MIS and normal readers.

Spatio temporal feature evaluation for action recognition

Spatio-Temporal interest points are the most popular feature representation in the field of action recognition. A variety of methods have been proposed to detect and describe local patches in video with several techniques reporting state of the art performance for action recognition. However, the reported results are obtained under different experimental settings with different datasets, making it difficult to compare the various approaches. As a result of this, we seek to comprehensively evaluate state of the art spatio- temporal features under a common evaluation framework with popular benchmark datasets (KTH, Weizmann) and more challenging datasets such as Hollywood2. The purpose of this work is to provide guidance for researchers, when selecting features for different applications with different environmental conditions. In this work we evaluate four popular descriptors (HOG, HOF, HOG/HOF, HOG3D) using a popular bag of visual features representation, and Support Vector Machines (SVM)for classification. Moreover, we provide an in-depth analysis of local feature descriptors and optimize the codebook sizes for different datasets with different descriptors. In this paper, we demonstrate that motion based features offer better performance than those that rely solely on spatial information, while features that combine both types of data are more consistent across a variety of conditions, but typically require a larger codebook for optimal performance.

Real-time video event detection in crowded scenes using MPEG derived features : a multiple instance learning approach

This paper presents an investigation into event detection in crowded scenes, where the event of interest co-occurs with other activities and only binary labels at the clip level are available. The proposed approach incorporates a fast feature descriptor from the MPEG domain, and a novel multiple instance learning (MIL) algorithm using sparse approximation and random sensing. MPEG motion vectors are used to build particle trajectories that represent the motion of objects in uniform video clips, and the MPEG DCT coefficients are used to compute a foreground map to remove background particles. Trajectories are transformed into the Fourier domain, and the Fourier representations are quantized into visual words using the K-Means algorithm. The proposed MIL algorithm models the scene as a linear combination of independent events, where each event is a distribution of visual words. Experimental results show that the proposed approaches achieve promising results for event detection compared to the state-of-the-art.

Involvement of target contrast in visual masking location tasks deficients in myopia

Purpose: There have been few studies of visual temporal processing of myopic eyes. This study investigated the visual performance of emmetropic and myopic eyes using a backward visual masking location task. Methods: Data were collected for 39 subjects (15 emmetropes, 12 stable myopes, 12 progressing myopes). In backward visual masking, a target’s visibility is reduced by a mask presented in quick succession ‘after’ the target. The target and mask stimuli were presented at different interstimulus intervals (from 12 to 300 ms). The task involved locating the position of a target letter with both a higher (seven per cent) and a lower (five per cent) contrast. Results: Emmetropic subjects had significantly better performance for the lower contrast location task than the myopes (F2,36 = 22.88; p < 0.001) but there was no difference between the progressing and stable myopic groups (p = 0.911). There were no differences between the groups for the higher contrast location task (F2,36 = 0.72, p = 0.495). No relationship between task performance and either the magnitude of myopia or axial length was found for either task. Conclusions: A location task deficit was observed in myopes only for lower contrast stimuli. Both emmetropic and myopic groups had better performance for the higher contrast task compared to the lower contrast task, with myopes showing considerable improvement. This suggests that five per cent contrast may be the contrast threshold required to bias the task towards the magnocellular system (where myopes have a temporal processing deficit). Alternatively, the task may be sensitive to the contrast sensitivity of the observer.

The effect of prolonged reading on visual functions and reading performance in students with low vision

This study is the first to investigate the effect of prolonged reading on reading performance and visual functions in students with low vision. The study focuses on one of the most common modes of achieving adequate magnification for reading by students with low vision, their close reading distance (proximal or relative distance magnification). Close reading distances impose high demands on near visual functions, such as accommodation and convergence. Previous research on accommodation in children with low vision shows that their accommodative responses are reduced compared to normal vision. In addition, there is an increased lag of accommodation for higher stimulus levels as may occur at close reading distance. Reduced accommodative responses in low vision and higher lag of accommodation at close reading distances together could impact on reading performance of students with low vision especially during prolonged reading tasks. The presence of convergence anomalies could further affect reading performance. Therefore, the aims of the present study were 1) To investigate the effect of prolonged reading on reading performance in students with low vision 2) To investigate the effect of prolonged reading on visual functions in students with low vision. This study was conducted as cross-sectional research on 42 students with low vision and a comparison group of 20 students with normal vision, aged 7 to 20 years. The students with low vision had vision impairments arising from a range of causes and represented a typical group of students with low vision, with no significant developmental delays, attending school in Brisbane, Australia. All participants underwent a battery of clinical tests before and after a prolonged reading task. An initial reading-specific history and pre-task measurements that included Bailey-Lovie distance and near visual acuities, Pelli-Robson contrast sensitivity, ocular deviations, sensory fusion, ocular motility, near point of accommodation (pull-away method), accuracy of accommodation (Monocular Estimation Method (MEM)) retinoscopy and Near Point of Convergence (NPC) (push-up method) were recorded for all participants. Reading performance measures were Maximum Oral Reading Rates (MORR), Near Text Visual Acuity (NTVA) and acuity reserves using Bailey-Lovie text charts. Symptoms of visual fatigue were assessed using the Convergence Insufficiency Symptom Survey (CISS) for all participants. Pre-task measurements of reading performance and accuracy of accommodation and NPC were compared with post-task measurements, to test for any effects of prolonged reading. The prolonged reading task involved reading a storybook silently for at least 30 minutes. The task was controlled for print size, contrast, difficulty level and content of the reading material. Silent Reading Rate (SRR) was recorded every 2 minutes during prolonged reading. Symptom scores and visual fatigue scores were also obtained for all participants. A visual fatigue analogue scale (VAS) was used to assess visual fatigue during the task, once at the beginning, once at the middle and once at the end of the task. In addition to the subjective assessments of visual fatigue, tonic accommodation was monitored using a photorefractor (PlusoptiX CR03™) every 6 minutes during the task, as an objective assessment of visual fatigue. Reading measures were done at the habitual reading distance of students with low vision and at 25 cms for students with normal vision. The initial history showed that the students with low vision read for significantly shorter periods at home compared to the students with normal vision. The working distances of participants with low vision ranged from 3-25 cms and half of them were not using any optical devices for magnification. Nearly half of the participants with low vision were able to resolve 8-point print (1M) at 25 cms. Half of the participants in the low vision group had ocular deviations and suppression at near. Reading rates were significantly reduced in students with low vision compared to those of students with normal vision. In addition, there were a significantly larger number of participants in the low vision group who could not sustain the 30-minute task compared to the normal vision group. However, there were no significant changes in reading rates during or following prolonged reading in either the low vision or normal vision groups. Individual changes in reading rates were independent of their baseline reading rates, indicating that the changes in reading rates during prolonged reading cannot be predicted from a typical clinical assessment of reading using brief reading tasks. Contrary to previous reports the silent reading rates of the students with low vision were significantly lower than their oral reading rates, although oral and silent reading was assessed using different methods. Although the visual acuity, contrast sensitivity, near point of convergence and accuracy of accommodation were significantly poorer for the low vision group compared to those of the normal vision group, there were no significant changes in any of these visual functions following prolonged reading in either group. Interestingly, a few students with low vision (n =10) were found to be reading at a distance closer than their near point of accommodation. This suggests a decreased sensitivity to blur. Further evaluation revealed that the equivalent intrinsic refractive errors (an estimate of the spherical dioptirc defocus which would be expected to yield a patient’s visual acuity in normal subjects) were significantly larger for the low vision group compared to those of the normal vision group. As expected, accommodative responses were significantly reduced for the low vision group compared to the expected norms, which is consistent with their close reading distances, reduced visual acuity and contrast sensitivity. For those in the low vision group who had an accommodative error exceeding their equivalent intrinsic refractive errors, a significant decrease in MORR was found following prolonged reading. The silent reading rates however were not significantly affected by accommodative errors in the present study. Suppression also had a significant impact on the changes in reading rates during prolonged reading. The participants who did not have suppression at near showed significant decreases in silent reading rates during and following prolonged reading. This impact of binocular vision at near on prolonged reading was possibly due to the high demands on convergence. The significant predictors of MORR in the low vision group were age, NTVA, reading interest and reading comprehension, accounting for 61.7% of the variances in MORR. SRR was not significantly influenced by any factors, except for the duration of the reading task sustained; participants with higher reading rates were able to sustain a longer reading duration. In students with normal vision, age was the only predictor of MORR. Participants with low vision also reported significantly greater visual fatigue compared to the normal vision group. Measures of tonic accommodation however were little influenced by visual fatigue in the present study. Visual fatigue analogue scores were found to be significantly associated with reading rates in students with low vision and normal vision. However, the patterns of association between visual fatigue and reading rates were different for SRR and MORR. The participants with low vision with higher symptom scores had lower SRRs and participants with higher visual fatigue had lower MORRs. As hypothesized, visual functions such as accuracy of accommodation and convergence did have an impact on prolonged reading in students with low vision, for students whose accommodative errors were greater than their equivalent intrinsic refractive errors, and for those who did not suppress one eye. Those students with low vision who have accommodative errors higher than their equivalent intrinsic refractive errors might significantly benefit from reading glasses. Similarly, considering prisms or occlusion for those without suppression might reduce the convergence demands in these students while using their close reading distances. The impact of these prescriptions on reading rates, reading interest and visual fatigue is an area of promising future research. Most importantly, it is evident from the present study that a combination of factors such as accommodative errors, near point of convergence and suppression should be considered when prescribing reading devices for students with low vision. Considering these factors would also assist rehabilitation specialists in identifying those students who are likely to experience difficulty in prolonged reading, which is otherwise not reflected during typical clinical reading assessments.