Selective combination of visual and thermal imaging for resilient localization in adverse conditions : day and night, smoke and fire

Long-term autonomy in robotics requires perception systems that are resilient to unusual but realistic conditions that will eventually occur during extended missions. For example, unmanned ground vehicles (UGVs) need to be capable of operating safely in adverse and low-visibility conditions, such as at night or in the presence of smoke. The key to a resilient UGV perception system lies in the use of multiple sensor modalities, e.g., operating at different frequencies of the electromagnetic spectrum, to compensate for the limitations of a single sensor type. In this paper, visual and infrared imaging are combined in a Visual-SLAM algorithm to achieve localization. We propose to evaluate the quality of data provided by each sensor modality prior to data combination. This evaluation is used to discard low-quality data, i.e., data most likely to induce large localization errors. In this way, perceptual failures are anticipated and mitigated. An extensive experimental evaluation is conducted on data sets collected with a UGV in a range of environments and adverse conditions, including the presence of smoke (obstructing the visual camera), fire, extreme heat (saturating the infrared camera), low-light conditions (dusk), and at night with sudden variations of artificial light. A total of 240 trajectory estimates are obtained using five different variations of data sources and data combination strategies in the localization method. In particular, the proposed approach for selective data combination is compared to methods using a single sensor type or combining both modalities without preselection. We show that the proposed framework allows for camera-based localization resilient to a large range of low-visibility conditions.

Selection et controle de modes de deplacement pour un robot mobile autonome en environnements naturels

Autonomous navigation and locomotion of a mobile robot in natural environments remain a rather open issue. Several functionalities are required to complete the usual perception/decision/action cycle. They can be divided in two main categories : navigation (perception and decision about the movement) and locomotion (movement execution). In order to be able to face the large range of possible situations in natural environments, it is essential to make use of various kinds of complementary functionalities, defining various navigation and locomotion modes. Indeed, a number of navigation and locomotion approaches have been proposed in the literature for the last years, but none can pretend being able to achieve autonomous navigation and locomotion in every situation. Thus, it seems relevant to endow an outdoor mobile robot with several complementary navigation and locomotion modes. Accordingly, the robot must also have means to select the most appropriate mode to apply. This thesis proposes the development of such a navigation/locomotion mode selection system, based on two types of data: an observation of the context to determine in what kind of situation the robot has to achieve its movement and an evaluation of the behavior of the current mode, made by monitors which influence the transitions towards other modes when the behavior of the current one is considered as non satisfying. Hence, this document introduces a probabilistic framework for the estimation of the mode to be applied, some navigation and locomotion modes used, a qualitative terrain representation method (based on the evaluation of a difficulty computed from the placement of the robot's structure on a digital elevation map), and monitors that check the behavior of the modes used (evaluation of rolling locomotion efficiency, robot's attitude and configuration watching. . .). Some experimental results obtained with those elements integrated on board two different outdoor robots are presented and discussed.

A probabilistic framework to monitor a multi-mode outdoor robot

This paper presents an approach to autonomously monitor the behavior of a robot endowed with several navigation and locomotion modes, adapted to the terrain to traverse. The mode selection process is done in two steps: the best suited mode is firstly selected on the basis of initial information or a qualitative map built on-line by the robot. Then, the motions of the robot are monitored by various processes that update mode transition probabilities in a Markov system. The paper focuses on this latter selection process: the overall approach is depicted, and preliminary experimental results are presented

Localization of glucocorticoid receptors at postsynaptic membranes in the lateral amygdala

Glucocorticoids, released in high concentrations from the adrenal cortex during stressful experiences, bind to glucocorticoid receptors in nuclear and peri-nuclear sites in neuronal somata. Their classically known mode of action is to induce gene promoter receptors to alter gene transcription. Nuclear glucocorticoid receptors are particularly dense in brain regions crucial for memory, including memory of stressful experiences, such as the hippocampus and amygdala. While it has been proposed that glucocorticoids may also act via membrane bound receptors, the existence of the latter remains controversial. Using electron microscopy, we found glucocorticoid receptors localized to non-genomic sites in rat lateral amygdala, glia processes, presynaptic terminals, neuronal dendrites, and dendritic spines including spine organelles and postsynaptic membrane densities. The lateral nucleus of the amygdala is a region specifically implicated in the formation of memories for stressful experiences. These newly observed glucocorticoid receptor immunoreactive sites were in addition to glucocorticoid receptor immunoreactive signals observed using electron and confocal microscopy in lateral amygdala principal neuron and GABA neuron soma and nuclei, cellular domains traditionally associated with glucocorticoid immunoreactivity. In lateral amygdala, glucocorticoid receptors are thus also localized to non-nuclear-membrane translocation sites, particularly dendritic spines, where they show an affinity for postsynaptic membrane densities, and may have a specialized role in modulating synaptic transmission plasticity related to fear and emotional memory.

Towards training-free appearance-based localization : probabilistic models for whole-image descriptors

Whole image descriptors have recently been shown to be remarkably robust to perceptual change especially compared to local features. However, whole-image-based localization systems typically rely on heuristic methods for determining appropriate matching thresholds in a particular environment. These environment-specific tuning requirements and the lack of a meaningful interpretation of these arbitrary thresholds limits the general applicability of these systems. In this paper we present a Bayesian model of probability for whole-image descriptors that can be seamlessly integrated into localization systems designed for probabilistic visual input. We demonstrate this method using CAT-Graph, an appearance-based visual localization system originally designed for a FAB-MAP-style probabilistic input. We show that using whole-image descriptors as visual input extends CAT-Graph’s functionality to environments that experience a greater amount of perceptual change. We also present a method of estimating whole-image probability models in an online manner, removing the need for a prior training phase. We show that this online, automated training method can perform comparably to pre-trained, manually tuned local descriptor methods.

Acoustic Masking of a Stealthy Outdoor Robot Tracking a Dynamic Target

This work is motivated by the desire to covertly track mobile targets, either animal or human, in previously unmapped outdoor natural environments using off-road robotic platforms with a non-negligible acoustic signature. The use of robots for stealthy surveillance is not new. Many studies exist but only consider the navigation problem to maintain visual covertness. However, robotic systems also have a significant acoustic footprint from the onboard sensors, motors, computers and cooling systems, and also from the wheels interacting with the terrain during motion. All these can jepordise any visual covertness. In this work, we experimentally explore the concepts of opportunistically utilizing naturally occurring sounds within outdoor environments to mask the motion of a robot, and being visually covert whilst maintaining constant observation of the target. Our experiments in a constrained outdoor built environment demonstrate the effectiveness of the concept by showing a reduced acoustic signature as perceived by a mobile target allowing the robot to covertly navigate to opportunistic vantage points for observation.

An efficient and effective localization method for networked disjoint top-view cameras

Disjoint top-view networked cameras are among the most commonly utilized networks in many applications. One of the open questions for these cameras' study is the computation of extrinsic parameters (positions and orientations), named extrinsic calibration or localization of cameras. Current approaches either rely on strict assumptions of the object motion for accurate results or fail to provide results of high accuracy without the requirement of the object motion. To address these shortcomings, we present a location-constrained maximum a posteriori (LMAP) approach by applying known locations in the surveillance area, some of which would be passed by the object opportunistically. The LMAP approach formulates the problem as a joint inference of the extrinsic parameters and object trajectory based on the cameras' observations and the known locations. In addition, a new task-oriented evaluation metric, named MABR (the Maximum value of All image points' Back-projected localization errors' L2 norms Relative to the area of field of view), is presented to assess the quality of the calibration results in an indoor object tracking context. Finally, results herein demonstrate the superior performance of the proposed method over the state-of-the-art algorithm based on the presented MABR and classical evaluation metric in simulations and real experiments.

Soluble laminin and arginine-glycine-aspartic acid containing peptides differentially regulate type IV collagenase messenger RNA, activation, and localization in testicular cell culture

Rat testicular cells in culture produce several metalloproteinases including type IV collagenases (Sang et al. Biol Reprod 1990; 43:946-955, 956-964). We have now investigated the regulation of testicular cell type IV collagenase and other metalloprotemases in vitro. Soluble laminin stimulated Sertoli cell type IV collagenase mRNA levels. However, three peptides corresponding to different domains of the laminin molecule (CSRAKQAASIKVASADR, FALRGDNP, CLQDGDVRV) did not influence type IV collagenase mENA levels. Zyniographic analysis of medium collected from these cultures revealed that neither soluble laminin nor any of the peptides influenced 72-Wa type IV collagenase protein levels. However, peptide FALRGDNP resulted in both, a selective increase in two higher molecular-weight metalloprotemnases (83 kDa and 110 Wa and in an activation of the 72-Wa rat type IV collagenase. Interleukin-1, phorbol ester, testosterone, and FSH did not affect collagenase activation, lmmunocytochemical studies demonstrated that the addition of soluble laminin resulted in a redistribution of type IV collagenase from intracellular vesicles to the cell-substrate region beneath the cells. Peptide FALRGDNP induced a change from a vesicular to peripheral plasma membrane type of staining pattern. Zymography of plasma membrane preparations demonstrated triton-soluble gelatinases of 76 Wa, 83 Wa, and 110 Wa and a triton-insoluble gelatinase of 225 Wa, These results indicate that testicular cell type IV collagenase mRNA levels, enzyme activation, and distribution are influenced by laminin and RGD-containing peptides.

Multiple map hypotheses for planning and navigating in non-stationary environments

This paper presents a method to enable a mobile robot working in non-stationary environments to plan its path and localize within multiple map hypotheses simultaneously. The maps are generated using a long-term and short-term memory mechanism that ensures only persistent configurations in the environment are selected to create the maps. In order to evaluate the proposed method, experimentation is conducted in an office environment. Compared to navigation systems that use only one map, our system produces superior path planning and navigation in a non-stationary environment where paths can be blocked periodically, a common scenario which poses significant challenges for typical planners.

Localization of fibronectin in the rat testis : effects of serum and dibutyryl cyclic AMP on fibronectin deposition by peritubular myoid cells in culture

The peritubular zone of the rat testis has an extensive extracellular matrix (ECM). Fibronectin (FN) is distributed primarily in the basal lamina of the seminiferous tubule boundary tissue and is synthesized by peritubular myoid cells. Several extracellular changes are mediated by growth factors and these changes occur at the time of hormone mediated testicular development, particularly in the peritubular zone. The effects of serum or dibutyryl cyclic AMP (cAMP) on FN production by the mesenchymal peritubular myoid cells were evaluated. Rats of various ages (10, 15, 20, 40 and 80 days) were employed for immunofluorescent localization of rat testicular FN in frozen sections. In all age groups tested, FN was primarily present in a broad layer around each seminiferous tubule, and blood vessel, and in variable distribution throughout the interstitial stroma. By day 20 there was no clear distinction in FN staining between the peritubular zone and the interstitial tissue. This indicates an involvement of FN in the ECM developments which occur in the peritubular zone of the testis at this time. The peritubular myoid cells were isolated from 20-22 day old rat testis and cultured on glass coverslips. These cells were grown to confluence with 10% fetal calf serum (FCS) in medium until day 4 and then subcultured to have secondary monocultures maintained with or without serum. By means of immunofluorescence and cytochemistry using avidin-biotin peroxidase complex it was observed that peritubular myoid cells were positive for FN and most of the FN was localized in the perinuclear region. Subcultured peritubular myoid cells maintained for 4 days in medium containing FCS developed an extensive interconnecting FN matrix. In the presence of 0.5 mM cAMP in culture, FN became localized along the filamentous process of peritubular myoid cells and more prominently in the areas of triangulated multi-cell aggregates as well as on the surface of the contracted small spherical cells. The addition of cAMP in the presence of FCS, also caused a noticeable change in the staining pattern; FN was detected along the filamentous process developing into a complex network of cells encased in an extensive matrix. It would appear that the translocation of FN in the cytoplasmic extensions of peritubular myoid cells may be a direct consequence of morphological changes associated with metabolic regulation of cAMP. This may also be related to the puberty associated development of in vivo changes in the ECM produced by peritubular myoid cells.

Vision based guidance for robot navigation in agriculture

This paper describes a novel vision based texture tracking method to guide autonomous vehicles in agricultural fields where the crop rows are challenging to detect. Existing methods require sufficient visual difference between the crop and soil for segmentation, or explicit knowledge of the structure of the crop rows. This method works by extracting and tracking the direction and lateral offset of the dominant parallel texture in a simulated overhead view of the scene and hence abstracts away crop-specific details such as colour, spacing and periodicity. The results demonstrate that the method is able to track crop rows across fields with extremely varied appearance during day and night. We demonstrate this method can autonomously guide a robot along the crop rows.

Vision-based simultaneous localization and mapping in changing outdoor environments

For robots operating in outdoor environments, a number of factors, including weather, time of day, rough terrain, high speeds, and hardware limitations, make performing vision-based simultaneous localization and mapping with current techniques infeasible due to factors such as image blur and/or underexposure, especially on smaller platforms and low-cost hardware. In this paper, we present novel visual place-recognition and odometry techniques that address the challenges posed by low lighting, perceptual change, and low-cost cameras. Our primary contribution is a novel two-step algorithm that combines fast low-resolution whole image matching with a higher-resolution patch-verification step, as well as image saliency methods that simultaneously improve performance and decrease computing time. The algorithms are demonstrated using consumer cameras mounted on a small vehicle in a mixed urban and vegetated environment and a car traversing highway and suburban streets, at different times of day and night and in various weather conditions. The algorithms achieve reliable mapping over the course of a day, both when incrementally incorporating new visual scenes from different times of day into an existing map, and when using a static map comprising visual scenes captured at only one point in time. Using the two-step place-recognition process, we demonstrate for the first time single-image, error-free place recognition at recall rates above 50% across a day-night dataset without prior training or utilization of image sequences. This place-recognition performance enables topologically correct mapping across day-night cycles.

Development of a 3500-tonne field robot

The mining industry is highly suitable for the application of robotics and automation technology, since the work is arduous, dangerous, and often repetitive. This paper presents a broad overview of the issues involved in the development of a physically large and complex field robotic system—a 3500-tonne mining machine (dragline). Draglines are “walking cranes” used in open-pit coal mining to remove the material covering a coal seam. The critical issues of robust load position sensing, modeling of the dynamics of the electrical drive system and the swinging load, control strategies, the operator interface, and automation system architecture are addressed. An important aspect of this system is that it must work cooperatively with a human operator, seamlessly passing control back and forth in order to achieve the main aim—increased productivity.

Connecting the person with dementia and family : a feasibility study of a telepresence robot

Background Maintenance of communication is important for people with dementia living in long-term care. The purpose of this study was to assess the feasibility of using “Giraff”, a telepresence robot to enhance engagement between family and a person with dementia living in long-term care. Methods A mixed-methods approach involving semi-structured interviews, call records and video observational data was used. Five people with dementia and their family member participated in a discussion via the Giraff robot for a minimum of six times over a six-week period. A feasibility framework was used to assess feasibility and included video analysis of emotional response and engagement. Results Twenty-six calls with an average duration of 23 mins took place. Residents showed a general state of positive emotions across the calls with a high level of engagement and a minimal level of negative emotions. Participants enjoyed the experience and families reported that the Giraff robot offered the opportunity to reduce social isolation. A number of software and hardware challenges were encountered. Conclusions Participants perceived this novel approach to engage families and people with dementia as a feasible option. Participants were observed and also reported to enjoy the experience. The technical challenges identified have been improved in a newer version of the robot. Future research should include a feasibility trial of longer duration, with a larger sample and a cost analysis.