Multi-modal object tracking using dynamic performance metrics

Intelligent surveillance systems typically use a single visual spectrum modality for their input. These systems work well in controlled conditions, but often fail when lighting is poor, or environmental effects such as shadows, dust or smoke are present. Thermal spectrum imagery is not as susceptible to environmental effects, however thermal imaging sensors are more sensitive to noise and they are only gray scale, making distinguishing between objects difficult. Several approaches to combining the visual and thermal modalities have been proposed, however they are limited by assuming that both modalities are perfuming equally well. When one modality fails, existing approaches are unable to detect the drop in performance and disregard the under performing modality. In this paper, a novel middle fusion approach for combining visual and thermal spectrum images for object tracking is proposed. Motion and object detection is performed on each modality and the object detection results for each modality are fused base on the current performance of each modality. Modality performance is determined by comparing the number of objects tracked by the system with the number detected by each mode, with a small allowance made for objects entering and exiting the scene. The tracking performance of the proposed fusion scheme is compared with performance of the visual and thermal modes individually, and a baseline middle fusion scheme. Improvement in tracking performance using the proposed fusion approach is demonstrated. The proposed approach is also shown to be able to detect the failure of an individual modality and disregard its results, ensuring performance is not degraded in such situations.

Unmanned Aerial Vehicles (UAVs) and artificial intelligence revolutionizing wildlife monitoring and conservation

Surveying threatened and invasive species to obtain accurate population estimates is an important but challenging task that requires a considerable investment in time and resources. Estimates using existing ground-based monitoring techniques, such as camera traps and surveys performed on foot, are known to be resource intensive, potentially inaccurate and imprecise, and difficult to validate. Recent developments in unmanned aerial vehicles (UAV), artificial intelligence and miniaturized thermal imaging systems represent a new opportunity for wildlife experts to inexpensively survey relatively large areas. The system presented in this paper includes thermal image acquisition as well as a video processing pipeline to perform object detection, classification and tracking of wildlife in forest or open areas. The system is tested on thermal video data from ground based and test flight footage, and is found to be able to detect all the target wildlife located in the surveyed area. The system is flexible in that the user can readily define the types of objects to classify and the object characteristics that should be considered during classification.

Próba zastosowania lotniczych termowizyjnych obrazów wideo do wykrywania miejsc nielegalnego składowania odpadów zwierzęcych

Access to the remote sensing data was increasing in Poland since 1989. This procccess had stimulating impact on scientific research in the scope of changes in the environment. Special attention should be given to the thermal imagery methods because of its information potential. Presented paper discusses the possibilities of using information from thermal images for detecting of places of illegal dumping of animal waste in the ground. On the basis of earlier survey and gathered data draft fl ight plan was created, covering the sorroundings of Śmiłowo (around 30 sq km). Theoretical thesis for the subject was an assumption that all disturbances of the ground and soil structure should give visible representation in both thermal and visible images. Moreover the process of decay of animal tissues should be the source of heat, which can be observed through thermal sensor. Several places of potential dumping of animal waste were selected. For detailed ground verifi cation eight of them were chosen. In these location geological drillings were performed and than analysis of the samples. Thermovision is a method with great potential for the monitoing of the environment, but its effectiveness depends on the access to another sources of geoinformation.

Variaciones termométricas en la planta del pie y piernas valorada en corredores antes y después de correr 30 km

La termometría es una técnica no invasiva que permite cuantificar los cambios en la temperatura cutánea y evaluarla de forma cuantitativa. El aumento significativo de la temperatura puede indicar la existencia de patología. Se ha demostrado que la actividad muscular induce procesos de transferencia de calor entre los músculos y las capas superficiales de tejido. En este estudio queremos cuantificar los cambios de temperatura que se producen en los músculos del pie y miembro inferior tras una carrera de 30 km, para ello hemos utilizado una cámara termográfica de alta resolución. Contamos con la colaboración voluntaria de 32 sujetos sanos a los que procedimos a tomar fotografías de la planta del pie, parte anterior de la pierna, parte posterior de la pierna, parte anterior del muslo y parte posterior del muslo en dos etapas, primero antes de la carrera y segunda toma después de la carrera de 30 km, de esta manera pudimos valorar si había o no variación de temperatura en las zonas seleccionadas. Tras el análisis de los datos obtenidos encontramos significativas variaciones térmicas en Talón, cabeza primer metatarsiano, cabeza segundo metatarsiano, cabeza tercer metatarsiano, cabeza cuarto metatarsiano, cabeza quinto metatarsiano, apófisis estiloides quinto metatarsiano, arco longitudinal interno, maléolo interno, maléolo externo, peroneo lateral largo, vasto interno, vasto externo, recto femoral, tensor de la fascia lata, inserción cuádriceps, gemelo interno, tendón de Aquiles y Biceps femoral.

Evaluation of remote sensing for the detection of landfill gas and leachate in an urban environment

The technique of remote sensing provides a unique view of the earth's surface and considerable areas can be surveyed in a short amount of time. The aim of this project was to evaluate whether remote sensing, particularly using the Airborne Thematic Mapper (ATM) with its wide spectral range, was capable of monitoring landfill sites within an urban environment with the aid of image processing and Geographical Information Systems (GIS) methods. The regions under study were in the West Midlands conurbation and consisted of a large area in what is locally known as the Black Country containing heavy industry intermingled with residential areas, and a large single active landfill in north Birmingham. When waste is collected in large volumes it decays and gives off pollutants. These pollutants, landfill gas and leachate (a liquid effluent), are known to be injurious to vegetation and can cause stress and death. Vegetation under stress can exhibit a physiological change, detectable by the remote sensing systems used. The chemical and biological reactions that create the pollutants are exothermic and the gas and leachate, if they leave the waste, can be warmer than their surroundings. Thermal imagery from the ATM (daylight and dawn) and thermal video were obtained and used to find thermal anomalies on the area under study. The results showed that vegetation stress is not a reliable indicator of landfill gas migration, as sites within an urban environment have a cover too complex for the effects to be identified. Gas emissions from two sites were successfully detected by all the thermal imagery with the thermal ATM being the best. Although the results were somewhat disappointing, recent technical advancements in the remote sensing systems used in this project would allow geo-registration of ATM imagery taken on different occasions and the elimination of the effects of solar insolation.

Improved detection and tracking of objects in surveillance video

Surveillance networks are typically monitored by a few people, viewing several monitors displaying the camera feeds. It is then very difficult for a human operator to effectively detect events as they happen. Recently, computer vision research has begun to address ways to automatically process some of this data, to assist human operators. Object tracking, event recognition, crowd analysis and human identification at a distance are being pursued as a means to aid human operators and improve the security of areas such as transport hubs. The task of object tracking is key to the effective use of more advanced technologies. To recognize an event people and objects must be tracked. Tracking also enhances the performance of tasks such as crowd analysis or human identification. Before an object can be tracked, it must be detected. Motion segmentation techniques, widely employed in tracking systems, produce a binary image in which objects can be located. However, these techniques are prone to errors caused by shadows and lighting changes. Detection routines often fail, either due to erroneous motion caused by noise and lighting effects, or due to the detection routines being unable to split occluded regions into their component objects. Particle filters can be used as a self contained tracking system, and make it unnecessary for the task of detection to be carried out separately except for an initial (often manual) detection to initialise the filter. Particle filters use one or more extracted features to evaluate the likelihood of an object existing at a given point each frame. Such systems however do not easily allow for multiple objects to be tracked robustly, and do not explicitly maintain the identity of tracked objects. This dissertation investigates improvements to the performance of object tracking algorithms through improved motion segmentation and the use of a particle filter. A novel hybrid motion segmentation / optical flow algorithm, capable of simultaneously extracting multiple layers of foreground and optical flow in surveillance video frames is proposed. The algorithm is shown to perform well in the presence of adverse lighting conditions, and the optical flow is capable of extracting a moving object. The proposed algorithm is integrated within a tracking system and evaluated using the ETISEO (Evaluation du Traitement et de lInterpretation de Sequences vidEO - Evaluation for video understanding) database, and significant improvement in detection and tracking performance is demonstrated when compared to a baseline system. A Scalable Condensation Filter (SCF), a particle filter designed to work within an existing tracking system, is also developed. The creation and deletion of modes and maintenance of identity is handled by the underlying tracking system; and the tracking system is able to benefit from the improved performance in uncertain conditions arising from occlusion and noise provided by a particle filter. The system is evaluated using the ETISEO database. The dissertation then investigates fusion schemes for multi-spectral tracking systems. Four fusion schemes for combining a thermal and visual colour modality are evaluated using the OTCBVS (Object Tracking and Classification in and Beyond the Visible Spectrum) database. It is shown that a middle fusion scheme yields the best results and demonstrates a significant improvement in performance when compared to a system using either mode individually. Findings from the thesis contribute to improve the performance of semi-automated video processing and therefore improve security in areas under surveillance.

Hand-held monocular SLAM in thermal-infrared

Thermal-infrared imagery is relatively robust to many of the failure conditions of visual and laser-based SLAM systems, such as fog, dust and smoke. The ability to use thermal-infrared video for localization is therefore highly appealing for many applications. However, operating in thermal-infrared is beyond the capacity of existing SLAM implementations. This paper presents the first known monocular SLAM system designed and tested for hand-held use in the thermal-infrared modality. The implementation includes a flexible feature detection layer able to achieve robust feature tracking in high-noise, low-texture thermal images. A novel approach for structure initialization is also presented. The system is robust to irregular motion and capable of handling the unique mechanical shutter interruptions common to thermal-infrared cameras. The evaluation demonstrates promising performance of the algorithm in several environments.

Investigating limitations of SURF approach for thermal imaging analysis and mapping

Wildlife conservation involves an understanding of a specific animal, its environment and the interaction within a local ecosystem. Unmanned Aerial Vehicles (UAVs) present cost effective, non-intrusive solution for detecting animals over large areas and the use thermal imaging cameras offer the ability detect animals that would otherwise be concealed to visible light cameras. This report examines some of limitations on using SURF for the development of large maps using multiple stills images extracted from the thermal imaging video camera which contain wildlife (eg. Koala in them).

On the thermal budget of Pahoehoe lava flows

In this thesis I investigate some aspects of the thermal budget of pahoehoe lava flows. This is done with a combination of general field observations, quantitative modeling, and specific field experiments. The results of this work apply to pahoehoe flows in general, even though the vast bulk of the work has been conducted on the lavas formed by the Pu'u 'O'o - Kupaianaha eruption of Kilauea Volcano on Hawai'i. The field observations rely heavily on discussions with the staff of the United States Geological Survey's Hawaiian Volcano Observatory (HVO), under whom I labored repeatedly in 1991-1993 for a period totaling about 10 months.

The quantitative models I have constructed are based on the physical processes observed by others and myself to be active on pahoehoe lava flows. By building up these models from the basic physical principles involved, this work avoids many of the pitfalls of earlier attempts to fit field observations with "intuitively appropriate" mathematical expressions. Unlike many earlier works, my model results can be analyzed in terms of the interactions between the different physical processes. I constructed models to: (1) describe the initial cooling of small pahoehoe flow lobes and (2) understand the thermal budget of lava tubes.

The field experiments were designed either to validate model results or to constrain key input parameters. In support of the cooling model for pahoehoe flow lobes, attempts were made to measure: (1) the cooling within the flow lobes, (2) the amount of heat transported away from the lava by wind, and (3) the growth of the crust on the lobes. Field data collected by Jones [1992], Hon et al. [1994b], and Denlinger [Keszthelyi and Denlinger, in prep.] were also particularly useful in constraining my cooling model for flow lobes. Most of the field observations I have used to constrain the thermal budget of lava tubes were collected by HVO (geological and geophysical monitoring) and the Jet Propulsion Laboratory (airborne infrared imagery [Realmuto et al., 1992]). I was able to assist HVO for part of their lava tube monitoring program and also to collect helicopterborne and ground-based IR video in collaboration with JPL [Keszthelyi et al., 1993].

The most significant results of this work are (1) the quantitative demonstration that the emplacement of pahoehoe and 'a'a flows are the fundamentally different, (2) confirmation that even the longest lava flows observed in our Solar System could have formed as low effusion rate, tube-fed pahoehoe flows, and (3) the recognition that the atmosphere plays a very important role throughout the cooling of history of pahoehoe lava flows. In addition to answering specific questions about the thermal budget of tube-fed pahoehoe lava flows, this thesis has led to some additional, more general, insights into the emplacement of these lava flows. This general understanding of the tube-fed pahoehoe lava flow as a system has suggested foci for future research in this part of physical volcanology.

Infrared thermography: A non-invasive window into thermal physiology

Infrared thermography is a non-invasive technique that measures mid to long-wave infrared radiation emanating from all objects and converts this to temperature. As an imaging technique, the value of modern infrared thermography is its ability to produce a digitized image or high speed video rendering a thermal map of the scene in false colour. Since temperature is an important environmental parameter influencing animal physiology and metabolic heat production an energetically expensive process, measuring temperature and energy exchange in animals is critical to understanding physiology, especially under field conditions. As a non-contact approach, infrared thermography provides a non-invasive complement to physiological data gathering. One caveat, however, is that only surface temperatures are measured, which guides much research to those thermal events occurring at the skin and insulating regions of the body. As an imaging technique, infrared thermal imaging is also subject to certain uncertainties that require physical modeling, which is typically done via built-in software approaches. Infrared thermal imaging has enabled different insights into the comparative physiology of phenomena ranging from thermogenesis, peripheral blood flow adjustments, evaporative cooling, and to respiratory physiology. In this review, I provide background and guidelines for the use of thermal imaging, primarily aimed at field physiologists and biologists interested in thermal biology. I also discuss some of the better known approaches and discoveries revealed from using thermal imaging with the objective of encouraging more quantitative assessment.

Temporally stable feature clusters for maritime object tracking in visible and thermal imagery

This paper describes a new approach to detect and track maritime objects in real time. The approach particularly addresses the highly dynamic maritime environment, panning cameras, target scale changes, and operates on both visible and thermal imagery. Object detection is based on agglomerative clustering of temporally stable features. Object extents are first determined based on persistence of detected features and their relative separation and motion attributes. An explicit cluster merging and splitting process handles object creation and separation. Stable object clus- ters are tracked frame-to-frame. The effectiveness of the approach is demonstrated on four challenging real-world public datasets.

Towards person authentication by fusing visual and thermal face biometrics

In this chapter we focus on face appearance-based biometrics. The cheap and readily available hardware used to acquire data, their non-invasiveness and the ease of employing them from a distance and without the awareness of the user, are just some of the reasons why these continue to be of great practical interest. However, a number of research challenges remain. Specifically, face biometrics have traditionally focused on images acquired in the visible light spectrum and these are greatly affected by such extrinsic factors such as the illumination, camera angle (or, equivalently, head pose) and occlusion. In practice, the effects of changing pose are usually least problematic and can oftentimes be overcome by acquiring data over a time period, e.g., by tracking a face in a surveillance video. Consequently, image sequence or image set matching has recently gained a lot of attention in the literature [137–139] and is the paradigm adopted in this chapter as well. In other words, we assume that the training image set for each individual contains some variability in pose, but is not obtained in scripted conditions or in controlled illumination. In contrast, illumination is much more difficult to deal with: the illumination setup is in most cases not practical to control and its physics is difficult to accurately model. Thermal spectrum imagery is useful in this regard as it is virtually insensitive to illumination changes, as illustrated in Fig. 6.1. On the other hand, it lacks much of the individual, discriminating facial detail contained in visual images. In this sense, the two modalities can be seen as complementing each other. The key idea behind the system presented in this chapter is that robustness to extreme illumination changes can be achieved by fusing the two. This paradigm will further prove useful when we consider the difficulty of recognition in the presence of occlusion caused by prescription glasses.

Image analysis for assessing broiler breeder behavior response to thermal environment

Fundação de Amparo à Pesquisa do Estado de São Paulo (FAPESP)

Digital monitoring of broiler breeder behavior for assessment of thermal welfare

The assessment of welfare issues has been a challenge for poultry producers, and lately welfare standards needs to be reached in order to agree with international market demand. This research proposes the use of continuous behavior monitoring in order to contribute for assessing welfare. A software was developed using the language Clarium. The software managed the recording of data as well as the data searching in the database Firebird. Both software and the observational methodology were tested in a trial conducted inside an environmental chamber, using three genetics of broiler breeders. Behavioral pattern was recorded and correlated to ambient thermal and aerial variation. Monitoring video cameras were placed on the roof facing the used for registering the bird's behavior. From video camera images were recorded during the total period when the ambient was bright, and for analyzing the video images a sample of 15min observation in the morning and 15 min in the afternoon was used, adding up to 30 min daily observation. A specific model so-called behavior was developed inside the software for counting specific behavior and its frequency of occurrence, as well as its duration. Electronic identification was recorded for 24h period. Behavioral video recording images was related to the data recorded using electronic identification.. Statistical analysis of data allowed to identify behavioral differences related to the change in thermal environment, and ultimately indicating thermal stress and departure from welfare conditions.