Data-driven clustering : new methods and applications

Abstract : This work is concerned with the development and application of novel unsupervised learning methods, having in mind two target applications: the analysis of forensic case data and the classification of remote sensing images. First, a method based on a symbolic optimization of the inter-sample distance measure is proposed to improve the flexibility of spectral clustering algorithms, and applied to the problem of forensic case data. This distance is optimized using a loss function related to the preservation of neighborhood structure between the input space and the space of principal components, and solutions are found using genetic programming. Results are compared to a variety of state-of--the-art clustering algorithms. Subsequently, a new large-scale clustering method based on a joint optimization of feature extraction and classification is proposed and applied to various databases, including two hyperspectral remote sensing images. The algorithm makes uses of a functional model (e.g., a neural network) for clustering which is trained by stochastic gradient descent. Results indicate that such a technique can easily scale to huge databases, can avoid the so-called out-of-sample problem, and can compete with or even outperform existing clustering algorithms on both artificial data and real remote sensing images. This is verified on small databases as well as very large problems. Résumé : Ce travail de recherche porte sur le développement et l'application de méthodes d'apprentissage dites non supervisées. Les applications visées par ces méthodes sont l'analyse de données forensiques et la classification d'images hyperspectrales en télédétection. Dans un premier temps, une méthodologie de classification non supervisée fondée sur l'optimisation symbolique d'une mesure de distance inter-échantillons est proposée. Cette mesure est obtenue en optimisant une fonction de coût reliée à la préservation de la structure de voisinage d'un point entre l'espace des variables initiales et l'espace des composantes principales. Cette méthode est appliquée à l'analyse de données forensiques et comparée à un éventail de méthodes déjà existantes. En second lieu, une méthode fondée sur une optimisation conjointe des tâches de sélection de variables et de classification est implémentée dans un réseau de neurones et appliquée à diverses bases de données, dont deux images hyperspectrales. Le réseau de neurones est entraîné à l'aide d'un algorithme de gradient stochastique, ce qui rend cette technique applicable à des images de très haute résolution. Les résultats de l'application de cette dernière montrent que l'utilisation d'une telle technique permet de classifier de très grandes bases de données sans difficulté et donne des résultats avantageusement comparables aux méthodes existantes.

Can we measure terrestrial photosynthesis from space directly, using spectral reflectance and fluorescence?

Attempts to estimate photosynthetic rate or gross primary productivity from remotely sensed absorbed solar radiation depend on knowledge of the light use efficiency (LUE). Early models assumed LUE to be constant, but now most researchers try to adjust it for variations in temperature and moisture stress. However, more exact methods are now required. Hyperspectral remote sensing offers the possibility of sensing the changes in the xanthophyll cycle, which is closely coupled to photosynthesis. Several studies have shown that an index (the photochemical reflectance index) based on the reflectance at 531 nm is strongly correlated with the LUE over hours, days and months. A second hyperspectral approach relies on the remote detection of fluorescence, which is a directly related to the efficiency of photosynthesis. We discuss the state of the art of the two approaches. Both have been demonstrated to be effective, but we specify seven conditions required before the methods can become operational.

Caracterização espectral da água do reservatório de Itupararanga, SP, a partir de imagens hiperespectrais hyperion e análise derivativa

The current accessibility to hyperspectral images of Hyperion/EO1 orbital sensor has brought new perspectives for studies of aquatic environments for allowing the remote estimative of several optically active constituents (OACs) in water body. The changes in the composition and concentration of OACs cause different patterns of absorption and scattering of electromagnetic radiation, likely to be detected using hyperspectral data. Therefore, an investigation was conducted taking into account the spectral characterization of water of a reservoir intended for public supply (Itupararanga Reservoir), from Hyperion/EO1 images and derivative analysis technique applied to spectral curves generated. Simultaneously to the acquisition of a Hyperion/EO1 image, a field campaign was carried out to collect limnological data in situ in georeferenced points. After radiometric correction of the image, reflectance curves of pixels were extracted for each station and the curves obtained were subjected to the technique of derivative analysis, which revealed features of absorption and scattering mainly associated to the presence of algal pigments. The results obtained show the presence of phytoplankton and algal activity, matching the field observation.

Potencial das imagens hiperespectrais orbitais na detecção de componentes opticamente ativos no reservatório de Itupararanga

Pós-graduação em Ciências Cartográficas - FCT

Caracterização espectral de espécies de Mata Atlântica de Interior em nível foliar e de copa

Conselho Nacional de Desenvolvimento Científico e Tecnológico (CNPq)

Mapping for environmental planning the basin of Córrego Rico, São Paulo

The landuse is usually an human phenomenon that occurs over the years, due to population increase. The territorial knowledge is needed, and is the first step for environmental planning to implement conservation practices on agricultural production system. This study aimed to develop thematic maps as: hydrography, soil, slope, land use, and subbasins to obtain the main geomorphic morphometric data (physical) of the Córrego Rico Watershed. The techniques of remote sensing and geographic information system were used to elaborate the maps and for calculating the geomorphological data, as area, altitude and length of the drainage net, which were submitted to multivariate statistics. The Córrego Rico Watershed has an area of 563 km2 . The predominant slopes were 3-8%, with 55.3% of the total area; and the main use was sugar cane. The soils that predominate in the area are Oxisols towards the Mogi-Guaçú river mouth and Ultisols at the upstream of the basin.

Sistemas multiespectrales e hiperespectrales para la observación del territorio : análisis y aplicación a la prospección de hidrocarburos.

La observación de la Tierra es una herramienta de gran utilidad en la actualidad para el estudio de los fenómenos que se dan en la misma. La observación se puede realizar a distintas escalas y por distintos métodos dependiendo del propósito. El actual Trabajo Final de Grado persigue exponer la observación del territorio mediante técnicas de Teledetección, o Detección Remota, y su aplicación en la exploración de hidrocarburos. Desde la Segunda Guerra Mundial el capturar imágenes aéreas de regiones de la Tierra estaba restringido a usos cartográficos en el sentido estricto. Desde aquellos tiempos, hasta ahora, ha acontecido una serie de avances científicos que permiten deducir características intrínsecas de la Tierra mediante mecanismos complejos que no apreciamos a simple vista, pero que, están configurados mediante determinados parámetros geométricos y electrónicos, que permiten generar series temporales de fenómenos físicos que se dan en la Tierra. Hoy en día se puede afirmar que el aprovechamiento del espectro electromagnético está en un punto máximo. Se ha pasado del análisis de la región del espectro visible al análisis del espectro en su totalidad. Esto supone el desarrollo de nuevos algoritmos, técnicas y procesos para extraer la mayor cantidad de información acerca de la interacción de la materia con la radiación electromagnética. La información que generan los sistemas de captura va a servir para la aplicación directa e indirecta de métodos de prospección de hidrocarburos. Las técnicas utilizadas en detección por sensores remotos, aplicadas en campañas geofísicas, son utilizadas para minimizar costes y maximizar resultados en investigaciones de campo. La predicción de anomalías en la zona de estudio depende del analista, quien diseña, calcula y evalúa las variaciones de la energía electromagnética reflejada o emitida por la superficie terrestre. Para dicha predicción se revisarán distintos programas espaciales, se evaluará la bondad de registro y diferenciación espectral mediante el uso de distintas clasificaciones (supervisadas y no supervisadas). Por su influencia directa sobre las observaciones realizadas, se realiza un estudio de la corrección atmosférica; se programan distintos modelos de corrección atmosférica para imágenes multiespectrales y se evalúan los métodos de corrección atmosférica en datos hiperespectrales. Se obtendrá temperatura de la zona de interés utilizando los sensores TM-4, ASTER y OLI, así como un Modelo Digital del Terreno generado por el par estereoscópico capturado por el sensor ASTER. Una vez aplicados estos procedimientos se aplicarán los métodos directos e indirectos, para la localización de zonas probablemente afectadas por la influencia de hidrocarburos y localización directa de hidrocarburos mediante teledetección hiperespectral. Para el método indirecto se utilizan imágenes capturadas por los sensores ETM+ y ASTER. Para el método directo se usan las imágenes capturadas por el sensor Hyperion. ABSTRACT The observation of the Earth is a wonderful tool for studying the different kind of phenomena that occur on its surface. The observation could be done by different scales and by different techniques depending on the information of interest. This Graduate Thesis is intended to expose the territory observation by remote sensing acquiring data systems and the analysis that can be developed to get information of interest. Since Second World War taking aerials photographs of scene was restricted only to a cartographic sense. From these days to nowadays, it have been developed many scientific advances that make capable the interpretation of the surface behavior trough complex systems that are configure by specific geometric and electronic parameters that make possible acquiring time series of the phenomena that manifest on the earth’s surface. Today it is possible to affirm that the exploitation of the electromagnetic spectrum is on a maxim value. In the past, analysis of the electromagnetic spectrum was carry in a narrow part of it, today it is possible to study entire. This implicates the development of new algorithms, process and techniques for the extraction of information about the interaction of matter with electromagnetic radiation. The information that has been acquired by remote sensing sensors is going to be a helpful tool for the exploration of hydrocarbon through direct and vicarious methods. The techniques applied in remote sensing, especially in geophysical campaigns, are employed to minimize costs and maximize results of ground-based geologic investigations. Forecasting of anomalies in the region of interest depends directly on the expertise data analyst who designs, computes and evaluates variations in the electromagnetic energy reflected or emanated from the earth’s surface. For an optimal prediction a review of the capture system take place; assess of the goodness in data acquisition and spectral separability, is carried out by mean of supervised and unsupervised classifications. Due to the direct influence of the atmosphere in the register data, a study of the minimization of its influence has been done; a script has been programed for the atmospheric correction in multispectral data; also, a review of hyperspectral atmospheric correction is conducted. Temperature of the region of interest is computed using the images captured by TM-4, ASTER and OLI, in addition to a Digital Terrain Model generated by a pair of stereo images taken by ASTER sensor. Once these procedures have finished, direct and vicarious methods are applied in order to find altered zones influenced by hydrocarbons, as well as pinpoint directly hydrocarbon presence by mean of hyperspectral remote sensing. For this purpose ETM+ and ASTER sensors are used to apply the vicarious method and Hyperion images are used to apply the direct method.

Analysis of spatial variation of tree heights in an urban area using LiDAR data.

Estimating with greater precision and accuracy the height of plants has been a challenge for the scientific community. The objective this study is to evaluate the spatial variation of tree heights at different spatial scales in areas of the city of Recife, Brazil, using LiDAR remote sensing data. The LiDAR data were processed in the QT Modeler (Quick Terrain Modeler v. 8.0.2) software from Applied Imagery. The TreeVaW software was utilized to estimate the heights and crown diameters of trees. The results obtained for tree height were consistent with field measurements.

GPU implementation of a hyperspectral coded aperture algorithm for compressive sensing

This paper presents a new parallel implementation of a previously hyperspectral coded aperture (HYCA) algorithm for compressive sensing on graphics processing units (GPUs). HYCA method combines the ideas of spectral unmixing and compressive sensing exploiting the high spatial correlation that can be observed in the data and the generally low number of endmembers needed in order to explain the data. The proposed implementation exploits the GPU architecture at low level, thus taking full advantage of the computational power of GPUs using shared memory and coalesced accesses to memory. The proposed algorithm is evaluated not only in terms of reconstruction error but also in terms of computational performance using two different GPU architectures by NVIDIA: GeForce GTX 590 and GeForce GTX TITAN. Experimental results using real data reveals signficant speedups up with regards to serial implementation.

A fast parallel hyperspectral coded aperture algorithm for compressive sensing using OpenCL

In this paper, we develop a fast implementation of an hyperspectral coded aperture (HYCA) algorithm on different platforms using OpenCL, an open standard for parallel programing on heterogeneous systems, which includes a wide variety of devices, from dense multicore systems from major manufactures such as Intel or ARM to new accelerators such as graphics processing units (GPUs), field programmable gate arrays (FPGAs), the Intel Xeon Phi and other custom devices. Our proposed implementation of HYCA significantly reduces its computational cost. Our experiments have been conducted using simulated data and reveal considerable acceleration factors. This kind of implementations with the same descriptive language on different architectures are very important in order to really calibrate the possibility of using heterogeneous platforms for efficient hyperspectral imaging processing in real remote sensing missions.

Hyperspectral sensing to detect the impact of herbicide drift on cotton growth and yield

Yield loss in crops is often associated with plant disease or external factors such as environment, water supply and nutrient availability. Improper agricultural practices can also introduce risks into the equation. Herbicide drift can be a combination of improper practices and environmental conditions which can create a potential yield loss. As traditional assessment of plant damage is often imprecise and time consuming, the ability of remote and proximal sensing techniques to monitor various bio-chemical alterations in the plant may offer a faster, non-destructive and reliable approach to predict yield loss caused by herbicide drift. This paper examines the prediction capabilities of partial least squares regression (PLS-R) models for estimating yield. Models were constructed with hyperspectral data of a cotton crop sprayed with three simulated doses of the phenoxy herbicide 2,4-D at three different growth stages. Fibre quality, photosynthesis, conductance, and two main hormones, indole acetic acid (IAA) and abscisic acid (ABA) were also analysed. Except for fibre quality and ABA, Spearman correlations have shown that these variables were highly affected by the chemical. Four PLS-R models for predicting yield were developed according to four timings of data collection: 2, 7, 14 and 28 days after the exposure (DAE). As indicated by the model performance, the analysis revealed that 7 DAE was the best time for data collection purposes (RMSEP = 2.6 and R2 = 0.88), followed by 28 DAE (RMSEP = 3.2 and R2 = 0.84). In summary, the results of this study show that it is possible to accurately predict yield after a simulated herbicide drift of 2,4-D on a cotton crop, through the analysis of hyperspectral data, thereby providing a reliable, effective and non-destructive alternative based on the internal response of the cotton leaves.

Detecting sugarcane 'orange rust' disease using EO-1 Hyperion hyperspectral imagery

This Letter evaluates several narrow-band indices from EO-1 Hyperion imagery in discriminating sugarcane areas affected by 'orange rust' ( Puccinia kuehnii ) disease. Forty spectral vegetation indices (SVIs), focusing on bands related to leaf pigments, leaf internal structure, and leaf water content, were generated from an image acquired over Mackay, Queensland, Australia. Discriminant function analysis was used to select an optimum set of indices based on their correlations with the discriminant function. The predictive ability of each index was also assessed based on the accuracy of classification. Results demonstrated that Hyperion imagery can be used to detect orange rust disease in sugarcane crops. While some indices that only used visible near-infrared (VNIR) bands (e.g. SIPI and R800/R680) offer separability, the combination of VNIR bands with the moisture-sensitive band (1660 nm) yielded increased separability of rust-affected areas. The newly formulated 'Disease-Water Stress Indices' (DWSI-1=R800/R1660; DSWI-2=R1660/R550; DWSI-5=(R800+R550)/(R1660+R680)) produced the largest correlations, indicating their superior ability to discriminate sugarcane areas affected by orange rust disease.

Learning dependent sources using mixtures of Dirichlet: applications on hyperspectral unmixing

This paper is an elaboration of the DECA algorithm [1] to blindly unmix hyperspectral data. The underlying mixing model is linear, meaning that each pixel is a linear mixture of the endmembers signatures weighted by the correspondent abundance fractions. The proposed method, as DECA, is tailored to highly mixed mixtures in which the geometric based approaches fail to identify the simplex of minimum volume enclosing the observed spectral vectors. We resort then to a statitistical framework, where the abundance fractions are modeled as mixtures of Dirichlet densities, thus enforcing the constraints on abundance fractions imposed by the acquisition process, namely non-negativity and constant sum. With respect to DECA, we introduce two improvements: 1) the number of Dirichlet modes are inferred based on the minimum description length (MDL) principle; 2) The generalized expectation maximization (GEM) algorithm we adopt to infer the model parameters is improved by using alternating minimization and augmented Lagrangian methods to compute the mixing matrix. The effectiveness of the proposed algorithm is illustrated with simulated and read data.

Hyperspectral subspace identification

Signal subspace identification is a crucial first step in many hyperspectral processing algorithms such as target detection, change detection, classification, and unmixing. The identification of this subspace enables a correct dimensionality reduction, yielding gains in algorithm performance and complexity and in data storage. This paper introduces a new minimum mean square error-based approach to infer the signal subspace in hyperspectral imagery. The method, which is termed hyperspectral signal identification by minimum error, is eigen decomposition based, unsupervised, and fully automatic (i.e., it does not depend on any tuning parameters). It first estimates the signal and noise correlation matrices and then selects the subset of eigenvalues that best represents the signal subspace in the least squared error sense. State-of-the-art performance of the proposed method is illustrated by using simulated and real hyperspectral images.