Enhanced PCA-based localization using depth maps with missing data

In this paper a new method for self-localization of mobile robots, based on a PCA positioning sensor to operate in unstructured environments, is proposed and experimentally validated. The proposed PCA extension is able to perform the eigenvectors computation from a set of signals corrupted by missing data. The sensor package considered in this work contains a 2D depth sensor pointed upwards to the ceiling, providing depth images with missing data. The positioning sensor obtained is then integrated in a Linear Parameter Varying mobile robot model to obtain a self-localization system, based on linear Kalman filters, with globally stable position error estimates. A study consisting in adding synthetic random corrupted data to the captured depth images revealed that this extended PCA technique is able to reconstruct the signals, with improved accuracy. The self-localization system obtained is assessed in unstructured environments and the methodologies are validated even in the case of varying illumination conditions.

Foreground segmentation in depth imagery using depth and spatial dynamic models for video surveillance applications

Low-cost systems that can obtain a high-quality foreground segmentation almostindependently of the existing illumination conditions for indoor environments are verydesirable, especially for security and surveillance applications. In this paper, a novelforeground segmentation algorithm that uses only a Kinect depth sensor is proposedto satisfy the aforementioned system characteristics. This is achieved by combininga mixture of Gaussians-based background subtraction algorithm with a new Bayesiannetwork that robustly predicts the foreground/background regions between consecutivetime steps. The Bayesian network explicitly exploits the intrinsic characteristics ofthe depth data by means of two dynamic models that estimate the spatial and depthevolution of the foreground/background regions. The most remarkable contribution is thedepth-based dynamic model that predicts the changes in the foreground depth distributionbetween consecutive time steps. This is a key difference with regard to visible imagery,where the color/gray distribution of the foreground is typically assumed to be constant.Experiments carried out on two different depth-based databases demonstrate that theproposed combination of algorithms is able to obtain a more accurate segmentation of theforeground/background than other state-of-the art approaches.

Real-time depth camera tracking with CAD models and ICP

In recent years, depth cameras have been widely utilized in camera tracking for augmented and mixed reality. Many of the studies focus on the methods that generate the reference model simultaneously with the tracking and allow operation in unprepared environments. However, methods that rely on predefined CAD models have their advantages. In such methods, the measurement errors are not accumulated to the model, they are tolerant to inaccurate initialization, and the tracking is always performed directly in reference model's coordinate system. In this paper, we present a method for tracking a depth camera with existing CAD models and the Iterative Closest Point (ICP) algorithm. In our approach, we render the CAD model using the latest pose estimate and construct a point cloud from the corresponding depth map. We construct another point cloud from currently captured depth frame, and find the incremental change in the camera pose by aligning the point clouds. We utilize a GPGPU-based implementation of the ICP which efficiently uses all the depth data in the process. The method runs in real-time, it is robust for outliers, and it does not require any preprocessing of the CAD models. We evaluated the approach using the Kinect depth sensor, and compared the results to a 2D edge-based method, to a depth-based SLAM method, and to the ground truth. The results show that the approach is more stable compared to the edge-based method and it suffers less from drift compared to the depth-based SLAM.

Percezione depth guidata mediante proiezione virtuale di pattern

L’utilizzo di informazioni di profondità è oggi di fondamentale utilità per molteplici settori applicativi come la robotica, la guida autonoma o assistita, la realtà aumentata e il monitoraggio ambientale. I sensori di profondità disponibili possono essere divisi in attivi e passivi, dove i sensori passivi ricavano le informazioni di profondità dall'ambiente senza emettere segnali, bensì utilizzando i segnali provenienti dall'ambiente (e.g., luce solare). Nei sensori depth passivi stereo è richiesto un algoritmo per elaborare le immagini delle due camere: la tecnica di stereo matching viene utilizzata appunto per stimare la profondità di una scena. Di recente la ricerca si è occupata anche della sinergia con sensori attivi al fine di migliorare la stima della depth ottenuta da un sensore stereo: si utilizzano i punti affidabili generati dal sensore attivo per guidare l'algoritmo di stereo matching verso la soluzione corretta. In questa tesi si è deciso di affrontare questa tematica da un punto di vista nuovo, utilizzando un sistema di proiezione virtuale di punti corrispondenti in immagini stereo: i pixel delle immagini vengono alterati per guidare l'algoritmo ottimizzando i costi. Un altro vantaggio della strategia proposta è la possibilità di iterare il processo, andando a cambiare il pattern in ogni passo: aggregando i passi in un unico risultato, è possibile migliorare il risultato finale. I punti affidabili sono ottenuti mediante sensori attivi (e.g. LiDAR, ToF), oppure direttamente dalle immagini, stimando la confidenza delle mappe prodotte dal medesimo sistema stereo: la confidenza permette di classificare la bontà di un punto fornito dall'algoritmo di matching. Nel corso della tesi sono stati utilizzati sensori attivi per verificare l'efficacia della proiezione virtuale, ma sono state anche effettuate analisi sulle misure di confidenza: lo scopo è verificare se le misure di confidenza possono rimpiazzare o assistere i sensori attivi.

Exploração da Point Cloud Library aplicada à percepção em sistemas autónomos

A navegação e a interpretação do meio envolvente por veículos autónomos em ambientes não estruturados continua a ser um grande desafio na actualidade. Sebastian Thrun, descreve em [Thr02], que o problema do mapeamento em sistemas robóticos é o da aquisição de um modelo espacial do meio envolvente do robô. Neste contexto, a integração de sistemas sensoriais em plataformas robóticas, que permitam a construção de mapas do mundo que as rodeia é de extrema importância. A informação recolhida desses dados pode ser interpretada, tendo aplicabilidade em tarefas de localização, navegação e manipulação de objectos. Até à bem pouco tempo, a generalidade dos sistemas robóticos que realizavam tarefas de mapeamento ou Simultaneous Localization And Mapping (SLAM), utilizavam dispositivos do tipo laser rangefinders e câmaras stereo. Estes equipamentos, para além de serem dispendiosos, fornecem apenas informação bidimensional, recolhidas através de cortes transversais 2D, no caso dos rangefinders. O paradigma deste tipo de tecnologia mudou consideravelmente, com o lançamento no mercado de câmaras RGB-D, como a desenvolvida pela PrimeSense TM e o subsequente lançamento da Kinect, pela Microsoft R para a Xbox 360 no final de 2010. A qualidade do sensor de profundidade, dada a natureza de baixo custo e a sua capacidade de aquisição de dados em tempo real, é incontornável, fazendo com que o sensor se tornasse instantaneamente popular entre pesquisadores e entusiastas. Este avanço tecnológico deu origem a várias ferramentas de desenvolvimento e interacção humana com este tipo de sensor, como por exemplo a Point Cloud Library [RC11] (PCL). Esta ferramenta tem como objectivo fornecer suporte para todos os blocos de construção comuns que uma aplicação 3D necessita, dando especial ênfase ao processamento de nuvens de pontos de n dimensões adquiridas a partir de câmaras RGB-D, bem como scanners laser, câmaras Time-of-Flight ou câmaras stereo. Neste contexto, é realizada nesta dissertação, a avaliação e comparação de alguns dos módulos e métodos constituintes da biblioteca PCL, para a resolução de problemas inerentes à construção e interpretação de mapas, em ambientes indoor não estruturados, utilizando os dados provenientes da Kinect. A partir desta avaliação, é proposta uma arquitectura de sistema que sistematiza o registo de nuvens de pontos, correspondentes a vistas parciais do mundo, num modelo global consistente. Os resultados da avaliação realizada à biblioteca PCL atestam a sua viabilidade, para a resolução dos problemas propostos. Prova da sua viabilidade, são os resultados práticos obtidos, da implementação da arquitectura de sistema proposta, que apresenta resultados de desempenho interessantes, como também boas perspectivas de integração deste tipo de conceitos e tecnologia em plataformas robóticas desenvolvidas no âmbito de projectos do Laboratório de Sistemas Autónomos (LSA).

Reconhecimento de língua gestual

Trabalho Final de Mestrado para obtenção do grau de Mestre em Engenharia Informática e Computadores

Semantic and fuzzy modelling for human behaviour recognition in smart spaces : a case study on ambient assisted living.

Human activity recognition in everyday environments is a critical, but challenging task in Ambient Intelligence applications to achieve proper Ambient Assisted Living, and key challenges still remain to be dealt with to realize robust methods. One of the major limitations of the Ambient Intelligence systems today is the lack of semantic models of those activities on the environment, so that the system can recognize the speci c activity being performed by the user(s) and act accordingly. In this context, this thesis addresses the general problem of knowledge representation in Smart Spaces. The main objective is to develop knowledge-based models, equipped with semantics to learn, infer and monitor human behaviours in Smart Spaces. Moreover, it is easy to recognize that some aspects of this problem have a high degree of uncertainty, and therefore, the developed models must be equipped with mechanisms to manage this type of information. A fuzzy ontology and a semantic hybrid system are presented to allow modelling and recognition of a set of complex real-life scenarios where vagueness and uncertainty are inherent to the human nature of the users that perform it. The handling of uncertain, incomplete and vague data (i.e., missing sensor readings and activity execution variations, since human behaviour is non-deterministic) is approached for the rst time through a fuzzy ontology validated on real-time settings within a hybrid data-driven and knowledgebased architecture. The semantics of activities, sub-activities and real-time object interaction are taken into consideration. The proposed framework consists of two main modules: the low-level sub-activity recognizer and the high-level activity recognizer. The rst module detects sub-activities (i.e., actions or basic activities) that take input data directly from a depth sensor (Kinect). The main contribution of this thesis tackles the second component of the hybrid system, which lays on top of the previous one, in a superior level of abstraction, and acquires the input data from the rst module's output, and executes ontological inference to provide users, activities and their in uence in the environment, with semantics. This component is thus knowledge-based, and a fuzzy ontology was designed to model the high-level activities. Since activity recognition requires context-awareness and the ability to discriminate among activities in di erent environments, the semantic framework allows for modelling common-sense knowledge in the form of a rule-based system that supports expressions close to natural language in the form of fuzzy linguistic labels. The framework advantages have been evaluated with a challenging and new public dataset, CAD-120, achieving an accuracy of 90.1% and 91.1% respectively for low and high-level activities. This entails an improvement over both, entirely data-driven approaches, and merely ontology-based approaches. As an added value, for the system to be su ciently simple and exible to be managed by non-expert users, and thus, facilitate the transfer of research to industry, a development framework composed by a programming toolbox, a hybrid crisp and fuzzy architecture, and graphical models to represent and con gure human behaviour in Smart Spaces, were developed in order to provide the framework with more usability in the nal application. As a result, human behaviour recognition can help assisting people with special needs such as in healthcare, independent elderly living, in remote rehabilitation monitoring, industrial process guideline control, and many other cases. This thesis shows use cases in these areas.

Evaluating perceptual maps of asymmetries for gait symmetry quantification and pathology detection

Le mouvement de la marche est un processus essentiel de l'activité humaine et aussi le résultat de nombreuses interactions collaboratives entre les systèmes neurologiques, articulaires et musculo-squelettiques fonctionnant ensemble efficacement. Ceci explique pourquoi une analyse de la marche est aujourd'hui de plus en plus utilisée pour le diagnostic (et aussi la prévention) de différents types de maladies (neurologiques, musculaires, orthopédique, etc.). Ce rapport présente une nouvelle méthode pour visualiser rapidement les différentes parties du corps humain liées à une possible asymétrie (temporellement invariante par translation) existant dans la démarche d'un patient pour une possible utilisation clinique quotidienne. L'objectif est de fournir une méthode à la fois facile et peu dispendieuse permettant la mesure et l'affichage visuel, d'une manière intuitive et perceptive, des différentes parties asymétriques d'une démarche. La méthode proposée repose sur l'utilisation d'un capteur de profondeur peu dispendieux (la Kinect) qui est très bien adaptée pour un diagnostique rapide effectué dans de petites salles médicales car ce capteur est d'une part facile à installer et ne nécessitant aucun marqueur. L'algorithme que nous allons présenter est basé sur le fait que la marche saine possède des propriétés de symétrie (relativement à une invariance temporelle) dans le plan coronal.

Zooplankton abundance and size structure in the North Atlantic from MSM26_133-14

Data on zooplankton abundance and biovolume were collected in concert with data on the biophysical environment at 9 stations in the North Atlantic, from the Iceland Basin in the East to the Labrador Sea in the West. The data were sampled along vertical profiles by a Laser Optical Plankton Counter (LOPC, Rolls Royce Canada Ltd.) that was mounted on a carousel water sampler together with a Conductivity-Temperature-Depth sensor (CTD, SBE19plusV2, Seabird Electronics, Inc., USA) and a fluorescence sensor (F, ECO Puck chlorophyll a fluorometer, WET Labs Inc., USA). Based on the LOPC data, abundance (individuals/m**3) and biovolume (mm3/m**3) were calculated as described in the LOPC Software Operation Manual [(Anonymous, 2006), http://www.brooke-ocean.com/index.html]. LOPC data were regrouped into 49 size groups of equal log10(body volume) increments, see Edvardsen et al. (2002, doi:10.3354/meps227205). LOPC data quality was checked as described in Basedow et al. (2013, doi:10.1016/j.pocean.2012.10.005). Fluorescence was roughly converted into chlorophyll based on filtered chlorophyll values obtained from station 10 in the Labrador Sea. Due to the low number of filtered samples that was used for the conversion the resulting chlorophyll values should be considered with care. CTD data were screened for erroneous (out of range) values and then averaged to the same frequency as the LOPC data (2 Hz). All data were processed using especially developed scripts in the python programming language. The LOPC is an optical instrument designed to count and measure particles (0.1 to 30 mm equivalent spherical diameter) in the water column, see Herman et al., (2004, doi:10.1093/plankt/fbh095). The size of particles as equivalent spherical diameter (ESD) was computed as described in the manual (Anonymous, 2006), and in more detail in Checkley et al. (2008, doi:10.4319/lo.2008.53.5_part_2.2123) and Gaardsted et al. (2010, doi:10.1111/j.1365-2419.2010.00558.x).

Zooplankton abundance and size structure in the North Atlantic from MSM26_135-21

Data on zooplankton abundance and biovolume were collected in concert with data on the biophysical environment at 9 stations in the North Atlantic, from the Iceland Basin in the East to the Labrador Sea in the West. The data were sampled along vertical profiles by a Laser Optical Plankton Counter (LOPC, Rolls Royce Canada Ltd.) that was mounted on a carousel water sampler together with a Conductivity-Temperature-Depth sensor (CTD, SBE19plusV2, Seabird Electronics, Inc., USA) and a fluorescence sensor (F, ECO Puck chlorophyll a fluorometer, WET Labs Inc., USA). Based on the LOPC data, abundance (individuals/m**3) and biovolume (mm3/m**3) were calculated as described in the LOPC Software Operation Manual [(Anonymous, 2006), http://www.brooke-ocean.com/index.html]. LOPC data were regrouped into 49 size groups of equal log10(body volume) increments, see Edvardsen et al. (2002, doi:10.3354/meps227205). LOPC data quality was checked as described in Basedow et al. (2013, doi:10.1016/j.pocean.2012.10.005). Fluorescence was roughly converted into chlorophyll based on filtered chlorophyll values obtained from station 10 in the Labrador Sea. Due to the low number of filtered samples that was used for the conversion the resulting chlorophyll values should be considered with care. CTD data were screened for erroneous (out of range) values and then averaged to the same frequency as the LOPC data (2 Hz). All data were processed using especially developed scripts in the python programming language. The LOPC is an optical instrument designed to count and measure particles (0.1 to 30 mm equivalent spherical diameter) in the water column, see Herman et al., (2004, doi:10.1093/plankt/fbh095). The size of particles as equivalent spherical diameter (ESD) was computed as described in the manual (Anonymous, 2006), and in more detail in Checkley et al. (2008, doi:10.4319/lo.2008.53.5_part_2.2123) and Gaardsted et al. (2010, doi:10.1111/j.1365-2419.2010.00558.x).

Zooplankton abundance and size structure in the North Atlantic from MSM26_127-22

Data on zooplankton abundance and biovolume were collected in concert with data on the biophysical environment at 9 stations in the North Atlantic, from the Iceland Basin in the East to the Labrador Sea in the West. The data were sampled along vertical profiles by a Laser Optical Plankton Counter (LOPC, Rolls Royce Canada Ltd.) that was mounted on a carousel water sampler together with a Conductivity-Temperature-Depth sensor (CTD, SBE19plusV2, Seabird Electronics, Inc., USA) and a fluorescence sensor (F, ECO Puck chlorophyll a fluorometer, WET Labs Inc., USA). Based on the LOPC data, abundance (individuals/m**3) and biovolume (mm3/m**3) were calculated as described in the LOPC Software Operation Manual [(Anonymous, 2006), http://www.brooke-ocean.com/index.html]. LOPC data were regrouped into 49 size groups of equal log10(body volume) increments, see Edvardsen et al. (2002, doi:10.3354/meps227205). LOPC data quality was checked as described in Basedow et al. (2013, doi:10.1016/j.pocean.2012.10.005). Fluorescence was roughly converted into chlorophyll based on filtered chlorophyll values obtained from station 10 in the Labrador Sea. Due to the low number of filtered samples that was used for the conversion the resulting chlorophyll values should be considered with care. CTD data were screened for erroneous (out of range) values and then averaged to the same frequency as the LOPC data (2 Hz). All data were processed using especially developed scripts in the python programming language. The LOPC is an optical instrument designed to count and measure particles (0.1 to 30 mm equivalent spherical diameter) in the water column, see Herman et al., (2004, doi:10.1093/plankt/fbh095). The size of particles as equivalent spherical diameter (ESD) was computed as described in the manual (Anonymous, 2006), and in more detail in Checkley et al. (2008, doi:10.4319/lo.2008.53.5_part_2.2123) and Gaardsted et al. (2010, doi:10.1111/j.1365-2419.2010.00558.x).

Zooplankton abundance and size structure in the North Atlantic from MSM26_134-16

Data on zooplankton abundance and biovolume were collected in concert with data on the biophysical environment at 9 stations in the North Atlantic, from the Iceland Basin in the East to the Labrador Sea in the West. The data were sampled along vertical profiles by a Laser Optical Plankton Counter (LOPC, Rolls Royce Canada Ltd.) that was mounted on a carousel water sampler together with a Conductivity-Temperature-Depth sensor (CTD, SBE19plusV2, Seabird Electronics, Inc., USA) and a fluorescence sensor (F, ECO Puck chlorophyll a fluorometer, WET Labs Inc., USA). Based on the LOPC data, abundance (individuals/m**3) and biovolume (mm3/m**3) were calculated as described in the LOPC Software Operation Manual [(Anonymous, 2006), http://www.brooke-ocean.com/index.html]. LOPC data were regrouped into 49 size groups of equal log10(body volume) increments, see Edvardsen et al. (2002, doi:10.3354/meps227205). LOPC data quality was checked as described in Basedow et al. (2013, doi:10.1016/j.pocean.2012.10.005). Fluorescence was roughly converted into chlorophyll based on filtered chlorophyll values obtained from station 10 in the Labrador Sea. Due to the low number of filtered samples that was used for the conversion the resulting chlorophyll values should be considered with care. CTD data were screened for erroneous (out of range) values and then averaged to the same frequency as the LOPC data (2 Hz). All data were processed using especially developed scripts in the python programming language. The LOPC is an optical instrument designed to count and measure particles (0.1 to 30 mm equivalent spherical diameter) in the water column, see Herman et al., (2004, doi:10.1093/plankt/fbh095). The size of particles as equivalent spherical diameter (ESD) was computed as described in the manual (Anonymous, 2006), and in more detail in Checkley et al. (2008, doi:10.4319/lo.2008.53.5_part_2.2123) and Gaardsted et al. (2010, doi:10.1111/j.1365-2419.2010.00558.x).