Exploração da Point Cloud Library aplicada à percepção em sistemas autónomos

A navegação e a interpretação do meio envolvente por veículos autónomos em ambientes não estruturados continua a ser um grande desafio na actualidade. Sebastian Thrun, descreve em [Thr02], que o problema do mapeamento em sistemas robóticos é o da aquisição de um modelo espacial do meio envolvente do robô. Neste contexto, a integração de sistemas sensoriais em plataformas robóticas, que permitam a construção de mapas do mundo que as rodeia é de extrema importância. A informação recolhida desses dados pode ser interpretada, tendo aplicabilidade em tarefas de localização, navegação e manipulação de objectos. Até à bem pouco tempo, a generalidade dos sistemas robóticos que realizavam tarefas de mapeamento ou Simultaneous Localization And Mapping (SLAM), utilizavam dispositivos do tipo laser rangefinders e câmaras stereo. Estes equipamentos, para além de serem dispendiosos, fornecem apenas informação bidimensional, recolhidas através de cortes transversais 2D, no caso dos rangefinders. O paradigma deste tipo de tecnologia mudou consideravelmente, com o lançamento no mercado de câmaras RGB-D, como a desenvolvida pela PrimeSense TM e o subsequente lançamento da Kinect, pela Microsoft R para a Xbox 360 no final de 2010. A qualidade do sensor de profundidade, dada a natureza de baixo custo e a sua capacidade de aquisição de dados em tempo real, é incontornável, fazendo com que o sensor se tornasse instantaneamente popular entre pesquisadores e entusiastas. Este avanço tecnológico deu origem a várias ferramentas de desenvolvimento e interacção humana com este tipo de sensor, como por exemplo a Point Cloud Library [RC11] (PCL). Esta ferramenta tem como objectivo fornecer suporte para todos os blocos de construção comuns que uma aplicação 3D necessita, dando especial ênfase ao processamento de nuvens de pontos de n dimensões adquiridas a partir de câmaras RGB-D, bem como scanners laser, câmaras Time-of-Flight ou câmaras stereo. Neste contexto, é realizada nesta dissertação, a avaliação e comparação de alguns dos módulos e métodos constituintes da biblioteca PCL, para a resolução de problemas inerentes à construção e interpretação de mapas, em ambientes indoor não estruturados, utilizando os dados provenientes da Kinect. A partir desta avaliação, é proposta uma arquitectura de sistema que sistematiza o registo de nuvens de pontos, correspondentes a vistas parciais do mundo, num modelo global consistente. Os resultados da avaliação realizada à biblioteca PCL atestam a sua viabilidade, para a resolução dos problemas propostos. Prova da sua viabilidade, são os resultados práticos obtidos, da implementação da arquitectura de sistema proposta, que apresenta resultados de desempenho interessantes, como também boas perspectivas de integração deste tipo de conceitos e tecnologia em plataformas robóticas desenvolvidas no âmbito de projectos do Laboratório de Sistemas Autónomos (LSA).

Characterisation of the LMS200 laser beam under the influence of blockage surfaces. Influence on 3D scanning of tree orchards

The geometric characterisation of tree orchards is a high-precision activity comprising the accurate measurement and knowledge of the geometry and structure of the trees. Different types of sensors can be used to perform this characterisation. In this work a terrestrial LIDAR sensor (SICK LMS200) whose emission source was a 905-nm pulsed laser diode was used. Given the known dimensions of the laser beam cross-section (with diameters ranging from 12 mm at the point of emission to 47.2 mm at a distance of 8 m), and the known dimensions of the elements that make up the crops under study (flowers, leaves, fruits, branches, trunks), it was anticipated that, for much of the time, the laser beam would only partially hit a foreground target/object, with the consequent problem of mixed pixels or edge effects. Understanding what happens in such situations was the principal objective of this work. With this in mind, a series of tests were set up to determine the geometry of the emitted beam and to determine the response of the sensor to different beam blockage scenarios. The main conclusions that were drawn from the results obtained were: (i) in a partial beam blockage scenario, the distance value given by the sensor depends more on the blocked radiant power than on the blocked surface area; (ii) there is an area that influences the measurements obtained that is dependent on the percentage of blockage and which ranges from 1.5 to 2.5 m with respect to the foreground target/object. If the laser beam impacts on a second target/object located within this range, this will affect the measurement given by the sensor. To interpret the information obtained from the point clouds provided by the LIDAR sensors, such as the volume occupied and the enclosing area, it is necessary to know the resolution and the process for obtaining this mesh of points and also to be aware of the problem associated with mixed pixels.

Innovative LIDAR 3D dynamic measurement system to estimate fruit-tree leaf area

In this work, a LIDAR-based 3D Dynamic Measurement System is presented and evaluated for the geometric characterization of tree crops. Using this measurement system, trees were scanned from two opposing sides to obtain two three-dimensional point clouds. After registration of the point clouds, a simple and easily obtainable parameter is the number of impacts received by the scanned vegetation. The work in this study is based on the hypothesis of the existence of a linear relationship between the number of impacts of the LIDAR sensor laser beam on the vegetation and the tree leaf area. Tests performed under laboratory conditions using an ornamental tree and, subsequently, in a pear tree orchard demonstrate the correct operation of the measurement system presented in this paper. The results from both the laboratory and field tests confirm the initial hypothesis and the 3D Dynamic Measurement System is validated in field operation. This opens the door to new lines of research centred on the geometric characterization of tree crops in the field of agriculture and, more specifically, in precision fruit growing.

Reconstruction of patient-specific 3D bone surface from 2D calibrated fluoroscopic images and point distribution model

Reconstruction of patient-specific 3D bone surface from 2D calibrated fluoroscopic images and a point distribution model is discussed. We present a 2D/3D reconstruction scheme combining statistical extrapolation and regularized shape deformation with an iterative image-to-model correspondence establishing algorithm, and show its application to reconstruct the surface of proximal femur. The image-to-model correspondence is established using a non-rigid 2D point matching process, which iteratively uses a symmetric injective nearest-neighbor mapping operator and 2D thin-plate splines based deformation to find a fraction of best matched 2D point pairs between features detected from the fluoroscopic images and those extracted from the 3D model. The obtained 2D point pairs are then used to set up a set of 3D point pairs such that we turn a 2D/3D reconstruction problem to a 3D/3D one. We designed and conducted experiments on 11 cadaveric femurs to validate the present reconstruction scheme. An average mean reconstruction error of 1.2 mm was found when two fluoroscopic images were used for each bone. It decreased to 1.0 mm when three fluoroscopic images were used.

3D visual sensing of human hand for remote operation of a robotic hand

New low cost sensors and open free libraries for 3D image processing are making important advances in robot vision applications possible, such as three-dimensional object recognition, semantic mapping, navigation and localization of robots, human detection and/or gesture recognition for human-machine interaction. In this paper, a novel method for recognizing and tracking the fingers of a human hand is presented. This method is based on point clouds from range images captured by a RGBD sensor. It works in real time and it does not require visual marks, camera calibration or previous knowledge of the environment. Moreover, it works successfully even when multiple objects appear in the scene or when the ambient light is changed. Furthermore, this method was designed to develop a human interface to control domestic or industrial devices, remotely. In this paper, the method was tested by operating a robotic hand. Firstly, the human hand was recognized and the fingers were detected. Secondly, the movement of the fingers was analysed and mapped to be imitated by a robotic hand.

3D model reconstruction using neural gas accelerated on GPU

In this work, we propose the use of the neural gas (NG), a neural network that uses an unsupervised Competitive Hebbian Learning (CHL) rule, to develop a reverse engineering process. This is a simple and accurate method to reconstruct objects from point clouds obtained from multiple overlapping views using low-cost sensors. In contrast to other methods that may need several stages that include downsampling, noise filtering and many other tasks, the NG automatically obtains the 3D model of the scanned objects. To demonstrate the validity of our proposal we tested our method with several models and performed a study of the neural network parameterization computing the quality of representation and also comparing results with other neural methods like growing neural gas and Kohonen maps or classical methods like Voxel Grid. We also reconstructed models acquired by low cost sensors that can be used in virtual and augmented reality environments for redesign or manipulation purposes. Since the NG algorithm has a strong computational cost we propose its acceleration. We have redesigned and implemented the NG learning algorithm to fit it onto Graphics Processing Units using CUDA. A speed-up of 180× faster is obtained compared to the sequential CPU version.

Evaluation of sampling method effects in 3D non-rigid registration

Since the beginning of 3D computer vision problems, the use of techniques to reduce the data to make it treatable preserving the important aspects of the scene has been necessary. Currently, with the new low-cost RGB-D sensors, which provide a stream of color and 3D data of approximately 30 frames per second, this is getting more relevance. Many applications make use of these sensors and need a preprocessing to downsample the data in order to either reduce the processing time or improve the data (e.g., reducing noise or enhancing the important features). In this paper, we present a comparison of different downsampling techniques which are based on different principles. Concretely, five different downsampling methods are included: a bilinear-based method, a normal-based, a color-based, a combination of the normal and color-based samplings, and a growing neural gas (GNG)-based approach. For the comparison, two different models have been used acquired with the Blensor software. Moreover, to evaluate the effect of the downsampling in a real application, a 3D non-rigid registration is performed with the data sampled. From the experimentation we can conclude that depending on the purpose of the application some kernels of the sampling methods can improve drastically the results. Bilinear- and GNG-based methods provide homogeneous point clouds, but color-based and normal-based provide datasets with higher density of points in areas with specific features. In the non-rigid application, if a color-based sampled point cloud is used, it is possible to properly register two datasets for cases where intensity data are relevant in the model and outperform the results if only a homogeneous sampling is used.

Contributions to 3D Data Registration and Representation

Nowadays, new computers generation provides a high performance that enables to build computationally expensive computer vision applications applied to mobile robotics. Building a map of the environment is a common task of a robot and is an essential part to allow the robots to move through these environments. Traditionally, mobile robots used a combination of several sensors from different technologies. Lasers, sonars and contact sensors have been typically used in any mobile robotic architecture, however color cameras are an important sensor due to we want the robots to use the same information that humans to sense and move through the different environments. Color cameras are cheap and flexible but a lot of work need to be done to give robots enough visual understanding of the scenes. Computer vision algorithms are computational complex problems but nowadays robots have access to different and powerful architectures that can be used for mobile robotics purposes. The advent of low-cost RGB-D sensors like Microsoft Kinect which provide 3D colored point clouds at high frame rates made the computer vision even more relevant in the mobile robotics field. The combination of visual and 3D data allows the systems to use both computer vision and 3D processing and therefore to be aware of more details of the surrounding environment. The research described in this thesis was motivated by the need of scene mapping. Being aware of the surrounding environment is a key feature in many mobile robotics applications from simple robotic navigation to complex surveillance applications. In addition, the acquisition of a 3D model of the scenes is useful in many areas as video games scene modeling where well-known places are reconstructed and added to game systems or advertising where once you get the 3D model of one room the system can add furniture pieces using augmented reality techniques. In this thesis we perform an experimental study of the state-of-the-art registration methods to find which one fits better to our scene mapping purposes. Different methods are tested and analyzed on different scene distributions of visual and geometry appearance. In addition, this thesis proposes two methods for 3d data compression and representation of 3D maps. Our 3D representation proposal is based on the use of Growing Neural Gas (GNG) method. This Self-Organizing Maps (SOMs) has been successfully used for clustering, pattern recognition and topology representation of various kind of data. Until now, Self-Organizing Maps have been primarily computed offline and their application in 3D data has mainly focused on free noise models without considering time constraints. Self-organising neural models have the ability to provide a good representation of the input space. In particular, the Growing Neural Gas (GNG) is a suitable model because of its flexibility, rapid adaptation and excellent quality of representation. However, this type of learning is time consuming, specially for high-dimensional input data. Since real applications often work under time constraints, it is necessary to adapt the learning process in order to complete it in a predefined time. This thesis proposes a hardware implementation leveraging the computing power of modern GPUs which takes advantage of a new paradigm coined as General-Purpose Computing on Graphics Processing Units (GPGPU). Our proposed geometrical 3D compression method seeks to reduce the 3D information using plane detection as basic structure to compress the data. This is due to our target environments are man-made and therefore there are a lot of points that belong to a plane surface. Our proposed method is able to get good compression results in those man-made scenarios. The detected and compressed planes can be also used in other applications as surface reconstruction or plane-based registration algorithms. Finally, we have also demonstrated the goodness of the GPU technologies getting a high performance implementation of a CAD/CAM common technique called Virtual Digitizing.

Virtual simulation of the postsurgical cosmetic outcome in patientswith pectus excavatum

Pectus excavatum is the most common congenital deformity of the anterior chest wall, in which several ribs and the sternum grow abnormally. Nowadays, the surgical correction is carried out in children and adults through Nuss technic. This technic has been shown to be safe with major drivers as cosmesis and the prevention of psychological problems and social stress. Nowadays, no application is known to predict the cosmetic outcome of the pectus excavatum surgical correction. Such tool could be used to help the surgeon and the patient in the moment of deciding the need for surgery correction. This work is a first step to predict postsurgical outcome in pectus excavatum surgery correction. Facing this goal, it was firstly determined a point cloud of the skin surface along the thoracic wall using Computed Tomography (before surgical correction) and the Polhemus FastSCAN (after the surgical correction). Then, a surface mesh was reconstructed from the two point clouds using a Radial Basis Function algorithm for further affine registration between the meshes. After registration, one studied the surgical correction influence area (SCIA) of the thoracic wall. This SCIA was used to train, test and validate artificial neural networks in order to predict the surgical outcome of pectus excavatum correction and to determine the degree of convergence of SCIA in different patients. Often, ANN did not converge to a satisfactory solution (each patient had its own deformity characteristics), thus invalidating the creation of a mathematical model capable of estimating, with satisfactory results, the postsurgical outcome

Optimization of structures modeled with a Meshfree approach

Radial basis functions are being used in different scientific areas in order to reproduce the geometrical modeling of an object/structure, as well as to predict its behavior. Due to its characteristics, these functions are well suited for meshfree modeling of physical quantities, which for instances can be associated to the data sets of 3D laser scanning point clouds. In the present work the geometry of a structure is modeled by using multiquadric radial basis functions, and its configuration is further optimized in order to obtain better performances concerning to its static and dynamic behavior. For this purpose the authors consider the particle swarm optimization technique. A set of case studies is presented to illustrate the adequacy of the meshfree model used, as well as its link to particle swarm optimization technique. © 2014 IEEE.

Estrutura a partir do Movimento em Sistemas Autónomos

Mestrado em Engenharia Electrotécnica e de Computadores - Ramo de Sistemas Autónomos

Investigation and monitoring of rock slope instabilities in Norway by terrestrial laser scanning

Long-range Terrestrial Laser Scanning (TLS) is widely used in studies on rock slope instabilities. TLS point clouds allow the creation of high-resolution digital elevation models for detailed mapping of landslide morphologies and the measurement of the orientation of main discontinuities. Multi-temporal TLS datasets enable the quantification of slope displacements and rockfall volumes. We present three case studies using TLS for the investigation and monitoring of rock slope instabilities in Norway: 1) the analysis of 3D displacement of the Oksfjellet rock slope failure (Troms, northern Norway); 2) the detection and quantification of rockfalls along the sliding surfaces and at the front of the Kvitfjellet rock slope instability (Møre og Romsdal, western Norway); 3) the analysis of discontinuities and rotational movements of an unstable block at Stampa (Sogn og Fjordane, western Norway). These case studies highlight the possibilities but also limitations of TLS in investigating and monitoring unstable rock slopes.

Detection, analysis and monitoring of slope movements by high-resolution digital elevation models

Summary Detection, analysis and monitoring of slope movements by high-resolution digital elevation modelsSlope movements, such as rockfalls, rockslides, shallow landslides or debris flows, are frequent in many mountainous areas. These natural hazards endanger the inhabitants and infrastructures making it necessary to assess the hazard and risk caused by these phenomena. This PhD thesis explores various approaches using digital elevation models (DEMs) - and particularly high-resolution DEMs created by aerial or terrestrial laser scanning (TLS) - that contribute to the assessment of slope movement hazard at regional and local scales.The regional detection of areas prone to rockfalls and large rockslides uses different morphologic criteria or geometric instability factors derived from DEMs, i.e. the steepness of the slope, the presence of discontinuities, which enable a sliding mechanism, and the denudation potential. The combination of these factors leads to a map of susceptibility to rockfall initiation that is in good agreement with field studies as shown with the example of the Little Mill Campground area (Utah, USA). Another case study in the Illgraben catchment in the Swiss Alps highlighted the link between areas with a high denudation potential and actual rockfall areas.Techniques for a detailed analysis and characterization of slope movements based on high-resolution DEMs have been developed for specific, localized sites, i.e. ancient slide scars, present active instabilities or potential slope instabilities. The analysis of the site's characteristics mainly focuses on rock slopes and includes structural analyses (orientation of discontinuities); estimation of spacing, persistence and roughness of discontinuities; failure mechanisms based on the structural setting; and volume calculations. For the volume estimation a new 3D approach was tested to reconstruct the topography before a landslide or to construct the basal failure surface of an active or potential instability. The rockslides at Åknes, Tafjord and Rundefjellet in western Norway were principally used as study sites to develop and test the different techniques.The monitoring of slope instabilities investigated in this PhD thesis is essentially based on multitemporal (or sequential) high-resolution DEMs, in particular sequential point clouds acquired by TLS. The changes in the topography due to slope movements can be detected and quantified by sequential TLS datasets, notably by shortest distance comparisons revealing the 3D slope movements over the entire region of interest. A detailed analysis of rock slope movements is based on the affine transformation between an initial and a final state of the rock mass and its decomposition into translational and rotational movements. Monitoring using TLS was very successful on the fast-moving Eiger rockslide in the Swiss Alps, but also on the active rockslides of Åknes and Nordnesfjellet (northern Norway). One of the main achievements on the Eiger and Aknes rockslides is to combine the site's morphology and structural setting with the measured slope movements to produce coherent instability models. Both case studies also highlighted a strong control of the structures in the rock mass on the sliding directions. TLS was also used to monitor slope movements in soils, such as landslides in sensitive clays in Québec (Canada), shallow landslides on river banks (Sorge River, Switzerland) and a debris flow channel (Illgraben).The PhD thesis underlines the broad uses of high-resolution DEMs and especially of TLS in the detection, analysis and monitoring of slope movements. Future studies should explore in more depth the different techniques and approaches developed and used in this PhD, improve them and better integrate the findings in current hazard assessment practices and in slope stability models.Résumé Détection, analyse et surveillance de mouvements de versant à l'aide de modèles numériques de terrain de haute résolutionDes mouvements de versant, tels que des chutes de blocs, glissements de terrain ou laves torrentielles, sont fréquents dans des régions montagneuses et mettent en danger les habitants et les infrastructures ce qui rend nécessaire d'évaluer le danger et le risque causé par ces phénomènes naturels. Ce travail de thèse explore diverses approches qui utilisent des modèles numériques de terrain (MNT) et surtout des MNT de haute résolution créés par scanner laser terrestre (SLT) ou aérien - et qui contribuent à l'évaluation du danger de mouvements de versant à l'échelle régionale et locale.La détection régionale de zones propices aux chutes de blocs ou aux éboulements utilise plusieurs critères morphologiques dérivés d'un MNT, tels que la pente, la présence de discontinuités qui permettent un mécanisme de glissement ou le potentiel de dénudation. La combinaison de ces facteurs d'instabilité mène vers une carte de susceptibilité aux chutes de blocs qui est en accord avec des travaux de terrain comme démontré avec l'exemple du Little Mill Campground (Utah, États-Unis). Un autre cas d'étude - l'Illgraben dans les Alpes valaisannes - a mis en évidence le lien entre les zones à fort potentiel de dénudation et les sources effectives de chutes de blocs et d'éboulements.Des techniques pour l'analyse et la caractérisation détaillée de mouvements de versant basées sur des MNT de haute résolution ont été développées pour des sites spécifiques et localisés, comme par exemple des cicatrices d'anciens éboulements et des instabilités actives ou potentielles. Cette analyse se focalise principalement sur des pentes rocheuses et comprend l'analyse structurale (orientation des discontinuités); l'estimation de l'espacement, la persistance et la rugosité des discontinuités; l'établissement des mécanismes de rupture; et le calcul de volumes. Pour cela une nouvelle approche a été testée en rétablissant la topographie antérieure au glissement ou en construisant la surface de rupture d'instabilités actuelles ou potentielles. Les glissements rocheux d'Åknes, Tafjord et Rundefjellet en Norvège ont été surtout utilisés comme cas d'étude pour développer et tester les diverses approches. La surveillance d'instabilités de versant effectuée dans cette thèse de doctorat est essentiellement basée sur des MNT de haute résolution multi-temporels (ou séquentiels), en particulier des nuages de points séquentiels acquis par SLT. Les changements topographiques dus aux mouvements de versant peuvent être détectés et quantifiés sur l'ensemble d'un glissement, notamment par comparaisons des distances les plus courtes entre deux nuages de points. L'analyse détaillée des mouvements est basée sur la transformation affine entre la position initiale et finale d'un bloc et sa décomposition en mouvements translationnels et rotationnels. La surveillance par SLT a démontré son potentiel avec l'effondrement d'un pan de l'Eiger dans les Alpes suisses, mais aussi aux glissements rocheux d'Aknes et Nordnesfjellet en Norvège. Une des principales avancées à l'Eiger et à Aknes est la création de modèles d'instabilité cohérents en combinant la morphologie et l'agencement structural des sites avec les mesures de déplacements. Ces deux cas d'étude ont aussi démontré le fort contrôle des structures existantes dans le massif rocheux sur les directions de glissement. Le SLT a également été utilisé pour surveiller des glissements dans des terrains meubles comme dans les argiles sensibles au Québec (Canada), sur les berges de la rivière Sorge en Suisse et dans le chenal à laves torrentielles de l'Illgraben.Cette thèse de doctorat souligne le vaste champ d'applications des MNT de haute résolution et particulièrement du SLT dans la détection, l'analyse et la surveillance des mouvements de versant. Des études futures devraient explorer plus en profondeur les différentes techniques et approches développées, les améliorer et mieux les intégrer dans des pratiques actuelles d'analyse de danger et surtout dans la modélisation de stabilité des versants.

Case study: object and ground point separation on a river bank using airborne based LIDAR data

In the past decade, airborne based LIght Detection And Ranging (LIDAR) has been recognised by both the commercial and public sectors as a reliable and accurate source for land surveying in environmental, engineering and civil applications. Commonly, the first task to investigate LIDAR point clouds is to separate ground and object points. Skewness Balancing has been proven to be an efficient non-parametric unsupervised classification algorithm to address this challenge. Initially developed for moderate terrain, this algorithm needs to be adapted to handle sloped terrain. This paper addresses the difficulty of object and ground point separation in LIDAR data in hilly terrain. A case study on a diverse LIDAR data set in terms of data provider, resolution and LIDAR echo has been carried out. Several sites in urban and rural areas with man-made structure and vegetation in moderate and hilly terrain have been investigated and three categories have been identified. A deeper investigation on an urban scene with a river bank has been selected to extend the existing algorithm. The results show that an iterative use of Skewness Balancing is suitable for sloped terrain.

Automatic Orientation of Multi-Scale Terrestrial Images for 3D Reconstruction

Fundação de Amparo à Pesquisa do Estado de São Paulo (FAPESP)