Modeling kinect sensor noise for improved 3D reconstruction and tracking

We contribute an empirically derived noise model for the Kinect sensor. We systematically measure both lateral and axial noise distributions, as a function of both distance and angle of the Kinect to an observed surface. The derived noise model can be used to filter Kinect depth maps for a variety of applications. Our second contribution applies our derived noise model to the KinectFusion system to extend filtering, volumetric fusion, and pose estimation within the pipeline. Qualitative results show our method allows reconstruction of finer details and the ability to reconstruct smaller objects and thinner surfaces. Quantitative results also show our method improves pose estimation accuracy. © 2012 IEEE.

Single camera pose estimation using Bayesian filtering and Kinect motion priors

Traditional approaches to upper body pose estimation using monocular vision rely on complex body models and a large variety of geometric constraints. We argue that this is not ideal and somewhat inelegant as it results in large processing burdens, and instead attempt to incorporate these constraints through priors obtained directly from training data. A prior distribution covering the probability of a human pose occurring is used to incorporate likely human poses. This distribution is obtained offline, by fitting a Gaussian mixture model to a large dataset of recorded human body poses, tracked using a Kinect sensor. We combine this prior information with a random walk transition model to obtain an upper body model, suitable for use within a recursive Bayesian filtering framework. Our model can be viewed as a mixture of discrete Ornstein-Uhlenbeck processes, in that states behave as random walks, but drift towards a set of typically observed poses. This model is combined with measurements of the human head and hand positions, using recursive Bayesian estimation to incorporate temporal information. Measurements are obtained using face detection and a simple skin colour hand detector, trained using the detected face. The suggested model is designed with analytical tractability in mind and we show that the pose tracking can be Rao-Blackwellised using the mixture Kalman filter, allowing for computational efficiency while still incorporating bio-mechanical properties of the upper body. In addition, the use of the proposed upper body model allows reliable three-dimensional pose estimates to be obtained indirectly for a number of joints that are often difficult to detect using traditional object recognition strategies. Comparisons with Kinect sensor results and the state of the art in 2D pose estimation highlight the efficacy of the proposed approach.

Avaliação automatizada de uma habilidade motora fundamental com o kinect

Pós-graduação em Desenvolvimento Humano e Tecnologias - IBRC

Augmenting depth camera output using photometric stereo

We present a system for augmenting depth camera output using multispectral photometric stereo. The technique is demonstrated using a Kinect sensor and is able to produce geometry independently for each frame. Improved reconstruction is demonstrated using the Kinect's inbuilt RGB camera and further improvements are achieved by introducing an additional high resolution camera. As well as qualitative improvements in reconstruction a quantitative reduction in temporal noise is shown. As part of the system an approach is presented for relaxing the assumption of multispectral photometric stereo that scenes are of constant chromaticity to the assumption that scenes contain multiple piecewise constant chromaticities.

Analyse quantifiée de l'asymétrie de la marche par application de Poincaré

La marche occupe un rôle important dans la vie quotidienne. Ce processus apparaît comme facile et naturel pour des gens en bonne santé. Cependant, différentes sortes de maladies (troubles neurologiques, musculaires, orthopédiques...) peuvent perturber le cycle de la marche à tel point que marcher devient fastidieux voire même impossible. Ce projet utilise l'application de Poincaré pour évaluer l'asymétrie de la marche d'un patient à partir d'une carte de profondeur acquise avec un senseur Kinect. Pour valider l'approche, 17 sujets sains ont marché sur un tapis roulant dans des conditions différentes : marche normale et semelle de 5 cm d'épaisseur placée sous l'un des pieds. Les descripteurs de Poincaré sont appliqués de façon à évaluer la variabilité entre un pas et le cycle complet de la marche. Les résultats montrent que la variabilité ainsi obtenue permet de discriminer significativement une marche normale d'une marche avec semelle. Cette méthode, à la fois simple à mettre en oeuvre et suffisamment précise pour détecter une asymétrie de la marche, semble prometteuse pour aider dans le diagnostic clinique.

Um método para reconstrução de superfícies baseado em nuvem de pontos visando representações em multirresolução

The representation of real objects in virtual environments has applications in many areas, such as cartography, mixed reality and reverse engineering. The generation of these objects can be performed in two ways: manually, with CAD (Computer Aided Design) tools, or automatically, by means of surface reconstruction techniques. The simpler the 3D model, the easier it is to process and store it. Multiresolution reconstruction methods can generate polygonal meshes in different levels of detail and, to improve the response time of a computer program, distant objects can be represented with few details, while more detailed models are used in closer objects. This work presents a new approach to multiresolution surface reconstruction, particularly interesting to noisy and low definition data, for example, point clouds captured with Kinect sensor

TECHNOLOGICAL ADVANCEMENT IN REMOTE SENSING AND INNOVATIVE APPLICATION TO VOLCANOLOGY

The integration of remote monitoring techniques at different scales is of crucial importance for monitoring of volcanoes and assessment of the associated hazard. In this optic, technological advancement and collaboration between research groups also play a key role. Vhub is a community cyberinfrastructure platform designed for collaboration in volcanology research. Within the Vhub framework, this dissertation focuses on two research themes, both representing novel applications of remotely sensed data in volcanology: advancement in the acquisition of topographic data via active techniques and application of passive multi-spectral satellite data to monitoring of vegetated volcanoes. Measuring surface deformation is a critical issue in analogue modelling of Earth science phenomena. I present a novel application of the Microsoft Kinect sensor to measurement of vertical and horizontal displacements in analogue models. Specifically, I quantified vertical displacement in a scaled analogue model of Nisyros volcano, Greece, simulating magmatic deflation and inflation and related surface deformation, and included the horizontal component to reconstruct 3D models of pit crater formation. The detection of active faults around volcanoes is of importance for seismic and volcanic hazard assessment, but not a simple task to be achieved using analogue models. I present new evidence of neotectonic deformation along a north-south trending fault from the Mt Shasta debris avalanche deposit (DAD), northern California. The fault was identified on an airborne LiDAR campaign of part of the region interested by the DAD and then confirmed in the field. High resolution LiDAR can be utilized also for geomorphological assessment of DADs, and I describe a size-distance analysis to document geomorphological aspects of hummock in the Shasta DAD. Relating the remote observations of volcanic passive degassing to conditions and impacts on the ground provides an increased understanding of volcanic degassing and how satellite-based monitoring can be used to inform hazard management strategies in nearreal time. Combining a variety of satellite-based spectral time series I aim to perform the first space-based assessment of the impacts of sulfur dioxide emissions from Turrialba volcano, Costa Rica, on vegetation in the surrounding environment, and establish whether vegetation indices could be used more broadly to detect volcanic unrest.

Three-dimensional mapping of real environments for cognitive robots navigation

Abstract The development of cognitive robots needs a strong “sensorial” support which should allow it to perceive the real world for interacting with it properly. Therefore the development of efficient visual-processing software to be equipped in effective artificial agents is a must. In this project we study and develop a visual-processing software that will work as the “eyes” of a cognitive robot. This software performs a three-dimensional mapping of the robot’s environment, providing it with the essential information required to make proper decisions during its navigation. Due to the complexity of this objective we have adopted the Scrum methodology in order to achieve an agile development process, which has allowed us to correct and improve in a fast way the successive versions of the product. The present project is structured in Sprints, which cover the different stages of the software development based on the requirements imposed by the robot and its real necessities. We have initially explored different commercial devices oriented to the acquisition of the required visual information, adopting the Kinect Sensor camera (Microsoft) as the most suitable option. Later on, we have studied the available software to manage the obtained visual information as well as its integration with the robot’s software, choosing the high-level platform Matlab as the common nexus to join the management of the camera, the management of the robot and the implementation of the behavioral algorithms. During the last stages the software has been developed to include the fundamental functionalities required to process the real environment, such as depth representation, segmentation, and clustering. Finally the software has been optimized to exhibit real-time processing and a suitable performance to fulfill the robot’s requirements during its operation in real situations.

Point clouds of measurements in the Dechen Cave near Iserlohn, Germany

Modeling natural phenomena from 3D information enhances our understanding of the environment. Dense 3D point clouds are increasingly used as highly detailed input datasets. In addition to the capturing techniques of point clouds with LiDAR, low-cost sensors have been released in the last few years providing access to new research fields and facilitating 3D data acquisition for a broader range of applications. This letter presents an analysis of different speleothem features using 3D point clouds acquired with the gaming device Microsoft® Kinect. We compare the Kinect sensor with terrestrial LiDAR reference measurements using the KinFu pipeline for capturing complete 3D objects (< 4m**3). The results demonstrate the suitability of the Kinect to capture flowstone walls and to derive morphometric parameters of cave features. Although the chosen capturing strategy (KinFu) reveals a high correlation (R2=0.92) of stalagmite morphometry along the vertical object axis, a systematic overestimation (22% for radii and 44% for volume) is found. The comparison of flowstone wall datasets predominantly shows low differences (mean of 1 mm with 7 mm standard deviation) of the order of the Kinect depth precision. For both objects the major differences occur at strongly varying and curved surface structures (e.g. with fine concave parts).

Redefinición de las TIC en el museo: del discurso invasivo al inclusivo

Las TIC son inseparables de la museografía in situ e imprescindibles en la museografía en red fija y móvil. En demasiados casos se han instalado prótesis tecnológicas para barnizar de modernidad el espacio cultural, olvidando que la tecnología debe estar al servicio de los contenidos de manera que resulte invisible y perfectamente imbricada con la museografía tradicional. Las interfaces móviles pueden fusionar museo in situ y en red y acompañar a las personas más allá del espacio físico. Esa fusión debe partir de una base de datos narrativa y abierta a obras materiales e inmateriales de otros museos de manera que no se trasladen las limitaciones del museo físico al virtual. En el museo in situ tienen sentido las instalaciones hipermedia inmersivas que faciliten experiencias culturales innovadoras. La interactividad (relaciones virtuales) debe convivir con la interacción (relaciones físicas y personales) y estar al servicio de todas las personas, partiendo de que todas, todos tenemos limitaciones. Trabajar interdisciplinarmente ayuda a comprender mejor el museo para ponerlo al servicio de las personas.

IntelliChair: a non-intrusive sitting posture and sitting activity recognition system

Current Ambient Intelligence and Intelligent Environment research focuses on the interpretation of a subject’s behaviour at the activity level by logging the Activity of Daily Living (ADL) such as eating, cooking, etc. In general, the sensors employed (e.g. PIR sensors, contact sensors) provide low resolution information. Meanwhile, the expansion of ubiquitous computing allows researchers to gather additional information from different types of sensor which is possible to improve activity analysis. Based on the previous research about sitting posture detection, this research attempts to further analyses human sitting activity. The aim of this research is to use non-intrusive low cost pressure sensor embedded chair system to recognize a subject’s activity by using their detected postures. There are three steps for this research, the first step is to find a hardware solution for low cost sitting posture detection, second step is to find a suitable strategy of sitting posture detection and the last step is to correlate the time-ordered sitting posture sequences with sitting activity. The author initiated a prototype type of sensing system called IntelliChair for sitting posture detection. Two experiments are proceeded in order to determine the hardware architecture of IntelliChair system. The prototype looks at the sensor selection and integration of various sensor and indicates the best for a low cost, non-intrusive system. Subsequently, this research implements signal process theory to explore the frequency feature of sitting posture, for the purpose of determining a suitable sampling rate for IntelliChair system. For second and third step, ten subjects are recruited for the sitting posture data and sitting activity data collection. The former dataset is collected byasking subjects to perform certain pre-defined sitting postures on IntelliChair and it is used for posture recognition experiment. The latter dataset is collected by asking the subjects to perform their normal sitting activity routine on IntelliChair for four hours, and the dataset is used for activity modelling and recognition experiment. For the posture recognition experiment, two Support Vector Machine (SVM) based classifiers are trained (one for spine postures and the other one for leg postures), and their performance evaluated. Hidden Markov Model is utilized for sitting activity modelling and recognition in order to establish the selected sitting activities from sitting posture sequences.2. After experimenting with possible sensors, Force Sensing Resistor (FSR) is selected as the pressure sensing unit for IntelliChair. Eight FSRs are mounted on the seat and back of a chair to gather haptic (i.e., touch-based) posture information. Furthermore, the research explores the possibility of using alternative non-intrusive sensing technology (i.e. vision based Kinect Sensor from Microsoft) and find out the Kinect sensor is not reliable for sitting posture detection due to the joint drifting problem. A suitable sampling rate for IntelliChair is determined according to the experiment result which is 6 Hz. The posture classification performance shows that the SVM based classifier is robust to “familiar” subject data (accuracy is 99.8% with spine postures and 99.9% with leg postures). When dealing with “unfamiliar” subject data, the accuracy is 80.7% for spine posture classification and 42.3% for leg posture classification. The result of activity recognition achieves 41.27% accuracy among four selected activities (i.e. relax, play game, working with PC and watching video). The result of this thesis shows that different individual body characteristics and sitting habits influence both sitting posture and sitting activity recognition. In this case, it suggests that IntelliChair is suitable for individual usage but a training stage is required.

Sistema de comercio electrónico para el sector textil: generar maniquíes 3D a partir de imágenes del sensor Kinect

Este proyecto consiste en analizar la viabilidad técnica un sistema de información para mejorar el comercio electrónico de compra –venta de ropa. El nuevo sistema de comercio electrónico ayudará al usuario a escoger el tamaño de la prenda con más precisión. El sistema consiste en calcular las medidas esenciales del cuerpo del cliente utilizando el dispositivo Kinect. Posteriormente, estas medidas se introducen en el programa Makehuman que genera un maniquí en 3D. Con la idea de que en el futuro se puede utilizar el maniquí para probar virtualmente prendas de vestir, se ha ideado un método, utilizando el programa Blender, que genera ropa en 3D con fotografías de prendas. Se generan dos ejemplos (pantalón y camiseta) para ilustrarlo.

On the use of a low-cost thermal sensor to improve kinect people detection in a mobile robot

[EN]Detecting people is a key capability for robots that operate in populated environments. In this paper, we have adopted a hierarchical approach that combines classifiers created using supervised learning in order to identify whether a person is in the view-scope of the robot or not. Our approach makes use of vision, depth and thermal sensors mounted on top of a mobile platform.

Desarrollo y evaluación de un sistema de teleoperación de robots basado en reconocimiento de gestos usando el sensor Kinect

En este Proyecto Fin de Carrera, se presenta un sistema de reconocimiento de gestos para teleoperar robots basado en el sensor Kinect. El proyecto se divide en dos partes, la primera relativa al diseño y evaluación de un sistema de reconocimiento de gestos basado en el sensor Kinect; y la segunda, relativa a la teleoperación de robots usando el sistema de reconocimiento de gestos desarrollado. En la primera parte, se enumeran las características y limitaciones del sensor Kinect. Posteriormente, se analiza la detección de movimiento y se presenta la máquina de estados propuesta para detectar el movimiento de un gesto. A continuación, se explican los posibles preprocesados de un esqueleto en 3 dimensiones para mejorar la detección de gestos y el algoritmo utilizado para la detección de gestos, el algoritmo de Alineamiento Temporal Dinámico (DTW). Por último, se expone con detalle el software desarrollado de reconocimiento y evaluación de gestos, el Evaluador de Gestos, y se realiza un análisis de varias evaluaciones realizadas con distintos perfiles de configuración donde se extraen las conclusiones de acierto, fiabilidad y precisión de cada configuración. En la segunda parte, se expone el sistema de teleoperación del robots y su integración con el evaluador de gestos: este sistema controla el robot Lego Mindstorm mediante la detección de gestos o el reconocimiento de voz. Por último, se exponen las conclusiones finales del proyecto.

Usando Kinect como sensor para una pulverización inteligente.

Este trabajo esta orientado a resolver el problema de la caracterización de la copa de arboles frutales para la aplicacion localizada de fitosanitarios. Esta propuesta utiliza un mapa de profundidad (Depth image) y una imagen RGB combinadas (RGB-D), proporcionados por el sensor Kinect de Microsoft, para aplicar pesticidas de forma localizada. A través del mapa de profundidad se puede estimar la densidad de la copa y a partir de esta información determinar qué boquillas se deben abrir en cada momento. Se desarrollaron algoritmos implementados en Matlab que permiten además de la adquisición de las imágenes RGB-D, aplicar plaguicidas sólo a hojas y/o frutos según se desee. Estos algoritmos fueron implementados en un software que se comunica con el entorno de desarrollo "Kinect Windows SDK", encargado de extraer las imágenes desde el sensor Kinect. Por otra parte, para identificar hojas, se implementaron algoritmos de clasificación e identificación. Los algoritmos de clasificación utilizados fueron "Fuzzy C-Means con Gustafson Kessel" (FCM-GK) y "K-Means". Los centroides o prototipos de cada clase generados por FCM-GK fueron usados como semilla para K-Means, para acelerar la convergencia del algoritmo y mantener la coherencia temporal en los grupos generados por K-Means. Los algoritmos de clasificación fueron aplicados sobre las imágenes transformadas al espacio de color L*a*b*; específicamente se emplearon los canales a*, b* (canales cromáticos) con el fin de reducir el efecto de la luz sobre los colores. Los algoritmos de clasificación fueron configurados para buscar cuatro grupos: hojas, porosidad, frutas y tronco. Una vez que el clasificador genera los prototipos de los grupos, un clasificador denominado Máquina de Soporte Vectorial, que utiliza como núcleo una función Gaussiana base radial, identifica la clase de interés (hojas). La combinación de estos algoritmos ha mostrado bajos errores de clasificación, rendimiento del 4% de error en la identificación de hojas. Además, estos algoritmos de procesamiento de hasta 8.4 imágenes por segundo, lo que permite su aplicación en tiempo real. Los resultados demuestran la viabilidad de utilizar el sensor "Kinect" para determinar dónde y cuándo aplicar pesticidas. Por otra parte, también muestran que existen limitaciones en su uso, impuesta por las condiciones de luz. En otras palabras, es posible usar "Kinect" en exteriores, pero durante días nublados, temprano en la mañana o en la noche con iluminación artificial, o añadiendo un parasol en condiciones de luz intensa.