A survey of human motion analysis using depth imagery

Analysis of human behaviour through visual information has been a highly active research topic in the computer vision community. This was previously achieved via images from a conventional camera, but recently depth sensors have made a new type of data available. This survey starts by explaining the advantages of depth imagery, then describes the new sensors that are available to obtain it. In particular, the Microsoft Kinect has made high-resolution real-time depth cheaply available. The main published research on the use of depth imagery for analysing human activity is reviewed. Much of the existing work focuses on body part detection and pose estimation. A growing research area addresses the recognition of human actions. The publicly available datasets that include depth imagery are listed, as are the software libraries that can acquire it from a sensor. This survey concludes by summarising the current state of work on this topic, and pointing out promising future research directions.

Why the hand motion proceeds the target in tracking experiment?

In the present study, to shed light on a role of positional error correction mechanism and prediction mechanism in the proactive control discovered earlier, we carried out a visual tracking experiment, in which the region where target was shown, was regulated in a circular orbit. Main results found in this research were following. Recognition of a time step, obtained from the environmental stimuli, is required for the predictive function. The period of the rhythm in the brain obtained from environmental stimuli is shortened about 10%, when the visual information is cut-off. The shortening of the period of the rhythm in the brain accelerates the motion as soon as the visual information is cut-off, and lets the hand motion precedes the target motion. Although the precedence of the hand in the blind region is reset by the environmental information when the target enters the visible region, the hand precedes in average the target when the predictive mechanism dominates the error-corrective mechanism.

Analysis of eye and head coordination in a visual peripheral recognition task

Coordinated eye and head movements simultaneously occur to scan the visual world for relevant targets. However, measuring both eye and head movements in experiments allowing natural head movements may be challenging. This paper provides an approach to study eye-head coordination: First, we demonstra- te the capabilities and limits of the eye-head tracking system used, and compare it to other technologies. Second, a beha- vioral task is introduced to invoke eye-head coordination. Third, a method is introduced to reconstruct signal loss in video- based oculography caused by cornea reflection artifacts in order to extend the tracking range. Finally, parameters of eye- head coordination are identified using EHCA (eye-head co- ordination analyzer), a MATLAB software which was developed to analyze eye-head shifts. To demonstrate the capabilities of the approach, a study with 11 healthy subjects was performed to investigate motion behavior. The approach presented here is discussed as an instrument to explore eye-head coordination, which may lead to further insights into attentional and motor symptoms of certain neurological or psychiatric diseases, e.g., schizophrenia.

Sparse representation based action and gesture recognition

In this paper we present a solution to the problem of action and gesture recognition using sparse representations. The dictionary is modelled as a simple concatenation of features computed for each action or gesture class from the training data, and test data is classified by finding sparse representation of the test video features over this dictionary. Our method does not impose any explicit training procedure on the dictionary. We experiment our model with two kinds of features, by projecting (i) Gait Energy Images (GEIs) and (ii) Motion-descriptors, to a lower dimension using Random projection. Experiments have shown 100% recognition rate on standard datasets and are compared to the results obtained with widely used SVM classifier.

Evaluation of Three State-of-the-Art Classifiers for Recognition of Activities of Daily Living from Smart Home Ambient Data

Smart homes for the aging population have recently started attracting the attention of the research community. The "health state" of smart homes is comprised of many different levels; starting with the physical health of citizens, it also includes longer-term health norms and outcomes, as well as the arena of positive behavior changes. One of the problems of interest is to monitor the activities of daily living (ADL) of the elderly, aiming at their protection and well-being. For this purpose, we installed passive infrared (PIR) sensors to detect motion in a specific area inside a smart apartment and used them to collect a set of ADL. In a novel approach, we describe a technology that allows the ground truth collected in one smart home to train activity recognition systems for other smart homes. We asked the users to label all instances of all ADL only once and subsequently applied data mining techniques to cluster in-home sensor firings. Each cluster would therefore represent the instances of the same activity. Once the clusters were associated to their corresponding activities, our system was able to recognize future activities. To improve the activity recognition accuracy, our system preprocessed raw sensor data by identifying overlapping activities. To evaluate the recognition performance from a 200-day dataset, we implemented three different active learning classification algorithms and compared their performance: naive Bayesian (NB), support vector machine (SVM) and random forest (RF). Based on our results, the RF classifier recognized activities with an average specificity of 96.53%, a sensitivity of 68.49%, a precision of 74.41% and an F-measure of 71.33%, outperforming both the NB and SVM classifiers. Further clustering markedly improved the results of the RF classifier. An activity recognition system based on PIR sensors in conjunction with a clustering classification approach was able to detect ADL from datasets collected from different homes. Thus, our PIR-based smart home technology could improve care and provide valuable information to better understand the functioning of our societies, as well as to inform both individual and collective action in a smart city scenario.

Spatial organization of retinal information about the direction of image motion.

The visual stimuli that elicit neural activity differ for different retinal ganglion cells and these cells have been categorized by the visual information that they transmit. If specific visual information is conveyed exclusively or primarily by a particular set of ganglion cells, one might expect the cells to be organized spatially so that their sampling of information from the visual field is complete but not redundant. In other words, the laterally spreading dendrites of the ganglion cells should completely cover the retinal plane without gaps or significant overlap. The first evidence for this sort of arrangement, which has been called a tiling or tessellation, was for the two types of "alpha" ganglion cells in cat retina. Other reports of tiling by ganglion cells have been made subsequently. We have found evidence of a particularly rigorous tiling for the four types of ganglion cells in rabbit retina that convey information about the direction of retinal image motion (the ON-OFF direction-selective cells). Although individual cells in the four groups are morphologically indistinguishable, they are organized as four overlaid tilings, each tiling consisting of like-type cells that respond preferentially to a particular direction of retinal image motion. These observations lend support to the hypothesis that tiling is a general feature of the organization of information outflow from the retina and clearly implicate mechanisms for recognition of like-type cells and establishment of mutually acceptable territories during retinal development.

Local form interference in biological motion perception

Peer reviewed

Field programmable gate array based target detection and gesture recognition

The move from Standard Definition (SD) to High Definition (HD) represents a six times increases in data, which needs to be processed. With expanding resolutions and evolving compression, there is a need for high performance with flexible architectures to allow for quick upgrade ability. The technology advances in image display resolutions, advanced compression techniques, and video intelligence. Software implementation of these systems can attain accuracy with tradeoffs among processing performance (to achieve specified frame rates, working on large image data sets), power and cost constraints. There is a need for new architectures to be in pace with the fast innovations in video and imaging. It contains dedicated hardware implementation of the pixel and frame rate processes on Field Programmable Gate Array (FPGA) to achieve the real-time performance. ^ The following outlines the contributions of the dissertation. (1) We develop a target detection system by applying a novel running average mean threshold (RAMT) approach to globalize the threshold required for background subtraction. This approach adapts the threshold automatically to different environments (indoor and outdoor) and different targets (humans and vehicles). For low power consumption and better performance, we design the complete system on FPGA. (2) We introduce a safe distance factor and develop an algorithm for occlusion occurrence detection during target tracking. A novel mean-threshold is calculated by motion-position analysis. (3) A new strategy for gesture recognition is developed using Combinational Neural Networks (CNN) based on a tree structure. Analysis of the method is done on American Sign Language (ASL) gestures. We introduce novel point of interests approach to reduce the feature vector size and gradient threshold approach for accurate classification. (4) We design a gesture recognition system using a hardware/ software co-simulation neural network for high speed and low memory storage requirements provided by the FPGA. We develop an innovative maximum distant algorithm which uses only 0.39% of the image as the feature vector to train and test the system design. Database set gestures involved in different applications may vary. Therefore, it is highly essential to keep the feature vector as low as possible while maintaining the same accuracy and performance^

The application of PCA as a movement pattern recognition technique: A proof of principle

Quantitative methods can help us understand how underlying attributes contribute to movement patterns. Applying principal components analysis (PCA) to whole-body motion data may provide an objective data-driven method to identify unique and statistically important movement patterns. Therefore, the primary purpose of this study was to determine if athletes’ movement patterns can be differentiated based on skill level or sport played using PCA. Motion capture data from 542 athletes performing three sport-screening movements (i.e. bird-dog, drop jump, T-balance) were analyzed. A PCA-based pattern recognition technique was used to analyze the data. Prior to analyzing the effects of skill level or sport on movement patterns, methodological considerations related to motion analysis reference coordinate system were assessed. All analyses were addressed as case-studies. For the first case study, referencing motion data to a global (lab-based) coordinate system compared to a local (segment-based) coordinate system affected the ability to interpret important movement features. Furthermore, for the second case study, where the interpretability of PCs was assessed when data were referenced to a stationary versus a moving segment-based coordinate system, PCs were more interpretable when data were referenced to a stationary coordinate system for both the bird-dog and T-balance task. As a result of the findings from case study 1 and 2, only stationary segment-based coordinate systems were used in cases 3 and 4. During the bird-dog task, elite athletes had significantly lower scores compared to recreational athletes for principal component (PC) 1. For the T-balance movement, elite athletes had significantly lower scores compared to recreational athletes for PC 2. In both analyses the lower scores in elite athletes represented a greater range of motion. Finally, case study 4 reported differences in athletes’ movement patterns who competed in different sports, and significant differences in technique were detected during the bird-dog task. Through these case studies, this thesis highlights the feasibility of applying PCA as a movement pattern recognition technique in athletes. Future research can build on this proof-of-principle work to develop robust quantitative methods to help us better understand how underlying attributes (e.g. height, sex, ability, injury history, training type) contribute to performance.

On covariate factor detection and removal for robust gait recognition

We propose a novel bolt-on module capable of boosting the robustness of various single compact 2D gait representations. Gait recognition is negatively influenced by covariate factors including clothing and time which alter the natural gait appearance and motion. Contrary to traditional gait recognition, our bolt-on module remedies this by a dedicated covariate factor detection and removal procedure which we quantitatively and qualitatively evaluate. The fundamental concept of the bolt-on module is founded on exploiting the pixel-wise composition of covariate factors. Results demonstrate how our bolt-on module is a powerful component leading to significant improvements across gait representations and datasets yielding state-of-the-art results.

Contributions to gait recognition using multiple views

Esta tesis se centra en la identificación de personas a través de la forma de caminar. El problema del reconocimiento del paso ha sido tratado mediante diferentes enfoques, en los dominios 2D y 3D, y usando una o varias vistas. Sin embargo, la dependencia con respecto al punto de vista, y por tanto de la trayectoria del sujeto al caminar sigue siendo aún un problema abierto. Se propone hacer frente al problema de la dependencia con respecto a la trayectoria por medio de reconstrucciones 3D de sujetos caminando. El uso de reconstrucciones varias ventajas que cabe destacar. En primer lugar, permite explotar una mayor cantidad de información en contraste con los métodos que extraen los descriptores de la marcha a partir de imágenes, en el dominio 2D. En segundo lugar, las reconstrucciones 3D pueden ser alineadas a lo largo de la marcha como si el sujeto hubiera caminado en una cinta andadora, proporcionando así una forma de analizar el paso independientemente de la trayectoria seguida. Este trabajo propone tres enfoques para resolver el problema de la dependencia a la vista: 1. Mediante la utilización de reconstrucciones volumétricas alineadas. 2. Mediante el uso de reconstrucciones volumétricas no alineadas. 3. Sin usar reconstrucciones. Se proponen además tres tipos de descriptores. El primero se centra en describir el paso mediante análisis morfológico de los volúmenes 3D alineados. El segundo hace uso del concepto de entropa de la información para describir la dinámica del paso humano. El tercero persigue capturar la dinámica de una forma invariante a rotación, lo cual lo hace especialmente interesante para ser aplicado tanto en trayectorias curvas como rectas, incluyendo cambios de dirección. Estos enfoques han sido probados sobre dos bases de datos públicas. Ambas están especialmente diseñadas para tratar el problema de la dependencia con respecto al punto de vista, y por tanto de la dependencia con respecto a la trayectoria. Los resultados experimentales muestran que para el enfoque basado en reconstrucciones volumétricas alineadas, el descriptor del paso basado en entropa consigue los mejores resultados, en comparación con métodos estrechamente relacionados del Estado del Arte actual. No obstante, el descriptor invariante a rotación consigue una tasa de reconocimiento que supera a los métodos actuales sin requerir la etapa previa de alineamiento de las reconstrucciones 3D.