A survey of human motion analysis using depth imagery

Analysis of human behaviour through visual information has been a highly active research topic in the computer vision community. This was previously achieved via images from a conventional camera, but recently depth sensors have made a new type of data available. This survey starts by explaining the advantages of depth imagery, then describes the new sensors that are available to obtain it. In particular, the Microsoft Kinect has made high-resolution real-time depth cheaply available. The main published research on the use of depth imagery for analysing human activity is reviewed. Much of the existing work focuses on body part detection and pose estimation. A growing research area addresses the recognition of human actions. The publicly available datasets that include depth imagery are listed, as are the software libraries that can acquire it from a sensor. This survey concludes by summarising the current state of work on this topic, and pointing out promising future research directions.

Video stimuli reduce object-directed imitation accuracy: a novel two-person motion-tracking approach

Imitation is an important form of social behavior, and research has aimed to discover and explain the neural and kinematic aspects of imitation. However, much of this research has featured single participants imitating in response to pre-recorded video stimuli. This is in spite of findings that show reduced neural activation to video vs. real life movement stimuli, particularly in the motor cortex. We investigated the degree to which video stimuli may affect the imitation process using a novel motion tracking paradigm with high spatial and temporal resolution. We recorded 14 positions on the hands, arms, and heads of two individuals in an imitation experiment. One individual freely moved within given parameters (moving balls across a series of pegs) and a second participant imitated. This task was performed with either simple (one ball) or complex (three balls) movement difficulty, and either face-to-face or via a live video projection. After an exploratory analysis, three dependent variables were chosen for examination: 3D grip position, joint angles in the arm, and grip aperture. A cross-correlation and multivariate analysis revealed that object-directed imitation task accuracy (as represented by grip position) was reduced in video compared to face-to-face feedback, and in complex compared to simple difficulty. This was most prevalent in the left-right and forward-back motions, relevant to the imitator sitting face-to-face with the actor or with a live projected video of the same actor. The results suggest that for tasks which require object-directed imitation, video stimuli may not be an ecologically valid way to present task materials. However, no similar effects were found in the joint angle and grip aperture variables, suggesting that there are limits to the influence of video stimuli on imitation. The implications of these results are discussed with regards to previous findings, and with suggestions for future experimentation.

Automatic Data Normalization and Parameterization for Optical Motion Tracking

Methods for optical motion capture often require timeconsuming manual processing before the data can be used for subsequent tasks such as retargeting or character animation. These processing steps restrict the applicability of motion capturing especially for dynamic VR-environments with real time requirements. To solve these problems, we present two additional, fast and automatic processing stages based on our motion capture pipeline presented in [HSK05]. A normalization step aligns the recorded coordinate systems with the skeleton structure to yield a common and intuitive data basis across different recording sessions. A second step computes a parameterization based on automatically extracted main movement axes to generate a compact motion description. Our method does not restrict the placement of marker bodies nor the recording setup, and only requires a short calibration phase.

A novel motion tracking system for evaluation of functional rehabilitation of the upper limbs

Upper limb function impairment is one of the most common sequelae of central nervous system injury, especially in stroke patients and when spinal cord injury produces tetraplegia. Conventional assessment methods cannot provide objective evaluation of patient performance and the tiveness of therapies. The most common assessment tools are based on rating scales, which are inefficient when measuring small changes and can yield subjective bias. In this study, we designed an inertial sensor-based monitoring system composed of five sensors to measure and analyze the complex movements of the upper limbs, which are common in activities of daily living. We developed a kinematic model with nine degrees of freedom to analyze upper limb and head movements in three dimensions. This system was then validated using a commercial optoelectronic system. These findings suggest that an inertial sensor-based motion tracking system can be used in patients who have upper limb impairment through data integration with a virtual reality-based neuroretation system.

Human Motion Analysis Based on Sequential Modeling of Radar Signal and Stereo Image Features

Falls are one of the greatest threats to elderly health in their daily living routines and activities. Therefore, it is very important to detect falls of an elderly in a timely and accurate manner, so that immediate response and proper care can be provided, by sending fall alarms to caregivers. Radar is an effective non-intrusive sensing modality which is well suited for this purpose, which can detect human motions in all types of environments, penetrate walls and fabrics, preserve privacy, and is insensitive to lighting conditions. Micro-Doppler features are utilized in radar signal corresponding to human body motions and gait to detect falls using a narrowband pulse-Doppler radar. Human motions cause time-varying Doppler signatures, which are analyzed using time-frequency representations and matching pursuit decomposition (MPD) for feature extraction and fall detection. The extracted features include MPD features and the principal components of the time-frequency signal representations. To analyze the sequential characteristics of typical falls, the extracted features are used for training and testing hidden Markov models (HMM) in different falling scenarios. Experimental results demonstrate that the proposed algorithm and method achieve fast and accurate fall detections. The risk of falls increases sharply when the elderly or patients try to exit beds. Thus, if a bed exit can be detected at an early stage of this motion, the related injuries can be prevented with a high probability. To detect bed exit for fall prevention, the trajectory of head movements is used for recognize such human motion. A head detector is trained using the histogram of oriented gradient (HOG) features of the head and shoulder areas from recorded bed exit images. A data association algorithm is applied on the head detection results to eliminate head detection false alarms. Then the three dimensional (3D) head trajectories are constructed by matching scale-invariant feature transform (SIFT) keypoints in the detected head areas from both the left and right stereo images. The extracted 3D head trajectories are used for training and testing an HMM based classifier for recognizing bed exit activities. The results of the classifier are presented and discussed in the thesis, which demonstrates the effectiveness of the proposed stereo vision based bed exit detection approach.

RF MIMO Systems for Wide-Area Indoor Human Motion Monitoring

Human motion monitoring is an important function in numerous applications. In this dissertation, two systems for monitoring motions of multiple human targets in wide-area indoor environments are discussed, both of which use radio frequency (RF) signals to detect, localize, and classify different types of human motion. In the first system, a coherent monostatic multiple-input multiple-output (MIMO) array is used, and a joint spatial-temporal adaptive processing method is developed to resolve micro-Doppler signatures at each location in a wide-area for motion mapping. The downranges are obtained by estimating time-delays from the targets, and the crossranges are obtained by coherently filtering array spatial signals. Motion classification is then applied to each target based on micro-Doppler analysis. In the second system, multiple noncoherent multistatic transmitters (Tx's) and receivers (Rx's) are distributed in a wide-area, and motion mapping is achieved by noncoherently combining bistatic range profiles from multiple Tx-Rx pairs. Also, motion classification is applied to each target by noncoherently combining bistatic micro-Doppler signatures from multiple Tx-Rx pairs. For both systems, simulation and real data results are shown to demonstrate the ability of the proposed methods for monitoring patient repositioning activities for pressure ulcer prevention.

PIV measurements combined with the motion tracking technique to analyze flow around a moving porous structure

To gain a better understanding of the fluid–structure interaction and especially when dealing with a flow around an arbitrarily moving body, it is essential to develop measurement tools enabling the instantaneous detection of moving deformable interface during the flow measurements. A particularly useful application is the determination of unsteady turbulent flow velocity field around a moving porous fishing net structure which is of great interest for selectivity and also for the numerical code validation which needs a realistic database. To do this, a representative piece of fishing net structure is used to investigate both the Turbulent Boundary Layer (TBL) developing over the horizontal porous moving fishing net structure and the turbulent flow passing through the moving porous structure. For such an investigation, Time Resolved PIV measurements are carried out and combined with a motion tracking technique allowing the measurement of the instantaneous motion of the deformable fishing net during PIV measurements. Once the two-dimensional motion of the porous structure is accessed, PIV velocity measurements are analyzed in connection with the detected motion. Finally, the TBL is characterized and the effect of the structure motion on the volumetric flow rate passing though the moving porous structure is clearly demonstrated.

Rao-blackwellized particle filter for human appearance and position tracking

In human motion analysis, the joint estimation of appearance, body pose and location parameters is not always tractable due to its huge computational cost. In this paper, we propose a Rao-Blackwellized Particle Filter for addressing the problem of human pose estimation and tracking. The advantage of the proposed approach is that Rao-Blackwellization allows the state variables to be splitted into two sets, being one of them analytically calculated from the posterior probability of the remaining ones. This procedure reduces the dimensionality of the Particle Filter, thus requiring fewer particles to achieve a similar tracking performance. In this manner, location and size over the image are obtained stochastically using colour and motion clues, whereas body pose is solved analytically applying learned human Point Distribution Models.

Tracking spatio temporal movement of human in terms of space utilization using media access control address data

Using Media-Access-Control (MAC) address for data collection and tracking is a capable and cost effective approach as the traditional ways such as surveys and video surveillance have numerous drawbacks and limitations. Positioning cell-phones by Global System for Mobile communication was considered an attack on people's privacy. MAC addresses just keep a unique log of a WiFi or Bluetooth enabled device for connecting to another device that has not potential privacy infringements. This paper presents the use of MAC address data collection approach for analysis of spatio-temporal dynamics of human in terms of shared space utilization. This paper firstly discuses the critical challenges and key benefits of MAC address data as a tracking technology for monitoring human movement. Here, proximity-based MAC address tracking is postulated as an effective methodology for analysing the complex spatio-temporal dynamics of human movements at shared zones such as lounge and office areas. A case study of university staff lounge area is described in detail and results indicates a significant added value of the methodology for human movement tracking. By analysis of MAC address data in the study area, clear statistics such as staff’s utilisation frequency, utilisation peak periods, and staff time spent is obtained. The analyses also reveal staff’s socialising profiles in terms of group and solo gathering. The paper is concluded with a discussion on why MAC address tracking offers significant advantages for tracking human behaviour in terms of shared space utilisation with respect to other and more prominent technologies, and outlines some of its remaining deficiencies.

Exploiting spatio-temporal constraints for robust 2D pose tracking

We present a Spatio-temporal 2D Models Framework (STMF) for 2D-Pose tracking. Space and time are discretized and a mixture of probabilistic "local models" is learnt associating 2D Shapes and 2D Stick Figures. Those spatio-temporal models generalize well for a particular viewpoint and state of the tracked action but some spatio-temporal discontinuities can appear along a sequence, as a direct consequence of the discretization. To overcome the problem, we propose to apply a Rao-Blackwellized Particle Filter (RBPF) in the 2D-Pose eigenspace, thus interpolating unseen data between view-based clusters. The fitness to the images of the predicted 2D-Poses is evaluated combining our STMF with spatio-temporal constraints. A robust, fast and smooth human motion tracker is obtained by tracking only the few most important dimensions of the state space and by refining deterministically with our STMF.

Generalized Laplacian Eigenmaps for Modeling and Tracking Human Motions

This paper presents generalized Laplacian eigenmaps, a novel dimensionality reduction approach designed to address stylistic variations in time series. It generates compact and coherent continuous spaces whose geometry is data-driven. This paper also introduces graph-based particle filter, a novel methodology conceived for efficient tracking in low dimensional space derived from a spectral dimensionality reduction method. Its strengths are a propagation scheme, which facilitates the prediction in time and style, and a noise model coherent with the manifold, which prevents divergence, and increases robustness. Experiments show that a combination of both techniques achieves state-of-the-art performance for human pose tracking in underconstrained scenarios.

Efficient tracking of human poses using a manifold hierarchy

In this paper a 3D human pose tracking framework is presented. A new dimensionality reduction method (Hierarchical Temporal Laplacian Eigenmaps) is introduced to represent activities in hierarchies of low dimensional spaces. Such a hierarchy provides increasing independence between limbs, allowing higher flexibility and adaptability that result in improved accuracy. Moreover, a novel deterministic optimisation method (Hierarchical Manifold Search) is applied to estimate efficiently the position of the corresponding body parts. Finally, evaluation on public datasets such as HumanEva demonstrates that our approach achieves a 62.5mm-65mm average joint error for the walking activity and outperforms state-of-the-art methods in terms of accuracy and computational cost.

Combining 3D tracking and surgical instrumentation to determine the stiffness of spinal motion segments: a validation study

The spine is a complex structure that provides motion in three directions: flexion and extension, lateral bending and axial rotation. So far, the investigation of the mechanical and kinematic behavior of the basic unit of the spine, a motion segment, is predominantly a domain of in vitro experiments on spinal loading simulators. Most existing approaches to measure spinal stiffness intraoperatively in an in vivo environment use a distractor. However, these concepts usually assume a planar loading and motion. The objective of our study was to develop and validate an apparatus, that allows to perform intraoperative in vivo measurements to determine both the applied force and the resulting motion in three dimensional space. The proposed setup combines force measurement with an instrumented distractor and motion tracking with an optoelectronic system. As the orientation of the applied force and the three dimensional motion is known, not only force-displacement, but also moment-angle relations could be determined. The validation was performed using three cadaveric lumbar ovine spines. The lateral bending stiffness of two motion segments per specimen was determined with the proposed concept and compared with the stiffness acquired on a spinal loading simulator which was considered to be gold standard. The mean values of the stiffness computed with the proposed concept were within a range of ±15% compared to data obtained with the spinal loading simulator under applied loads of less than 5 Nm.

Development and characterization of a scaffold-free 3D spheroid model of iPSC-derived human cardiomyocytes

Purpose: Cardiomyocytes are terminally differentiated cells in the adult heart and ischemia and cardiotoxic compounds can lead to cell death and irreversible decline of cardiac function. As testing platforms, isolated organs and primary cells from rodents have been the standard in research and toxicology, but there is a need for better models that more faithfully recapitulate native human biology. Hence, a new in vitro model comprising the advantages of 3D cell culture and the availability of induced pluripotent stem cells (iPSC) from human origin was developed and characterized. Methods: Human cardiomyocytes (CMs) derived from induced pluripotent stem cells (iPSCs) were studied in standard 2D culture and as cardiac microtissues (MTs) formed in hanging drops. 2D cultures were examined using immunofluorescence microscopy and Western blotting while the cardiac MTs were subjected to immunofluorescence, contractility, and pharmacological investigations. Results: iPSC-derived CMs in 2D culture showed well-formed myofibrils, cell-cell contacts positive for connexin-43, and other typical cardiac proteins. The cells reacted to pro-hypertrophic growth factors with a substantial increase in myofibrils and sarcomeric proteins. In hanging drop cultures, iPSC-derived cardiomyocytes formed spheroidal MTs within 4 days showing a homogeneous tissue structure with well-developed myofibrils extending throughout the whole spheroid without a necrotic core. MTs showed spontaneous contractions for more than 4 weeks that were recorded by optical motion tracking, sensitive to temperature, and responsive to electrical pacing. Contractile pharmacology was tested with several agents known to modulate cardiac rate and viability. Calcium-transients underlay the contractile activity and were also responsive to electrical stimulation, caffeine-induced Ca2+-release, extracellular calcium levels. Conclusions: 3D culture using iPSC-derived human cardiomyocytes provides an organoid human-based cellular platform that is free of necrosis and recapitulates vital cardiac functionality, thereby providing new and promising relevant model for the evaluation and development of new therapies and detection of cardiotoxicity.