Real-time hand tracking for rehabilitation and character animation

Hand and finger tracking has a major importance in healthcare, for rehabilitation of hand function required due to a neurological disorder, and in virtual environment applications, like characters animation for on-line games or movies. Current solutions consist mostly of motion tracking gloves with embedded resistive bend sensors that most often suffer from signal drift, sensor saturation, sensor displacement and complex calibration procedures. More advanced solutions provide better tracking stability, but at the expense of a higher cost. The proposed solution aims to provide the required precision, stability and feasibility through the combination of eleven inertial measurements units (IMUs). Each unit captures the spatial orientation of the attached body. To fully capture the hand movement, each finger encompasses two units (at the proximal and distal phalanges), plus one unit at the back of the hand. The proposed glove was validated in two distinct steps: a) evaluation of the sensors’ accuracy and stability over time; b) evaluation of the bending trajectories during usual finger flexion tasks based on the intra-class correlation coefficient (ICC). Results revealed that the glove was sensitive mainly to magnetic field distortions and sensors tuning. The inclusion of a hard and soft iron correction algorithm and accelerometer and gyro drift and temperature compensation methods provided increased stability and precision. Finger trajectories evaluation yielded high ICC values with an overall reliability within application’s satisfying limits. The developed low cost system provides a straightforward calibration and usability, qualifying the device for hand and finger tracking in healthcare and animation industries.

Hand tracking in high-speed camera videos

Visual object tracking has been one of the most popular research topics in the field of computer vision recently. Specifically, hand tracking has attracted significant attention since it would enable many useful practical applications. However, hand tracking is still a very challenging problem which cannot be considered solved. The fact that almost every aspect of hand appearance can change is the fundamental reason for this difficulty. This thesis focused on 2D-based hand tracking in high-speed camera videos. During the project, a toolbox for this purpose was collected which contains nine different tracking methods. In the experiments, these methods were tested and compared against each other with both high-speed videos recorded during the project and publicly available normal speed videos. The results revealed that tracking accuracies varied considerably depending on the video and the method. Therefore, no single method was clearly the best in all videos, but three methods, CT, HT, and TLD, performed better than the others overall. Moreover, the results provide insights about the suitability of each method to different types and situations of hand tracking.

Transition from an anti-phase error-correction-mode to a synchronization mode in the mutual hand tracking

Proactive motion in hand tracking and in finger bending, in which the body motion occurs prior to the reference signal, was reported by the preceding researchers when the target signals were shown to the subjects at relatively high speed or high frequencies. These phenomena indicate that the human sensory-motor system tends to choose an anticipatory mode rather than a reactive mode, when the target motion is relatively fast. The present research was undertaken to study what kind of mode appears in the sensory-motor system when two persons were asked to track the hand position of the partner with each other at various mean tracking frequency. The experimental results showed a transition from a mutual error-correction mode to a synchronization mode occurred in the same region of the tracking frequency with that of the transition from a reactive error-correction mode to a proactive anticipatory mode in the mechanical target tracking experiments. Present research indicated that synchronization of body motion occurred only when both of the pair subjects operated in a proactive anticipatory mode. We also presented mathematical models to explain the behavior of the error-correction mode and the synchronization mode.

Efficient and precise interactive hand tracking through joint, continuous optimization of pose and correspondences

Fully articulated hand tracking promises to enable fundamentally new interactions with virtual and augmented worlds, but the limited accuracy and efficiency of current systems has prevented widespread adoption. Today's dominant paradigm uses machine learning for initialization and recovery followed by iterative model-fitting optimization to achieve a detailed pose fit. We follow this paradigm, but make several changes to the model-fitting, namely using: (1) a more discriminative objective function; (2) a smooth-surface model that provides gradients for non-linear optimization; and (3) joint optimization over both the model pose and the correspondences between observed data points and the model surface. While each of these changes may actually increase the cost per fitting iteration, we find a compensating decrease in the number of iterations. Further, the wide basin of convergence means that fewer starting points are needed for successful model fitting. Our system runs in real-time on CPU only, which frees up the commonly over-burdened GPU for experience designers. The hand tracker is efficient enough to run on low-power devices such as tablets. We can track up to several meters from the camera to provide a large working volume for interaction, even using the noisy data from current-generation depth cameras. Quantitative assessments on standard datasets show that the new approach exceeds the state of the art in accuracy. Qualitative results take the form of live recordings of a range of interactive experiences enabled by this new approach.

Post-processing and analysis of tracked hand trajectories

In this thesis, the suitability of different trackers for finger tracking in high-speed videos was studied. Tracked finger trajectories from the videos were post-processed and analysed using various filtering and smoothing methods. Position derivatives of the trajectories, speed and acceleration were extracted for the purposes of hand motion analysis. Overall, two methods, Kernelized Correlation Filters and Spatio-Temporal Context Learning tracking, performed better than the others in the tests. Both achieved high accuracy for the selected high-speed videos and also allowed real-time processing, being able to process over 500 frames per second. In addition, the results showed that different filtering methods can be applied to produce more appropriate velocity and acceleration curves calculated from the tracking data. Local Regression filtering and Unscented Kalman Smoother gave the best results in the tests. Furthermore, the results show that tracking and filtering methods are suitable for high-speed hand-tracking and trajectory-data post-processing.

Efficient Bimanual Symmetric 3D Manipulation for Bare-Handed Interaction

Recently, stable markerless 6 DOF video based handtracking devices became available. These devices simultaneously track the positions and orientations of both user hands in different postures with at least 25 frames per second. Such hand-tracking allows for using the human hands as natural input devices. However, the absence of physical buttons for performing click actions and state changes poses severe challenges in designing an efficient and easy to use 3D interface on top of such a device. In particular, for coupling and decoupling a virtual object’s movements to the user’s hand (i.e. grabbing and releasing) a solution has to be found. In this paper, we introduce a novel technique for efficient two-handed grabbing and releasing objects and intuitively manipulating them in the virtual space. This technique is integrated in a novel 3D interface for virtual manipulations. A user experiment shows the superior applicability of this new technique. Last but not least, we describe how this technique can be exploited in practice to improve interaction by integrating it with RTT DeltaGen, a professional CAD/CAS visualization and editing tool.

A Comparative Evaluation of Three Skin Color Detection Approaches

Skin segmentation is a challenging task due to several influences such as unknown lighting conditions, skin colored background, and camera limitations. A lot of skin segmentation approaches were proposed in the past including adaptive (in the sense of updating the skin color online) and non-adaptive approaches. In this paper, we compare three skin segmentation approaches that are promising to work well for hand tracking, which is our main motivation for this work. Hand tracking can widely be used in VR/AR e.g. navigation and object manipulation. The first skin segmentation approach is a well-known non-adaptive approach. It is based on a simple, pre-computed skin color distribution. Methods two and three adaptively estimate the skin color in each frame utilizing clustering algorithms. The second approach uses a hierarchical clustering for a simultaneous image and color space segmentation, while the third approach is a pure color space clustering, but with a more sophisticated clustering approach. For evaluation, we compared the segmentation results of the approaches against a ground truth dataset. To obtain the ground truth dataset, we labeled about 500 images captured under various conditions.

Tracking the invisible hand: Convergence of double auctions to competitive equilibrium

Economics is the science of want and scarcity. We show that want andscarcity, operating within a simple exchange institution (double auction),are sufficient for an economy consisting of multiple inter--related marketsto attain competitive equilibrium (CE). We generalize Gode and Sunder's(1993a, 1993b) single--market finding to multi--market economies, andexplore the role of the scarcity constraint in convergence of economies to CE.When the scarcity constraint is relaxed by allowing arbitrageurs in multiple markets to enter speculative trades, prices still converge to CE,but allocative efficiency of the economy drops. \\Optimization by individual agents, often used to derive competitive equilibria,are unnecessary for an actual economy to approximately attain such equilibria.From the failure of humans to optimize in complex tasks, one need not concludethat the equilibria derived from the competitive model are descriptivelyirrelevant. We show that even in complex economic systems, such equilibriacan be attained under a range of surprisingly weak assumptions about agentbehavior.

Why the hand motion proceeds the target in tracking experiment?

In the present study, to shed light on a role of positional error correction mechanism and prediction mechanism in the proactive control discovered earlier, we carried out a visual tracking experiment, in which the region where target was shown, was regulated in a circular orbit. Main results found in this research were following. Recognition of a time step, obtained from the environmental stimuli, is required for the predictive function. The period of the rhythm in the brain obtained from environmental stimuli is shortened about 10%, when the visual information is cut-off. The shortening of the period of the rhythm in the brain accelerates the motion as soon as the visual information is cut-off, and lets the hand motion precedes the target motion. Although the precedence of the hand in the blind region is reset by the environmental information when the target enters the visible region, the hand precedes in average the target when the predictive mechanism dominates the error-corrective mechanism.

Tracking cost savings from competitive tendering in the short and long run

A major initiative of the Thatcher and Major Conservative administrations was that public sector ancillary and professional services provided by incumbent direct service organisations [DSOs] be put out to tender. Analyses of this initiative, in the UK and elsewhere, found costs were often reduced in the short run. However, few if any studies went beyond the first round of tendering. We analyze data collected over successive rounds of tendering for cleaning and catering services of Scottish hospitals in order to assess the long term consequences of this initiative. The experience of the two services was very different. Cost savings for cleaning services tended to increase with each additional round of tendering and became increasingly stable. In accordance with previous results in the literature, DSOs produced smaller cost reductions than private contractors: probably an inevitable consequence of the tendering process at the time. Cost savings from DSOs tended to disappear during the first round of tendering, but they appear to have been more permanent in successive rounds. Cost savings for catering, on the other hand, tended to be much smaller, and these were not sustained.

Thalamic connectivity in patients with essential tremor treated with MR imaging-guided focused ultrasound: in vivo fiber tracking by using diffusion-tensor MR imaging.

PURPOSE: To use diffusion-tensor (DT) magnetic resonance (MR) imaging in patients with essential tremor who were treated with transcranial MR imaging-guided focused ultrasound lesion inducement to identify the structural connectivity of the ventralis intermedius nucleus of the thalamus and determine how DT imaging changes correlated with tremor changes after lesion inducement. MATERIALS AND METHODS: With institutional review board approval, and with prospective informed consent, 15 patients with medication-refractory essential tremor were enrolled in a HIPAA-compliant pilot study and were treated with transcranial MR imaging-guided focused ultrasound surgery targeting the ventralis intermedius nucleus of the thalamus contralateral to their dominant hand. Fourteen patients were ultimately included. DT MR imaging studies at 3.0 T were performed preoperatively and 24 hours, 1 week, 1 month, and 3 months after the procedure. Fractional anisotropy (FA) maps were calculated from the DT imaging data sets for all time points in all patients. Voxels where FA consistently decreased over time were identified, and FA change in these voxels was correlated with clinical changes in tremor over the same period by using Pearson correlation. RESULTS: Ipsilateral brain structures that showed prespecified negative correlation values of FA over time of -0.5 or less included the pre- and postcentral subcortical white matter in the hand knob area; the region of the corticospinal tract in the centrum semiovale, in the posterior limb of the internal capsule, and in the cerebral peduncle; the thalamus; the region of the red nucleus; the location of the central tegmental tract; and the region of the inferior olive. The contralateral middle cerebellar peduncle and bilateral portions of the superior vermis also showed persistent decrease in FA over time. There was strong correlation between decrease in FA and clinical improvement in hand tremor 3 months after lesion inducement (P < .001). CONCLUSION: DT MR imaging after MR imaging-guided focused ultrasound thalamotomy depicts changes in specific brain structures. The magnitude of the DT imaging changes after thalamic lesion inducement correlates with the degree of clinical improvement in essential tremor.

Real-time Image-based RGB-D Camera Motion Tracking and Environment Mapping

In this work, image based estimation methods, also known as direct methods, are studied which avoid feature extraction and matching completely. Cost functions use raw pixels as measurements and the goal is to produce precise 3D pose and structure estimates. The cost functions presented minimize the sensor error, because measurements are not transformed or modified. In photometric camera pose estimation, 3D rotation and translation parameters are estimated by minimizing a sequence of image based cost functions, which are non-linear due to perspective projection and lens distortion. In image based structure refinement, on the other hand, 3D structure is refined using a number of additional views and an image based cost metric. Image based estimation methods are particularly useful in conditions where the Lambertian assumption holds, and the 3D points have constant color despite viewing angle. The goal is to improve image based estimation methods, and to produce computationally efficient methods which can be accomodated into real-time applications. The developed image-based 3D pose and structure estimation methods are finally demonstrated in practise in indoor 3D reconstruction use, and in a live augmented reality application.

The influence of drug-induced dyskinesias on manual tracking in Parkinson's disease

The influence of peak-dose drug-induced dyskinesia (DID) on manual tracking (MT) was examined in 10 dyskinetic patients (OPO), and compared to 10 age/gendermatched non-dyskinetic patients (NDPD) and 10 healthy controls. Whole body movement (WBM) and MT were recorded with a 6-degrees of freedom magnetic motion tracker and forearm rotation sensors, respectively. Subjects were asked to match the length of a computer-generated line with a line controlled via wrist rotation. Results show that OPO patients had greater WBM displacement and velocity than other groups. All groups displayed increased WBM from rest to MT, but only DPD and NDPO patients demonstrated a significant increase in WBM displacement and velocity. In addition, OPO patients exhibited excessive increase in WBM suggesting overflow DID. When two distinct target pace segments were examined (FAST/SLOW), all groups had slight increases in WBM displacement and velocity from SLOW to FAST, but only OPO patients showed significantly increased WBM displacement and velocity from SLOW to FAST. Therefore, it can be suggested that overflow DID was further increased with increased task speed. OPO patients also showed significantly greater ERROR matching target velocity, but no significant difference in ERROR in displacement, indicating that significantly greater WBM displacement in the OPO group did not have a direct influence on tracking performance. Individual target and performance traces demonstrated this relatively good tracking performance with the exception of distinct deviations from the target trace that occurred suddenly, followed by quick returns to the target coherent in time with increased performance velocity. In addition, performance hand velocity was not correlated with WBM velocity in DPO patients, suggesting that increased ERROR in velocity was not a direct result of WBM velocity. In conclusion, we propose that over-excitation of motor cortical areas, reported to be present in DPO patients, resulted in overflow DID during voluntary movement. Furthermore, we propose that the increased ERROR in velocity was the result of hypermetric voluntary movements also originating from the over-excitation of motor cortical areas.

Learning-Based Approach to Real Time Tracking and Analysis of Faces

This paper describes a trainable system capable of tracking faces and facialsfeatures like eyes and nostrils and estimating basic mouth features such as sdegrees of openness and smile in real time. In developing this system, we have addressed the twin issues of image representation and algorithms for learning. We have used the invariance properties of image representations based on Haar wavelets to robustly capture various facial features. Similarly, unlike previous approaches this system is entirely trained using examples and does not rely on a priori (hand-crafted) models of facial features based on optical flow or facial musculature. The system works in several stages that begin with face detection, followed by localization of facial features and estimation of mouth parameters. Each of these stages is formulated as a problem in supervised learning from examples. We apply the new and robust technique of support vector machines (SVM) for classification in the stage of skin segmentation, face detection and eye detection. Estimation of mouth parameters is modeled as a regression from a sparse subset of coefficients (basis functions) of an overcomplete dictionary of Haar wavelets.