Classifying human actions using an incomplete real-time pose skeleton

Currently, most human action recognition systems are trained with feature sets that have no missing data. Unfortunately, the use of human pose estimation models to provide more descriptive features also entails an increased sensitivity to occlusions, meaning that incomplete feature information will be unavoidable for realistic scenarios. To address this, our approach is to shift the responsibility for dealing with occluded pose data away from the pose estimator and onto the action classifier. This allows the use of a simple, real-time pose estimation (stick-figure) that does not estimate the positions of limbs it cannot find quickly. The system tracks people via background subtraction and extracts the (possibly incomplete) pose skeleton from their silhouette. Hidden Markov Models modified to handle missing data are then used to successfully classify several human actions using the incomplete pose features.