Learning and detecting activities from movement trajectories using the hierarchical hidden Markov model

Directly modeling the inherent hierarchy and shared structures of human behaviors, we present an application of the hierarchical hidden Markov model (HHMM) for the problem of activity recognition. We argue that to robustly model and recognize complex human activities, it is crucial to exploit both the natural hierarchical decomposition and shared semantics embedded in the movement trajectories. To this end, we propose the use of the HHMM, a rich stochastic model that has been recently extended to handle shared structures, for representing and recognizing a set of complex indoor activities. Furthermore, in the need of real-time recognition, we propose a Rao-Blackwellised particle filter (RBPF) that efficiently computes the filtering distribution at a constant time complexity for each new observation arrival. The main contributions of this paper lie in the application of the shared-structure HHMM, the estimation of the model's parameters at all levels simultaneously, and a construction of an RBPF approximate inference scheme. The experimental results in a real-world environment have confirmed our belief that directly modeling shared structures not only reduces computational cost, but also improves recognition accuracy when compared with the tree HHMM and the flat HMM.

Spatial activity recognition in a smart home environment using a chemotactic model

Spatial activity recognition is challenging due to the amount of noise incorporated during video tracking in everyday environments. We address the spatial recognition problem with a biologically-inspired chemotactic model that is capable of handling noisy data. The model is based on bacterial chemotaxis, a process that allows bacteria to change motile behaviour in relation to environmental gradients. Through adoption of chemotactic principles, we propose the chemotactic model and evaluate its performance in a smart house environment. The model exhibits greater than 99% recognition performance with a diverse six class dataset and outperforms the Hidden Markov Model (HMM). The approach also maintains high accuracy (90-99%) with small training sets of one to five sequences. Importantly, unlike other low-level spatial activity recognition models, we show that the chemotactic model is capable of recognising simple interwoven activities.

Automatically learning structural units in educational videos with the hierarchical hidden Markov models

In this paper we present a coherent approach using the hierarchical HMM with shared structures to extract the structural units that form the building blocks of an education/training video. Rather than using hand-crafted approaches to define the structural units, we use the data from nine training videos to learn the parameters of the HHMM, and thus naturally extract the hierarchy. We then study this hierarchy and examine the nature of the structure at different levels of abstraction. Since the observable is continuous, we also show how to extend the parameter learning in the HHMM to deal with continuous observations.

Human action segmentation via controlled use of missing data in HMMs

Segmentation of individual actions from a stream of human motion is an open problem in computer vision. This paper approaches the problem of segmenting higher-level activities into their component sub-actions using Hidden Markov Models modified to handle missing data in the observation vector. By controlling the use of missing data, action labels can be inferred from the observation vector during inferencing, thus performing segmentation and classification simultaneously. The approach is able to segment both prominent and subtle actions, even when subtle actions are grouped together. The advantage of this method over sliding windows and Viterbi state sequence interrogation is that segmentation is performed as a trainable task, and the temporal relationship between actions is encoded in the model and used as evidence for action labelling.

Hierarchical monitoring of people's behaviors in complex environments using multiple cameras

We present a distributed, surveillance system that works in large and complex indoor environments. To track and recognize behaviors of people, we propose the use of the Abstract Hidden Markov Model (AHMM), which can be considered as an extension of the Hidden Markov Model (HMM), where the single Markov chain in the HMM is replaced by a hierarchy of Markov policies. In this policy hierarchy, each behavior can be represented as a policy at the corresponding level of abstraction. The noisy observations are handled in the same way as an HMM and an efficient Rao-Blackwellised particle filter method is used to compute the probabilities of the current policy at different levels of the hierarchy The novelty of the paper lies in the implementation of a scalable framework in the context of both the scale of behaviors and the size of the environment, making it ideal for distributed surveillance. The results of the system demonstrate the ability to answer queries about people's behaviors at different levels of details using multiple cameras in a large and complex indoor environment.

Content structure discovery in educational videos using shared structures in the hierarchical hidden Markov models

In this paper, we present an application of the hierarchical HMM for structure discovery in educational videos. The HHMM has recently been extended to accommodate the concept of shared structure, ie: a state might multiply inherit from more than one parents. Utilising the expressiveness of this model, we concentrate on a specific class of video -educational videos - in which the hierarchy of semantic units is simpler and clearly defined in terms of topics and its subunits. We model the hierarchy of topical structures by an HHMM and demonstrate the usefulness of the model in detecting topic transitions.

Topic transition detection using hierarchical hidden Markov and semi-Markov models

In this paper we introduce a probabilistic framework to exploit hierarchy, structure sharing and duration information for topic transition detection in videos. Our probabilistic detection framework is a combination of a shot classification step and a detection phase using hierarchical probabilistic models. We consider two models in this paper: the extended Hierarchical Hidden Markov Model (HHMM) and the Coxian Switching Hidden semi-Markov Model (S-HSMM) because they allow the natural decomposition of semantics in videos, including shared structures, to be modeled directly, and thus enabling efficient inference and reducing the sample complexity in learning. Additionally, the S-HSMM allows the duration information to be incorporated, consequently the modeling of long-term dependencies in videos is enriched through both hierarchical and duration modeling. Furthermore, the use of the Coxian distribution in the S-HSMM makes it tractable to deal with long sequences in video. Our experimentation of the proposed framework on twelve educational and training videos shows that both models outperform the baseline cases (flat HMM and HSMM) and performances reported in earlier work in topic detection. The superior performance of the S-HSMM over the HHMM verifies our belief that duration information is an important factor in video content modeling.

Discovery of activity structures using the hierarchical hidden Markov model

In building a surveillance system for monitoring people behaviours, it is important to understand the typical patterns of people's movement in the environment. This task is difficult when dealing with high-level behaviours. The flat model such as the hidden Markov model (HMM) is inefficient in differentiating between signatures of such behaviours. This paper examines structure learning for high-level behaviours using the hierarchical hidden Markov model (HHMM).We propose a two-phase learning algorithm in which the parameters of the behaviours at low levels are estimated first and then the structures and parameters of the behaviours at high levels are learned from multi-camera training data. Our algorithm is then evaluated using data from a real environment, demonstrating the robustness of the learned structure in recognising people's behaviour.

Efficient Coxian duration modelling for activity recognition in smart environment with the hidden semi-Markov model

In this paper, we exploit the discrete Coxian distribution and propose a novel form of stochastic model, termed as the Coxian hidden semi-Makov model (Cox-HSMM), and apply it to the task of recognising activities of daily living (ADLs) in a smart house environment. The use of the Coxian has several advantages over traditional parameterization (e.g. multinomial or continuous distributions) including the low number of free parameters needed, its computational efficiency, and the existing of closed-form solution. To further enrich the model in real-world applications, we also address the problem of handling missing observation for the proposed Cox-HSMM. In the domain of ADLs, we emphasize the importance of the duration information and model it via the Cox-HSMM. Our experimental results have shown the superiority of the Cox-HSMM in all cases when compared with the standard HMM. Our results have further shown that outstanding recognition accuracy can be achieved with relatively low number of phases required in the Coxian, thus making the Cox-HSMM particularly suitable in recognizing ADLs whose movement trajectories are typically very long in nature.

Long-term species, sexual and individual variations in foraging strategies of fur seals revealed by stable isotopes in whiskers

Background: Individual variations in the use of the species niche are an important component of diversity in trophic interactions. A challenge in testing consistency of individual foraging strategy is the repeated collection of information on the same individuals.

Methodology/Principal Findings: The foraging strategies of sympatric fur seals (Arctocephalus gazella and A. tropicalis) were examined using the stable isotope signature of serially sampled whiskers. Most whiskers exhibited synchronous delta C-13 and delta N-15 oscillations that correspond to the seal annual movements over the long term (up to 8 years). delta C-13 and delta N-15 values were spread over large ranges, with differences between species, sexes and individuals. The main segregating mechanism operates at the spatial scale. Most seals favored foraging in subantarctic waters (where the Crozet Islands are located) where they fed on myctophids. However, A. gazella dispersed in the Antarctic Zone and A. tropicalis more in the subtropics. Gender differences in annual time budget shape the seal movements. Males that do not perform any parental care exhibited large isotopic oscillations reflecting broad annual migrations, while isotopic values of females confined to a limited foraging range during lactation exhibited smaller changes. Limited inter-individual isotopic variations occurred in female seals and in male A. tropicalis. In contrast, male A. gazella showed large inter-individual variations, with some males migrating repeatedly to high-Antarctic waters where they fed on krill, thus meaning that individual specialization occurred over years.

Conclusions/Significance: Whisker isotopic signature yields unique long-term information on individual behaviour that integrates the spatial, trophic and temporal dimensions of the ecological niche. The method allows depicting the entire realized niche of the species, including some of its less well-known components such as age-, sex-, individual- and migration-related changes. It highlights intrapopulation heterogeneity in foraging strategies that could have important implications for likely demographic responses to environmental variability.

A generic classifier-ensemble approach for biomedical named entity recognition

In named entity recognition (NER) for biomedical literature, approaches based on combined classifiers have demonstrated great performance improvement compared to a single (best) classifier. This is mainly owed to sufficient level of diversity exhibited among classifiers, which is a selective property of classifier set. Given a large number of classifiers, how to select different classifiers to put into a classifier-ensemble is a crucial issue of multiple classifier-ensemble design. With this observation in mind, we proposed a generic genetic classifier-ensemble method for the classifier selection in biomedical NER. Various diversity measures and majority voting are considered, and disjoint feature subsets are selected to construct individual classifiers. A basic type of individual classifier – Support Vector Machine (SVM) classifier is adopted as SVM-classifier committee. A multi-objective Genetic algorithm (GA) is employed as the classifier selector to facilitate the ensemble classifier to improve the overall sample classification accuracy. The proposed approach is tested on the benchmark dataset – GENIA version 3.02 corpus, and compared with both individual best SVM classifier and SVM-classifier ensemble algorithm as well as other machine learning methods such as CRF, HMM and MEMM. The results show that the proposed approach outperforms other classification algorithms and can be a useful method for the biomedical NER problem.

Generative models for automatic recognition of human daily activities from a single triaxial accelerometer

In this work, we compare two generative models including Gaussian Mixture Model (GMM) and Hidden Markov Model (HMM) with Support Vector Machine (SVM) classifier for the recognition of six human daily activity (i.e., standing, walking, running, jumping, falling, sitting-down) from a single waist-worn tri-axial accelerometer signals through 4-fold cross-validation and testing on a total of thirteen subjects, achieving an average recognition accuracy of 96.43% and 98.21% in the first experiment and 95.51% and 98.72% in the second, respectively. The results demonstrate that both HMM and GMM are not only able to learn but also capable of generalization while the former outperformed the latter in the recognition of daily activities from a single waist worn tri-axial accelerometer. In addition, these two generative models enable the assessment of human activities based on acceleration signals with varying lengths.

Preparing for WIL : online career development for IT students

This paper reports on an online unit that enhances IT students’ generic skills. Frequently IT students, even those with a strong technical background and a high academic record, can be unsuccessful at obtaining work placements as they stumble at the interview stage due to a lack of social or professional skills. A simulation was created that enables students to enhance their employability and to prepare for transition to work integrated learning (WIL) through realistic interview preparation. The simulation utilizes a synchronous communication tool to conduct behavioural group interviews with expert careers advisors. The impact of this new initiative is explored and feedback received from faculty, careers advisors and students during three trimesters is discussed. The findings suggest that incorporating WIL through the simulation has been a success by at least raising students’ awareness of the importance and significance of being well prepared for job interviews.