Facial expression recognition using facial movement features

Facial expression is an important channel for human communication and can be applied in many real applications. One critical step for facial expression recognition (FER) is to accurately extract emotional features. Current approaches on FER in static images have not fully considered and utilized the features of facial element and muscle movements, which represent static and dynamic, as well as geometric and appearance characteristics of facial expressions. This paper proposes an approach to solve this limitation using ‘salient’ distance features, which are obtained by extracting patch-based 3D Gabor features, selecting the ‘salient’ patches, and performing patch matching operations. The experimental results demonstrate high correct recognition rate (CRR), significant performance improvements due to the consideration of facial element and muscle movements, promising results under face registration errors, and fast processing time. The comparison with the state-of-the-art performance confirms that the proposed approach achieves the highest CRR on the JAFFE database and is among the top performers on the Cohn-Kanade (CK) database.

Improving the performance of facial expression recognition using dynamic, subtle and regional features

Human facial expression is a complex process characterized of dynamic, subtle and regional emotional features. State-of-the-art approaches on facial expression recognition (FER) have not fully utilized this kind of features to improve the recognition performance. This paper proposes an approach to overcome this limitation using patch-based ‘salient’ Gabor features. A set of 3D patches are extracted to represent the subtle and regional features, and then inputted into patch matching operations for capturing the dynamic features. Experimental results show a significant performance improvement of the proposed approach due to the use of the dynamic features. Performance comparison with pervious work also confirms that the proposed approach achieves the highest CRR reported to date on the JAFFE database and a top-level performance on the Cohn-Kanade (CK) database.

Car following model improvement for traffic safety metrics reproduction

Car Following models have a critical role in all microscopic traffic simulation models. Current microscopic simulation models are unable to mimic the unsafe behaviour of drivers as most are based on presumptions about the safe behaviour of drivers. Gipps model is a widely used car following model embedded in different micro-simulation models. This paper examines the Gipps car following model to investigate ways of improving the model for safety studies application. The paper puts forward some suggestions to modify the Gipps model to improve its capabilities to simulate unsafe vehicle movements (vehicles with safety indicators below critical thresholds). The result of the paper is one step forward to facilitate assessing and predicting safety at motorways using microscopic simulation. NGSIM as a rich source of vehicle trajectory data for a motorway is used to extract its relatively risky events. Short following headways and Time To Collision are used to assess critical safety event within traffic flow. The result shows that the modified proposed car following to a certain extent predicts the unsafe trajectories with smaller error values than the generic Gipps model.

Evaluation of texture and geometry for dimensional facial expression recognition

Facial expression recognition (FER) algorithms mainly focus on classification into a small discrete set of emotions or representation of emotions using facial action units (AUs). Dimensional representation of emotions as continuous values in an arousal-valence space is relatively less investigated. It is not fully known whether fusion of geometric and texture features will result in better dimensional representation of spontaneous emotions. Moreover, the performance of many previously proposed approaches to dimensional representation has not been evaluated thoroughly on publicly available databases. To address these limitations, this paper presents an evaluation framework for dimensional representation of spontaneous facial expressions using texture and geometric features. SIFT, Gabor and LBP features are extracted around facial fiducial points and fused with FAP distance features. The CFS algorithm is adopted for discriminative texture feature selection. Experimental results evaluated on the publicly accessible NVIE database demonstrate that fusion of texture and geometry does not lead to a much better performance than using texture alone, but does result in a significant performance improvement over geometry alone. LBP features perform the best when fused with geometric features. Distributions of arousal and valence for different emotions obtained via the feature extraction process are compared with those obtained from subjective ground truth values assigned by viewers. Predicted valence is found to have a more similar distribution to ground truth than arousal in terms of covariance or Bhattacharya distance, but it shows a greater distance between the means.

Sparse temporal representations for facial expression recognition

In automatic facial expression recognition, an increasing number of techniques had been proposed for in the literature that exploits the temporal nature of facial expressions. As all facial expressions are known to evolve over time, it is crucially important for a classifier to be capable of modelling their dynamics. We establish that the method of sparse representation (SR) classifiers proves to be a suitable candidate for this purpose, and subsequently propose a framework for expression dynamics to be efficiently incorporated into its current formulation. We additionally show that for the SR method to be applied effectively, then a certain threshold on image dimensionality must be enforced (unlike in facial recognition problems). Thirdly, we determined that recognition rates may be significantly influenced by the size of the projection matrix \Phi. To demonstrate these, a battery of experiments had been conducted on the CK+ dataset for the recognition of the seven prototypic expressions - anger, contempt, disgust, fear, happiness, sadness and surprise - and comparisons have been made between the proposed temporal-SR against the static-SR framework and state-of-the-art support vector machine.

No evidence for subliminal affective priming with emotional facial expression primes

The present study investigated whether facial expressions of emotion presented outside consciousness awareness will elicit evaluative responses as assessed in affective priming. Participants were asked to evaluate pleasant and unpleasant target words that were preceded by masked or unmasked schematic (Experiment 1) or photographic faces (Experiments 1 and 2) with happy or angry expressions. They were either required to perform the target evaluation only or to perform the target evaluation and to name the emotion expressed by the face prime. Prime-target interval was 300 ms in Experiment 1 and 80 ms in Experiment 2. Naming performance confirmed the effectiveness of the masking procedure. Affective priming was evident after unmasked primes in tasks that required naming of the facial expressions for schematic and photographic faces and after unmasked primes in tasks that did not require naming for photographic faces. No affective priming was found after masked primes. The present study failed to provide evidence for affective priming with masked face primes, however, it indicates that voluntary attention to the primes enhances affective priming.

Discovering the best feature extraction and selection algorithms for spontaneous facial expression recognition

Feature extraction and selection are critical processes in developing facial expression recognition (FER) systems. While many algorithms have been proposed for these processes, direct comparison between texture, geometry and their fusion, as well as between multiple selection algorithms has not been found for spontaneous FER. This paper addresses this issue by proposing a unified framework for a comparative study on the widely used texture (LBP, Gabor and SIFT) and geometric (FAP) features, using Adaboost, mRMR and SVM feature selection algorithms. Our experiments on the Feedtum and NVIE databases demonstrate the benefits of fusing geometric and texture features, where SIFT+FAP shows the best performance, while mRMR outperforms Adaboost and SVM. In terms of computational time, LBP and Gabor perform better than SIFT. The optimal combination of SIFT+FAP+mRMR also exhibits a state-of-the-art performance.

Improved facial expression recognition via uni-hyperplane classification

Large margin learning approaches, such as support vector machines (SVM), have been successfully applied to numerous classification tasks, especially for automatic facial expression recognition. The risk of such approaches however, is their sensitivity to large margin losses due to the influence from noisy training examples and outliers which is a common problem in the area of affective computing (i.e., manual coding at the frame level is tedious so coarse labels are normally assigned). In this paper, we leverage the relaxation of the parallel-hyperplanes constraint and propose the use of modified correlation filters (MCF). The MCF is similar in spirit to SVMs and correlation filters, but with the key difference of optimizing only a single hyperplane. We demonstrate the superiority of MCF over current techniques on a battery of experiments.

Robust facial feature extraction and matching

Automated feature extraction and correspondence determination is an extremely important problem in the face recognition community as it often forms the foundation of the normalisation and database construction phases of many recognition and verification systems. This paper presents a completely automatic feature extraction system based upon a modified volume descriptor. These features form a stable descriptor for faces and are utilised in a reversible jump Markov chain Monte Carlo correspondence algorithm to automatically determine correspondences which exist between faces. The developed system is invariant to changes in pose and occlusion and results indicate that it is also robust to minor face deformations which may be present with variations in expression.

Towards spontaneous facial expression recognition in real-world video

Facial expression is an important channel of human social communication. Facial expression recognition (FER) aims to perceive and understand emotional states of humans based on information in the face. Building robust and high performance FER systems that can work in real-world video is still a challenging task, due to the various unpredictable facial variations and complicated exterior environmental conditions, as well as the difficulty of choosing a suitable type of feature descriptor for extracting discriminative facial information. Facial variations caused by factors such as pose, age, gender, race and occlusion, can exert profound influence on the robustness, while a suitable feature descriptor largely determines the performance. Most present attention on FER has been paid to addressing variations in pose and illumination. No approach has been reported on handling face localization errors and relatively few on overcoming facial occlusions, although the significant impact of these two variations on the performance has been proved and highlighted in many previous studies. Many texture and geometric features have been previously proposed for FER. However, few comparison studies have been conducted to explore the performance differences between different features and examine the performance improvement arisen from fusion of texture and geometry, especially on data with spontaneous emotions. The majority of existing approaches are evaluated on databases with posed or induced facial expressions collected in laboratory environments, whereas little attention has been paid on recognizing naturalistic facial expressions on real-world data. This thesis investigates techniques for building robust and high performance FER systems based on a number of established feature sets. It comprises of contributions towards three main objectives: (1) Robustness to face localization errors and facial occlusions. An approach is proposed to handle face localization errors and facial occlusions using Gabor based templates. Template extraction algorithms are designed to collect a pool of local template features and template matching is then performed to covert these templates into distances, which are robust to localization errors and occlusions. (2) Improvement of performance through feature comparison, selection and fusion. A comparative framework is presented to compare the performance between different features and different feature selection algorithms, and examine the performance improvement arising from fusion of texture and geometry. The framework is evaluated for both discrete and dimensional expression recognition on spontaneous data. (3) Evaluation of performance in the context of real-world applications. A system is selected and applied into discriminating posed versus spontaneous expressions and recognizing naturalistic facial expressions. A database is collected from real-world recordings and is used to explore feature differences between standard database images and real-world images, as well as between real-world images and real-world video frames. The performance evaluations are based on the JAFFE, CK, Feedtum, NVIE, Semaine and self-collected QUT databases. The results demonstrate high robustness of the proposed approach to the simulated localization errors and occlusions. Texture and geometry have different contributions to the performance of discrete and dimensional expression recognition, as well as posed versus spontaneous emotion discrimination. These investigations provide useful insights into enhancing robustness and achieving high performance of FER systems, and putting them into real-world applications.

Studies of assisted reproduction in the spotted grass frog Limnodynastes tasmaniensis: ovulation, early development and microinjection (ICSI)

Assisted Reproductive Technologies (ART) offer a wide range of techniques that have the potential to augment efforts to conserve and manage endangered amphibians and improve wild and captive population numbers. Gametes and tissues of species nearing endangered or extinct status can be cryopreserved and stored in gene banks, to provide material that can be utilised in the future as ART methods are refined. The Spotted Grass Frog, Limnodynastes tasmaniensis, is an abundant amphibian species in South-Eastern Australia of the family Myobatrachidae, that is suitable for the development of ART systems that can be applied to the threatened and endangered myobatrachid and other amphibian species native to Australia. The aim of this study was to advance the understanding of ovulation, fertilisation and embryo nic development of Lim. tasmaniensis and in vitro manipulations of reproduction and development for use in the development of advanced ART procedures such as intracytoplasmic spermatozoon injection (ICSI), androgenesis and nuclear transfer. Ovulation in amphibians can be induced by protocols utilising natural or synthetic hormones. All protocols tested on Lim. tasmaniensis in this study required two injections and the most effective protocols continued to require a first injection of pituitary extracts to induce ovulation. The second injection was, however, successfully replaced by synthetic chorionic gonadotrophin at a threshold dosage of 100 iu and halved the number of cane toads required to source the pituitaries. A combination of LHRH and Pimozide offered a less effective protocol, that did not require the use of pituitary extracts, and avoided the risk of pathogen transfer associated with unsterilised pituitary extracts. Unfertilised eggs of Lim. tasmaniensis were exposed to media of various osmolalities to determine media effects on eggs and their surrounding jelly layers that might impact on egg viability and fertilisability. Osmolality had no effect upon the egg diameter, however, rapid swelling of the jelly layers occurred within 15 minutes of exposure to various media treatments and plateaued from 30-90 minutes without further expansion. Swelling of the jelly layers was increased in hypotonic media (2.5% SAR, H2O) and minimised in the isotonic media (100% SAR). The optimal conditions for the culture of Lim. tasmaniensis eggs were identified as a holding media of 100% SAR, followed by a medium change to 2.5% SAR at insemination. This sequence of media minimised the rate of swelling of the jelly layers prior to contact with the spermatozoa, and maximised the activation of spermatozoa and eggs throughout fertilisation and embryonic development. Embryos of Lim. tasmaniensis were cultured at four temperatures (13 C, 17 C, 23 C and 29 C), to determine the effect of temperature on cleavage and embryonic development rates. Embryonic development progressed through a sequence of stages that were not altered by changes in temperature. However cleavage rates were affected by changes in temperature as compared with normal embryonic growth at 23 C. Embryonic development was suspended at the lowest temperature (13 C) while embryonic viability was maintained. A moderate decrease in temperature (17 C) slowed cleavage, while the highest temperature (29 C) increased the cleavage rate, but decreased the embryo survival. Rates of embryonic development can be manipulated by changes in temperature and this method can be used to source blastomeres of a specific size/stage at a predetermined age or halt cleavage at specific stages for embryos or embryo derived cells to be included in ART procedures. This study produced the first report of the application of Intracytoplasmic Spermatozoon Injection (ICSI) in an Australian amphibian. Eggs that were activated by microinjection with a single spermatozoon (n=50) formed more deep, but abnormal, cleavage furrows post-injection (18/50, 36%), than surface changes (12/50, 24%). This result is in contrast to eggs injected without a spermatozoon (n=42), where the majority of eggs displayed limited surface changes (36/42, 86%), and few deep, abnormal furrows (3/42, 7%). Three advanced embryos (3/50, 6%) were produced by ICSI that developed to various stages within the culture system. Technical difficulties were encountered that prevented the generation of any metamorphs from ICSI tadpoles. Nevertheless, when these blocks to ICSI are overcome, the ICSI procedure will be both directly useful as an ART procedure in its own right, and the associated refinement of micromanipulation procedures will assist in the development of other ART procedures in Lim. tasmaniensis. A greater understanding of basic reproductive and developmental biology in Lim. tasmaniensis would greatly facilitate refinement of fertilisation by ICSI. Assisted Reproductive Technologies, in conjunction with gene banks may in the future regenerate extinct amphibian species, and assist in the recovery of declining amphibian populations nationally and worldwide.