Robust facial expression analysis for affect-based interaction

Dissertação para obtenção do Grau de Mestre em Engenharia Informática

Interaction is moving towards new and more natural approaches. Human Computer Interaction (HCI) is increasingly expanding towards more modalities of human expression such as gestures, body movements and other natural interactions. In this thesis, we propose to extend existing interaction paradigms by including the face as an affect-based input. Affective interaction methods can greatly change the way computers interact with humans; these methods can detect displays of user moods, such as frustration or engagement and adapt the experience accordingly. We have created an affect-based framework that encompasses face detection, face recognition and facial expression recognition and applied it in a computer game. ImEmotion is a two-player game where the player who best mimics an expression wins. The game combines face detection with facial expression recognition to recognize and rate an expression in real time. A controlled evaluation of the framework algorithms and a game trial with 46 users showed the potential of the framework and success of the usage of affect-based interaction based on facial expressions in the game. Despite the novelty of the interaction approach and the limitations of computer vision algorithms, players adapted and became competitive easily.