Impact Study of Nonverbal Facial Cues on Spontaneous Chatting with Virtual Humans

Non-verbal communication (NVC) is considered to represent more than 90 percent of everyday communication. In virtual world, this important aspect of interaction between virtual humans (VH) is strongly neglected. This paper presents a user-test study to demonstrate the impact of automatically generated graphics-based NVC expression on the dialog quality: first, we wanted to compare impassive and emotion facial expression simulation for impact on the chatting. Second, we wanted to see whether people like chatting within a 3D graphical environment. Our model only proposes facial expressions and head movements induced from spontaneous chatting between VHs. Only subtle facial expressions are being used as nonverbal cues - i.e. related to the emotional model. Motion capture animations related to hand gestures, such as cleaning glasses, were randomly used to make the virtual human lively. After briefly introducing the technical architecture of the 3D-chatting system, we focus on two aspects of chatting through VHs. First, what is the influence of facial expressions that are induced from text dialog? For this purpose, we exploited an emotion engine extracting an emotional content from a text and depicting it into a virtual character developed previously [GAS11]. Second, as our goal was not addressing automatic generation of text, we compared the impact of nonverbal cues in conversation with a chatbot or with a human operator with a wizard of oz approach. Among main results, the within group study -involving 40 subjects- suggests that subtle facial expressions impact significantly not only on the quality of experience but also on dialog understanding.

Impaired gesture performance in schizophrenia: particular vulnerability of meaningless pantomimes

Schizophrenia patients frequently present with subtle motor impairments, including higher order motor function such as hand gesture performance. Using cut off scores from a standardized gesture test, we previously reported gesture deficits in 40% of schizophrenia patients irrespective of the gesture content. However, these findings were based on normative data from an older control group. Hence, we now aimed at determining cut-off scores in an age and gender matched control group. Furthermore, we wanted to explore whether gesture categories are differentially affected in Schizophrenia. Gesture performance data of 30 schizophrenia patients and data from 30 matched controls were compared. Categories included meaningless, intransitive (communicative) and transitive (object related) hand gestures, which were either imitated or pantomimed, i.e. produced on verbal command. Cut-off scores of the age matched control group were higher than the previous cut-off scores in an older control group. An ANOVA tested effects of group, domain (imitation or pantomime), and semantic category (meaningless, transitive or intransitive), as well as their interaction. According to the new cut-off scores, 67% of the schizophrenia patients demonstrated gestural deficits. Patients performed worse in all gesture categories, however meaningless gestures on verbal command were particularly impaired (p = 0.008). This category correlated with poor frontal lobe function (p < 0.001). In conclusion, gestural deficits in schizophrenia are even more frequent than previously reported. Gesture categories that pose higher demands on planning and selection such as pantomime of meaningless gestures are predominantly affected and associated with the well-known frontal lobe dysfunction.

Disorganization and timing of motor behavior : insight from gesture impairments and movement patterns in schizophrenia

Background: Motor symptoms are frequent phenomena across the entire course of schizophrenia1. Some have argued that disorganized behavior was associated with aberrant motor behavior. We have studied the association of motor disturbances and disorganization in two projects focusing on the timing of movements. Method: In two studies, we assessed motor behavior and psychopathology. The first study applied a validated test of upper limb apraxia in 30 schizophrenia patients2,3. We used standardized video assessments of hand gestures by a blinded rater. The second study tested the stability of movement patterns using time series analysis in actigraphy data of 100 schizophrenia patients4. Both stability of movement patterns and the overall amount of movement were calculated from data of two hours with high degrees of social interaction comparable across the 100 subjects. Results: In total, 67% of the patients had gesture performance deficits3. Most frequently, they made spatial, temporal and body-part-as-object errors. Gesture performance relied on frontal lobe function2. Poor gesture performance was associated with increased disorganization scores. In the second study, we found disorganization to be predicted only by more irregular movement patterns irrespective of the overall amount of movement4. Conclusion : Both studies provide evidence for a link between aberrant timing of motor behavior and disorganization. Disturbed movement control seems critical for disorganized behavior in schizophrenia.

User-independent accelerometer-based gesture recognition for mobile devices

Many mobile devices embed nowadays inertial sensors. This enables new forms of human-computer interaction through the use of gestures (movements performed with the mobile device) as a way of communication. This paper presents an accelerometer-based gesture recognition system for mobile devices which is able to recognize a collection of 10 different hand gestures. The system was conceived to be light and to operate in a user -independent manner in real time. The recognition system was implemented in a smart phone and evaluated through a collection of user tests, which showed a recognition accuracy similar to other state-of-the art techniques and a lower computational complexity. The system was also used to build a human -robot interface that enables controlling a wheeled robot with the gestures made with the mobile phone.

Controlador MIDI basado en Arduino

El presente proyecto tiene como objetivo la creación de un controlador MIDI económico que haga uso de la tecnología actual, y partiendo de la idea del instrumento clásico, el Theremin, desarrollado por Lev Serguéievich Termen. Para ello se ha dividido el proyecto en dos principales bloques, el primero, hardware y el segundo, software. En la parte del hardware, se explica cual ha sido la razón de la utilización del microprocesador Arduino Uno, sus características técnicas y el uso de sensores de ultrasonido, ya que proporcionan la característica de poder interactuar con el controlador a través de gestos con las manos, al igual que un Theremin clásico. Se explica el montaje de los dispositivos que conforman el controlador, así como la mejora realizada, con la utilización de 4 de estos sensores, para dar más capacidades de interactuación con el controlador MIDI. También se ve en ese apartado, como se programa la tarjeta de Arduino, para que se encargue de realizar medidas con los sensores y enviarlas por el puerto serial USB. En el apartado del software se da una introducción al entorno de programación Max/MSP. Se ve el plug in desarrollado con este lenguaje, para poder comunicar el controlador MIDI con un software de audio profesional (Ableton Live) y se explica con detalle los bloques que conforman el plug in de control de sensores y como es transformada la información que entrega el microprocesador Arduino por el puerto USB, en datos MIDI. También, se da una explicación sobre el manejo correcto del controlador a la hora de mover las manos sobre los sensores y de donde situar el instrumento para que no se produzcan problemas de interferencias con las señales que envían los ultrasonidos. Además, se proporciona un presupuesto del coste de los materiales, y otro del coste del desarrollo realizado por el ingeniero. ABSTRACT The aim of this Project is the creation of an economical MIDI controller that uses nowadays technology and that is based on the idea of the Theremin, a classical instrument conceived by Lev Serguéievich Termen. In order to accomplish this, the project has been divided into two sections: hardware and software. The hardware section explains why the microprocessor Arduino Uno has been chosen, sets out its technical specifications and the use of ultrasonic sensors. These sensors enable the user to interact with the controller through hand gestures like the Theremin. The assembly of the devices is exposed as well as the improvements made with the use of four of these sensors to offer more interactive capabilities with the MIDI controller. The Arduino singleboard programming that performs the measurements with the sensors and sends these measurements through the USB serial port is also explained here. The software section introduces Max/MSP programming environment as well as the plug in developed with this language that connects the MIDI controller with professional audio software (Ableton Live). The blocks that build the sensor controller plug in are explained in detail along with the way the Arduino delivers the information through the USB port into MIDI data. In addition, an explanation of the correct handling of the MIDI controller is given focusing on how the user should move his hands above the sensors and where to place the instrument to avoid interference problems with the signals sent. Also, a cost estimation of both materials and engineering is provided.

New generation of human machine interfaces for controlling UAV through depth based gesture recognition

New forms of natural interactions between human operators and UAVs (Unmanned Aerial Vehicle) are demanded by the military industry to achieve a better balance of the UAV control and the burden of the human operator. In this work, a human machine interface (HMI) based on a novel gesture recognition system using depth imagery is proposed for the control of UAVs. Hand gesture recognition based on depth imagery is a promising approach for HMIs because it is more intuitive, natural, and non-intrusive than other alternatives using complex controllers. The proposed system is based on a Support Vector Machine (SVM) classifier that uses spatio-temporal depth descriptors as input features. The designed descriptor is based on a variation of the Local Binary Pattern (LBP) technique to efficiently work with depth video sequences. Other major consideration is the especial hand sign language used for the UAV control. A tradeoff between the use of natural hand signs and the minimization of the inter-sign interference has been established. Promising results have been achieved in a depth based database of hand gestures especially developed for the validation of the proposed system.

Reconocimiento de gestos basado en acelerómetros

En los últimos años, ha crecido de forma significativa el interés por la utilización de dispositivos capaces de reconocer gestos humanos. En este trabajo, se pretenden reconocer gestos manuales colocando sensores en la mano de una persona. El reconocimiento de gestos manuales puede ser implementado para diversos usos y bajo diversas plataformas: juegos (Wii), control de brazos robóticos, etc. Como primer paso, se realizará un estudio de las actuales técnicas de reconocimiento de gestos que utilizan acelerómetros como sensor de medida. En un segundo paso, se estudiará como los acelerómetros pueden utilizarse para intentar reconocer los gestos que puedan realizar una persona (mover el brazo hacia un lado, girar la mano, dibujar un cuadrado, etc.) y los problemas que de su utilización puedan derivarse. Se ha utilizado una IMU (Inertial Measurement Unit) como sensor de medida. Está compuesta por tres acelerómetros y tres giróscopos (MTi-300 de Xsens). Con las medidas que proporcionan estos sensores se realiza el cálculo de la posición y orientación de la mano, representando esta última en función de los ángulos de Euler. Un aspecto importante a destacar será el efecto de la gravedad en las medidas de las aceleraciones. A través de diversos cálculos y mediante la ayuda de los giróscopos se podrá corregir dicho efecto. Por último, se desarrollará un sistema que identifique la posición y orientación de la mano como gestos reconocidos utilizando lógica difusa. Tanto para la adquisición de las muestras, como para los cálculos de posicionamiento, se ha desarrollado un código con el programa Matlab. También, con este mismo software, se ha implementado un sistema de lógica difusa con la que se realizará el reconocimiento de los gestos, utilizando la herramienta FIS Editor. Las pruebas realizadas han consistido en la ejecución de nueve gestos por diferentes personas teniendo una tasa de reconocimiento comprendida entre el 90 % y 100 % dependiendo del gesto a identificar. ABSTRACT In recent years, it has grown significantly interest in the use of devices capable of recognizing human gestures. In this work, we aim to recognize hand gestures placing sensors on the hand of a person. The recognition of hand gestures can be implemented for different applications on different platforms: games (Wii), control of robotic arms ... As a first step, a study of current gesture recognition techniques that use accelerometers and sensor measurement is performed. In a second step, we study how accelerometers can be used to try to recognize the gestures that can make a person (moving the arm to the side, rotate the hand, draw a square, etc...) And the problems of its use can be derived. We used an IMU (Inertial Measurement Unit) as a measuring sensor. It comprises three accelerometers and three gyroscopes (Xsens MTI-300). The measures provided by these sensors to calculate the position and orientation of the hand are made, with the latter depending on the Euler angles. An important aspect to note is the effect of gravity on the measurements of the accelerations. Through various calculations and with the help of the gyroscopes can correct this effect. Finally, a system that identifies the position and orientation of the hand as recognized gestures developed using fuzzy logic. Both the acquisition of samples to calculate position, a code was developed with Matlab program. Also, with the same software, has implemented a fuzzy logic system to be held with the recognition of gestures using the FIS Editor. Tests have involved the execution of nine gestures by different people having a recognition rate between 90% and 100% depending on the gesture to identify.

Control and Guidance of Low-Cost Robots via Gesture Perception for Monitoring Activities in the Home

This paper describes the development of a low-cost mini-robot that is controlled by visual gestures. The prototype allows a person with disabilities to perform visual inspections indoors and in domestic spaces. Such a device could be used as the operator's eyes obviating the need for him to move about. The robot is equipped with a motorised webcam that is also controlled by visual gestures. This camera is used to monitor tasks in the home using the mini-robot while the operator remains quiet and motionless. The prototype was evaluated through several experiments testing the ability to use the mini-robot’s kinematics and communication systems to make it follow certain paths. The mini-robot can be programmed with specific orders and can be tele-operated by means of 3D hand gestures to enable the operator to perform movements and monitor tasks from a distance.

Low-cost real-time gesture recognition

A major impediment to developing real-time computer vision systems has been the computational power and level of skill required to process video streams in real-time. This has meant that many researchers have either analysed video streams off-line or used expensive dedicated hardware acceleration techniques. Recent software and hardware developments have greatly eased the development burden of realtime image analysis leading to the development of portable systems using cheap PC hardware and software exploiting the Multimedia Extension (MMX) instruction set of the Intel Pentium chip. This paper describes the implementation of a computationally efficient computer vision system for recognizing hand gestures using efficient coding and MMX-acceleration to achieve real-time performance on low cost hardware.

Dynamic Human Robot Interaction Framework Using Deep Learning and Robot Operating System (ROS): a practical approach

Trying to explain to a robot what to do is a difficult undertaking, and only specific types of people have been able to do so far, such as programmers or operators who have learned how to use controllers to communicate with a robot. My internship's goal was to create and develop a framework that would make that easier. The system uses deep learning techniques to recognize a set of hand gestures, both static and dynamic. Then, based on the gesture, it sends a command to a robot. To be as generic as feasible, the communication is implemented using Robot Operating System (ROS). Furthermore, users can add new recognizable gestures and link them to new robot actions; a finite state automaton enforces the users' input verification and correct action sequence. Finally, the users can create and utilize a macro to describe a sequence of actions performable by a robot.

Cultural differences in nonverbal communication and their implications in public service interpreting

This paper intends to underline the importance of nonverbal cues in intercultural communication and especially in the profession of the interpreter in the public services. The first chapter focuses on giving an overview on the different components of nonverbal communication by providing a definition and examples. The chapter then analyses the different functions carried out by nonverbal language. The second chapter addresses cultural differences related to nonverbal language and provides examples for every component of nonverbal language. It also explains the distinction between high context cultures and low context cultures. Lastly, the third chapter first gives an overview on Public Service Interpreting (PSI), explaining the main features and characteristics. It then provides real examples of cultural misunderstandings related to nonverbal language which occurred in public services and gives the techniques used by interpreters to solve them. This chapter includes examples of various nonverbal cues including eye contact, touch, silence, and hand gestures.

Motor Unit Spike Trains Recontruction from sEMG Signals for Gesture Classification on a Low-Power Processing Platform

Hand gesture recognition based on surface electromyography (sEMG) signals is a promising approach for the development of intuitive human-machine interfaces (HMIs) in domains such as robotics and prosthetics. The sEMG signal arises from the muscles' electrical activity, and can thus be used to recognize hand gestures. The decoding from sEMG signals to actual control signals is non-trivial; typically, control systems map sEMG patterns into a set of gestures using machine learning, failing to incorporate any physiological insight. This master thesis aims at developing a bio-inspired hand gesture recognition system based on neuromuscular spike extraction rather than on simple pattern recognition. The system relies on a decomposition algorithm based on independent component analysis (ICA) that decomposes the sEMG signal into its constituent motor unit spike trains, which are then forwarded to a machine learning classifier. Since ICA does not guarantee a consistent motor unit ordering across different sessions, 3 approaches are proposed: 2 ordering criteria based on firing rate and negative entropy, and a re-calibration approach that allows the decomposition model to retain information about previous sessions. Using a multilayer perceptron (MLP), the latter approach results in an accuracy up to 99.4% in a 1-subject, 1-degree of freedom scenario. Afterwards, the decomposition and classification pipeline for inference is parallelized and profiled on the PULP platform, achieving a latency < 50 ms and an energy consumption < 1 mJ. Both the classification models tested (a support vector machine and a lightweight MLP) yielded an accuracy > 92% in a 1-subject, 5-classes (4 gestures and rest) scenario. These results prove that the proposed system is suitable for real-time execution on embedded platforms and also capable of matching the accuracy of state-of-the-art approaches, while also giving some physiological insight on the neuromuscular spikes underlying the sEMG.

Hand gesture recognition system based in computer vision and machine learning : applications on human-machine interaction

Sendo uma forma natural de interação homem-máquina, o reconhecimento de gestos implica uma forte componente de investigação em áreas como a visão por computador e a aprendizagem computacional. O reconhecimento gestual é uma área com aplicações muito diversas, fornecendo aos utilizadores uma forma mais natural e mais simples de comunicar com sistemas baseados em computador, sem a necessidade de utilização de dispositivos extras. Assim, o objectivo principal da investigação na área de reconhecimento de gestos aplicada à interacção homemmáquina é o da criação de sistemas, que possam identificar gestos específicos e usálos para transmitir informações ou para controlar dispositivos. Para isso as interfaces baseados em visão para o reconhecimento de gestos, necessitam de detectar a mão de forma rápida e robusta e de serem capazes de efetuar o reconhecimento de gestos em tempo real. Hoje em dia, os sistemas de reconhecimento de gestos baseados em visão são capazes de trabalhar com soluções específicas, construídos para resolver um determinado problema e configurados para trabalhar de uma forma particular. Este projeto de investigação estudou e implementou soluções, suficientemente genéricas, com o recurso a algoritmos de aprendizagem computacional, permitindo a sua aplicação num conjunto alargado de sistemas de interface homem-máquina, para reconhecimento de gestos em tempo real. A solução proposta, Gesture Learning Module Architecture (GeLMA), permite de forma simples definir um conjunto de comandos que pode ser baseado em gestos estáticos e dinâmicos e que pode ser facilmente integrado e configurado para ser utilizado numa série de aplicações. É um sistema de baixo custo e fácil de treinar e usar, e uma vez que é construído unicamente com bibliotecas de código. As experiências realizadas permitiram mostrar que o sistema atingiu uma precisão de 99,2% em termos de reconhecimento de gestos estáticos e uma precisão média de 93,7% em termos de reconhecimento de gestos dinâmicos. Para validar a solução proposta, foram implementados dois sistemas completos. O primeiro é um sistema em tempo real capaz de ajudar um árbitro a arbitrar um jogo de futebol robótico. A solução proposta combina um sistema de reconhecimento de gestos baseada em visão com a definição de uma linguagem formal, o CommLang Referee, à qual demos a designação de Referee Command Language Interface System (ReCLIS). O sistema identifica os comandos baseados num conjunto de gestos estáticos e dinâmicos executados pelo árbitro, sendo este posteriormente enviado para um interface de computador que transmite a respectiva informação para os robôs. O segundo é um sistema em tempo real capaz de interpretar um subconjunto da Linguagem Gestual Portuguesa. As experiências demonstraram que o sistema foi capaz de reconhecer as vogais em tempo real de forma fiável. Embora a solução implementada apenas tenha sido treinada para reconhecer as cinco vogais, o sistema é facilmente extensível para reconhecer o resto do alfabeto. As experiências também permitiram mostrar que a base dos sistemas de interação baseados em visão pode ser a mesma para todas as aplicações e, deste modo facilitar a sua implementação. A solução proposta tem ainda a vantagem de ser suficientemente genérica e uma base sólida para o desenvolvimento de sistemas baseados em reconhecimento gestual que podem ser facilmente integrados com qualquer aplicação de interface homem-máquina. A linguagem formal de definição da interface pode ser redefinida e o sistema pode ser facilmente configurado e treinado com um conjunto de gestos diferentes de forma a serem integrados na solução final.

Hand gesture recognition for human computer interaction : a comparative study of different image features

Hand gesture recognition for human computer interaction, being a natural way of human computer interaction, is an area of active research in computer vision and machine learning. This is an area with many different possible applications, giving users a simpler and more natural way to communicate with robots/systems interfaces, without the need for extra devices. So, the primary goal of gesture recognition research is to create systems, which can identify specific human gestures and use them to convey information or for device control. For that, vision-based hand gesture interfaces require fast and extremely robust hand detection, and gesture recognition in real time. In this study we try to identify hand features that, isolated, respond better in various situations in human-computer interaction. The extracted features are used to train a set of classifiers with the help of RapidMiner in order to find the best learner. A dataset with our own gesture vocabulary consisted of 10 gestures, recorded from 20 users was created for later processing. Experimental results show that the radial signature and the centroid distance are the features that when used separately obtain better results, with an accuracy of 91% and 90,1% respectively obtained with a Neural Network classifier. These to methods have also the advantage of being simple in terms of computational complexity, which make them good candidates for real-time hand gesture recognition.

Hand gesture recognition system based in computer vision and machine learning

"Lecture notes in computational vision and biomechanics series, ISSN 2212-9391, vol. 19"