Accurate and Efficient Gesture Spotting via Pruning and Subgesture Reasoning

Gesture spotting is the challenging task of locating the start and end frames of the video stream that correspond to a gesture of interest, while at the same time rejecting non-gesture motion patterns. This paper proposes a new gesture spotting and recognition algorithm that is based on the continuous dynamic programming (CDP) algorithm, and runs in real-time. To make gesture spotting efficient a pruning method is proposed that allows the system to evaluate a relatively small number of hypotheses compared to CDP. Pruning is implemented by a set of model-dependent classifiers, that are learned from training examples. To make gesture spotting more accurate a subgesture reasoning process is proposed that models the fact that some gesture models can falsely match parts of other longer gestures. In our experiments, the proposed method with pruning and subgesture modeling is an order of magnitude faster and 18% more accurate compared to the original CDP algorithm.

Spatiotemporal Gesture Segmentation

Spotting patterns of interest in an input signal is a very useful task in many different fields including medicine, bioinformatics, economics, speech recognition and computer vision. Example instances of this problem include spotting an object of interest in an image (e.g., a tumor), a pattern of interest in a time-varying signal (e.g., audio analysis), or an object of interest moving in a specific way (e.g., a human's body gesture). Traditional spotting methods, which are based on Dynamic Time Warping or hidden Markov models, use some variant of dynamic programming to register the pattern and the input while accounting for temporal variation between them. At the same time, those methods often suffer from several shortcomings: they may give meaningless solutions when input observations are unreliable or ambiguous, they require a high complexity search across the whole input signal, and they may give incorrect solutions if some patterns appear as smaller parts within other patterns. In this thesis, we develop a framework that addresses these three problems, and evaluate the framework's performance in spotting and recognizing hand gestures in video. The first contribution is a spatiotemporal matching algorithm that extends the dynamic programming formulation to accommodate multiple candidate hand detections in every video frame. The algorithm finds the best alignment between the gesture model and the input, and simultaneously locates the best candidate hand detection in every frame. This allows for a gesture to be recognized even when the hand location is highly ambiguous. The second contribution is a pruning method that uses model-specific classifiers to reject dynamic programming hypotheses with a poor match between the input and model. Pruning improves the efficiency of the spatiotemporal matching algorithm, and in some cases may improve the recognition accuracy. The pruning classifiers are learned from training data, and cross-validation is used to reduce the chance of overpruning. The third contribution is a subgesture reasoning process that models the fact that some gesture models can falsely match parts of other, longer gestures. By integrating subgesture reasoning the spotting algorithm can avoid the premature detection of a subgesture when the longer gesture is actually being performed. Subgesture relations between pairs of gestures are automatically learned from training data. The performance of the approach is evaluated on two challenging video datasets: hand-signed digits gestured by users wearing short sleeved shirts, in front of a cluttered background, and American Sign Language (ASL) utterances gestured by ASL native signers. The experiments demonstrate that the proposed method is more accurate and efficient than competing approaches. The proposed approach can be generally applied to alignment or search problems with multiple input observations, that use dynamic programming to find a solution.

Kawamoto’s "Dojoji Temple": Puppetry, Gesture and Myth

Dojoji Temple ( Dōjōji, 1976) is a short puppet animation directed by Kihachirō Kawamoto. Influenced by Bunraku (Japanese puppet plays), emaki (painted scroll), Noh theatre and Japanese myth, Dojoji Temple tells of a woman’s unrequited love for a young priest. Heartbroken, she then transforms into a sea serpent and goes after the priest for revenge. While Kawamoto’s animation is rich with Japanese aesthetics and tragedy, his animation is peopled by puppets who do not speak. Limited and restrained though the puppets may be, their animated gestures speak volumes of powerful emotions. For our article, we will select several scenes from the animation, and interpret their actions so that we can further understand the mythical world of Dojoji Temple and the essential being of puppetry. Our gesture analysis will take into account cinematographic compositions, sound and bodily attires, among other elements.

Categorizations of physical gesture in piano teaching: A preliminary enquiry

The significance of the “physicality” involved in learning to play a musical instrument and the essential role of teachers are areas in need of research. This article explores the role of gesture within teacher–student communicative interaction in one-to-one piano lessons. Three teachers were required to teach a pre-selected repertoire of two contrasting pieces to three students studying piano grade 1. The data was collected by video recordings of piano lessons and analysis based on the type and frequency of gestures employed by teachers in association to teaching behaviours specifying where gestures fit under (or evade) predefined classifications. Spontaneous co-musical gestures were observed in the process of piano tuition emerging with similar general communicative purposes as spontaneous co-verbal gestures and were essential for the process of musical communication between teachers and students. Observed frequencies of categorized gestures varied significantly between different teaching behaviours and between the three teachers. Parallels established between co-verbal and co-musical spontaneous gestures lead to an argument for extension of McNeill’s (2005) ideas of imagery–language–dialectic to imagery–music–dialectic with relevant implications for piano pedagogy and fields of study invested in musical communication.

Communicating musical knowledge through gesture: Piano teachers’ gestural behaviours across different levels of student proficiency

Teachers’ communication of musical knowledge through physical gesture represents a valuable pedagogical field in need of investigation. This exploratory case study compares the gestural behaviour of three piano teachers while giving individual lessons to students who differed according to piano proficiency levels. The data was collected by video recordings of one-to-one piano lessons and gestures were categorized using two gesture classifications: the spontaneous co-verbal gesture classification (McNeill, 1992; 2005) and spontaneous co-musical gesture classification (Simones, Schroeder & Rodger, 2013). Poisson regression analysis and qualitative observation suggest a relationship between teachers’ didactic intentions and the types of gesture they produced while teaching, as shown by differences in gestural category frequency between teaching students of higher and lower levels of proficiency. Such reported agreement between teachers’ gestural approach in relation to student proficiency levels indicates a teachers’ gestural scaffolding approach whereby teachers adapted gestural communicative channels to suit students’ specific conceptual skill levels.

Gesture interaction in music for instruments and electroacoustic sounds

Esta tese apresenta alguns aspectos em como o fenómeno do gesto musical pode ser compreendido na percepção da interação musical na música para instrumentos e sons electroacústicos. Através de exemplos de análise, classificação e categorização de diferentes relacões gestuais entre instrumentos e sons electroacústicos, pretende-se estabelecer modelos específicos de interacção que podem ser aplicados como método analítico assim como na composição musical. A pesquisa parte de uma variedade de definições sobre gesto musical na música em geral, na música contemporânea e na música electroacústica em particular, para subsequentemente incluir as relações entre dois eventos sonoros com características diferentes - o electroacústico e o instrumental. São essencialmente abordadas as relações entre gestos musicais através da análise de diversas características: altura, ritmo, timbre, dinâmica, características contrapontísticas, espectromorfológicas, semânticas e espaciais. O resultado da pesquisa teórica serviu de suporte à composição de diversas obras, onde estes aspectos são explorados sob o ponto de vista da criação musical.

Multi-scale cortical keypoints for realtime hand tracking and gesture recognition

Human-robot interaction is an interdisciplinary research area which aims at integrating human factors, cognitive psychology and robot technology. The ultimate goal is the development of social robots. These robots are expected to work in human environments, and to understand behavior of persons through gestures and body movements. In this paper we present a biological and realtime framework for detecting and tracking hands. This framework is based on keypoints extracted from cortical V1 end-stopped cells. Detected keypoints and the cells’ responses are used to classify the junction type. By combining annotated keypoints in a hierarchical, multi-scale tree structure, moving and deformable hands can be segregated, their movements can be obtained, and they can be tracked over time. By using hand templates with keypoints at only two scales, a hand’s gestures can be recognized.

Natural hand-gesture interaction (Interacción natural con gestos manuales).

Tesis (Doctor en Ingeniería con Acentuación en Computación y Mecatrónica) UANL, 2013.

Maternal label and gesture use affects acquisition of specific object names

Ten mothers were observed prospectively, interacting with their infants aged 0 ; 10 in two contexts (picture description and noun description). Maternal communicative behaviours were coded for volubility, gestural production and labelling style. Verbal labelling events were categorized into three exclusive categories: label only; label plus deictic gesture; label plus iconic gesture. We evaluated the predictive relations between maternal communicative style and children's subsequent acquisition of ten target nouns. Strong relations were observed between maternal communicative style and children's acquisition of the target nouns. Further, even controlling for maternal volubility and maternal labelling, maternal use of iconic gestures predicted the timing of acquisition of nouns in comprehension. These results support the proposition that maternal gestural input facilitates linguistic development, and suggest that such facilitation may be a function of gesture type.