Deep learning architecture for gesture recognition

Pós-graduação em Engenharia Mecânica - FEG

Eyewear selector

[EN]The widespread availability of portable computing power and inexpensive digital cameras is opening up new possibilities for retailers. One example is in optical shops, where a number of systems exist that facilitate eyeglasses selection. These systems are now more necessary as the market is saturated with an increasingly complex array of lenses, frames, coatings, tints, photochromic and polarizing treatments, etc. Research challenges encompass Computer Vision, Multimedia and Human-Computer Interaction. Cost factors are also of importance for widespread product acceptance. This paper describes a low-cost system that allows the user to visualize di erent spectacle models in live video. The user can also move the spectacles to adjust its position on the face. Experiments show the potential of the system.

Constructing a user experience user experience-based mobile learning environment

Mobile learning, in the past defined as learning with mobile devices, now refers to any type of learning-on-the-go or learning that takes advantage of mobile technologies. This new definition shifted its focus from the mobility of technology to the mobility of the learner (O'Malley and Stanton 2002; Sharples, Arnedillo-Sanchez et al. 2009). Placing emphasis on the mobile learner’s perspective requires studying “how the mobility of learners augmented by personal and public technology can contribute to the process of gaining new knowledge, skills, and experience” (Sharples, Arnedillo-Sanchez et al. 2009). The demands of an increasingly knowledge based society and the advances in mobile phone technology are combining to spur the growth of mobile learning. Around the world, mobile learning is predicted to be the future of online learning, and is slowly entering the mainstream education. However, for mobile learning to attain its full potential, it is essential to develop more advanced technologies that are tailored to the needs of this new learning environment. A research field that allows putting the development of such technologies onto a solid basis is user experience design, which addresses how to improve usability and therefore user acceptance of a system. Although there is no consensus definition of user experience, simply stated it focuses on how a person feels about using a product, system or service. It is generally agreed that user experience adds subjective attributes and social aspects to a space that has previously concerned itself mainly with ease-of-use. In addition, it can include users’ perceptions of usability and system efficiency. Recent advances in mobile and ubiquitous computing technologies further underline the importance of human-computer interaction and user experience (feelings, motivations, and values) with a system. Today, there are plenty of reports on the limitations of mobile technologies for learning (e.g., small screen size, slow connection), but there is a lack of research on user experience with mobile technologies. This dissertation will fill in this gap by a new approach in building a user experience-based mobile learning environment. The optimized user experience we suggest integrates three priorities, namely a) content, by improving the quality of delivered learning materials, b) the teaching and learning process, by enabling live and synchronous learning, and c) the learners themselves, by enabling a timely detection of their emotional state during mobile learning. In detail, the contributions of this thesis are as follows: • A video codec optimized for screencast videos which achieves an unprecedented compression rate while maintaining a very high video quality, and a novel UI layout for video lectures, which together enable truly mobile access to live lectures. • A new approach in HTTP-based multimedia delivery that exploits the characteristics of live lectures in a mobile context and enables a significantly improved user experience for mobile live lectures. • A non-invasive affective learning model based on multi-modal emotion detection with very high recognition rates, which enables real-time emotion detection and subsequent adaption of the learning environment on mobile devices. The technology resulting from the research presented in this thesis is in daily use at the School of Continuing Education of Shanghai Jiaotong University (SOCE), a blended-learning institution with 35.000 students.

April 4, 1968: Death, Difference, and Dialogue

Robert Kennedy's announcement of the assassination of Martin Luther King, Jr., in an Indianapolis urban community that did not revolt in riots on April 4, 1968, provides one significant example in which feelings, energy, and bodily risk resonate alongside the articulated message. The relentless focus on Kennedy's spoken words, in historical biographies and other critical research, presents a problem of isolated effect because the power really comes from elements outside the speech act. Thus, this project embraces the complexities of rhetorical effectivity, which involves such things as the unique situational context, all participants (both Kennedy and his audience) of the speech act, aesthetic argument, and the ethical implications. This version of the story embraces the many voices of the participants through first hand interviews and new oral history reports. Using evidence provided from actual participants in the 1968 Indianapolis event, this project reflects critically upon the world disclosure of the event as it emerges from those remembrances. Phenomenology provides one answer to the constitutive dilemma of rhetorical effectivity that stems from a lack of a framework that gets at questions of ethics, aesthetics, feelings, energy, etc. Thus, this work takes a pedagogical shift away from discourse (verbal/written) as the primary place to render judgments about the effects of communication interaction. With a turn to explore extra-sensory reasoning, by way of the physical, emotional, and numinous, a multi-dimensional look at public address is delivered. The rhetorician will be interested in new ways of assessing effects. The communication ethicist will appreciate the work as concepts like answerability, emotional-volitional tone, and care for the other, come to life via application and consideration of Kennedy's appearance. For argumentation scholars, the interest comes forth in a re-thinking of how we do argumentation. And the critical cultural scholar will find this story ripe with opportunities to uncover the politics of representation, racialized discourse, privilege, power, ideological hegemony, and reconciliation. Through an approach of multiple layers this real-life tale will expose the power of the presence among audience and speaker, emotive argument, as well as the magical turn of fate which all contributes the possibility of a dialogic rhetoric.

Facilitating communication in children with multiple disabilities: three case studies of girls with Rett syndrome

Purpose: To investigate the effect of cueing on communicative responses of children with multiple disabilities in an educational setting. It was hypothesized that differences would exist in teacher interactional styles and the use of orienting cues would increase the communicative responses of the participants. Method: A naturalistic observation research method was employed in order to examine the interaction of three student-teacher dyads in three special schools. Three different activity types were videotaped from which interactions were coded and analysed. Results: Multi-modal cueing facilitated communicative responses of children with Rett syndrome. However, increased communication opportunities provided by caregivers did not elicit increased responses from the girls. Conclusion: There is a difference in cueing by teachers in their interactions with children with multiple disabilities. Also, more frequent communicative interactions did not necessarily lead to increased student responses. It is suggested that amount and type of cueing may need to be considered to be effective in generating student responses. The small number of participants, however, means findings should be viewed cautiously and that more research is indicated.

3D navigation with six degrees-of-freedom using a multi-touch display

With the introduction of new input devices, such as multi-touch surface displays, the Nintendo WiiMote, the Microsoft Kinect, and the Leap Motion sensor, among others, the field of Human-Computer Interaction (HCI) finds itself at an important crossroads that requires solving new challenges. Given the amount of three-dimensional (3D) data available today, 3D navigation plays an important role in 3D User Interfaces (3DUI). This dissertation deals with multi-touch, 3D navigation, and how users can explore 3D virtual worlds using a multi-touch, non-stereo, desktop display. ^ The contributions of this dissertation include a feature-extraction algorithm for multi-touch displays (FETOUCH), a multi-touch and gyroscope interaction technique (GyroTouch), a theoretical model for multi-touch interaction using high-level Petri Nets (PeNTa), an algorithm to resolve ambiguities in the multi-touch gesture classification process (Yield), a proposed technique for navigational experiments (FaNS), a proposed gesture (Hold-and-Roll), and an experiment prototype for 3D navigation (3DNav). The verification experiment for 3DNav was conducted with 30 human-subjects of both genders. The experiment used the 3DNav prototype to present a pseudo-universe, where each user was required to find five objects using the multi-touch display and five objects using a game controller (GamePad). For the multi-touch display, 3DNav used a commercial library called GestureWorks in conjunction with Yield to resolve the ambiguity posed by the multiplicity of gestures reported by the initial classification. The experiment compared both devices. The task completion time with multi-touch was slightly shorter, but the difference was not statistically significant. The design of experiment also included an equation that determined the level of video game console expertise of the subjects, which was used to break down users into two groups: casual users and experienced users. The study found that experienced gamers performed significantly faster with the GamePad than casual users. When looking at the groups separately, casual gamers performed significantly better using the multi-touch display, compared to the GamePad. Additional results are found in this dissertation.^

3D Navigation with Six Degrees-of-Freedom using a Multi-Touch Display

With the introduction of new input devices, such as multi-touch surface displays, the Nintendo WiiMote, the Microsoft Kinect, and the Leap Motion sensor, among others, the field of Human-Computer Interaction (HCI) finds itself at an important crossroads that requires solving new challenges. Given the amount of three-dimensional (3D) data available today, 3D navigation plays an important role in 3D User Interfaces (3DUI). This dissertation deals with multi-touch, 3D navigation, and how users can explore 3D virtual worlds using a multi-touch, non-stereo, desktop display. The contributions of this dissertation include a feature-extraction algorithm for multi-touch displays (FETOUCH), a multi-touch and gyroscope interaction technique (GyroTouch), a theoretical model for multi-touch interaction using high-level Petri Nets (PeNTa), an algorithm to resolve ambiguities in the multi-touch gesture classification process (Yield), a proposed technique for navigational experiments (FaNS), a proposed gesture (Hold-and-Roll), and an experiment prototype for 3D navigation (3DNav). The verification experiment for 3DNav was conducted with 30 human-subjects of both genders. The experiment used the 3DNav prototype to present a pseudo-universe, where each user was required to find five objects using the multi-touch display and five objects using a game controller (GamePad). For the multi-touch display, 3DNav used a commercial library called GestureWorks in conjunction with Yield to resolve the ambiguity posed by the multiplicity of gestures reported by the initial classification. The experiment compared both devices. The task completion time with multi-touch was slightly shorter, but the difference was not statistically significant. The design of experiment also included an equation that determined the level of video game console expertise of the subjects, which was used to break down users into two groups: casual users and experienced users. The study found that experienced gamers performed significantly faster with the GamePad than casual users. When looking at the groups separately, casual gamers performed significantly better using the multi-touch display, compared to the GamePad. Additional results are found in this dissertation.

Deep Head Pose: Gaze-Direction Estimation in Multimodal Video

In this paper we present a convolutional neuralnetwork (CNN)-based model for human head pose estimation inlow-resolution multi-modal RGB-D data. We pose the problemas one of classification of human gazing direction. We furtherfine-tune a regressor based on the learned deep classifier. Next wecombine the two models (classification and regression) to estimateapproximate regression confidence. We present state-of-the-artresults in datasets that span the range of high-resolution humanrobot interaction (close up faces plus depth information) data tochallenging low resolution outdoor surveillance data. We buildupon our robust head-pose estimation and further introduce anew visual attention model to recover interaction with theenvironment. Using this probabilistic model, we show thatmany higher level scene understanding like human-human/sceneinteraction detection can be achieved. Our solution runs inreal-time on commercial hardware

Taste-odour interaction as a strategy to reduce sugar in yogurt.

2014

2D-3D Face Recognition Based on PCA and Feature Modelling

Hybrid face recognition, using image (2D) and structural (3D) information, has explored the fusion of Nearest Neighbour classifiers. This paper examines the effectiveness of feature modelling for each individual modality, 2D and 3D. Furthermore, it is demonstrated that the fusion of feature modelling techniques for the 2D and 3D modalities yields performance improvements over the individual classifiers. By fusing the feature modelling classifiers for each modality with equal weights the average Equal Error Rate improves from 12.60% for the 2D classifier and 12.10% for the 3D classifier to 7.38% for the Hybrid 2D+3D clasiffier.

Dancing between diversity and consistency : refining assessment in postgraduate degrees in dance

This publication is the culmination of a 2 year Australian Learning and Teaching Council's Project Priority Programs Research Grant which investigates key issues and challenges in developing flexible guidelines lines for best practice in Australian Doctoral and Masters by Research Examination, encompassing the two modes of investigation, written and multi-modal (practice-led/based) theses, their distinctiveness and their potential interplay. The aims of the project were to address issues of assessment legitimacy raised by the entry of practice-orientated dance studies into Australian higher degrees; examine literal embodiment and presence, as opposed to cultural studies about states of embodiment; foreground the validity of questions around subjectivity and corporeal intelligence/s and the reliability of artistic/aesthetic communications, and finally to celebrate ‘performance mastery’(Melrose 2003) as a rigorous and legitimate mode of higher research. The project began with questions which centred around: the functions of higher degree dance research; concepts of 'master-ness’ and ‘doctorateness’; the kinds of languages, structures and processes which may guide candidates, supervisors, examiners and research personnel; the purpose of evaluation/examination; addressing positive and negative attributes of examination. Finally the study examined ways in which academic/professional, writing/dancing, tradition/creation and diversity/consistency relationships might be fostered to embrace change. Over two years, the authors undertook a qualitative national study encompassing a triangulation of semi-structured face to face interviews and industry forums to gather views from the profession, together with an analysis of existing guidelines, and recent literature in the field. The most significant primary data emerged from 74 qualitative interviews with supervisors, examiners, research deans and administrators, and candidates in dance and more broadly across the creative arts. Qualitative data gathered from the two primary sources, was coded and analysed using the NVivo software program. Further perspectives were drawn from international consultant and dance researcher Susan Melrose, as well as publications in the field, and initial feedback from a draft document circulated at the World Dance Alliance Global Summit in July 2008 in Brisbane. Refinement of data occurred in a continual sifting process until the final publication was produced. This process resulted in a set of guidelines in the form of a complex dynamic system for both product and process oriented outcomes of multi-modal theses, along with short position papers on issues which arose from the research such as contested definitions, embodiment and ephemerality, ‘liveness’ in performance research higher degrees, dissolving theory/practice binaries, the relationship between academe and industry, documenting practices and a re-consideration of the viva voce.

Dynamic Lock Synchronisation for collaborative 3D applications

In a typical collaborative application, users contends for common resources by mutual exclusion. The introduction of multi-modal environment, however, introduced problems such as frequent dropping of connection or limited connectivity speed of mobile users. This paper target 3D resources which require additional considerations such as dependency of users' manipulation command. This paper introduces Dynamic Locking Synchronisation technique to enable seamless and collaborative environment for large number of user, by combining the contention-free concepts of locking mechanism and the seamless nature of lockless design.

Beyond the intercultural to the accented body : an Australian perspective

'Beyond the intercultural to the Accented Body’ foregrounds contemporary choreography as a multi-modal practice which is increasingly interdisciplinary and engages with interactive technologies. These concepts are explored in the context of intercultural dance and performance practices particularly in relation to issues of identity, hybridity, the diaspora and transformation. Four models of intercultural choreography are proposed: in-country immersion; collaborative international exchanges through sharing of culturally diverse practices; hybrid practices of diasporic artists; and implicit intercultural connections. The latter model is investigated via a case study of an interactive, multi-site and interdisciplinary collaboration Accented Body.

Choreographing research : supervising the dancing thesis

The dancing doctorate is an interrogative endeavour which can but nurture the art form and forge a beneficial dynamism between those who seek and those who assess the emerging knowledges of dance’. (Vincs, 2009) From 2006-2008 three dance academics from Perth, Brisbane and Melbourne undertook a research project entitled Dancing between Diversity and Consistency: Refining Assessment in Postgraduate Degrees in Dance, funded by the ALTC Priority Projects Program. Although assessment rather than supervision was the primary focus of this research, interviews with 40 examiner/supervisors, 7 research deans and 32 candidates across Australia and across the creative arts, primarily in dance, provide an insight into what might be considered best practice in preparing students for higher research degrees, and the challenges that embodied and experiential knowledges present for supervision. The study also gained the industry perspectives of dance professionals in a series of national forums in 5 cities, based around the value of higher degrees in dance. The qualitative data gathered from these two primary sources was coded and analysed using the NVivo system. Further perspectives were drawn from international consultant and dance researcher Susan Melrose, as well as recent publications in the field. Dance is a young addition to academia and consequently there tends to be a close liaison between the academy and the industry, with a relational fluidity that is both beneficial and problematic. This partially explains why dance research higher degrees are predominantly practice-led (or multi-modal, referring to those theses where practice comprises the substantial examinable component). As a physical, embodied art form, dance engages with the contested territory of legitimising alternative forms of knowledge that do not sit comfortably with accepted norms of research. In supporting research students engaged with dance practice, supervisors traverse the tricky terrain of balancing university academic requirements with studies that are emergent, not only in the practice and attendant theory but in their methodologies and open-ended outcomes; and in an art form in which originality and new knowledge also arises from collaborative creative processes. Formal supervisor accreditation through training is now mandatory in most Australian universities, but it tends to be generic and not address supervisory specificity. This paper offers the kind of alternative proposed by Edwards (2002) that improving postgraduate supervision will be effective if supervisors are empowered to generate their own standards and share best practice; in this case, in ways appropriate to the needs of their discipline and alternative modes of thesis presentation. In order to frame the qualities and processes conducive to this goal, this paper will draw on both the experiences of interviewees and on philosophical premises which underpin the research findings of our study. These include the ongoing challenge of dissolving the binary oppositions of theory and practice, especially in creative arts practice where theory resides in and emerges from the doing as much as in articulating reflection about the doing through what Melrose (2003) terms ‘mixed mode disciplinary practices’. In guiding practitioners through research higher degrees, how do supervisors deal with not only different forms of knowledge but indeed differing modes of knowledge? How can they navigate tensions that occur between the ‘incompatible competencies’ (Candlin, 2000) of the ‘spectating’ academic experts with their ‘irrepressible drive ... to inscribe, interpret, and hence to practise temporal closure’, and practitioner experts who create emergent works of ‘residual unfinishedness’ (Melrose 2006) which are not only embodied but ephemeral, as in the case of live performance?

Dancing the Thesis : Potentials and Pitfalls in Practice-led Research

Practice-led or multi modal theses (describing examinable outcomes of postgraduate study which comprise the practice of dancing/choreography with an accompanying exegesis) are an emerging strength of dance scholarship; a form of enquiry that has been gaining momentum for over a decade, particularly in Australia and the United Kingdom. It has been strongly argued that, in this form of research, legitimate claims to new knowledge are embodied predominantly within the practice itself (Pakes, 2003) and that these findings are emergent, contingent and often interstitial, contained within both the material form of the practice and in the symbolic languages surrounding the form. In a recent study on ‘dancing’ theses Phillips, Stock, Vincs (2009) found that there was general agreement from academics and artists that ‘there could be more flexibility in matching written language with conceptual thought expressed in practice’. The authors discuss how the seemingly intangible nature of danced / embodied research, reliant on what Melrose (2003) terms ‘performance mastery’ by the ‘expert practitioner’ (2006, Point 4) involving ‘expert’ intuition (2006, Point 5), might be accessed, articulated and validated in terms of alternative ways of knowing through exploring an ongoing dialogue in which the danced practice develops emergent theory. They also propose ways in which the danced thesis can be ‘converted’ into the required ‘durable’ artefact which the ephemerality of live performance denies, drawing on the work of Rye’s ‘multi-view’ digital record (2003) and Stapleton’s ‘multi-voiced audio visual document’(2006, 82). Building on a two-year research project (2007-2008) Dancing Between Diversity and Consistency: Refining Assessment in Postgraduate Degrees in Dance, which examined such issues in relation to assessment in an Australian context, the three researchers have further explored issues around interdisciplinarity, cultural differences and documentation through engaging with the following questions:  How do we represent research in which understandings, meanings and findings are situated within the body of the dancer/choreographer?  Do these need a form of ‘translating’ into textual form in order to be accessed as research?  What kind of language structures can be developed to effect this translation: metaphor, allusion, symbol?  How important is contextualising the creative practice?  How do we incorporate differing cultural inflections and practices into our reading and evaluation?  What kind of layered documentation can assist in producing a ‘durable’ research artefact from a non-reproduce-able live event?