Movements in gesture interfaces

There is a mismatch between the kinds of movements used in gesture interfaces and our existing theoretical understandings of gesture. We need to re-examine the assumptions of gesture research and develop theory more suited to gesture interface design. In addition to improved theory, we need to develop ways for participants in the process of design to adapt, extend and develop theory for their own design contexts. Gesture interface designers should approach theory as a contingent resource for design actions that is responsive to the needs of the design process.

Hand pose detection for vision-based gesture interfaces

[EN]Vision-based applications designed for humanmachine interaction require fast and accurate hand detection. However, previous works on this field assume different constraints, like a limitation in the number of detected gestures, because hands are highly complex objects to locate. This paper presents an approach which changes the detection target without limiting the number of detected gestures. Using a cascade classifier we detect hands based on their wrists. With this approach, we introduce two main contributions: (1) a reliable segmentation, independently of the gesture being made and (2) a training phase faster than previous cascade classifier based methods. The paper includes experimental evaluations with different video streams that illustrate the efficiency and suitability for perceptual interfaces.

Framing Movements for Gesture Interface Design

Gesture interfaces are an attractive avenue for human-computer interaction, given the range of expression that people are able to engage when gesturing. Consequently, there is a long running stream of research into gesture as a means of interaction in the field of human-computer interaction. However, most of this research has focussed on the technical challenges of detecting and responding to people’s movements, or on exploring the interaction possibilities opened up by technical developments. There has been relatively little research on how to actually design gesture interfaces, or on the kinds of understandings of gesture that might be most useful to gesture interface designers. Running parallel to research in gesture interfaces, there is a body of research into human gesture, which would seem a useful source to draw knowledge that could inform gesture interface design. However, there is a gap between the ways that ‘gesture’ is conceived of in gesture interface research compared to gesture research. In this dissertation, I explore this gap and reflect on the appropriateness of existing research into human gesturing for the needs of gesture interface design. Through a participatory design process, I designed, prototyped and evaluated a gesture interface for the work of the dental examination. Against this grounding experience, I undertook an analysis of the work of the dental examination with particular focus on the roles that gestures play in the work to compare and discuss existing gesture research. I take the work of the gesture researcher McNeill as a point of focus, because he is widely cited within gesture interface research literature. I show that although McNeill’s research into human gesture can be applied to some important aspects of the gestures of dentistry, there remain range of gestures that McNeill’s work does not deal with directly, yet which play an important role in the work and could usefully be responded to with gesture interface technologies. I discuss some other strands of gesture research, which are less widely cited within gesture interface research, but offer a broader conception of gesture that would be useful for gesture interface design. Ultimately, I argue that the gap in conceptions of gesture between gesture interface research and gesture research is an outcome of the different interests that each community brings to bear on the research. What gesture interface research requires is attention to the problems of designing gesture interfaces for authentic context of use and assessment of existing theory in light of this.

Using clustering techniques for intelligent camera-based user interfaces

The area of Human-Machine Interface is growing fast due to its high importance in all technological systems. The basic idea behind designing human-machine interfaces is to enrich the communication with the technology in a natural and easy way. Gesture interfaces are a good example of transparent interfaces. Such interfaces must identify properly the action the user wants to perform, so the proper gesture recognition is of the highest importance. However, most of the systems based on gesture recognition use complex methods requiring high-resource devices. In this work, we propose to model gestures capturing their temporal properties, which significantly reduce storage requirements, and use clustering techniques, namely self-organizing maps and unsupervised genetic algorithm, for their classification. We further propose to train a certain number of algorithms with different parameters and combine their decision using majority voting in order to decrease the false positive rate. The main advantage of the approach is its simplicity, which enables the implementation using devices with limited resources, and therefore low cost. The testing results demonstrate its high potential.

Meaning in movement : a gestural design game

This paper describes a design game that we called 'Meaning in Movement'. The purpose was to explore notions of professional dental practice with dental practioners in terms of gestures, actions and movements. The game represents a first step towards involving gestures, actions and movements in a design dialog with practioners for the purpose of designing future interactive systems which are more appropriate to the type of skilful actions and richly structured environments of dentists and dental assistants.

基于视觉的手势界面关键技术研究

针对视觉手势界面存在的问题,提出了一套行之有效的解决方案.首先,为了解决视觉手势交互中的Midas Touch问题,以人类注意的信息加工模型为理论依据提出了一个可扩展的视觉手势交互模型,该模型将手势交互过程分为选择性处理、分配性处理和集中处理3个不同阶段;然后,基于该模型提出了一个视觉手势识别框架,并结合认知心理学从手势检测、跟踪和识别3个方面对该框架的各个组成模块的关键技术进行了阐述,其中手势检测模块和识别管理模块能够辅助系统在复杂的背景中滤除掉不相关信息而选择性地搜索人手并根据上下文信息对手势识别任务重定向,从而避免了系统时刻都处于激活状态并对所有的手势动作都进行识别分析,有效解决了Midas Touch问题.文中介绍了使用该方法实现的IEToolkit手势界面工具平台,并基于一个视觉手势交互系统进行了实验测试与评估,结果验证了文中方法的可用性.

基于手势的视觉用户界面研究

随着人机交互技术的发展，各种新的交互手段不断涌现，使人机交互朝着更加自然、高效和智能化的方向前进。基于手势的视觉用户界面是post-WIMP时代的一种重要的界面形式，与传统的WIMP交互方式相比视觉手势交互能够使用户摆脱鼠标键盘的束缚而采用一种更加自然、无约束的交互方式，从而提供给用户更大的交互空间、更多的交互自由度和更逼真的交互体验，具有较高的应用价值和良好的应用前景，因此被国内外越来越多的研究者所关注，迅速成为了人机交互领域一个热门的研究方向，并被广泛应用于虚拟/增强现实、普适计算、智能空间以及基于计算机的互动游戏等多个领域。 视觉交互是自然人机交互的核心和热点研究内容之一，视觉交互可以通过手势、目光、头部运动或者面部表情等多种方式进行。其中，手势是人类进行视觉交互的主要手段，手势所能表达的语义信息十分丰富。在人机交互中使用视觉手势完成交互任务不仅自然、直观和方便，而且从计算机的角度出发来看系统实现起来也较为容易。因此，基于手势的视觉用户界面得到了广泛的关注并取得了许多研究成果。从目前的研究现状来看，在基于手势的视觉用户界面研究中，仍然存在着以下的问题： （1）传统的WIMP界面模型已经不适合描述视觉手势界面这种post-WIMP界面的特点，需要对传统的用户界面模型加以扩展，构造适于描述视觉交互特征的界面模型帮助用户清晰、准确地分析和表达界面功能及其变化，描述出用户与系统的交互过程，指导软件系统的设计和实现； （2）具体的交互设计过程中还存在着许多关键技术没有解决好，例如手势的正确理解问题、算法的鲁棒性问题以及手势的可扩展性问题等； （3）传统视觉工具箱复杂难用的问题。需要降低开发难度，开发出便捷、易用、可扩展的软件界面开发工具及相应的开发方法，支持领域内非专家用户快速开发出视觉手势原型系统。 本文正是从以上问题出发，围绕着视觉手势界面交互技术，从理论、方法与应用等几个方面展开了深入的研究。首先论述了用户界面的发展历程，继而对视觉界面研究现状进行了综述。在充分研究和对比国内外相关研究成果的基础上提出了一种基于手势的视觉用户界面模型UIDT。接下来针对视觉手势交互中存在的难题，以认知心理学为理论依据提出了一种可扩展的手势交互状态转移模型，并在此基础上构建了视觉手势处理框架。随后对该框架中的关键技术进行了针对性的研究。针对非专家用户在构建具有个性化视觉手势界面过程中所遇到的问题，设计开发了一个支持视觉手势交互的开发工具IEToolkit并给出了一套基于该工具的通用的软件开发方法。最后，将上述成果应用于互动娱乐领域，取得了满意的效果。 本文的创新点主要表现在以下几个方面： 1. 提出了一种基于手势的视觉用户界面模型UIDT 在充分分析视觉交互特征的基础上，以传统的用户界面模型为基础提出了一种基于手势交互的视觉界面模型UIDT。该模型从用户模型、任务模型、设备模型和交互模型等几个方面对视觉手势交互进行了深入分析和描述，给出了各个模型的形式化定义，介绍了模型的各个组成模块以及它们之间的相互关系，讨论了视觉手势界面设计中应该遵循的设计规范。评估结果表明，该模型具有较强的通用性，有助于设计者对VBI的任务、用户、设备以及交互的不同层次进行抽象描述，使用户界面满足可用性要求，提高界面设计和原型开发的效率。 2. 提出了一个可扩展的视觉手势交互模型及一种新的视觉手势识别处理框架，围绕着该框架提出了一种新的视觉手势跟踪和识别方法 首先，提出了一个可扩展的视觉手势交互模型。根据认知心理学原理将视觉手势交互处理过程细分为选择性处理、分配性处理和集中处理三个不同的阶段，有效解决了Midas Touch问题；基于该模型提出了一个视觉手势识别框架，并结合认知心理学从手势检测、跟踪和识别三个方面对该框架的各个组成模块的关键技术进行了阐述。其中手势检测模块和识别管理模块能够辅助系统在复杂的背景中滤除掉不相关信息而选择性地搜索人手并根据上下文信息对手势识别任务重定向，从而避免了系统时刻都处于激活状态并对所有的手势动作都进行识别分析，有效解决了Midas Touch问题；为了提高系统的性能，提出了一种鲁棒的面向实时交互的变形手势跟踪方法和基于小样本学习的模板匹配方法用于动态手势识别。评估结果表明，上述技术有效地提高了系统的实时性、准确性和鲁棒性。 3. 设计开发了一个支持快速原型开发的视觉手势工具箱系统 针对目前大多数视觉手势工具箱复杂难用的问题，设计开发了一个简单、易用、可扩展的手势工具箱系统IEToolkit，实现了本文所述的各种关键交互技术。它包含了构造一个基于视觉手势的交互系统所需要的方方面面，从事件模型、交互模型、数据流模型等几个方面对工具箱的组成结构进行了描述，并给出了一个基于IEToolkit的通用的软件系统开发流程，开发人员可以将更多的精力集中在具体的高层逻辑语义处理上，而不需要过多考虑底层的技术细节与支撑结构。应用实例及评估结果表明IEToolkit能够降低开发门槛，较好地支持基于视觉手势交互的应用系统的快速构造，具有较高的应用价值。 4. 基于上述研究成果设计开发了一系列典型的视觉手势交互系统 将上述理论模型与具体的交互技术进行有效结合，在视觉手势开发工具平台基础上开发了一系列典型的互动娱乐系统，在实践中对本文的研究成果进行了有效性验证。

New generation of human machine interfaces for controlling UAV through depth based gesture recognition

New forms of natural interactions between human operators and UAVs (Unmanned Aerial Vehicle) are demanded by the military industry to achieve a better balance of the UAV control and the burden of the human operator. In this work, a human machine interface (HMI) based on a novel gesture recognition system using depth imagery is proposed for the control of UAVs. Hand gesture recognition based on depth imagery is a promising approach for HMIs because it is more intuitive, natural, and non-intrusive than other alternatives using complex controllers. The proposed system is based on a Support Vector Machine (SVM) classifier that uses spatio-temporal depth descriptors as input features. The designed descriptor is based on a variation of the Local Binary Pattern (LBP) technique to efficiently work with depth video sequences. Other major consideration is the especial hand sign language used for the UAV control. A tradeoff between the use of natural hand signs and the minimization of the inter-sign interference has been established. Promising results have been achieved in a depth based database of hand gestures especially developed for the validation of the proposed system.

Work at hand : an exploration of gesture in the context of work and everyday life to inform the design of gestural input devices

This research explores gestures used in the context of activities in the workplace and in everyday life in order to understand requirements and devise concepts for the design of gestural information applicances. A collaborative method of video interaction analysis devised to suit design explorations, the Video Card Game, was used to capture and analyse how gesture is used in the context of six different domains: the dentist's office; PDA and mobile phone use; the experimental biologist's laboratory; a city ferry service; a video cassette player repair shop; and a factory flowmeter assembly station. Findings are presented in the form of gestural themes, derived from the tradition of qualitative analysis but bearing some similarity to Alexandrian patterns. Implications for the design of gestural devices are discussed.

An intuitive multi-touch surface and gesture based interaction for video surveillance systems

This paper discusses the idea and demonstrates an early prototype of a novel method of interacting with security surveillance footage using natural user interfaces in place of traditional mouse and keyboard interaction. Current surveillance monitoring stations and systems provide the user with a vast array of video feeds from multiple locations on a video wall, relying on the user’s ability to distinguish locations of the live feeds from experience or list based key-value pair of location and camera IDs. During an incident, this current method of interaction may cause the user to spend increased amounts time obtaining situational and location awareness, which is counter-productive. The system proposed in this paper demonstrates how a multi-touch screen and natural interaction can enable the surveillance monitoring station users to quickly identify the location of a security camera and efficiently respond to an incident.

Social NUI: Social perspectives in natural user interfaces

Natural User Interfaces (NUI) offer rich ways for interacting with the digital world that make innovative use of existing human capabilities. They include and often combine different input modalities such as voice, gesture, eye gaze, body interactions, touch and touchless interactions. However much of the focus of NUI research and development has been on enhancing the experience of individuals interacting with technology. Effective NUIs must also acknowledge our innately social characteristics, and support how we communicate with each other, play together, learn together and collaboratively work together. This workshop concerns the social aspects of NUI. The workshop seeks to better understand the social uses and applications of these new NUI technologies -- how we design these technologies for new social practices and how we understand the use of these technologies in key social contexts.

A software development process model for gesture-based interface

Gesture-based applications have particularities, since users interact in a natural way, much as they interact in the non-digital world. Hence, new requirements are needed on the software design process. This paper shows a software development process model for these applications, including requirement specification, design, implementation, and testing procedures. The steps and activities of the proposed model were tested through a game case study, which is a puzzle game. The puzzle is completed when all pieces of a painting are correctly positioned by the drag and drop action of users hand gesture. It also shows the results obtained of applying a heuristic evaluation on this game. © 2012 IEEE.

Gesture-based teleoperation using a holonomic robot

This work presents the development and integration of an user interface (UI) framework based on various current input devices that take advantage of our ergonomics. The purpose is to teleoperate a holonomic robot using upper member gestures and postures for studying the suitable of such interfaces when programming and interacting with a mobile robot. As performance vary from UI to UI the framework is focused to be used as a complementary industrial or didactic tool thus, changing how inexperience users tackle their first impressions when working with mobile robots while performing simple gesture-based teleoperation tasks. © 2012 ICROS.

Deep learning architecture for gesture recognition

Pós-graduação em Engenharia Mecânica - FEG