A multimodal and adaptive real-time human -computer interface for peoples with severe motor disability

This dissertation introduces the design of a multimodal, adaptive real-time assistive system as an alternate human computer interface that can be used by individuals with severe motor disabilities. The proposed design is based on the integration of a remote eye-gaze tracking system, voice recognition software, and a virtual keyboard. The methodology relies on a user profile that customizes eye gaze tracking using neural networks. The user profiling feature facilitates the notion of universal access to computing resources for a wide range of applications such as web browsing, email, word processing and editing. ^ The study is significant in terms of the integration of key algorithms to yield an adaptable and multimodal interface. The contributions of this dissertation stem from the following accomplishments: (a) establishment of the data transport mechanism between the eye-gaze system and the host computer yielding to a significantly low failure rate of 0.9%; (b) accurate translation of eye data into cursor movement through congregate steps which conclude with calibrated cursor coordinates using an improved conversion function; resulting in an average reduction of 70% of the disparity between the point of gaze and the actual position of the mouse cursor, compared with initial findings; (c) use of both a moving average and a trained neural network in order to minimize the jitter of the mouse cursor, which yield an average jittering reduction of 35%; (d) introduction of a new mathematical methodology to measure the degree of jittering of the mouse trajectory; (e) embedding an onscreen keyboard to facilitate text entry, and a graphical interface that is used to generate user profiles for system adaptability. ^ The adaptability nature of the interface is achieved through the establishment of user profiles, which may contain the jittering and voice characteristics of a particular user as well as a customized list of the most commonly used words ordered according to the user's preferences: in alphabetical or statistical order. This allows the system to successfully provide the capability of interacting with a computer. Every time any of the sub-system is retrained, the accuracy of the interface response improves even more. ^

Computer interaction : its effect on attitude and performance in older adults

This study examined the interaction of age, attitude, and performance within the context of an interactive computer testing experience. Subjects were 13 males and 47 females between the ages of 55 and 82, with a minimum of a high school education. Initial attitudes toward computers, as measured by the Cybernetics Attitude Scale (CAS), demonstrated overall equivalence between these older subjects and previously tested younger subjects. Post-intervention scores on the CAS indicated that attitudes toward computers were unaffected by either a "fun" or a "challenging" computer interaction experience. The differential effects of a computerized vs. a paperand- pencil presentation format of a 20-item, multiple choice vocabulary test were examined. Results indicated no significant differences in the performance of subjects in the two conditions, and no interaction effect between attitude and performance. These findings suggest that the attitudes of older adults towards computers do not affect their computerized testing performance, at least for short term testing of verbal abilities. A further implication is that, under the conditions presented here, older subjects appear to be unaffected by mode of testing. The impact of recent advanced in technology on older adults is discussed.

A Digital Signal Processing Approach for Affective Sensing of a Computer User through Pupil Diameter Monitoring

Recent research has indicated that the pupil diameter (PD) in humans varies with their affective states. However, this signal has not been fully investigated for affective sensing purposes in human-computer interaction systems. This may be due to the dominant separate effect of the pupillary light reflex (PLR), which shrinks the pupil when light intensity increases. In this dissertation, an adaptive interference canceller (AIC) system using the H∞ time-varying (HITV) adaptive algorithm was developed to minimize the impact of the PLR on the measured pupil diameter signal. The modified pupil diameter (MPD) signal, obtained from the AIC was expected to reflect primarily the pupillary affective responses (PAR) of the subject. Additional manipulations of the AIC output resulted in a processed MPD (PMPD) signal, from which a classification feature, PMPDmean, was extracted. This feature was used to train and test a support vector machine (SVM), for the identification of stress states in the subject from whom the pupil diameter signal was recorded, achieving an accuracy rate of 77.78%. The advantages of affective recognition through the PD signal were verified by comparatively investigating the classification of stress and relaxation states through features derived from the simultaneously recorded galvanic skin response (GSR) and blood volume pulse (BVP) signals, with and without the PD feature. The discriminating potential of each individual feature extracted from GSR, BVP and PD was studied by analysis of its receiver operating characteristic (ROC) curve. The ROC curve found for the PMPDmean feature encompassed the largest area (0.8546) of all the single-feature ROCs investigated. The encouraging results seen in affective sensing based on pupil diameter monitoring were obtained in spite of intermittent illumination increases purposely introduced during the experiments. Therefore, these results confirmed the benefits of using the AIC implementation with the HITV adaptive algorithm to isolate the PAR and the potential of using PD monitoring to sense the evolving affective states of a computer user.

A Virtual Joystick as a Novel Interaction Modality for Surgeon-Computer Interaction During Computer-Assisted Surgical Procedures

Interacting with a computer system in the operating room (OR) can be a frustrating experience for a surgeon, who currently has to verbally delegate to an assistant every computer interaction task. This indirect mode of interaction is time consuming, error prone and can lead to poor usability of OR computer systems. This thesis describes the design and evaluation of a joystick-like device that allows direct surgeon control of the computer in the OR. The device was tested extensively in comparison to a mouse and delegated dictation with seven surgeons, eleven residents, and five graduate students. The device contains no electronic parts, is easy to use, is unobtrusive, has no physical connection to the computer and makes use of an existing tool in the OR. We performed a user study to determine its effectiveness in allowing a user to perform all the tasks they would be expected to perform on an OR computer system during a computer-assisted surgery. Dictation was found to be superior to the joystick in qualitative measures, but the joystick was preferred over dictation in user satisfaction responses. The mouse outperformed both joystick and dictation, but it is not a readily accepted modality in the OR.

Comparative study of human age estimation based on hand-crafted and deep face features

In the past few years, human facial age estimation has drawn a lot of attention in the computer vision and pattern recognition communities because of its important applications in age-based image retrieval, security control and surveillance, biomet- rics, human-computer interaction (HCI) and social robotics. In connection with these investigations, estimating the age of a person from the numerical analysis of his/her face image is a relatively new topic. Also, in problems such as Image Classification the Deep Neural Networks have given the best results in some areas including age estimation. In this work we use three hand-crafted features as well as five deep features that can be obtained from pre-trained deep convolutional neural networks. We do a comparative study of the obtained age estimation results with these features.

INNOVATIVE HUMAN-COMPUTER INTERACTIONS TO SUPPORT COGNITIVE COLLABORATIVE GEOSPATIAL ENVIRONMENTS

In the current world geospatial information is being demanded in almost real time, which requires the speed at which this data is processed and made available to the user to be at an all-time high. In order to keep up with this ever increasing speed, analysts must find ways to increase their productivity. At the same time the demand for new analysts is high, and current methods of training are long and can be costly. Through the use of human computer interactions and basic networking systems, this paper explores new ways to increase efficiency in data processing and analyst training.

A NATURALISTIC COMPUTATIONAL MODEL OF HUMAN BEHAVIOR IN NAVIGATION AND SEARCH TASKS

Planning, navigation, and search are fundamental human cognitive abilities central to spatial problem solving in search and rescue, law enforcement, and military operations. Despite a wealth of literature concerning naturalistic spatial problem solving in animals, literature on naturalistic spatial problem solving in humans is comparatively lacking and generally conducted by separate camps among which there is little crosstalk. Addressing this deficiency will allow us to predict spatial decision making in operational environments, and understand the factors leading to those decisions. The present dissertation is comprised of two related efforts, (1) a set of empirical research studies intended to identify characteristics of planning, execution, and memory in naturalistic spatial problem solving tasks, and (2) a computational modeling effort to develop a model of naturalistic spatial problem solving. The results of the behavioral studies indicate that problem space hierarchical representations are linear in shape, and that human solutions are produced according to multiple optimization criteria. The Mixed Criteria Model presented in this dissertation accounts for global and local human performance in a traditional and naturalistic Traveling Salesman Problem. The results of the empirical and modeling efforts hold implications for basic and applied science in domains such as problem solving, operations research, human-computer interaction, and artificial intelligence.

UGI: a multi-dimensional ultrasonic-based interaction approach

We are currently witnessing an era where interaction with computers is no longer limited to conventional methods (i.e. keyboard and mouse). Human Computer Interaction (HCI) as a progressive field of research, has opened up alternatives to the traditional interaction techniques. Embedded Infrared (IR) sensors, Accelerometers and RGBD cameras have become common inputs for devices to recognize gestures and body movements. These sensors are vision based and as a result the devices that incorporate them will be reliant on presence of light. Ultrasonic sensors on the other hand do not suffer this limitation as they utilize properties of sound waves. These sensors however, have been mainly used for distance detection and not with HCI devices. This paper presents our approach in developing a multi-dimensional interaction input method and tool Ultrasonic Gesture-based Interaction (UGI) that utilizes ultrasonic sensors. We demonstrate how these sensors can detect object movements and recognize gestures. We present our approach in building the device and demonstrate sample interactions with it. We have also conducted a user study to evaluate our tool and its distance and micro gesture detection accuracy. This paper reports these results and outlines our future work in the area.

Situating Design Research within Social Framework

This paper explores the expertise in industrial (product) design and contribution of knowledge generated trough the design research. Within this approach the research is situated within the social structure that constitutes people, activity, context and culture where an artifact is seen to be a mediator for the generation of new knowledge and its application. The paper concludes about the importance of research and practice integration and points out that situating the research around the artifacts, as mediators of knowledge, is transferable to Human-Computer Interaction field and any other area of the design research

Designing usable ubiquitous computing

This paper demonstrates that in order to design successful ubiquitous computing, designers must consider concurrently both the end user interactions in the context of use and the sustainability of the technology and its underlying infrastructure. We describe methods used to create more useful collaboration and communication between users, designers and engineers in designing ubiquitous computing systems. We tested these methods in a real domain in an attempt to create a system that is affordable, minimally disrupts the end-user's workplace and improves human-computer interaction.

The lens of ludic engagement : evaluating participation in interactive art installations

Designers and artists have integrated recent advances in interactive, tangible and ubiquitous computing technologies to create new forms of interactive environments in the domains of work, recreation, culture and leisure. Many designs of technology systems begin with the workplace in mind, and with function, ease of use, and efficiency high on the list of priorities. [1] These priorities do not fit well with works designed for an interactive art environment, where the aims are many, and where the focus on utility and functionality is to support a playful, ambiguous or even experimental experience for the participants. To evaluate such works requires an integration of art-criticism techniques with more recent Human Computer Interaction (HCI) methods, and an understanding of the different nature of engagement in these environments. This paper begins a process of mapping a set of priorities for amplifying engagement in interactive art installations. I first define the concept of ludic engagement and its usefulness as a lens for both design and evaluation in these settings. I then detail two fieldwork evaluations I conducted within two exhibitions of interactive artworks, and discuss their outcomes and the future directions of this research.

The Figmentum Project: appropriating information and communication technologies to animate our urban fabric

This paper explores how we may design located information and communication technologies (ICTs) to foster community sentiment. It focuses explicitly on possibilities for ICTs to create new modalities of place through exploring key factors such as shared experiences, shared knowledge and shared authorship. To contextualise this discussion in a real world setting, this paper presents FIGMENTUM, a situated generative art application that was developed for and installed in a new urban development. FIGMENTUM is a non-authoritative, non-service based application that aims to trigger emotional and representational place-based communities. Out of this practice-led research comes a theory and a process for designing creative place-based ICT’s to animate our urban communities.

Bridging technical and HCI research : creating usable ubiquitous computing

This paper describes methods used to support collaboration and communication between practitioners, designers and engineers when designing ubiquitous computing systems. We tested methods such as “Wizard of Oz” and design games in a real domain, the dental surgery, in an attempt to create a system that is: affordable; minimally disruptive of the natural flow of work; and improves human-computer interaction. In doing so we found that such activities allowed the practitioners to be on a ‘level playing ground’ with designers and engineers. The findings we present suggest that dentists are willing to engage in detailed exploration and constructive critique of technical design possibilities if the design ideas and prototypes are presented in the context of their work practice and are of a resolution and relevance that allow them to jointly explore and question with the design time. This paper is an extension of a short paper submitted to the Participatory Design Conference, 2004.

Construction defect reporting using mobile and digital workbench technologies

Sending data between the construction site and an off-site design office is one of the more problematic areas in information technology for construction automation, particularly for construction defect management. The aim of this research is to investigate how mobile computing and new forms of human-computer interaction can be brought to bear on specific problems in construction management. The construction defect reporting system is one such application. Combining mobile and wireless computing technologies with a digital workbench, we have developed a system to facilitate remote telecollaboration between a construction site and an off-site engineering office. The application reported in this paper demonstrates how construction defect reporting can be streamlined by field collection of construction defect information using a mobile device and visualising the defect in a CAD model on a digital workbench in an engineering office. This paper reports on the design of the system and our tests of sending images from the construction site to the engineer’s office and positional accuracy of GPS for localization of the defect.

The Brisbane media map : participatory design and authentic learning to link students and industry

The Brisbane Media Map is both an online resource and a tertiary-level authentic learning project. The Brisbane Media Map is an online database which provides a detailed overview of about 600 media industry organisations in Brisbane, Australia. In addition to providing contact details and synopses for each organisation’s profile, the Brisbane Media Map also includes supplementary information on current issues, trends, and individuals in the media and communication industry sectors. This resource is produced and updated annually by final-year undergraduate Media and Communication students. This article introduces the Brisbane Media Map, its functionality and systems design approach, as well as its alignment with key learning infrastructures. It examines authentic learning as the pedagogical framework underpinning the ongoing development work of the resource and highlights some synergies of this framework with participatory design principles. The Brisbane Media Map is a useful example of an authentic learning approach that successfully engages students of non-traditional and non-design areas of study in human-computer interaction, usability, and participatory design activities.