Interaction techniques with novel multimodal feedback for addressing gesture-sensing systems

Users need to be able to address in-air gesture systems, which means finding where to perform gestures and how to direct them towards the intended system. This is necessary for input to be sensed correctly and without unintentionally affecting other systems. This thesis investigates novel interaction techniques which allow users to address gesture systems properly, helping them find where and how to gesture. It also investigates audio, tactile and interactive light displays for multimodal gesture feedback; these can be used by gesture systems with limited output capabilities (like mobile phones and small household controls), allowing the interaction techniques to be used by a variety of device types. It investigates tactile and interactive light displays in greater detail, as these are not as well understood as audio displays. Experiments 1 and 2 explored tactile feedback for gesture systems, comparing an ultrasound haptic display to wearable tactile displays at different body locations and investigating feedback designs. These experiments found that tactile feedback improves the user experience of gesturing by reassuring users that their movements are being sensed. Experiment 3 investigated interactive light displays for gesture systems, finding this novel display type effective for giving feedback and presenting information. It also found that interactive light feedback is enhanced by audio and tactile feedback. These feedback modalities were then used alongside audio feedback in two interaction techniques for addressing gesture systems: sensor strength feedback and rhythmic gestures. Sensor strength feedback is multimodal feedback that tells users how well they can be sensed, encouraging them to find where to gesture through active exploration. Experiment 4 found that they can do this with 51mm accuracy, with combinations of audio and interactive light feedback leading to the best performance. Rhythmic gestures are continuously repeated gesture movements which can be used to direct input. Experiment 5 investigated the usability of this technique, finding that users can match rhythmic gestures well and with ease. Finally, these interaction techniques were combined, resulting in a new single interaction for addressing gesture systems. Using this interaction, users could direct their input with rhythmic gestures while using the sensor strength feedback to find a good location for addressing the system. Experiment 6 studied the effectiveness and usability of this technique, as well as the design space for combining the two types of feedback. It found that this interaction was successful, with users matching 99.9% of rhythmic gestures, with 80mm accuracy from target points. The findings show that gesture systems could successfully use this interaction technique to allow users to address them. Novel design recommendations for using rhythmic gestures and sensor strength feedback were created, informed by the experiment findings.

Hierarchically grouped 2D local features applied to edge contour localisation

One of the most significant research topics in computer vision is object detection. Most of the reported object detection results localise the detected object within a bounding box, but do not explicitly label the edge contours of the object. Since object contours provide a fundamental diagnostic of object shape, some researchers have initiated work on linear contour feature representations for object detection and localisation. However, linear contour feature-based localisation is highly dependent on the performance of linear contour detection within natural images, and this can be perturbed significantly by a cluttered background. In addition, the conventional approach to achieving rotation-invariant features is to rotate the feature receptive field to align with the local dominant orientation before computing the feature representation. Grid resampling after rotation adds extra computational cost and increases the total time consumption for computing the feature descriptor. Though it is not an expensive process if using current computers, it is appreciated that if each step of the implementation is faster to compute especially when the number of local features is increasing and the application is implemented on resource limited ”smart devices”, such as mobile phones, in real-time. Motivated by the above issues, a 2D object localisation system is proposed in this thesis that matches features of edge contour points, which is an alternative method that takes advantage of the shape information for object localisation. This is inspired by edge contour points comprising the basic components of shape contours. In addition, edge point detection is usually simpler to achieve than linear edge contour detection. Therefore, the proposed localization system could avoid the need for linear contour detection and reduce the pathological disruption from the image background. Moreover, since natural images usually comprise many more edge contour points than interest points (i.e. corner points), we also propose new methods to generate rotation-invariant local feature descriptors without pre-rotating the feature receptive field to improve the computational efficiency of the whole system. In detail, the 2D object localisation system is achieved by matching edge contour points features in a constrained search area based on the initial pose-estimate produced by a prior object detection process. The local feature descriptor obtains rotation invariance by making use of rotational symmetry of the hexagonal structure. Therefore, a set of local feature descriptors is proposed based on the hierarchically hexagonal grouping structure. Ultimately, the 2D object localisation system achieves a very promising performance based on matching the proposed features of edge contour points with the mean correct labelling rate of the edge contour points 0.8654 and the mean false labelling rate 0.0314 applied on the data from Amsterdam Library of Object Images (ALOI). Furthermore, the proposed descriptors are evaluated by comparing to the state-of-the-art descriptors and achieve competitive performances in terms of pose estimate with around half-pixel pose error.

Investigating assistive technology to support memory for people with cognitive impairments

Technologies such as automobiles or mobile phones allow us to perform beyond our physical capabilities and travel faster or communicate over long distances. Technologies such as computers and calculators can also help us perform beyond our mental capabilities by storing and manipulating information that we would be unable to process or remember. In recent years there has been a growing interest in assistive technology for cognition (ATC) which can help people compensate for cognitive impairments. The aim of this thesis was to investigate ATC for memory to help people with memory difficulties which impacts independent functioning during everyday life. Chapter one argues that using both neuropsychological and human computing interaction theory and approaches is crucial when developing and researching ATC. Chapter two describes a systematic review and meta-analysis of studies which tested technology to aid memory for groups with ABI, stroke or degenerative disease. Good evidence was found supporting the efficacy of prompting devices which remind the user about a future intention at a set time. Chapter three looks at the prevalence of technologies and memory aids in current use by people with ABI and dementia and the factors that predicted this use. Pre-morbid use of technology, current use of non-tech aids and strategies and age (ABI group only) were the best predictors of this use. Based on the results, chapter four focuses on mobile phone based reminders for people with ABI. Focus groups were held with people with memory impairments after ABI and ABI caregivers (N=12) which discussed the barriers to uptake of mobile phone based reminding. Thematic analysis revealed six key themes that impact uptake of reminder apps; Perceived Need, Social Acceptability, Experience/Expectation, Desired Content and Functions, Cognitive Accessibility and Sensory/Motor Accessibility. The Perceived need theme described the difficulties with insight, motivation and memory which can prevent people from initially setting reminders on a smartphone. Chapter five investigates the efficacy and acceptability of unsolicited prompts (UPs) from a smartphone app (ForgetMeNot) to encourage people with ABI to set reminders. A single-case experimental design study evaluated use of the app over four weeks by three people with severe ABI living in a post-acute rehabilitation hospital. When six UPs were presented through the day from ForgetMeNot, daily reminder-setting and daily memory task completion increased compared to when using the app without the UPs. Chapter six investigates another barrier from chapter 4 – cognitive and sensory accessibility. A study is reported which shows that an app with ‘decision tree’ interface design (ApplTree) leads to more accurate reminder setting performance with no compromise of speed or independence (amount of guidance required) for people with ABI (n=14) compared to a calendar based interface. Chapter seven investigates the efficacy of a wearable reminding device (smartwatch) as a tool for delivering reminders set on a smartphone. Four community dwelling participants with memory difficulties following ABI were included in an ABA single case experimental design study. Three of the participants successfully used the smartwatch throughout the intervention weeks and these participants gave positive usability ratings. Two participants showed improved memory performance when using the smartwatch and all participants had marked decline in memory performance when the technology was removed. Chapter eight is a discussion which highlights the implications of these results for clinicians, researchers and designers.