Investigating assistive technology to support memory for people with cognitive impairments

Technologies such as automobiles or mobile phones allow us to perform beyond our physical capabilities and travel faster or communicate over long distances. Technologies such as computers and calculators can also help us perform beyond our mental capabilities by storing and manipulating information that we would be unable to process or remember. In recent years there has been a growing interest in assistive technology for cognition (ATC) which can help people compensate for cognitive impairments. The aim of this thesis was to investigate ATC for memory to help people with memory difficulties which impacts independent functioning during everyday life. Chapter one argues that using both neuropsychological and human computing interaction theory and approaches is crucial when developing and researching ATC. Chapter two describes a systematic review and meta-analysis of studies which tested technology to aid memory for groups with ABI, stroke or degenerative disease. Good evidence was found supporting the efficacy of prompting devices which remind the user about a future intention at a set time. Chapter three looks at the prevalence of technologies and memory aids in current use by people with ABI and dementia and the factors that predicted this use. Pre-morbid use of technology, current use of non-tech aids and strategies and age (ABI group only) were the best predictors of this use. Based on the results, chapter four focuses on mobile phone based reminders for people with ABI. Focus groups were held with people with memory impairments after ABI and ABI caregivers (N=12) which discussed the barriers to uptake of mobile phone based reminding. Thematic analysis revealed six key themes that impact uptake of reminder apps; Perceived Need, Social Acceptability, Experience/Expectation, Desired Content and Functions, Cognitive Accessibility and Sensory/Motor Accessibility. The Perceived need theme described the difficulties with insight, motivation and memory which can prevent people from initially setting reminders on a smartphone. Chapter five investigates the efficacy and acceptability of unsolicited prompts (UPs) from a smartphone app (ForgetMeNot) to encourage people with ABI to set reminders. A single-case experimental design study evaluated use of the app over four weeks by three people with severe ABI living in a post-acute rehabilitation hospital. When six UPs were presented through the day from ForgetMeNot, daily reminder-setting and daily memory task completion increased compared to when using the app without the UPs. Chapter six investigates another barrier from chapter 4 – cognitive and sensory accessibility. A study is reported which shows that an app with ‘decision tree’ interface design (ApplTree) leads to more accurate reminder setting performance with no compromise of speed or independence (amount of guidance required) for people with ABI (n=14) compared to a calendar based interface. Chapter seven investigates the efficacy of a wearable reminding device (smartwatch) as a tool for delivering reminders set on a smartphone. Four community dwelling participants with memory difficulties following ABI were included in an ABA single case experimental design study. Three of the participants successfully used the smartwatch throughout the intervention weeks and these participants gave positive usability ratings. Two participants showed improved memory performance when using the smartwatch and all participants had marked decline in memory performance when the technology was removed. Chapter eight is a discussion which highlights the implications of these results for clinicians, researchers and designers.

Machine learning techniques for implicit interaction using mobile sensors

Interactions in mobile devices normally happen in an explicit manner, which means that they are initiated by the users. Yet, users are typically unaware that they also interact implicitly with their devices. For instance, our hand pose changes naturally when we type text messages. Whilst the touchscreen captures finger touches, hand movements during this interaction however are unused. If this implicit hand movement is observed, it can be used as additional information to support or to enhance the users’ text entry experience. This thesis investigates how implicit sensing can be used to improve existing, standard interaction technique qualities. In particular, this thesis looks into enhancing front-of-device interaction through back-of-device and hand movement implicit sensing. We propose the investigation through machine learning techniques. We look into problems on how sensor data via implicit sensing can be used to predict a certain aspect of an interaction. For instance, one of the questions that this thesis attempts to answer is whether hand movement during a touch targeting task correlates with the touch position. This is a complex relationship to understand but can be best explained through machine learning. Using machine learning as a tool, such correlation can be measured, quantified, understood and used to make predictions on future touch position. Furthermore, this thesis also evaluates the predictive power of the sensor data. We show this through a number of studies. In Chapter 5 we show that probabilistic modelling of sensor inputs and recorded touch locations can be used to predict the general area of future touches on touchscreen. In Chapter 7, using SVM classifiers, we show that data from implicit sensing from general mobile interactions is user-specific. This can be used to identify users implicitly. In Chapter 6, we also show that touch interaction errors can be detected from sensor data. In our experiment, we show that there are sufficient distinguishable patterns between normal interaction signals and signals that are strongly correlated with interaction error. In all studies, we show that performance gain can be achieved by combining sensor inputs.