A multimodal and adaptive real-time human -computer interface for peoples with severe motor disability

This dissertation introduces the design of a multimodal, adaptive real-time assistive system as an alternate human computer interface that can be used by individuals with severe motor disabilities. The proposed design is based on the integration of a remote eye-gaze tracking system, voice recognition software, and a virtual keyboard. The methodology relies on a user profile that customizes eye gaze tracking using neural networks. The user profiling feature facilitates the notion of universal access to computing resources for a wide range of applications such as web browsing, email, word processing and editing. ^ The study is significant in terms of the integration of key algorithms to yield an adaptable and multimodal interface. The contributions of this dissertation stem from the following accomplishments: (a) establishment of the data transport mechanism between the eye-gaze system and the host computer yielding to a significantly low failure rate of 0.9%; (b) accurate translation of eye data into cursor movement through congregate steps which conclude with calibrated cursor coordinates using an improved conversion function; resulting in an average reduction of 70% of the disparity between the point of gaze and the actual position of the mouse cursor, compared with initial findings; (c) use of both a moving average and a trained neural network in order to minimize the jitter of the mouse cursor, which yield an average jittering reduction of 35%; (d) introduction of a new mathematical methodology to measure the degree of jittering of the mouse trajectory; (e) embedding an onscreen keyboard to facilitate text entry, and a graphical interface that is used to generate user profiles for system adaptability. ^ The adaptability nature of the interface is achieved through the establishment of user profiles, which may contain the jittering and voice characteristics of a particular user as well as a customized list of the most commonly used words ordered according to the user's preferences: in alphabetical or statistical order. This allows the system to successfully provide the capability of interacting with a computer. Every time any of the sub-system is retrained, the accuracy of the interface response improves even more. ^

Functional near infrared spectroscopy study of language, joint attention and motor skills

Near infrared spectroscopy (NIRS) is an emerging non-invasive optical neuro imaging technique that monitors the hemodynamic response to brain activation with ms-scale temporal resolution and sub-cm spatial resolution. The overall goal of my dissertation was to develop and apply NIRS towards investigation of neurological response to language, joint attention and planning and execution of motor skills in healthy adults. Language studies were performed to investigate the hemodynamic response, synchrony and dominance feature of the frontal and fronto-temporal cortex of healthy adults in response to language reception and expression. The mathematical model developed based on granger causality explicated the directional flow of information during the processing of language stimuli by the fronto-temporal cortex. Joint attention and planning/ execution of motor skill studies were performed to investigate the hemodynamic response, synchrony and dominance feature of the frontal cortex of healthy adults and in children (5-8 years old) with autism (for joint attention studies) and individuals with cerebral palsy (for planning/execution of motor skills studies). The joint attention studies on healthy adults showed differences in activation as well as intensity and phase dependent connectivity in the frontal cortex during joint attention in comparison to rest. The joint attention studies on typically developing children showed differences in frontal cortical activation in comparison to that in children with autism. The planning and execution of motor skills studies on healthy adults and individuals with cerebral palsy (CP) showed difference in the frontal cortical dominance, that is, bilateral and ipsilateral dominance, respectively. The planning and execution of motor skills studies also demonstrated the plastic and learning behavior of brain wherein correlation was found between the relative change in total hemoglobin in the frontal cortex and the kinematics of the activity performed by the participants. Thus, during my dissertation the NIRS neuroimaging technique was successfully implemented to investigate the neurological response of language, joint attention and planning and execution of motor skills in healthy adults as well as preliminarily on children with autism and individuals with cerebral palsy. These NIRS studies have long-term potential for the design of early stage interventions in children with autism and customized rehabilitation in individuals with cerebral palsy.

Funtional Near Infrared Spectroscopy Study of Language, Joint Attention and Motor Skills

Near infrared spectroscopy (NIRS) is an emerging non-invasive optical neuro imaging technique that monitors the hemodynamic response to brain activation with ms-scale temporal resolution and sub-cm spatial resolution. The overall goal of my dissertation was to develop and apply NIRS towards investigation of neurological response to language, joint attention and planning and execution of motor skills in healthy adults. Language studies were performed to investigate the hemodynamic response, synchrony and dominance feature of the frontal and fronto-temporal cortex of healthy adults in response to language reception and expression. The mathematical model developed based on granger causality explicated the directional flow of information during the processing of language stimuli by the fronto-temporal cortex. Joint attention and planning/ execution of motor skill studies were performed to investigate the hemodynamic response, synchrony and dominance feature of the frontal cortex of healthy adults and in children (5-8 years old) with autism (for joint attention studies) and individuals with cerebral palsy (for planning/execution of motor skills studies). The joint attention studies on healthy adults showed differences in activation as well as intensity and phase dependent connectivity in the frontal cortex during joint attention in comparison to rest. The joint attention studies on typically developing children showed differences in frontal cortical activation in comparison to that in children with autism. The planning and execution of motor skills studies on healthy adults and individuals with cerebral palsy (CP) showed difference in the frontal cortical dominance, that is, bilateral and ipsilateral dominance, respectively. The planning and execution of motor skills studies also demonstrated the plastic and learning behavior of brain wherein correlation was found between the relative change in total hemoglobin in the frontal cortex and the kinematics of the activity performed by the participants. Thus, during my dissertation the NIRS neuroimaging technique was successfully implemented to investigate the neurological response of language, joint attention and planning and execution of motor skills in healthy adults as well as preliminarily on children with autism and individuals with cerebral palsy. These NIRS studies have long-term potential for the design of early stage interventions in children with autism and customized rehabilitation in individuals with cerebral palsy.