Neurorehabilitation of hand functions using brain computer interface

Introduction: Brain computer interface (BCI) is a promising new technology with possible application in neurorehabilitation after spinal cord injury. Movement imagination or attempted movement-based BCI coupled with functional electrical stimulation (FES) enables the simultaneous activation of the motor cortices and the muscles they control. When using the BCI- coupled with FES (known as BCI-FES), the subject activates the motor cortex using attempted movement or movement imagination of a limb. The BCI system detects the motor cortex activation and activates the FES attached to the muscles of the limb the subject is attempting or imaging to move. In this way the afferent and the efferent pathways of the nervous system are simultaneously activated. This simultaneous activation encourages Hebbian type learning which could be beneficial in functional rehabilitation after spinal cord injury (SCI). The FES is already in use in several SCI rehabilitation units but there is currently not enough clinical evidence to support the use of BCI-FES for rehabilitation. Aims: The main aim of this thesis is to assess outcomes in sub-acute tetraplegic patients using BCI-FES for functional hand rehabilitation. In addition, the thesis explores different methods for assessing neurological rehabilitation especially after BCI-FES therapy. The thesis also investigated mental rotation as a possible rehabilitation method in SCI. Methods: Following investigation into applicable methods that can be used to implement rehabilitative BCI, a BCI based on attempted movement was built. Further, the BCI was used to build a BCI-FES system. The BCI-FES system was used to deliver therapy to seven sub-acute tetraplegic patients who were scheduled to receive the therapy over a total period of 20 working days. These seven patients are in a 'BCI-FES' group. Five more patients were also recruited and offered equivalent FES quantity without the BCI. These further five patients are in a 'FES-only' group. Neurological and functional measures were investigated and used to assess both patient groups before and after therapy. Results: The results of the two groups of patients were compared. The patients in the BCI-FES group had better improvements. These improvements were found with outcome measures assessing neurological changes. The neurological changes following the use of the BCI-FES showed that during movement attempt, the activation of the motor cortex areas of the SCI patients became closer to the activation found in healthy individuals. The intensity of the activation and its spatial localisation both improved suggesting desirable cortical reorganisation. Furthermore, the responses of the somatosensory cortex during sensory stimulation were of clear evidence of better improvement in patients who used the BCI-FES. Missing somatosensory evoked potential peaks returned more for the BCI-FES group while there was no overall change in the FES-only group. Although the BCI-FES group had better neurological improvement, they did not show better functional improvement than the FES-only group. This was attributed mainly to the short duration of the study where therapies were only delivered for 20 working days. Conclusions: The results obtained from this study have shown that BCI-FES may induce cortical changes in the desired direction at least faster than FES alone. The observation of better improvement in the patients who used the BCI-FES is a good result in neurorehabilitation and it shows the potential of thought-controlled FES as a neurorehabilitation tool. These results back other studies that have shown the potential of BCI-FES in rehabilitation following neurological injuries that lead to movement impairment. Although the results are promising, further studies are necessary given the small number of subjects in the current study.

Integrated visual perception architecture for robotic clothes perception and manipulation

This thesis proposes a generic visual perception architecture for robotic clothes perception and manipulation. This proposed architecture is fully integrated with a stereo vision system and a dual-arm robot and is able to perform a number of autonomous laundering tasks. Clothes perception and manipulation is a novel research topic in robotics and has experienced rapid development in recent years. Compared to the task of perceiving and manipulating rigid objects, clothes perception and manipulation poses a greater challenge. This can be attributed to two reasons: firstly, deformable clothing requires precise (high-acuity) visual perception and dexterous manipulation; secondly, as clothing approximates a non-rigid 2-manifold in 3-space, that can adopt a quasi-infinite configuration space, the potential variability in the appearance of clothing items makes them difficult to understand, identify uniquely, and interact with by machine. From an applications perspective, and as part of EU CloPeMa project, the integrated visual perception architecture refines a pre-existing clothing manipulation pipeline by completing pre-wash clothes (category) sorting (using single-shot or interactive perception for garment categorisation and manipulation) and post-wash dual-arm flattening. To the best of the author’s knowledge, as investigated in this thesis, the autonomous clothing perception and manipulation solutions presented here were first proposed and reported by the author. All of the reported robot demonstrations in this work follow a perception-manipulation method- ology where visual and tactile feedback (in the form of surface wrinkledness captured by the high accuracy depth sensor i.e. CloPeMa stereo head or the predictive confidence modelled by Gaussian Processing) serve as the halting criteria in the flattening and sorting tasks, respectively. From scientific perspective, the proposed visual perception architecture addresses the above challenges by parsing and grouping 3D clothing configurations hierarchically from low-level curvatures, through mid-level surface shape representations (providing topological descriptions and 3D texture representations), to high-level semantic structures and statistical descriptions. A range of visual features such as Shape Index, Surface Topologies Analysis and Local Binary Patterns have been adapted within this work to parse clothing surfaces and textures and several novel features have been devised, including B-Spline Patches with Locality-Constrained Linear coding, and Topology Spatial Distance to describe and quantify generic landmarks (wrinkles and folds). The essence of this proposed architecture comprises 3D generic surface parsing and interpretation, which is critical to underpinning a number of laundering tasks and has the potential to be extended to other rigid and non-rigid object perception and manipulation tasks. The experimental results presented in this thesis demonstrate that: firstly, the proposed grasp- ing approach achieves on-average 84.7% accuracy; secondly, the proposed flattening approach is able to flatten towels, t-shirts and pants (shorts) within 9 iterations on-average; thirdly, the proposed clothes recognition pipeline can recognise clothes categories from highly wrinkled configurations and advances the state-of-the-art by 36% in terms of classification accuracy, achieving an 83.2% true-positive classification rate when discriminating between five categories of clothes; finally the Gaussian Process based interactive perception approach exhibits a substantial improvement over single-shot perception. Accordingly, this thesis has advanced the state-of-the-art of robot clothes perception and manipulation.