Brain-computer interface (BCI): a methodological proposal to assess the impacts of medical applications in 2022

Based on the report for “Project IV” unit of the PhD programme on Technology Assessment (Doctoral Conference) at Universidade Nova de Lisboa (December 2011). This thesis research has the supervision of António Moniz (FCT-UNL and ITAS-KIT) and Armin Grunwald (Karlsruhe Institute of Technology-ITAS, Germany). Other members of the thesis committee are Mário Forjaz Secca (FCT-UNL) and Femke Nijboer (University of Twente, Netherlands).

Hand gesture recognition system based in computer vision and machine learning

"Lecture notes in computational vision and biomechanics series, ISSN 2212-9391, vol. 19"

Hand gesture recognition system based in computer vision and machine learning: Applications on human-machine interaction

Tese de Doutoramento em Engenharia de Eletrónica e de Computadores

Therapeutic potential of computer to cerebral cortex implantable devices

In this article, an overview of some of the latest developments in the field of cerebral cortex to computer interfacing (CCCI) is given. This is posed in the more general context of Brain-Computer Interfaces in order to assess advantages and disadvantages. The emphasis is clearly placed on practical studies that have been undertaken and reported on, as opposed to those speculated, simulated or proposed as future projects. Related areas are discussed briefly only in the context of their contribution to the studies being undertaken. The area of focus is notably the use of invasive implant technology, where a connection is made directly with the cerebral cortex and/or nervous system. Tests and experimentation which do not involve human subjects are invariably carried out a priori to indicate the eventual possibilities before human subjects are themselves involved. Some of the more pertinent animal studies from this area are discussed. The paper goes on to describe human experimentation, in which neural implants have linked the human nervous system bidirectionally with technology and the internet. A view is taken as to the prospects for the future for CCCI, in terms of its broad therapeutic role.

Gravity well visibility in haptic interfaces for motion-impaired

Haptic computer interfaces provide users with feedback through the sense of touch, thereby allowing users to feel a graphical user interface. Force feedback gravity wells, i.e. attractive basins that can pull the cursor toward a target, are one type of haptic effect that have been shown to provide improvements in "point and click" tasks. For motion-impaired users, gravity wells could improve times by as much as 50%. It has been reported that the presentation of information to multiple sensory modalities, e.g. haptics and vision, can provide performance benefits. However, previous studies investigating the use of force feedback gravity wells have generally not provided visual representations of the haptic effect. Where force fields extend beyond clickable targets, the addition of visual cues may affect performance. This paper investigates how the performance of motion-impaired computer users is affected by having visual representations of force feedback gravity wells presented on-screen. Results indicate that the visual representation does not affect times and errors in a "point and click" task involving multiple targets.

Motor imagery induced EEG patterns in spinal cord injury patients and their impact on Brain-Computer Interface accuracy

OBJECTIVE: Assimilating the diagnosis complete spinal cord injury (SCI) takes time and is not easy, as patients know that there is no 'cure' at the present time. Brain-computer interfaces (BCIs) can facilitate daily living. However, inter-subject variability demands measurements with potential user groups and an understanding of how they differ to healthy users BCIs are more commonly tested with. Thus, a three-class motor imagery (MI) screening (left hand, right hand, feet) was performed with a group of 10 able-bodied and 16 complete spinal-cord-injured people (paraplegics, tetraplegics) with the objective of determining what differences were present between the user groups and how they would impact upon the ability of these user groups to interact with a BCI. APPROACH: Electrophysiological differences between patient groups and healthy users are measured in terms of sensorimotor rhythm deflections from baseline during MI, electroencephalogram microstate scalp maps and strengths of inter-channel phase synchronization. Additionally, using a common spatial pattern algorithm and a linear discriminant analysis classifier, the classification accuracy was calculated and compared between groups. MAIN RESULTS: It is seen that both patient groups (tetraplegic and paraplegic) have some significant differences in event-related desynchronization strengths, exhibit significant increases in synchronization and reach significantly lower accuracies (mean (M) = 66.1%) than the group of healthy subjects (M = 85.1%). SIGNIFICANCE: The results demonstrate significant differences in electrophysiological correlates of motor control between healthy individuals and those individuals who stand to benefit most from BCI technology (individuals with SCI). They highlight the difficulty in directly translating results from healthy subjects to participants with SCI and the challenges that, therefore, arise in providing BCIs to such individuals.

An optimized ERP Brain-computer interface based on facial expression changes

OBJECTIVE: Interferences from spatially adjacent non-target stimuli are known to evoke event-related potentials (ERPs) during non-target flashes and, therefore, lead to false positives. This phenomenon was commonly seen in visual attention-based brain-computer interfaces (BCIs) using conspicuous stimuli and is known to adversely affect the performance of BCI systems. Although users try to focus on the target stimulus, they cannot help but be affected by conspicuous changes of the stimuli (such as flashes or presenting images) which were adjacent to the target stimulus. Furthermore, subjects have reported that conspicuous stimuli made them tired and annoyed. In view of this, the aim of this study was to reduce adjacent interference, annoyance and fatigue using a new stimulus presentation pattern based upon facial expression changes. Our goal was not to design a new pattern which could evoke larger ERPs than the face pattern, but to design a new pattern which could reduce adjacent interference, annoyance and fatigue, and evoke ERPs as good as those observed during the face pattern. APPROACH: Positive facial expressions could be changed to negative facial expressions by minor changes to the original facial image. Although the changes are minor, the contrast is big enough to evoke strong ERPs. In this paper, a facial expression change pattern between positive and negative facial expressions was used to attempt to minimize interference effects. This was compared against two different conditions, a shuffled pattern containing the same shapes and colours as the facial expression change pattern, but without the semantic content associated with a change in expression, and a face versus no face pattern. Comparisons were made in terms of classification accuracy and information transfer rate as well as user supplied subjective measures. MAIN RESULTS: The results showed that interferences from adjacent stimuli, annoyance and the fatigue experienced by the subjects could be reduced significantly (p < 0.05) by using the facial expression change patterns in comparison with the face pattern. The offline results show that the classification accuracy of the facial expression change pattern was significantly better than that of the shuffled pattern (p < 0.05) and the face pattern (p < 0.05). SIGNIFICANCE: The facial expression change pattern presented in this paper reduced interference from adjacent stimuli and decreased the fatigue and annoyance experienced by BCI users significantly (p < 0.05) compared to the face pattern.

Educação matemática através de interfaces computacionais: o papel de componentes interativos na postura exploratória e na aprendizagem

Neste trabalho, investigamos o papel de componentes interativos, frequentemente utilizados na construção de interfaces computacionais educativas, na postura exploratória do estudante e na aprendizagem de conceitos matemáticos. Selecionamos para esta pesquisa os seguintes componentes: caixa de combinações (combo box) e campo de texto (text field). Do ponto de vista educacional, estes componentes têm papéis distintos: o primeiro orienta as escolhas do estudante durante um processo exploratório, enquanto que o segundo não oferece qualquer orientação. Para comparar o papel desses componentes, desenvolvemos duas interfaces computacionais interativas através das quais o estudante pode explorar o comportamento gráfico de uma função do primeiro grau. Ambas as interfaces são idênticas entre si, a menos do componente interativo empregado: em uma delas foi utilizado a caixa de combinações e em outra o campo de texto. Tanto a postura exploratória quanto o desempenho em testes de conhecimento foram avaliados a partir de medidas diretas registradas pelas próprias interfaces. A postura exploratória foi avaliada através do número e do tipo de interações do estudante com o componente interativo, sendo este registro uma das características singulares desta pesquisa, pois permite a observação de alguns comportamentos do estudante durante o processo de interação com a interface, e não somente antes e após a interação. Dentro da limitação da ferramenta de coleta de dados da presente pesquisa, a aprendizagem foi medida através da comparação do desempenho em testes de conhecimento aplicados antes e depois do uso dos componentes interativos pelos estudantes. Neste contexto, diferenças significativas no papel de cada componente na postura exploratória e na aprendizagem foram então observadas.

Mental states monitoring through passive Brain-Computer Interface systems

The monitoring of cognitive functions aims at gaining information about the current cognitive state of the user by decoding brain signals. In recent years, this approach allowed to acquire valuable information about the cognitive aspects regarding the interaction of humans with external world. From this consideration, researchers started to consider passive application of brain–computer interface (BCI) in order to provide a novel input modality for technical systems solely based on brain activity. The objective of this thesis is to demonstrate how the passive Brain Computer Interfaces (BCIs) applications can be used to assess the mental states of the users, in order to improve the human machine interaction. Two main studies has been proposed. The first one allows to investigate whatever the Event Related Potentials (ERPs) morphological variations can be used to predict the users’ mental states (e.g. attentional resources, mental workload) during different reactive BCI tasks (e.g. P300-based BCIs), and if these information can predict the subjects’ performance in performing the tasks. In the second study, a passive BCI system able to online estimate the mental workload of the user by relying on the combination of the EEG and the ECG biosignals has been proposed. The latter study has been performed by simulating an operative scenario, in which the occurrence of errors or lack of performance could have significant consequences. The results showed that the proposed system is able to estimate online the mental workload of the subjects discriminating three different difficulty level of the tasks ensuring a high reliability.

Supplement to CAMAC (computer automated measurement and control) standards and reports /

Issued Mar. 1978.

Adaptive Brain-Computer Interface Systems For Communication in People with Severe Neuromuscular Disabilities

Brain-computer interfaces (BCI) have the potential to restore communication or control abilities in individuals with severe neuromuscular limitations, such as those with amyotrophic lateral sclerosis (ALS). The role of a BCI is to extract and decode relevant information that conveys a user's intent directly from brain electro-physiological signals and translate this information into executable commands to control external devices. However, the BCI decision-making process is error-prone due to noisy electro-physiological data, representing the classic problem of efficiently transmitting and receiving information via a noisy communication channel.

This research focuses on P300-based BCIs which rely predominantly on event-related potentials (ERP) that are elicited as a function of a user's uncertainty regarding stimulus events, in either an acoustic or a visual oddball recognition task. The P300-based BCI system enables users to communicate messages from a set of choices by selecting a target character or icon that conveys a desired intent or action. P300-based BCIs have been widely researched as a communication alternative, especially in individuals with ALS who represent a target BCI user population. For the P300-based BCI, repeated data measurements are required to enhance the low signal-to-noise ratio of the elicited ERPs embedded in electroencephalography (EEG) data, in order to improve the accuracy of the target character estimation process. As a result, BCIs have relatively slower speeds when compared to other commercial assistive communication devices, and this limits BCI adoption by their target user population. The goal of this research is to develop algorithms that take into account the physical limitations of the target BCI population to improve the efficiency of ERP-based spellers for real-world communication.

In this work, it is hypothesised that building adaptive capabilities into the BCI framework can potentially give the BCI system the flexibility to improve performance by adjusting system parameters in response to changing user inputs. The research in this work addresses three potential areas for improvement within the P300 speller framework: information optimisation, target character estimation and error correction. The visual interface and its operation control the method by which the ERPs are elicited through the presentation of stimulus events. The parameters of the stimulus presentation paradigm can be modified to modulate and enhance the elicited ERPs. A new stimulus presentation paradigm is developed in order to maximise the information content that is presented to the user by tuning stimulus paradigm parameters to positively affect performance. Internally, the BCI system determines the amount of data to collect and the method by which these data are processed to estimate the user's target character. Algorithms that exploit language information are developed to enhance the target character estimation process and to correct erroneous BCI selections. In addition, a new model-based method to predict BCI performance is developed, an approach which is independent of stimulus presentation paradigm and accounts for dynamic data collection. The studies presented in this work provide evidence that the proposed methods for incorporating adaptive strategies in the three areas have the potential to significantly improve BCI communication rates, and the proposed method for predicting BCI performance provides a reliable means to pre-assess BCI performance without extensive online testing.

Dynamic Image Precompensation for Improving Visual Performance of Computer Users with Ocular Aberrations

With the progress of computer technology, computers are expected to be more intelligent in the interaction with humans, presenting information according to the user's psychological and physiological characteristics. However, computer users with visual problems may encounter difficulties on the perception of icons, menus, and other graphical information displayed on the screen, limiting the efficiency of their interaction with computers. In this dissertation, a personalized and dynamic image precompensation method was developed to improve the visual performance of the computer users with ocular aberrations. The precompensation was applied on the graphical targets before presenting them on the screen, aiming to counteract the visual blurring caused by the ocular aberration of the user's eye. A complete and systematic modeling approach to describe the retinal image formation of the computer user was presented, taking advantage of modeling tools, such as Zernike polynomials, wavefront aberration, Point Spread Function and Modulation Transfer Function. The ocular aberration of the computer user was originally measured by a wavefront aberrometer, as a reference for the precompensation model. The dynamic precompensation was generated based on the resized aberration, with the real-time pupil diameter monitored. The potential visual benefit of the dynamic precompensation method was explored through software simulation, with the aberration data from a real human subject. An "artificial eye'' experiment was conducted by simulating the human eye with a high-definition camera, providing objective evaluation to the image quality after precompensation. In addition, an empirical evaluation with 20 human participants was also designed and implemented, involving image recognition tests performed under a more realistic viewing environment of computer use. The statistical analysis results of the empirical experiment confirmed the effectiveness of the dynamic precompensation method, by showing significant improvement on the recognition accuracy. The merit and necessity of the dynamic precompensation were also substantiated by comparing it with the static precompensation. The visual benefit of the dynamic precompensation was further confirmed by the subjective assessments collected from the evaluation participants.

Smart object for 3D interaction

This paper reports on the creation of an interface for 3D virtual environments, computer-aided design applications or computer games. Standard computer interfaces are bound to 2D surfaces, e.g., computer mouses, keyboards, touch pads or touch screens. The Smart Object is intended to provide the user with a 3D interface by using sensors that register movement (inertial measurement unit), touch (touch screen) and voice (microphone). The design and development process as well as the tests and results are presented in this paper. The Smart Object was developed by a team of four third-year engineering students from diverse scientific backgrounds and nationalities during one semester.

Implementação de uma Interface Homem-máquina baseada em Eletroculografia

A constante evolução da tecnologia permitiu ao ser humano a utilização de dispositivos electrónicos nas suas rotinas diárias. Estas podem ser afetadas quando os utilizadores sofrem de deficiências ou doenças que afetam as suas capacidades motoras. Com o intuito de minimizar este obstáculo surgiram as Interfaces Homem-Computador (HCI). É neste panorama que os sistemas HCI baseados em Eletroculografia (EOG) assumem um papel preponderante na melhoria da qualidade de vida destes indivíduos. A Eletroculografia é o resultado da aquisição do movimento ocular, que pode ser adquirido através de diversos métodos. Os métodos mais convencionais utilizam elétrodos de superfície para aquisição dos sinais elétricos, ou então, utilizam sistemas de gravação de vídeo, que gravam o movimento ocular. O objetivo desta tese é desenvolver um sistema HCI baseado em Eletroculografia, que adquire o sinal elétrico do movimento ocular através de elétrodos de superfície. Para tal desenvolveu-se um circuito eletrónico para a aquisição do sinal de EOG, bem como um algoritmo em Python para análise do mesmo. O circuito foi desenvolvido recorrendo a seis módulos diferentes, cada um deles com uma função específica. Para cada módulo foi necessário desenhar e implementar placas de circuito impresso, que quando conectadas entre si permitem filtrar, amplificar e digitalizar os sinais elétricos, adquiridos através de elétrodos de superfície, originados pelo movimento ocular. O algoritmo criado em Python permite analisar os dados provenientes do circuito e converte-os para coordenadas. Através destas foi possível determinar o sentido e a amplitude do movimento ocular.