Off-line and on-line affective recognition of a computer user through a biosignal processing approach

Physiological signals, which are controlled by the autonomic nervous system (ANS), could be used to detect the affective state of computer users and therefore find applications in medicine and engineering. The Pupil Diameter (PD) seems to provide a strong indication of the affective state, as found by previous research, but it has not been investigated fully yet. ^ In this study, new approaches based on monitoring and processing the PD signal for off-line and on-line affective assessment ("relaxation" vs. "stress") are proposed. Wavelet denoising and Kalman filtering methods are first used to remove abrupt changes in the raw Pupil Diameter (PD) signal. Then three features (PDmean, PDmax and PDWalsh) are extracted from the preprocessed PD signal for the affective state classification. In order to select more relevant and reliable physiological data for further analysis, two types of data selection methods are applied, which are based on the paired t-test and subject self-evaluation, respectively. In addition, five different kinds of the classifiers are implemented on the selected data, which achieve average accuracies up to 86.43% and 87.20%, respectively. Finally, the receiver operating characteristic (ROC) curve is utilized to investigate the discriminating potential of each individual feature by evaluation of the area under the ROC curve, which reaches values above 0.90. ^ For the on-line affective assessment, a hard threshold is implemented first in order to remove the eye blinks from the PD signal and then a moving average window is utilized to obtain the representative value PDr for every one-second time interval of PD. There are three main steps for the on-line affective assessment algorithm, which are preparation, feature-based decision voting and affective determination. The final results show that the accuracies are 72.30% and 73.55% for the data subsets, which were respectively chosen using two types of data selection methods (paired t-test and subject self-evaluation). ^ In order to further analyze the efficiency of affective recognition through the PD signal, the Galvanic Skin Response (GSR) was also monitored and processed. The highest affective assessment classification rate obtained from GSR processing is only 63.57% (based on the off-line processing algorithm). The overall results confirm that the PD signal should be considered as one of the most powerful physiological signals to involve in future automated real-time affective recognition systems, especially for detecting the "relaxation" vs. "stress" states.^

Effects of real -time captioning and sign language interpreting on the learning of college students who are deaf or hard of hearing

College personnel are required to provide accommodations for students who are deaf and hard of hearing (D/HoH), but few empirical studies have been conducted on D/HoH students as they learn under the various accommodation conditions (sign language interpreting, SLI, real-time captioning, RTC, and both). Guided by the experiences of students who are D/HoH at Miami-Dade College (MDC) who requested RTC in addition to SLI as accommodations, the researcher adopted Merten’s transformative-emancipatory theoretical framework that values perceptions and voice of students who are D/HoH. A mixed methods design addressed two research questions: Did student learning differ for each accommodation? What did students experience while learning through accommodations? Participants included 30 students who were D/HoH (60% women). They represented MDC’s majority minority population: 10% White (non-Hispanic), 20% Black (non-Hispanic, including Haitian/Caribbean), 67% Hispanic, and 3% other. Hearing loss, ranged from severe-profound (70%) to mild-moderate (30%). All were able to communicate with American Sign Language: Learning was measured while students who were D/HoH viewed three lectures under three accommodation conditions (SLI, RTC, SLI+RTC). The learning measure was defined as the difference in pre- and post-test scores on tests of the content presented in the lectures. Using repeated measure ANOVA and ANCOVA, confounding variables of fluency in American Sign Language and literacy skills were treated as covariates. Perceptions were obtained through interviews and verbal protocol analysis that were signed, videotaped, transcribed, coded, and examined for common themes and metacognitive strategies. No statistically significant differences were found among the three accommodations on the learning measure. Students who were D/HoH expressed thoughts about five different aspects of their learning while they viewed lectures: (a) comprehending the information, (b) feeling a part of the classroom environment, (c) past experiences with an accommodation, (d) individual preferences for an accommodation, (e) suggestions for improving an accommodation. They exhibited three metacognitive strategies: (a) constructing knowledge, (b) monitoring comprehension, and (c) evaluating information. No patterns were found in the types of metacognitive strategies used for any particular accommodation. The researcher offers recommendations for flexible applications of the standard accommodations used with students who are D/HoH.

Off-line and On-line Affective Recognition of a Computer User through A Biosignal Processing Approach

Physiological signals, which are controlled by the autonomic nervous system (ANS), could be used to detect the affective state of computer users and therefore find applications in medicine and engineering. The Pupil Diameter (PD) seems to provide a strong indication of the affective state, as found by previous research, but it has not been investigated fully yet. In this study, new approaches based on monitoring and processing the PD signal for off-line and on-line affective assessment (“relaxation” vs. “stress”) are proposed. Wavelet denoising and Kalman filtering methods are first used to remove abrupt changes in the raw Pupil Diameter (PD) signal. Then three features (PDmean, PDmax and PDWalsh) are extracted from the preprocessed PD signal for the affective state classification. In order to select more relevant and reliable physiological data for further analysis, two types of data selection methods are applied, which are based on the paired t-test and subject self-evaluation, respectively. In addition, five different kinds of the classifiers are implemented on the selected data, which achieve average accuracies up to 86.43% and 87.20%, respectively. Finally, the receiver operating characteristic (ROC) curve is utilized to investigate the discriminating potential of each individual feature by evaluation of the area under the ROC curve, which reaches values above 0.90. For the on-line affective assessment, a hard threshold is implemented first in order to remove the eye blinks from the PD signal and then a moving average window is utilized to obtain the representative value PDr for every one-second time interval of PD. There are three main steps for the on-line affective assessment algorithm, which are preparation, feature-based decision voting and affective determination. The final results show that the accuracies are 72.30% and 73.55% for the data subsets, which were respectively chosen using two types of data selection methods (paired t-test and subject self-evaluation). In order to further analyze the efficiency of affective recognition through the PD signal, the Galvanic Skin Response (GSR) was also monitored and processed. The highest affective assessment classification rate obtained from GSR processing is only 63.57% (based on the off-line processing algorithm). The overall results confirm that the PD signal should be considered as one of the most powerful physiological signals to involve in future automated real-time affective recognition systems, especially for detecting the “relaxation” vs. “stress” states.