A digital signal processing approach for affective sensing of a computer user through pupil diameter monitoring

Recent research has indicated that the pupil diameter (PD) in humans varies with their affective states. However, this signal has not been fully investigated for affective sensing purposes in human-computer interaction systems. This may be due to the dominant separate effect of the pupillary light reflex (PLR), which shrinks the pupil when light intensity increases. In this dissertation, an adaptive interference canceller (AIC) system using the H∞ time-varying (HITV) adaptive algorithm was developed to minimize the impact of the PLR on the measured pupil diameter signal. The modified pupil diameter (MPD) signal, obtained from the AIC was expected to reflect primarily the pupillary affective responses (PAR) of the subject. Additional manipulations of the AIC output resulted in a processed MPD (PMPD) signal, from which a classification feature, PMPDmean, was extracted. This feature was used to train and test a support vector machine (SVM), for the identification of stress states in the subject from whom the pupil diameter signal was recorded, achieving an accuracy rate of 77.78%. The advantages of affective recognition through the PD signal were verified by comparatively investigating the classification of stress and relaxation states through features derived from the simultaneously recorded galvanic skin response (GSR) and blood volume pulse (BVP) signals, with and without the PD feature. The discriminating potential of each individual feature extracted from GSR, BVP and PD was studied by analysis of its receiver operating characteristic (ROC) curve. The ROC curve found for the PMPDmean feature encompassed the largest area (0.8546) of all the single-feature ROCs investigated. The encouraging results seen in affective sensing based on pupil diameter monitoring were obtained in spite of intermittent illumination increases purposely introduced during the experiments. Therefore, these results confirmed the benefits of using the AIC implementation with the HITV adaptive algorithm to isolate the PAR and the potential of using PD monitoring to sense the evolving affective states of a computer user.

A Digital Signal Processing Approach for Affective Sensing of a Computer User through Pupil Diameter Monitoring

Recent research has indicated that the pupil diameter (PD) in humans varies with their affective states. However, this signal has not been fully investigated for affective sensing purposes in human-computer interaction systems. This may be due to the dominant separate effect of the pupillary light reflex (PLR), which shrinks the pupil when light intensity increases. In this dissertation, an adaptive interference canceller (AIC) system using the H∞ time-varying (HITV) adaptive algorithm was developed to minimize the impact of the PLR on the measured pupil diameter signal. The modified pupil diameter (MPD) signal, obtained from the AIC was expected to reflect primarily the pupillary affective responses (PAR) of the subject. Additional manipulations of the AIC output resulted in a processed MPD (PMPD) signal, from which a classification feature, PMPDmean, was extracted. This feature was used to train and test a support vector machine (SVM), for the identification of stress states in the subject from whom the pupil diameter signal was recorded, achieving an accuracy rate of 77.78%. The advantages of affective recognition through the PD signal were verified by comparatively investigating the classification of stress and relaxation states through features derived from the simultaneously recorded galvanic skin response (GSR) and blood volume pulse (BVP) signals, with and without the PD feature. The discriminating potential of each individual feature extracted from GSR, BVP and PD was studied by analysis of its receiver operating characteristic (ROC) curve. The ROC curve found for the PMPDmean feature encompassed the largest area (0.8546) of all the single-feature ROCs investigated. The encouraging results seen in affective sensing based on pupil diameter monitoring were obtained in spite of intermittent illumination increases purposely introduced during the experiments. Therefore, these results confirmed the benefits of using the AIC implementation with the HITV adaptive algorithm to isolate the PAR and the potential of using PD monitoring to sense the evolving affective states of a computer user.

Consumer (dis)satisfaction response to disconfirmation: An examination of the form and the underlying process

The purpose of this dissertation was to examine the form of the consumer satisfaction/dissatisfaction (CS/D) response to disconfirmation. In addition, the cognitive and affective processes underlying the response were also explored. ^ Respondents were provided with information from a prior market research study about a new brand of printer that was being tested. This market research information helped set prior expectations regarding the print quality. Subjects were randomly assigned to an experimental condition that manipulated prior expectations to be either positive or negative. Respondents were then provided with printouts that had performance quality that was either worse (negative disconfirmation) or better (positive disconfirmation) than the prior expectations. In other words, for each level of expectation, respondents were assigned to either positive or negative disconfirmation condition. Subjects were also randomly assigned to a condition of either a high or low level of outcome involvement. ^ Analyses of variance indicated that positive disconfirmation led to a more intense CS/D response than negative disconfirmation, even though there was no significant difference in the intensity for positive and negative disconfirmation. Intensity of CS/D was measured by the distance of the CS/D rating from the midpoint of the scale. The study also found that although outcome involvement did not influence the polarity of the CS/D response, the more direct measures of processing involvement such as the subjects' concentration, attention and care in evaluating the printout did have a significant positive effect on CS/D intensity. ^ Analyses of covariance also indicated that the relationship between the intensity of the CS/D response and the intensity of the disconfirmation was mediated by the intensity of affective responses. Positive disconfirmation led to more intense affective responses than negative disconfirmation. ^

Signal processing of physiological signals for automated assessment of stress in computer users

The need to provide computers with the ability to distinguish the affective state of their users is a major requirement for the practical implementation of affective computing concepts. This dissertation proposes the application of signal processing methods on physiological signals to extract from them features that can be processed by learning pattern recognition systems to provide cues about a person's affective state. In particular, combining physiological information sensed from a user's left hand in a non-invasive way with the pupil diameter information from an eye-tracking system may provide a computer with an awareness of its user's affective responses in the course of human-computer interactions. In this study an integrated hardware-software setup was developed to achieve automatic assessment of the affective status of a computer user. A computer-based "Paced Stroop Test" was designed as a stimulus to elicit emotional stress in the subject during the experiment. Four signals: the Galvanic Skin Response (GSR), the Blood Volume Pulse (BVP), the Skin Temperature (ST) and the Pupil Diameter (PD), were monitored and analyzed to differentiate affective states in the user. Several signal processing techniques were applied on the collected signals to extract their most relevant features. These features were analyzed with learning classification systems, to accomplish the affective state identification. Three learning algorithms: Naïve Bayes, Decision Tree and Support Vector Machine were applied to this identification process and their levels of classification accuracy were compared. The results achieved indicate that the physiological signals monitored do, in fact, have a strong correlation with the changes in the emotional states of the experimental subjects. These results also revealed that the inclusion of pupil diameter information significantly improved the performance of the emotion recognition system. ^

The role of contingency and gaze direction in the emergence of social referencing

The current study assessed the importance of infant detection of contingency and head and eye gaze direction in the emergence of social referencing. Five- to six-month-old infants' detection of affect-object relations and subsequent manual preferences for objects paired with positive expressions were assessed. In particular, the role of contingency between toys' movements and an actress's emotional expressions as well as the role of gaze direction toward the toys' location were examined. Infants were habituated to alternating films of two toys each paired with an actress's affective expression (happy and fearful) under contingent or noncontingent and gaze congruent or gaze incongruent conditions. Results indicated that gaze congruence and contingency between toys' movements and a person's affective expressions were important for infant perception of affect-object relations. Furthermore, infant perception of the relation between affective expressions and toys translated to their manual preferences for the 3-dimensional toys. Infants who received contingent affective responses to the movements of the toys spent more time touching the toy that was previously paired with the positive expression. These findings demonstrate the role of contingency and gaze direction in the emergence of social referencing in the first half year of life.^

Adolescent readers' self-perceptions of affective factors related to performance on standardized tests

The purpose of this study was to investigate which affective factors of adolescent high school readers were related to high-level readers, middle-level readers and low-level readers. The research problem was to determine the relationship between adolescent high school students' self-perceived reading self-efficacy factors and the students' reading performance on a standardized reading assessment considering demographic factors of age, gender and socio-economic status as covariates. The research design was ex post facto making inferences without direct intervention. The sample was obtained from one large, diverse, urban high school, consisting of 9th and 10th grade adolescent students (N = 176). Students voluntarily completed a self-report, reading self-efficacy survey. School records were used to obtain standardized reading level scores, age, gender, and socio-economic status data. An exploratory factor analysis of the self-efficacy survey responses resulted in the identification of 7 underlying factors. The striving (low-level) readers had significantly lower self-perceptions on 5 of the 7 affective factors than the middle-level readers, and strong (high-level) readers, p < .05. The 5 affective factors on which the striving readers had significantly lower self-perceptions were: (a) Observational Comparison, (b) Progress, (c) Lack of Progress, (d) Lack of Anxiety, and (e) Positive Social Feedback. The 2 affective factors which were not significantly different for reader level were Anxiety and Negative Social Feedback. Girls had significantly less anxiety than boys for both of the factors in the Anxiety category. Statistical results showed that none of the demographic covariates tested; age, gender, or socio-economic status, moderated the relationship between affective reader self-efficacy factors and reader level. This study concluded that there were distinguishable differences for striving, middle, and strong readers' self-efficacy factors. Determining affective factors related to reading can be used to create better instructional environments and instruction for adolescent students.