A digital signal processing approach for affective sensing of a computer user through pupil diameter monitoring

Recent research has indicated that the pupil diameter (PD) in humans varies with their affective states. However, this signal has not been fully investigated for affective sensing purposes in human-computer interaction systems. This may be due to the dominant separate effect of the pupillary light reflex (PLR), which shrinks the pupil when light intensity increases. In this dissertation, an adaptive interference canceller (AIC) system using the H∞ time-varying (HITV) adaptive algorithm was developed to minimize the impact of the PLR on the measured pupil diameter signal. The modified pupil diameter (MPD) signal, obtained from the AIC was expected to reflect primarily the pupillary affective responses (PAR) of the subject. Additional manipulations of the AIC output resulted in a processed MPD (PMPD) signal, from which a classification feature, PMPDmean, was extracted. This feature was used to train and test a support vector machine (SVM), for the identification of stress states in the subject from whom the pupil diameter signal was recorded, achieving an accuracy rate of 77.78%. The advantages of affective recognition through the PD signal were verified by comparatively investigating the classification of stress and relaxation states through features derived from the simultaneously recorded galvanic skin response (GSR) and blood volume pulse (BVP) signals, with and without the PD feature. The discriminating potential of each individual feature extracted from GSR, BVP and PD was studied by analysis of its receiver operating characteristic (ROC) curve. The ROC curve found for the PMPDmean feature encompassed the largest area (0.8546) of all the single-feature ROCs investigated. The encouraging results seen in affective sensing based on pupil diameter monitoring were obtained in spite of intermittent illumination increases purposely introduced during the experiments. Therefore, these results confirmed the benefits of using the AIC implementation with the HITV adaptive algorithm to isolate the PAR and the potential of using PD monitoring to sense the evolving affective states of a computer user.

Digital signal processing implementation for near real-time wavelet transformation system for binary images

Communication has become an essential function in our civilization. With the increasing demand for communication channels, it is now necessary to find ways to optimize the use of their bandwidth. One way to achieve this is by transforming the information before it is transmitted. This transformation can be performed by several techniques. One of the newest of these techniques is the use of wavelets. Wavelet transformation refers to the act of breaking down a signal into components called details and trends by using small waveforms that have a zero average in the time domain. After this transformation the data can be compressed by discarding the details, transmitting the trends. In the receiving end, the trends are used to reconstruct the image. In this work, the wavelet used for the transformation of an image will be selected from a library of available bases. The accuracy of the reconstruction, after the details are discarded, is dependent on the wavelets chosen from the wavelet basis library. The system developed in this thesis takes a 2-D image and decomposes it using a wavelet bank. A digital signal processor is used to achieve near real-time performance in this transformation task. A contribution of this thesis project is the development of DSP-based test bed for the future development of new real-time wavelet transformation algorithms.

A Digital Signal Processing Approach for Affective Sensing of a Computer User through Pupil Diameter Monitoring

Recent research has indicated that the pupil diameter (PD) in humans varies with their affective states. However, this signal has not been fully investigated for affective sensing purposes in human-computer interaction systems. This may be due to the dominant separate effect of the pupillary light reflex (PLR), which shrinks the pupil when light intensity increases. In this dissertation, an adaptive interference canceller (AIC) system using the H∞ time-varying (HITV) adaptive algorithm was developed to minimize the impact of the PLR on the measured pupil diameter signal. The modified pupil diameter (MPD) signal, obtained from the AIC was expected to reflect primarily the pupillary affective responses (PAR) of the subject. Additional manipulations of the AIC output resulted in a processed MPD (PMPD) signal, from which a classification feature, PMPDmean, was extracted. This feature was used to train and test a support vector machine (SVM), for the identification of stress states in the subject from whom the pupil diameter signal was recorded, achieving an accuracy rate of 77.78%. The advantages of affective recognition through the PD signal were verified by comparatively investigating the classification of stress and relaxation states through features derived from the simultaneously recorded galvanic skin response (GSR) and blood volume pulse (BVP) signals, with and without the PD feature. The discriminating potential of each individual feature extracted from GSR, BVP and PD was studied by analysis of its receiver operating characteristic (ROC) curve. The ROC curve found for the PMPDmean feature encompassed the largest area (0.8546) of all the single-feature ROCs investigated. The encouraging results seen in affective sensing based on pupil diameter monitoring were obtained in spite of intermittent illumination increases purposely introduced during the experiments. Therefore, these results confirmed the benefits of using the AIC implementation with the HITV adaptive algorithm to isolate the PAR and the potential of using PD monitoring to sense the evolving affective states of a computer user.

Digital surveillance based on video CODEC System-on-a-Chip (SoC) platforms

Today, most conventional surveillance networks are based on analog system, which has a lot of constraints like manpower and high-bandwidth requirements. It becomes the barrier for today's surveillance network development. This dissertation describes a digital surveillance network architecture based on the H.264 coding/decoding (CODEC) System-on-a-Chip (SoC) platform. The proposed digital surveillance network architecture includes three major layers: software layer, hardware layer, and the network layer. The following outlines the contributions to the proposed digital surveillance network architecture. (1) We implement an object recognition system and an object categorization system on the software layer by applying several Digital Image Processing (DIP) algorithms. (2) For better compression ratio and higher video quality transfer, we implement two new modules on the hardware layer of the H.264 CODEC core, i.e., the background elimination module and the Directional Discrete Cosine Transform (DDCT) module. (3) Furthermore, we introduce a Digital Signal Processor (DSP) sub-system on the main bus of H.264 SoC platforms as the major hardware support system for our software architecture. Thus we combine the software and hardware platforms to be an intelligent surveillance node. Lab results show that the proposed surveillance node can dramatically save the network resources like bandwidth and storage capacity.

Development of an open architecture motion control system

A multipurpose open architecture motion control system was developed with three platforms for control and monitoring. The Visual Basic user interface communicated with the operator and gave instructions to the electronic components. The first platform had a BASIC Stamp based controller and three stepping motors. The second platform had a controller, amplifiers and two DC servomotors. The third platform had a DSP module. In this study, each platform was used on machine tools either to move the table or to evaluate the incoming signal. The study indicated that by using advanced microcontrollers, which use high-level languages, motor controllers, DSPs (Digital Signal Processor) and microcomputers, the motion control of different systems could be realized in a short time. Although, the proposed systems had some limitations, their jobs were performed effectively. ^

A nanoliter volume nuclear magnetic resonance (NMR) system using tunneling magneto-resistive (TMR) sensors to recognize biomolecules

The need to incorporate advanced engineering tools in biology, biochemistry and medicine is in great demand. Many of the existing instruments and tools are usually expensive and require special facilities.^ With the advent of nanotechnology in the past decade, new approaches to develop devices and tools have been generated by academia and industry. ^ One such technology, NMR spectroscopy, has been used by biochemists for more than 2 decades to study the molecular structure of chemical compounds. However, NMR spectrometers are very expensive and require special laboratory rooms for their proper operation. High magnetic fields with strengths in the order of several Tesla make these instruments unaffordable to most research groups.^ This doctoral research proposes a new technology to develop NMR spectrometers that can operate at field strengths of less than 0.5 Tesla using an inexpensive permanent magnet and spin dependent nanoscale magnetic devices. This portable NMR system is intended to analyze samples as small as a few nanoliters.^ The main problem to resolve when downscaling the variables is to obtain an NMR signal with high Signal-To-Noise-Ratio (SNR). A special Tunneling Magneto-Resistive (TMR) sensor design was developed to achieve this goal. The minimum specifications for each component of the proposed NMR system were established. A complete NMR system was designed based on these minimum requirements. The goat was always to find cost effective realistic components. The novel design of the NMR system uses technologies such as Direct Digital Synthesis (DDS), Digital Signal Processing (DSP) and a special Backpropagation Neural Network that finds the best match of the NMR spectrum. The system was designed, calculated and simulated with excellent results.^ In addition, a general method to design TMR Sensors was developed. The technique was automated and a computer program was written to help the designer perform this task interactively.^

Efficiency and power density improvement of grid-connected hybrid renewable energy systems utilizing high frequency-based power converters

High efficiency of power converters placed between renewable energy sources and the utility grid is required to maximize the utilization of these sources. Power quality is another aspect that requires large passive elements (inductors, capacitors) to be placed between these sources and the grid. The main objective is to develop higher-level high frequency-based power converter system (HFPCS) that optimizes the use of hybrid renewable power injected into the power grid. The HFPCS provides high efficiency, reduced size of passive components, higher levels of power density realization, lower harmonic distortion, higher reliability, and lower cost. The dynamic modeling for each part in this system is developed, simulated and tested. The steady-state performance of the grid-connected hybrid power system with battery storage is analyzed. Various types of simulations were performed and a number of algorithms were developed and tested to verify the effectiveness of the power conversion topologies. A modified hysteresis-control strategy for the rectifier and the battery charging/discharging system was developed and implemented. A voltage oriented control (VOC) scheme was developed to control the energy injected into the grid. The developed HFPCS was compared experimentally with other currently available power converters. The developed HFPCS was employed inside a microgrid system infrastructure, connecting it to the power grid to verify its power transfer capabilities and grid connectivity. Grid connectivity tests verified these power transfer capabilities of the developed converter in addition to its ability of serving the load in a shared manner. In order to investigate the performance of the developed system, an experimental setup for the HF-based hybrid generation system was constructed. We designed a board containing a digital signal processor chip on which the developed control system was embedded. The board was fabricated and experimentally tested. The system's high precision requirements were verified. Each component of the system was built and tested separately, and then the whole system was connected and tested. The simulation and experimental results confirm the effectiveness of the developed converter system for grid-connected hybrid renewable energy systems as well as for hybrid electric vehicles and other industrial applications.

Development of New Structural Health Monitoring Techniques

During the past two decades, many researchers have developed methods for the detection of structural defects at the early stages to operate the aerospace vehicles safely and to reduce the operating costs. The Surface Response to Excitation (SuRE) method is one of these approaches developed at FIU to reduce the cost and size of the equipment. The SuRE method excites the surface at a series of frequencies and monitors the propagation characteristics of the generated waves. The amplitude of the waves reaching to any point on the surface varies with frequency; however, it remains consistent as long as the integrity and strain distribution on the part is consistent. These spectral characteristics change when cracks develop or the strain distribution changes. The SHM methods may be used for many applications, from the detection of loose screws to the monitoring of manufacturing operations. A scanning laser vibrometer was used in this study to investigate the characteristics of the spectral changes at different points on the parts. The study started with detecting a load on a plate and estimating its location. The modifications on the part with manufacturing operations were detected and the Part-Based Manufacturing Process Performance Monitoring (PbPPM) method was developed. Hardware was prepared to demonstrate the feasibility of the proposed methods in real time. Using low-cost piezoelectric elements and the non-contact scanning laser vibrometer successfully, the data was collected for the SuRE and PbPPM methods. Locational force, loose bolts and material loss could be easily detected by comparing the spectral characteristics of the arriving waves. On-line methods used fast computational methods for estimating the spectrum and detecting the changing operational conditions from sum of the squares of the variations. Neural networks classified the spectrums when the desktop – DSP combination was used. The results demonstrated the feasibility of the SuRE and PbPPM methods.

Development of New Structural Health Monitoring Techniques

During the past two decades, many researchers have developed methods for the detection of structural defects at the early stages to operate the aerospace vehicles safely and to reduce the operating costs. The Surface Response to Excitation (SuRE) method is one of these approaches developed at FIU to reduce the cost and size of the equipment. The SuRE method excites the surface at a series of frequencies and monitors the propagation characteristics of the generated waves. The amplitude of the waves reaching to any point on the surface varies with frequency; however, it remains consistent as long as the integrity and strain distribution on the part is consistent. These spectral characteristics change when cracks develop or the strain distribution changes. The SHM methods may be used for many applications, from the detection of loose screws to the monitoring of manufacturing operations. A scanning laser vibrometer was used in this study to investigate the characteristics of the spectral changes at different points on the parts. The study started with detecting a load on a plate and estimating its location. The modifications on the part with manufacturing operations were detected and the Part-Based Manufacturing Process Performance Monitoring (PbPPM) method was developed. Hardware was prepared to demonstrate the feasibility of the proposed methods in real time. Using low-cost piezoelectric elements and the non-contact scanning laser vibrometer successfully, the data was collected for the SuRE and PbPPM methods. Locational force, loose bolts and material loss could be easily detected by comparing the spectral characteristics of the arriving waves. On-line methods used fast computational methods for estimating the spectrum and detecting the changing operational conditions from sum of the squares of the variations. Neural networks classified the spectrums when the desktop – DSP combination was used. The results demonstrated the feasibility of the SuRE and PbPPM methods.

Optimization of adaptive traffic signal control with transit signal priority at isolated intersections using parallel genetic algorithms

Optimization of adaptive traffic signal timing is one of the most complex problems in traffic control systems. This dissertation presents a new method that applies the parallel genetic algorithm (PGA) to optimize adaptive traffic signal control in the presence of transit signal priority (TSP). The method can optimize the phase plan, cycle length, and green splits at isolated intersections with consideration for the performance of both the transit and the general vehicles. Unlike the simple genetic algorithm (GA), PGA can provide better and faster solutions needed for real-time optimization of adaptive traffic signal control. ^ An important component in the proposed method involves the development of a microscopic delay estimation model that was designed specifically to optimize adaptive traffic signal with TSP. Macroscopic delay models such as the Highway Capacity Manual (HCM) delay model are unable to accurately consider the effect of phase combination and phase sequence in delay calculations. In addition, because the number of phases and the phase sequence of adaptive traffic signal may vary from cycle to cycle, the phase splits cannot be optimized when the phase sequence is also a decision variable. A "flex-phase" concept was introduced in the proposed microscopic delay estimation model to overcome these limitations. ^ The performance of PGA was first evaluated against the simple GA. The results show that PGA achieved both faster convergence and lower delay for both under- or over-saturated traffic conditions. A VISSIM simulation testbed was then developed to evaluate the performance of the proposed PGA-based adaptive traffic signal control with TSP. The simulation results show that the PGA-based optimizer for adaptive TSP outperformed the fully actuated NEMA control in all test cases. The results also show that the PGA-based optimizer was able to produce TSP timing plans that benefit the transit vehicles while minimizing the impact of TSP on the general vehicles. The VISSIM testbed developed in this research provides a powerful tool to design and evaluate different TSP strategies under both actuated and adaptive signal control. ^

Signal processing of physiological signals for automated assessment of stress in computer users

The need to provide computers with the ability to distinguish the affective state of their users is a major requirement for the practical implementation of affective computing concepts. This dissertation proposes the application of signal processing methods on physiological signals to extract from them features that can be processed by learning pattern recognition systems to provide cues about a person's affective state. In particular, combining physiological information sensed from a user's left hand in a non-invasive way with the pupil diameter information from an eye-tracking system may provide a computer with an awareness of its user's affective responses in the course of human-computer interactions. In this study an integrated hardware-software setup was developed to achieve automatic assessment of the affective status of a computer user. A computer-based "Paced Stroop Test" was designed as a stimulus to elicit emotional stress in the subject during the experiment. Four signals: the Galvanic Skin Response (GSR), the Blood Volume Pulse (BVP), the Skin Temperature (ST) and the Pupil Diameter (PD), were monitored and analyzed to differentiate affective states in the user. Several signal processing techniques were applied on the collected signals to extract their most relevant features. These features were analyzed with learning classification systems, to accomplish the affective state identification. Three learning algorithms: Naïve Bayes, Decision Tree and Support Vector Machine were applied to this identification process and their levels of classification accuracy were compared. The results achieved indicate that the physiological signals monitored do, in fact, have a strong correlation with the changes in the emotional states of the experimental subjects. These results also revealed that the inclusion of pupil diameter information significantly improved the performance of the emotion recognition system. ^

Signal processing and information fusing algorithms for the synthesis of an alternative electromyogram/eye gaze tracking computer cursor control system

This research pursued the conceptualization and real-time verification of a system that allows a computer user to control the cursor of a computer interface without using his/her hands. The target user groups for this system are individuals who are unable to use their hands due to spinal dysfunction or other afflictions, and individuals who must use their hands for higher priority tasks while still requiring interaction with a computer. ^ The system receives two forms of input from the user: Electromyogram (EMG) signals from muscles in the face and point-of-gaze coordinates produced by an Eye Gaze Tracking (EGT) system. In order to produce reliable cursor control from the two forms of user input, the development of this EMG/EGT system addressed three key requirements: an algorithm was created to accurately translate EMG signals due to facial movements into cursor actions, a separate algorithm was created that recognized an eye gaze fixation and provided an estimate of the associated eye gaze position, and an information fusion protocol was devised to efficiently integrate the outputs of these algorithms. ^ Experiments were conducted to compare the performance of EMG/EGT cursor control to EGT-only control and mouse control. These experiments took the form of two different types of point-and-click trials. The data produced by these experiments were evaluated using statistical analysis, Fitts' Law analysis and target re-entry (TRE) analysis. ^ The experimental results revealed that though EMG/EGT control was slower than EGT-only and mouse control, it provided effective hands-free control of the cursor without a spatial accuracy limitation, and it also facilitated a reliable click operation. This combination of qualities is not possessed by either EGT-only or mouse control, making EMG/EGT cursor control a unique and practical alternative for a user's cursor control needs. ^

Neuroendocrine control of a dynamic communication signal

Multiple physiological systems regulate the electric communication signal of the weakly electric gymnotiform fish, Brachyhypopomus pinnicaudatus. Fish were injected with neuroendocrine probes which identified pharmacologically relevant serotonin (5-HT) receptors similar to the mammalian 5-HT1AR and 5-HT2AR. Peptide hormones of the hypothalamic-pituitary-adrenal/interrenal axis also augment the electric waveform. These results indicate that the central serotonergic system interacts with the hypothalamic-pituitary-interrenal system to regulate communication signals in this species. The same neuroendocrine probes were tested in females before and after introducing androgens to examine the relationship between sex steroid hormones, the serotonergic system, melanocortin peptides, and EOD modulations. Androgens caused an increase in female B. pinnicaudatus responsiveness to other pharmacological challenges, particularly to the melanocortin peptide adrenocorticotropic hormone (ACTH). A forced social challenge paradigm was administered to determine if androgens are responsible for controlling the signal modulations these fish exhibit when they encounter conspecifics. Males and females responded similarly to this social challenge construct, however introducing androgens caused implanted females to produce more exaggerated responses. These results confirm that androgens enhance an individual's capacity to produce an exaggerated response to challenge, however another unidentified factor appears to regulate sex-specific behaviors in this species. These results suggest that the rapid electric waveform modulations B. pinnicaudatus produces in response to conspecifics are situation-specific and controlled by activation of different serotonin receptor types and the subsequent effect on release of pituitary hormones.

Social and environmental regulation of signal plasticity and signal reliability in the electric fish Brachyhypopomus gauderio

The balance between the costs and benefits of conspicuous signals ensures that the expression of those signals is related to the quality of the bearer. Plastic signals could enable males to maximize conspicuous traits to impress mates and competitors, but reduce the expression of those traits to minimize signaling costs, potentially compromising the information conveyed by the signals. ^ I investigated the effect of signal enhancement on the information coded by the biphasic electric signal pulse of the gymnotiform fish Brachyhypopomus gauderio. Increases in population density drive males to enhance the amplitude of their signals. I found that signal amplitude enhancement improves the information about the signaler's size. Furthermore, I found that the elongation of the signal's second phase conveys information about androgen levels in both sexes, gonad size in males and estrogen levels in females. Androgens link the duration of the signal's second phase to other androgen-mediated traits making the signal an honest indicator of reproductive state and aggressive motivation. ^ Signal amplitude enhancement facilitates the assessment of the signaler's resource holding potential, important for male-male interactions, while signal duration provides information about aggressive motivation to same-sex competitors and reproductive state to the opposite sex. Moreover, I found that female signals also change in accordance to the social environment. Females also increase the amplitude of their signal when population density increases and elongate the duration of their signal's second phase when the sex ratio becomes female-biased. Indicating that some degree of sexual selection operates in females. ^ I studied whether male B. gauderio use signal plasticity to reduce the cost of reproductive signaling when energy is limited. Surprisingly, I found that food limitation promotes the investment in reproduction manifested as signal enhancement and elevated androgen levels. The short lifespan and single breeding season of B. gauderio diminishes the advantage of energy savings and gives priority to sustaining reproduction. I conclude that the electric signal of B. gauderio provides reliable information about the signaler, the quality of this information is reinforced rather than degraded with signal enhancement.^

A theoretical analysis of eyewitness identification: Dual -process theory, signal detection theory and eyewitness confidence

Given the growing number of wrongful convictions involving faulty eyewitness evidence and the strong reliance by jurors on eyewitness testimony, researchers have sought to develop safeguards to decrease erroneous identifications. While decades of eyewitness research have led to numerous recommendations for the collection of eyewitness evidence, less is known regarding the psychological processes that govern identification responses. The purpose of the current research was to expand the theoretical knowledge of eyewitness identification decisions by exploring two separate memory theories: signal detection theory and dual-process theory. This was accomplished by examining both system and estimator variables in the context of a novel lineup recognition paradigm. Both theories were also examined in conjunction with confidence to determine whether it might add significantly to the understanding of eyewitness memory. ^ In two separate experiments, both an encoding and a retrieval-based manipulation were chosen to examine the application of theory to eyewitness identification decisions. Dual-process estimates were measured through the use of remember-know judgments (Gardiner & Richardson-Klavehn, 2000). In Experiment 1, the effects of divided attention and lineup presentation format (simultaneous vs. sequential) were examined. In Experiment 2, perceptual distance and lineup response deadline were examined. Overall, the results indicated that discrimination and remember judgments (recollection) were generally affected by variations in encoding quality and response criterion and know judgments (familiarity) were generally affected by variations in retrieval options. Specifically, as encoding quality improved, discrimination ability and judgments of recollection increased; and as the retrieval task became more difficult there was a shift toward lenient choosing and more reliance on familiarity. ^ The application of signal detection theory and dual-process theory in the current experiments produced predictable results on both system and estimator variables. These theories were also compared to measures of general confidence, calibration, and diagnosticity. The application of the additional confidence measures in conjunction with signal detection theory and dual-process theory gave a more in-depth explanation than either theory alone. Therefore, the general conclusion is that eyewitness identifications can be understood in a more complete manor by applying theory and examining confidence. Future directions and policy implications are discussed. ^