Learning Data-Driven Models of Non-Verbal Behaviors for Building Rapport Using an Intelligent Virtual Agent

There is a growing societal need to address the increasing prevalence of behavioral health issues, such as obesity, alcohol or drug use, and general lack of treatment adherence for a variety of health problems. The statistics, worldwide and in the USA, are daunting. Excessive alcohol use is the third leading preventable cause of death in the United States (with 79,000 deaths annually), and is responsible for a wide range of health and social problems. On the positive side though, these behavioral health issues (and associated possible diseases) can often be prevented with relatively simple lifestyle changes, such as losing weight with a diet and/or physical exercise, or learning how to reduce alcohol consumption. Medicine has therefore started to move toward finding ways of preventively promoting wellness, rather than solely treating already established illness. Evidence-based patient-centered Brief Motivational Interviewing (BMI) interven- tions have been found particularly effective in helping people find intrinsic motivation to change problem behaviors after short counseling sessions, and to maintain healthy lifestyles over the long-term. Lack of locally available personnel well-trained in BMI, however, often limits access to successful interventions for people in need. To fill this accessibility gap, Computer-Based Interventions (CBIs) have started to emerge. Success of the CBIs, however, critically relies on insuring engagement and retention of CBI users so that they remain motivated to use these systems and come back to use them over the long term as necessary. Because of their text-only interfaces, current CBIs can therefore only express limited empathy and rapport, which are the most important factors of health interventions. Fortunately, in the last decade, computer science research has progressed in the design of simulated human characters with anthropomorphic communicative abilities. Virtual characters interact using humans’ innate communication modalities, such as facial expressions, body language, speech, and natural language understanding. By advancing research in Artificial Intelligence (AI), we can improve the ability of artificial agents to help us solve CBI problems. To facilitate successful communication and social interaction between artificial agents and human partners, it is essential that aspects of human social behavior, especially empathy and rapport, be considered when designing human-computer interfaces. Hence, the goal of the present dissertation is to provide a computational model of rapport to enhance an artificial agent’s social behavior, and to provide an experimental tool for the psychological theories shaping the model. Parts of this thesis were already published in [LYL+12, AYL12, AL13, ALYR13, LAYR13, YALR13, ALY14].

Learning data-driven models of non-verbal behaviors for building rapport using an intelligent virtual agent

There is a growing societal need to address the increasing prevalence of behavioral health issues, such as obesity, alcohol or drug use, and general lack of treatment adherence for a variety of health problems. The statistics, worldwide and in the USA, are daunting. Excessive alcohol use is the third leading preventable cause of death in the United States (with 79,000 deaths annually), and is responsible for a wide range of health and social problems. On the positive side though, these behavioral health issues (and associated possible diseases) can often be prevented with relatively simple lifestyle changes, such as losing weight with a diet and/or physical exercise, or learning how to reduce alcohol consumption. Medicine has therefore started to move toward finding ways of preventively promoting wellness, rather than solely treating already established illness.^ Evidence-based patient-centered Brief Motivational Interviewing (BMI) interventions have been found particularly effective in helping people find intrinsic motivation to change problem behaviors after short counseling sessions, and to maintain healthy lifestyles over the long-term. Lack of locally available personnel well-trained in BMI, however, often limits access to successful interventions for people in need. To fill this accessibility gap, Computer-Based Interventions (CBIs) have started to emerge. Success of the CBIs, however, critically relies on insuring engagement and retention of CBI users so that they remain motivated to use these systems and come back to use them over the long term as necessary.^ Because of their text-only interfaces, current CBIs can therefore only express limited empathy and rapport, which are the most important factors of health interventions. Fortunately, in the last decade, computer science research has progressed in the design of simulated human characters with anthropomorphic communicative abilities. Virtual characters interact using humans’ innate communication modalities, such as facial expressions, body language, speech, and natural language understanding. By advancing research in Artificial Intelligence (AI), we can improve the ability of artificial agents to help us solve CBI problems.^ To facilitate successful communication and social interaction between artificial agents and human partners, it is essential that aspects of human social behavior, especially empathy and rapport, be considered when designing human-computer interfaces. Hence, the goal of the present dissertation is to provide a computational model of rapport to enhance an artificial agent’s social behavior, and to provide an experimental tool for the psychological theories shaping the model. Parts of this thesis were already published in [LYL+12, AYL12, AL13, ALYR13, LAYR13, YALR13, ALY14].^

Detecting deception: Identifying differences in liars' and truth tellers' verbal strategies

Deception research has traditionally focused on three methods of identifying liars and truth tellers: observing non-verbal or behavioral cues, analyzing verbal cues, and monitoring changes in physiological arousal during polygraph tests. Research shows that observers are often incapable of discriminating between liars and truth tellers with better than chance accuracy when they use these methods. One possible explanation for observers' poor performance is that they are not properly applying existing lie detection methods. An alternative explanation is that the cues on which these methods — and observers' judgments — are based do not reliably discriminate between liars and truth tellers. It may be possible to identify more reliable cues, and potentially improve observers' ability to discriminate, by developing a better understanding of how liars and truth tellers try to tell a convincing story. ^ This research examined (a) the verbal strategies used by truthful and deceptive individuals during interviews concerning an assigned activity, and (b) observers' ability to discriminate between them based on their verbal strategies. In Experiment I, pre-interview instructions manipulated participants' expectations regarding verifiability; each participant was led to believe that the interviewer could check some types of details, but not others, before deciding whether the participant was being truthful or deceptive. Interviews were then transcribed and scored for quantity and type of information provided. In Experiment II, observers listened to a random sample of the Experiment I interviews and rendered veracity judgments; half of the observers were instructed to judge the interviews according to the verbal strategies used by liars and truth tellers and the other half were uninstructed. ^ Results of Experiment I indicate that liars and truth tellers use different verbal strategies, characterized by a differential amount of detail. Overall, truthful participants provided more information than deceptive participants. This effect was moderated by participants' expectations regarding verifiability such that truthful participants provided more information only with regard to verifiable details. Results of Experiment II indicate that observers instructed about liars' and truth tellers' verbal strategies identify them with greater accuracy than uninstructed observers. ^

Interpersonal Techniques Boost your "Hospitality Batting Average"

The ability to listen and comprehend the intrinsic meaning behind the words people are saying is an important skill for those in the hospitality industry. The author provides some prescriptions for "winning friends and influencing people.”

The differential effects of *American and British trial proceedings on juror decision-making

Seven basic elements differentiate British from American trial procedures: confining attorneys to their tables; dealing with objections outside the presence of the jury; resolving disagreements between attorneys prior to objections being made; presenting the defense opening statement at the close of the prosecution case; the judge directly questions witnesses and has a wider latitude in controlling the evidence; and the judge gives a summation of all the evidence presented to the jury (Fulero & Turner, 1997). The present experiment examined the influence of these different courtroom procedures, judges' non-verbal behavior, and evidence strength on juror decision-making. Using models of persuasion to understand how the varying elements may effect juror decision-making, it was predicted that trials following American courtroom procedures would be more distracting for jurors and as such, they would be more likely to rely on the peripheral cue of the judge's expectations for trial outcome as expressed in his nonverbal behavior. In trials following British procedures jurors should be less distracted and better able to scrutinize the strength of the evidence that in turn should minimize the influence of the judge's nonverbal behavior. Two hundred forty-five participants viewed a mock civil trial in which courtroom procedure, judge's nonverbal behavior, and evidence strength were varied. Analyses suggest that courtroom procedure and evidence strength influenced the direction of participants' verdicts, but that judge's nonverbal behavior did not have a direct impact on verdict preference. Judge's nonverbal behavior appeared to influence other measures related to verdict. Participants were more confident in their verdicts when they agreed with judge's nonverbal behavior and when they viewed British courtroom procedures. Participants were more likely to return estimates of the defendant's liability that reflected judge's nonverbal behavior and a congruency with evidence strength. Participants also recalled more facts in the British conditions than in the American conditions. These findings are interpreted as indicating the importance of the impact of trial procedures and of nonverbal influence. ^

Two -way direction of influence in mother-infant interaction as denoted by concurrent reciprocal learning processes

This series of 5 single-subject studies used the operant conditioning paradigm to investigate, within the two-way influence process, how (a) contingent infant attention can reinforce maternal verbal behaviors during a period of mother-infant interaction and under subsequent experimental manipulation. Differential reinforcement was used to determine if it is possible that an infant attending to the mother (denoted by head-turns towards the image of the mother plus eye contact) increases (reinforces) the mother's verbal response (to a cue from the infant) upon which the infant behavior is contingent. There was also (b) an evaluation during the contrived parent-infant interaction for concurrent operant learning of infant vocal behavior via contingent verbal responding (reinforcement) implemented by the mother. Further, it was noted (c) whether or not the mother reported being aware that her responses were influenced by the infant's behavior. Findings showed: the operant conditioning of the maternal verbal behaviors were reinforced by contingent infant attention; and the operant conditioning of infant vocalizations was reinforced by contingent maternal verbal behaviors. No parent reported (1) being aware of the increase in their verbal response reinforced during operant conditioning of parental behavior nor a decrease in those responses during the DRA reversal phase, or (2) noticing a contingency between infant's and mother's response. By binomial 1-tail tests, the verbal-behavior patterns of the 5 mothers were conditioned by infant reinforcement (p < 0.02) and, concurrently, the vocal-response patterns of the 5 infants were conditioned by maternal reinforcement (p < 0.02). A program of systematic empirical research on the determinants of concurrent conditioning within mother-child interaction may provide a way to evaluate the differential effectiveness of interventions aimed at improving parent-child interactions. The work conducted in the present study is one step in this direction. ^

Drawing as a technique to facilitate childrens memory

This study examined a technique to assist children to recall more information about witnessed events. Thirty-eight fourth-grade children from a public grade school in Miami Florida participated in the experiment. The participants watched a Red Cross demonstration and were interviewed one week later about details of the demonstration. All of the children were interviewed using a police style interview. In addition, half of the children were instructed to draw during the interview. The current study supported previous findings that the instruction to draw increased the amount of information recalled. The effect of drawing was greatest for high-visual events. In addition, the instruction to draw prompted an increase in non-verbal information, which had an unusually high accuracy rate.

An Analysis of the Proposition Density, Sentence and Clause Types, and Non-Finite Verbal Usage in Two College Textbooks

This study investigated the proposition density, sentence and clause type usage and non-finite verbal usage in two college textbooks. The teaching implications are presented.

Influence of religious coping on the substance use and HIV risk behaviors of recent Latino immigrants

This study examines the influence of acculturative stress on substance use and HIV risk behaviors among recent Latino immigrants. The central hypothesis of the study is that specific religious coping mechanisms influence the relationship that acculturative stress has on the substance use and HIV-risk behaviors of recent Latino immigrants. Within the Latino culture religiosity is a pervasive force, guiding attitudes, behaviors, and even social interactions. When controlling for education and socioeconomic status, Latinos have been found to use religious coping mechanisms more frequently than their Non-Latino White counterparts. In addition, less acculturated Latinos use religious coping strategies more frequently than those with higher levels of acculturation. Given its prominent role in Latino culture, it appears probable that this mechanism may prove to be influential during difficult life transitions, such as those experienced during the immigration process. This study examines the moderating influence of specific religious coping mechanisms on the relationship between acculturative stress and substance use/HIV risk behaviors of recent Latino immigrants. Analyses for the present study were conducted with wave 2 data from an ongoing longitudinal study investigating associations between pre-immigration factors and health behavior trajectories of recent Latino immigrants. Structural equation and zero-inflated Poisson modeling were implemented to test the specified models and examine the nature of the relationship among the variables. Moderating effects were found for negative religious coping. Higher levels of negative religious coping strengthened an inverse relationship between acculturative stress and substance use. Results also indicated direct relationships between religious coping mechanisms and substance use. External and positive religious coping were inversely related to substance use. Negative religious coping was positively related to substance use. This study aims to contribute knowledge of how religious coping influence's the adaptation process of recent Latino immigrants. Expanding scientific understanding as to the function and effect of these coping mechanisms could lead to enhanced culturally relevant approaches in service delivery among Latino populations. Furthermore this knowledge could inform research about specific cognitions and behaviors that need to be targeted in prevention and treatment programs with this population.

Evoking non-repertory verbal behavior across operant classes : the effects of motor echoic sign language training within the context of a motivating operation

The individual effects that echoic, mand, and sign language training procedures have on the acquisition of verbal behavior have been widely demonstrated, but more efficient strategies are still needed. This study combined all three treatment strategies into one treatment intervention in order to investigate the joint effects they may have on verbal behavior. Six participants took part in the study. Intervention totaled 1 hour/day for 5 days/week until mastery criterion for motor echoic behavior was achieved. Although motor echoic behavior were solely targeted for acquisition, significant increases in spontaneous motor mands were noted in all treatment participants. Additionally, 4 treatment participants also demonstrated significant gains in vocal echoics and spontaneous vocal mands. No significant increases were noted for the control participant. Results suggest that the aforementioned procedure may provide more efficient results as a first-step to teaching a functional repertoire of verbal behavior to developmentally delayed children.

An investigation of the relationships among self-care behaviors, health beliefs, and metabolic control in the non-insulin-dependent diabetic adult

Self-care and health beliefs have been found to be important concepts in the management of chronic diseases such as diabetes mellitus. Poor metabolic control has been associated with a higher incidence of complications in diabetic patients. This study sought to explore any relationships among perceptions of self-care behaviors, health beliefs and metabolic control. The sample consisted of 52 outpatients with non-insulin-dependent diabetes from a large teaching medical center. Interviews were done to obtain the patients' perceptions of their self-care behaviors, and their health beliefs concerning diabetes. Results of glycosylated hemoglobin and/or serum glucose levels were obtained from the medical records. Data were analyzed using Cochran-Mantel-Haenzel statistics, and Pearson's r. Results indicated no significant relationships among perceptions of self-care behaviors, health beliefs and metabolic control. Ethnicity, education and gender were found to be significantly associated with self-care behaviors and health beliefs.

Modeling of a Heat-Induced Buckling of Plates Using the Mesh-free Method

In the process of engineering design of structural shapes, the flat plate analysis results can be generalized to predict behaviors of complete structural shapes. In this case, the purpose of this project is to analyze a thin flat plate under conductive heat transfer and to simulate the temperature distribution, thermal stresses, total displacements, and buckling deformations. The current approach in these cases has been using the Finite Element Method (FEM), whose basis is the construction of a conforming mesh. In contrast, this project uses the mesh-free Scan Solve Method. This method eliminates the meshing limitation using a non-conforming mesh. I implemented this modeling process developing numerical algorithms and software tools to model thermally induced buckling. In addition, convergence analysis was achieved, and the results were compared with FEM. In conclusion, the results demonstrate that the method gives similar solutions to FEM in quality, but it is computationally less time consuming.

Ionized cluster beam deposition: Modeling, cluster size measurement and applications

Clusters are aggregations of atoms or molecules, generally intermediate in size between individual atoms and aggregates that are large enough to be called bulk matter. Clusters can also be called nanoparticles, because their size is on the order of nanometers or tens of nanometers. A new field has begun to take shape called nanostructured materials which takes advantage of these atom clusters. The ultra-small size of building blocks leads to dramatically different properties and it is anticipated that such atomically engineered materials will be able to be tailored to perform as no previous material could.^ The idea of ionized cluster beam (ICB) thin film deposition technique was first proposed by Takagi in 1972. It was based upon using a supersonic jet source to produce, ionize and accelerate beams of atomic clusters onto substrates in a vacuum environment. Conditions for formation of cluster beams suitable for thin film deposition have only recently been established following twenty years of effort. Zinc clusters over 1,000 atoms in average size have been synthesized both in our lab and that of Gspann. More recently, other methods of synthesizing clusters and nanoparticles, using different types of cluster sources, have come under development.^ In this work, we studied different aspects of nanoparticle beams. The work includes refinement of a model of the cluster formation mechanism, development of a new real-time, in situ cluster size measurement method, and study of the use of ICB in the fabrication of semiconductor devices.^ The formation process of the vaporized-metal cluster beam was simulated and investigated using classical nucleation theory and one dimensional gas flow equations. Zinc cluster sizes predicted at the nozzle exit are in good quantitative agreement with experimental results in our laboratory.^ A novel in situ real-time mass, energy and velocity measurement apparatus has been designed, built and tested. This small size time-of-flight mass spectrometer is suitable to be used in our cluster deposition systems and does not suffer from problems related to other methods of cluster size measurement like: requirement for specialized ionizing lasers, inductive electrical or electromagnetic coupling, dependency on the assumption of homogeneous nucleation, limits on the size measurement and non real-time capability. Measured ion energies using the electrostatic energy analyzer are in good accordance with values obtained from computer simulation. The velocity (v) is measured by pulsing the cluster beam and measuring the time of delay between the pulse and analyzer output current. The mass of a particle is calculated from m = (2E/v$\sp2).$ The error in the measured value of background gas mass is on the order of 28% of the mass of one N$\sb2$ molecule which is negligible for the measurement of large size clusters. This resolution in cluster size measurement is very acceptable for our purposes.^ Selective area deposition onto conducting patterns overlying insulating substrates was demonstrated using intense, fully-ionized cluster beams. Parameters influencing the selectivity are ion energy, repelling voltage, the ratio of the conductor to insulator dimension, and substrate thickness. ^

Without women: Masculinities, gay male sexual culture and sexual behaviors in Miami, Florida

Recent studies have reported alarmingly high rates of HIV infection and risky sexual behaviors among gay men in Miami, Florida. Previous research has suggested that the risky sexual behaviors of many gay men reflect the pursuit of intimacy and love, and that barriers to intimate relationships among gay men may stem from traditional masculinity norms. This dissertation examines the meanings which gay men ascribe to their sexual behaviors, as well as the intersections of those meanings with both traditional masculinity constructions and Miami's gay male sexual culture. ^ The study is based upon participant observation, print media content analysis, surveys and ethnographic interviews of a purposive snowball sample of 30 Cuban American, Puerto Rican, African American and Anglo gay men who reside in Miami-Dade County, Florida. Analysis of research questions was accomplished through grounded theory methods and descriptive and non-parametric statistics, including Pearson chi-square, Fisher's Exact and Mann-Whitney U tests. ^ The study shows that culturally-specified masculinity norms vary in the relative importance ascribed to heterosexual prowess, economic providership and competitiveness. These cultural differences appear important not only to the timing of sexual awareness and to the strength of homosexual stereotyping as effeminacy, but also to men's strategies in coming out as gay. The meanings men attributed to their sexual behaviors were, however, constructed in response to both inherited masculinity norms and the hypermasculine structure of Miami's gay male sexual culture. In addition to providing an ethnographic account of this subculture, the study elaborates men's issues relative to casual sex and committed relationships. Unprotected anal intercourse with casual partners during the previous twelve months was associated with growing up without one's father in the home, having been teased for effeminacy during childhood, being defensive about one's masculinity, not trusting men, having been cheated on by boyfriends, and believing that long-term gay male relationships are problematic. ^ It is concluded that the continuing epidemic of HIV infections among local gay men, as well as the hypermasculine form of the gay sexual subculture itself, are nihilistic symptoms embedded in the masculinist gender structure of the larger society. ^

Parental behaviors in child anxiety: An observational study

The increasing awareness of the prevalence, impairment, and long-term consequences of childhood anxiety disorders have led investigators to explore psychosocial factors in the etiology of these disorders. Recent investigations have begun to focus on family-level processes in the etiology and/or maintenance of childhood anxiety disorders, specifically patterns of parent-child interaction. The present study compared parent-child interactions across three problem-solving tasks of clinically anxious children and their mothers versus non-referred children and their mothers in terms of (1) direct observation measures, (2) children's, mothers', and independent observer's subjective ratings, (3) and children's evaluations using videotape-aided thought reconstruction. ^ Results suggested that the mothers of clinically anxious children engaged in fewer positive and more negative parenting strategies as compared to the mothers of non-referred children across three tasks. Although not significant, trends were evident among the subjective ratings reported by the clinically anxious children, mothers, and observer for the global perception indices. When videotape-aided thought reconstruction was used as a cue to elicit children's perceptions of the parent-child interactions, clinically anxious children reported less mother-referent positive statements and more mother-referent negative statements than non-referred children. ^