Learning Data-Driven Models of Non-Verbal Behaviors for Building Rapport Using an Intelligent Virtual Agent

There is a growing societal need to address the increasing prevalence of behavioral health issues, such as obesity, alcohol or drug use, and general lack of treatment adherence for a variety of health problems. The statistics, worldwide and in the USA, are daunting. Excessive alcohol use is the third leading preventable cause of death in the United States (with 79,000 deaths annually), and is responsible for a wide range of health and social problems. On the positive side though, these behavioral health issues (and associated possible diseases) can often be prevented with relatively simple lifestyle changes, such as losing weight with a diet and/or physical exercise, or learning how to reduce alcohol consumption. Medicine has therefore started to move toward finding ways of preventively promoting wellness, rather than solely treating already established illness. Evidence-based patient-centered Brief Motivational Interviewing (BMI) interven- tions have been found particularly effective in helping people find intrinsic motivation to change problem behaviors after short counseling sessions, and to maintain healthy lifestyles over the long-term. Lack of locally available personnel well-trained in BMI, however, often limits access to successful interventions for people in need. To fill this accessibility gap, Computer-Based Interventions (CBIs) have started to emerge. Success of the CBIs, however, critically relies on insuring engagement and retention of CBI users so that they remain motivated to use these systems and come back to use them over the long term as necessary. Because of their text-only interfaces, current CBIs can therefore only express limited empathy and rapport, which are the most important factors of health interventions. Fortunately, in the last decade, computer science research has progressed in the design of simulated human characters with anthropomorphic communicative abilities. Virtual characters interact using humans’ innate communication modalities, such as facial expressions, body language, speech, and natural language understanding. By advancing research in Artificial Intelligence (AI), we can improve the ability of artificial agents to help us solve CBI problems. To facilitate successful communication and social interaction between artificial agents and human partners, it is essential that aspects of human social behavior, especially empathy and rapport, be considered when designing human-computer interfaces. Hence, the goal of the present dissertation is to provide a computational model of rapport to enhance an artificial agent’s social behavior, and to provide an experimental tool for the psychological theories shaping the model. Parts of this thesis were already published in [LYL+12, AYL12, AL13, ALYR13, LAYR13, YALR13, ALY14].

Learning data-driven models of non-verbal behaviors for building rapport using an intelligent virtual agent

There is a growing societal need to address the increasing prevalence of behavioral health issues, such as obesity, alcohol or drug use, and general lack of treatment adherence for a variety of health problems. The statistics, worldwide and in the USA, are daunting. Excessive alcohol use is the third leading preventable cause of death in the United States (with 79,000 deaths annually), and is responsible for a wide range of health and social problems. On the positive side though, these behavioral health issues (and associated possible diseases) can often be prevented with relatively simple lifestyle changes, such as losing weight with a diet and/or physical exercise, or learning how to reduce alcohol consumption. Medicine has therefore started to move toward finding ways of preventively promoting wellness, rather than solely treating already established illness.^ Evidence-based patient-centered Brief Motivational Interviewing (BMI) interventions have been found particularly effective in helping people find intrinsic motivation to change problem behaviors after short counseling sessions, and to maintain healthy lifestyles over the long-term. Lack of locally available personnel well-trained in BMI, however, often limits access to successful interventions for people in need. To fill this accessibility gap, Computer-Based Interventions (CBIs) have started to emerge. Success of the CBIs, however, critically relies on insuring engagement and retention of CBI users so that they remain motivated to use these systems and come back to use them over the long term as necessary.^ Because of their text-only interfaces, current CBIs can therefore only express limited empathy and rapport, which are the most important factors of health interventions. Fortunately, in the last decade, computer science research has progressed in the design of simulated human characters with anthropomorphic communicative abilities. Virtual characters interact using humans’ innate communication modalities, such as facial expressions, body language, speech, and natural language understanding. By advancing research in Artificial Intelligence (AI), we can improve the ability of artificial agents to help us solve CBI problems.^ To facilitate successful communication and social interaction between artificial agents and human partners, it is essential that aspects of human social behavior, especially empathy and rapport, be considered when designing human-computer interfaces. Hence, the goal of the present dissertation is to provide a computational model of rapport to enhance an artificial agent’s social behavior, and to provide an experimental tool for the psychological theories shaping the model. Parts of this thesis were already published in [LYL+12, AYL12, AL13, ALYR13, LAYR13, YALR13, ALY14].^

Detecting deception: Identifying differences in liars' and truth tellers' verbal strategies

Deception research has traditionally focused on three methods of identifying liars and truth tellers: observing non-verbal or behavioral cues, analyzing verbal cues, and monitoring changes in physiological arousal during polygraph tests. Research shows that observers are often incapable of discriminating between liars and truth tellers with better than chance accuracy when they use these methods. One possible explanation for observers' poor performance is that they are not properly applying existing lie detection methods. An alternative explanation is that the cues on which these methods — and observers' judgments — are based do not reliably discriminate between liars and truth tellers. It may be possible to identify more reliable cues, and potentially improve observers' ability to discriminate, by developing a better understanding of how liars and truth tellers try to tell a convincing story. ^ This research examined (a) the verbal strategies used by truthful and deceptive individuals during interviews concerning an assigned activity, and (b) observers' ability to discriminate between them based on their verbal strategies. In Experiment I, pre-interview instructions manipulated participants' expectations regarding verifiability; each participant was led to believe that the interviewer could check some types of details, but not others, before deciding whether the participant was being truthful or deceptive. Interviews were then transcribed and scored for quantity and type of information provided. In Experiment II, observers listened to a random sample of the Experiment I interviews and rendered veracity judgments; half of the observers were instructed to judge the interviews according to the verbal strategies used by liars and truth tellers and the other half were uninstructed. ^ Results of Experiment I indicate that liars and truth tellers use different verbal strategies, characterized by a differential amount of detail. Overall, truthful participants provided more information than deceptive participants. This effect was moderated by participants' expectations regarding verifiability such that truthful participants provided more information only with regard to verifiable details. Results of Experiment II indicate that observers instructed about liars' and truth tellers' verbal strategies identify them with greater accuracy than uninstructed observers. ^

Interpersonal Techniques Boost your "Hospitality Batting Average"

The ability to listen and comprehend the intrinsic meaning behind the words people are saying is an important skill for those in the hospitality industry. The author provides some prescriptions for "winning friends and influencing people.”

The differential effects of *American and British trial proceedings on juror decision-making

Seven basic elements differentiate British from American trial procedures: confining attorneys to their tables; dealing with objections outside the presence of the jury; resolving disagreements between attorneys prior to objections being made; presenting the defense opening statement at the close of the prosecution case; the judge directly questions witnesses and has a wider latitude in controlling the evidence; and the judge gives a summation of all the evidence presented to the jury (Fulero & Turner, 1997). The present experiment examined the influence of these different courtroom procedures, judges' non-verbal behavior, and evidence strength on juror decision-making. Using models of persuasion to understand how the varying elements may effect juror decision-making, it was predicted that trials following American courtroom procedures would be more distracting for jurors and as such, they would be more likely to rely on the peripheral cue of the judge's expectations for trial outcome as expressed in his nonverbal behavior. In trials following British procedures jurors should be less distracted and better able to scrutinize the strength of the evidence that in turn should minimize the influence of the judge's nonverbal behavior. Two hundred forty-five participants viewed a mock civil trial in which courtroom procedure, judge's nonverbal behavior, and evidence strength were varied. Analyses suggest that courtroom procedure and evidence strength influenced the direction of participants' verdicts, but that judge's nonverbal behavior did not have a direct impact on verdict preference. Judge's nonverbal behavior appeared to influence other measures related to verdict. Participants were more confident in their verdicts when they agreed with judge's nonverbal behavior and when they viewed British courtroom procedures. Participants were more likely to return estimates of the defendant's liability that reflected judge's nonverbal behavior and a congruency with evidence strength. Participants also recalled more facts in the British conditions than in the American conditions. These findings are interpreted as indicating the importance of the impact of trial procedures and of nonverbal influence. ^

Drawing as a technique to facilitate childrens memory

This study examined a technique to assist children to recall more information about witnessed events. Thirty-eight fourth-grade children from a public grade school in Miami Florida participated in the experiment. The participants watched a Red Cross demonstration and were interviewed one week later about details of the demonstration. All of the children were interviewed using a police style interview. In addition, half of the children were instructed to draw during the interview. The current study supported previous findings that the instruction to draw increased the amount of information recalled. The effect of drawing was greatest for high-visual events. In addition, the instruction to draw prompted an increase in non-verbal information, which had an unusually high accuracy rate.

An Analysis of the Proposition Density, Sentence and Clause Types, and Non-Finite Verbal Usage in Two College Textbooks

This study investigated the proposition density, sentence and clause type usage and non-finite verbal usage in two college textbooks. The teaching implications are presented.

Evoking non-repertory verbal behavior across operant classes : the effects of motor echoic sign language training within the context of a motivating operation

The individual effects that echoic, mand, and sign language training procedures have on the acquisition of verbal behavior have been widely demonstrated, but more efficient strategies are still needed. This study combined all three treatment strategies into one treatment intervention in order to investigate the joint effects they may have on verbal behavior. Six participants took part in the study. Intervention totaled 1 hour/day for 5 days/week until mastery criterion for motor echoic behavior was achieved. Although motor echoic behavior were solely targeted for acquisition, significant increases in spontaneous motor mands were noted in all treatment participants. Additionally, 4 treatment participants also demonstrated significant gains in vocal echoics and spontaneous vocal mands. No significant increases were noted for the control participant. Results suggest that the aforementioned procedure may provide more efficient results as a first-step to teaching a functional repertoire of verbal behavior to developmentally delayed children.

Factors associated with critical care nurses' communication with non-communicative patients in the ICU

Hospitalized individuals are isolated from their familiar environment at the onset of illness. Those individuals who are non-communicative are detached from the world and from life, as they previously knew it. Although nurses have long since recognized the importance of communication, patients still report the lack of iy. This study was done to identify factors influencing critical care nurses to communicate with their noncommunicative patients. The overall results of the study indicate that nurses are aware of the importance of verbal communication with patients who may be intubated, paralyzed, unconscious, comatose or neurologically impaired and are not deterred by them. Despite these results, some significant observations emerged identified. CCRN certified nurses and nurses with more years of experience were less likely to have verbal communication with noncommunicative patients. Nurses with children, spouses and those working full-time were more likely to communicate with non-communicative patients.