Learning Data-Driven Models of Non-Verbal Behaviors for Building Rapport Using an Intelligent Virtual Agent

There is a growing societal need to address the increasing prevalence of behavioral health issues, such as obesity, alcohol or drug use, and general lack of treatment adherence for a variety of health problems. The statistics, worldwide and in the USA, are daunting. Excessive alcohol use is the third leading preventable cause of death in the United States (with 79,000 deaths annually), and is responsible for a wide range of health and social problems. On the positive side though, these behavioral health issues (and associated possible diseases) can often be prevented with relatively simple lifestyle changes, such as losing weight with a diet and/or physical exercise, or learning how to reduce alcohol consumption. Medicine has therefore started to move toward finding ways of preventively promoting wellness, rather than solely treating already established illness. Evidence-based patient-centered Brief Motivational Interviewing (BMI) interven- tions have been found particularly effective in helping people find intrinsic motivation to change problem behaviors after short counseling sessions, and to maintain healthy lifestyles over the long-term. Lack of locally available personnel well-trained in BMI, however, often limits access to successful interventions for people in need. To fill this accessibility gap, Computer-Based Interventions (CBIs) have started to emerge. Success of the CBIs, however, critically relies on insuring engagement and retention of CBI users so that they remain motivated to use these systems and come back to use them over the long term as necessary. Because of their text-only interfaces, current CBIs can therefore only express limited empathy and rapport, which are the most important factors of health interventions. Fortunately, in the last decade, computer science research has progressed in the design of simulated human characters with anthropomorphic communicative abilities. Virtual characters interact using humans’ innate communication modalities, such as facial expressions, body language, speech, and natural language understanding. By advancing research in Artificial Intelligence (AI), we can improve the ability of artificial agents to help us solve CBI problems. To facilitate successful communication and social interaction between artificial agents and human partners, it is essential that aspects of human social behavior, especially empathy and rapport, be considered when designing human-computer interfaces. Hence, the goal of the present dissertation is to provide a computational model of rapport to enhance an artificial agent’s social behavior, and to provide an experimental tool for the psychological theories shaping the model. Parts of this thesis were already published in [LYL+12, AYL12, AL13, ALYR13, LAYR13, YALR13, ALY14].

Learning data-driven models of non-verbal behaviors for building rapport using an intelligent virtual agent

There is a growing societal need to address the increasing prevalence of behavioral health issues, such as obesity, alcohol or drug use, and general lack of treatment adherence for a variety of health problems. The statistics, worldwide and in the USA, are daunting. Excessive alcohol use is the third leading preventable cause of death in the United States (with 79,000 deaths annually), and is responsible for a wide range of health and social problems. On the positive side though, these behavioral health issues (and associated possible diseases) can often be prevented with relatively simple lifestyle changes, such as losing weight with a diet and/or physical exercise, or learning how to reduce alcohol consumption. Medicine has therefore started to move toward finding ways of preventively promoting wellness, rather than solely treating already established illness.^ Evidence-based patient-centered Brief Motivational Interviewing (BMI) interventions have been found particularly effective in helping people find intrinsic motivation to change problem behaviors after short counseling sessions, and to maintain healthy lifestyles over the long-term. Lack of locally available personnel well-trained in BMI, however, often limits access to successful interventions for people in need. To fill this accessibility gap, Computer-Based Interventions (CBIs) have started to emerge. Success of the CBIs, however, critically relies on insuring engagement and retention of CBI users so that they remain motivated to use these systems and come back to use them over the long term as necessary.^ Because of their text-only interfaces, current CBIs can therefore only express limited empathy and rapport, which are the most important factors of health interventions. Fortunately, in the last decade, computer science research has progressed in the design of simulated human characters with anthropomorphic communicative abilities. Virtual characters interact using humans’ innate communication modalities, such as facial expressions, body language, speech, and natural language understanding. By advancing research in Artificial Intelligence (AI), we can improve the ability of artificial agents to help us solve CBI problems.^ To facilitate successful communication and social interaction between artificial agents and human partners, it is essential that aspects of human social behavior, especially empathy and rapport, be considered when designing human-computer interfaces. Hence, the goal of the present dissertation is to provide a computational model of rapport to enhance an artificial agent’s social behavior, and to provide an experimental tool for the psychological theories shaping the model. Parts of this thesis were already published in [LYL+12, AYL12, AL13, ALYR13, LAYR13, YALR13, ALY14].^

From Clerk and Cashier to Guest Service Agent

In their study - From Clerk and Cashier to Guest Service Agent - by Nancy J. Allin, Director of Quality Assurance and Training and Kelly Halpine, Assistant Director of Quality Assurance and Training, The Waldorf-Astoria, New York, the authors state at the outset: “The Waldorf-Astoria has taken the positions of registration clerk and cashier and combined them to provide excellent guest service and efficient systems operation. The authors tell how and why the combination works. That thesis statement defines the article, and puts it squarely in the crosshairs of the service category. Allin and Halpine use their positions at the Waldorf-Astoria in New York City to frame their observations “The allocation of staff hours has been a challenge to many front office managers who try their hardest to schedule for the norm but provide excellent, efficient service throughout the peaks,” Allin and Halpine allude. “…the decision [to combine the positions of registration clerk and cashier] was driven by a desire to improve guest service where its impact is most obvious, at the front desk. Cross-trained employees speed the check-in and check-out process by performing both functions, as the traffic at the desk dictates,” the authors say. Making such a move has resulted in positive benefits for both the guests and the hotel. “Benefits to the hotel, in addition to those brought to bear by increased guest satisfaction, include greater flexibility in weekly scheduling and in granting vacations while maintaining adequate staffing at the desk,” say Allin and Halpine . “Another expected outcome, net payroll savings, should also be realized as a consequence of the ability to schedule more efficiently.” The authors point to communication as the key to designing a successful combination such as this, with the least amount of service disruption. They bullet-point what that communication should entail. Issues of seniority, wage and salary rates, organizational charting, filing, scheduling, possible probationary periods, position titles, and physical layouts are all discussed. “It is critical that each of the management issues be addressed and resolved before any training is begun,” Allin and Halpine suggest. “Unresolved issues project confusion and lack of conviction to line employees and the result is frustration and a lack of commitment to the combination process,” they push the thought Allin and Halpine insist: “Once begun, training must be ongoing and consistent.” In the practical sense, the authors provide that authorizing overtime is helpful in accomplishing training. “Training must address the fact that employees will be faced with guest situations which are new to them, for example: an employee previously functioning as a cashier will be faced with walking guests. Specific exercises should be included to address these needs,” say the authors.