Nonverbal social sensing in action: Unobtrusive recording and extracting of nonverbal behavior in social interactions illustrated with a research example

Nonverbal behavior coding is typically conducted by "hand". To remedy this time and resource intensive undertaking, we illustrate how nonverbal social sensing, defined as the automated recording and extracting of nonverbal behavior via ubiquitous social sensing platforms, can be achieved. More precisely, we show how and what kind of nonverbal cues can be extracted and to what extent automated extracted nonverbal cues can be validly obtained with an illustrative research example. In a job interview, the applicant's vocal and visual nonverbal immediacy behavior was automatically sensed and extracted. Results show that the applicant's nonverbal behavior can be validly extracted. Moreover, both visual and vocal applicant nonverbal behavior predict recruiter hiring decision, which is in line with previous findings on manually coded applicant nonverbal behavior. Finally, applicant average turn duration, tempo variation, and gazing best predict recruiter hiring decision. Results and implications of such a nonverbal social sensing for future research are discussed.

Hire me: Computational inference of hirability in employment interviews based on nonverbal behavior

Understanding the basis on which recruiters form hirability impressions for a job applicant is a key issue in organizational psychology and can be addressed as a social computing problem. We approach the problem from a face-to-face, nonverbal perspective where behavioral feature extraction and inference are automated. This paper presents a computational framework for the automatic prediction of hirability. To this end, we collected an audio-visual dataset of real job interviews where candidates were applying for a marketing job. We automatically extracted audio and visual behavioral cues related to both the applicant and the interviewer. We then evaluated several regression methods for the prediction of hirability scores and showed the feasibility of conducting such a task, with ridge regression explaining 36.2% of the variance. Feature groups were analyzed, and two main groups of behavioral cues were predictive of hirability: applicant audio features and interviewer visual cues, showing the predictive validity of cues related not only to the applicant, but also to the interviewer. As a last step, we analyzed the predictive validity of psychometric questionnaires often used in the personnel selection process, and found that these questionnaires were unable to predict hirability, suggesting that hirability impressions were formed based on the interaction during the interview rather than on questionnaire data.

Talking bodies: nonverbal behavior in the assessment of depression severity

BACKGROUND: Evaluations of clinical depression are traditionally based on verbal information. Nonverbal expressive behavior, however, being associated with a person's reflexive responses, may reveal negative emotional or social processes that are not under complete control of the patients. However, investigations of nonverbal behavior in the evaluation of depressed patients are still scarce. This study examines the nonverbal behaviors of a group of Brazilian patients, associating their nonverbal behavior with severity of depression. METHODS: Forty depressed patients were evaluated at baseline (T0) and after a two-week transcranial direct current stimulation treatment (T1), according to rating scales and through a 21-category Ethogram for assessment of the frequency of nonverbal behaviors displayed during an interview. RESULTS: Behaviors that were related to negative feelings and social disinterest decreased with corresponding clinical improvement and were associated with increased severity of symptoms at T0 and greater negative affect and dissatisfaction at T1. Pro-social behaviors were associated with milder symptoms at T0 and increased after treatment. Facial, head and hand expressive movements stood out as important indicators because of their associations with severity of depression. LIMITATIONS: Duration of behaviors was not assessed and there was not a healthy control group with which to compare the findings. CONCLUSIONS: These results support the usefulness of nonverbal behavior as an evaluation technique in the assessment of clinical depression.

Attachment insecurity as a filter in the decoding and encoding of nonverbal behavior in close relationships

In this article, I have focused my comments on the possible associations between the cognitions related to different attachment styles, and the impact that those cognitions are likely to have on nonverbal encoding and decoding. I see attachment insecurity as acting as a filter, distorting both encoding and decoding processes. In terms of decoding, an insecure individual may appraise the situation as more threatening than it actually is, may see the attachment figure as more or less available than he or she actually is, and may make an inappropriate decision about the viability or desirability of seeking proximity to the attachment figure. Attachment insecurity is also likely to inhibit the distressed individual from expressing their distress in a way that is understood by the attachment figure and that increases the likelihood that the attachment figure will engage in supportive behavior.

Reliability and validity of nonverbal thin slices in social interactions

Four studies investigated the reliability and validity of thin slices of nonverbal behavior from social interactions including (1) how well individual slices of a given behavior predict other slices in the same interaction; (2) how well a slice of a given behavior represents the entirety of that behavior within an interaction; (3) how long a slice is necessary to sufficiently represent the entirety of a behavior within an interaction; (4) which slices best capture the entirety of behavior, across different behaviors; and (5) which behaviors (of six measured behaviors) are best captured by slices. Notable findings included strong reliability and validity for thin slices of gaze and nods, and that a 1.5 min slice from the start of an interaction may adequately represent some behaviors. Results provide useful information to researchers making decisions about slice measurement of behavior.

Explicit vs implicit affiliation motives predict different social behavior in sports competition

Objectives The present study investigated the predictive value of the explicit and implicit affiliation motive for social behavior in sport competitions. From an information processing perspective, an explicit motive is linked to verbal cues and respondent behavior. The implicit motive in turn is linked to nonverbal stimuli and operant behavior (McClelland, Koestner, & Weinberger, 1989; Schultheiss, 2008). Both respondent affiliative behavior (e.g., verbal interactions with teammates) and operant nonverbal social behavior (e.g., pleasant to opponents) can be observed in racquet sports team competitions. Design & Methods Fifty-two male racquet sportsmen completed the Personality Research Form (explicit affiliation motive) and the Operant Motive Test (implicit affiliation motive). Motive measures were used to predict social behavior during competitions using multiple regression analyses. To this aim real competitive matches were videotaped and analyzed. Results Results show that the explicit affiliation motive is associated with time spent in verbal team contact. The implicit affiliation motive, by contrast, is linked to pleasant nonverbal behavior shown toward opponents. Conclusions Findings suggest that implicit and explicit affiliation motives predict different kinds of social behavior in sports competition respectively. Indirect motive measures may be of additional predictive value for different behavior in real sports settings.

Nonverbal synchrony in social interactions of patients with schizophrenia indicates socio-communicative deficits

Background Disordered interpersonal communication can be a serious problem in schizophrenia. Recent advances in computer-based measures allow reliable and objective quantification of nonverbal behavior. Research using these novel measures has shown that objective amounts of body and head movement in patients with schizophrenia during social interactions are closely related to the symptom profiles of these patients. In addition to and above mere amounts of movement, the degree of synchrony, or imitation, between patients and normal interactants may be indicative of core deficits underlying various problems in domains related to interpersonal communication, such as symptoms, social competence, and social functioning. Methods Nonverbal synchrony was assessed objectively using Motion Energy Analysis (MEA) in 378 brief, videotaped role-play scenes involving 27 stabilized outpatients diagnosed with paranoid-type schizophrenia. Results Low nonverbal synchrony was indicative of symptoms, low social competence, impaired social functioning, and low self-evaluation of competence. These relationships remained largely significant when correcting for the amounts of patients‘ movement. When patients showed reduced imitation of their interactants’ movements, negative symptoms were likely to be prominent. Conversely, positive symptoms were more prominent in patients when their interaction partners’ imitation of their movements was reduced. Conclusions Nonverbal synchrony can be an objective and sensitive indicator of the severity of patients’ problems. Furthermore, quantitative analysis of nonverbal synchrony may provide novel insights into specific relationships between symptoms, cognition, and core communicative problems in schizophrenia.

Strategies of accommodation: Development of a coding system for conversational interaction

This study describes a coding system developed to operationalize the sociolinguistic strategies proposed by communication accommodation theory (CAT) in an academic context. Fifty interactions between two students (of Australian or Chinese ethnic background) or a student and faculty member were videotaped. A turn- and episode-based coding system was developed, focusing on verbal and nonverbal behavior. The development of this system is described in detail, before results are presented. Results indicated that status was the main influence on choice of strategies, particularly the extent and type of discourse management and interpersonal control. Participants' sew and ethnicity also played a role: Male participants made more use of interpretability (largely questions), whereas female participants used discourse management to develop a shared perspective. The results make clear that there is no automatic correspondence between behaviors and the strategies they constitute, and they point to the appropriateness of conceptualizing behavior and strategies separately in CAT.

The Body Action and Posture Coding System (BAP): Development and Reliability

Several methods are available for coding body movement in nonverbal behavior research, but there is no consensus on a reliable coding system that can be used for the study of emotion expression. Adopting an integrative approach, we developed a new method, the Body Action and Posture (BAP) coding system, for the time-aligned micro description of body movement on an anatomical level (different articulations of body parts), a form level (direction and orientation of movement), and a functional level (communicative and self-regulatory functions). We applied the system to a new corpus of acted emotion portrayals, examined its comprehensiveness and demonstrated intercoder reliability at three levels: a) occurrence, b) temporal precision and c) segmentation. We discuss issues for further validation and propose some research applications.

Emotion expression in body action and posture

Emotion communication research strongly focuses on the face and voice as expressive modalities, leaving the rest of the body relatively understudied. Contrary to the early assumption that body movement only indicates emotional intensity, recent studies show that body movement and posture also convey emotion specific information. However, a deeper understanding of the underlying mechanisms is hampered by a lack of production studies informed by a theoretical framework. In this research we adopted the Body Action and Posture (BAP) coding system to examine the types and patterns of body movement that are employed by 10 professional actors to portray a set of 12 emotions. We investigated to what extent these expression patterns support explicit or implicit predictions from basic emotion theory, bi-dimensional theory, and componential appraisal theory. The overall results showed partial support for the different theoretical approaches. They revealed that several patterns of body movement systematically occur in portrayals of specific emotions, allowing emotion differentiation. While a few emotions were prototypically encoded by one particular pattern, most emotions were variably expressed by multiple patterns, many of which can be explained as reflecting functional components of emotion such as modes of appraisal and action readiness. It is concluded that further work in this largely underdeveloped area should be guided by an appropriate theoretical framework to allow a more systematic design of experiments and clear hypothesis testing.

The vertical dimension of social relations and accurate interpersonal perception: A meta-analysis (Publishd online: 24 Oct. 2014)

There is little consensus regarding how verticality (social power, dominance, and status) is related to accurate interpersonal perception. The relation could be either positive or negative, and there could be many causal processes at play. The present article discusses the theoretical possibilities and presents a meta-analysis of this question. In studies using a standard test of interpersonal accuracy, higher socioeconomic status (SES) predicted higher accuracy defined as accurate inference about the meanings of cues; also, higher experimentally manipulated vertical position predicted higher accuracy defined as accurate recall of others' words. In addition, although personality dominance did not predict accurate inference overall, the type of personality dominance did, such that empathic/responsible dominance had a positive relation and egoistic/aggressive dominance had a negative relation to accuracy. In studies involving live interaction, higher experimentally manipulated vertical position produced lower accuracy defined as accurate inference about cues; however, methodological problems place this result in doubt.