The SEMAINE Database: Annotated Multimodal Records of Emotionally Colored Conversations between a Person and a Limited Agent

SEMAINE has created a large audiovisual database as a part of an iterative approach to building Sensitive Artificial Listener (SAL) agents that can engage a person in a sustained, emotionally colored conversation. Data used to build the agents came from interactions between users and an operator simulating a SAL agent, in different configurations: Solid SAL (designed so that operators displayed an appropriate nonverbal behavior) and Semi-automatic SAL (designed so that users' experience approximated interacting with a machine). We then recorded user interactions with the developed system, Automatic SAL, comparing the most communicatively competent version to versions with reduced nonverbal skills. High quality recording was provided by five high-resolution, high-framerate cameras, and four microphones, recorded synchronously. Recordings total 150 participants, for a total of 959 conversations with individual SAL characters, lasting approximately 5 minutes each. Solid SAL recordings are transcribed and extensively annotated: 6-8 raters per clip traced five affective dimensions and 27 associated categories. Other scenarios are labeled on the same pattern, but less fully. Additional information includes FACS annotation on selected extracts, identification of laughs, nods, and shakes, and measures of user engagement with the automatic system. The material is available through a web-accessible database. © 2010-2012 IEEE.

Robust Multimodal Person Identification With Limited Training Data

This paper presents a novel method of audio-visual feature-level fusion for person identification where both the speech and facial modalities may be corrupted, and there is a lack of prior knowledge about the corruption. Furthermore, we assume there are limited amount of training data for each modality (e.g., a short training speech segment and a single training facial image for each person). A new multimodal feature representation and a modified cosine similarity are introduced to combine and compare bimodal features with limited training data, as well as vastly differing data rates and feature sizes. Optimal feature selection and multicondition training are used to reduce the mismatch between training and testing, thereby making the system robust to unknown bimodal corruption. Experiments have been carried out on a bimodal dataset created from the SPIDRE speaker recognition database and AR face recognition database with variable noise corruption of speech and occlusion in the face images. The system's speaker identification performance on the SPIDRE database, and facial identification performance on the AR database, is comparable with the literature. Combining both modalities using the new method of multimodal fusion leads to significantly improved accuracy over the unimodal systems, even when both modalities have been corrupted. The new method also shows improved identification accuracy compared with the bimodal systems based on multicondition model training or missing-feature decoding alone.

Device-Free Person Detection and Ranging in UWB Networks

We present a novel device-free stationary person detection and ranging method, that is applicable to ultra-wide bandwidth (UWB) networks. The method utilizes a fixed UWB infrastructure and does not require a training database of template waveforms. Instead, the method capitalizes on the fact that a human presence induces small low-frequency variations that stand out against the background signal, which is mainly affected by wideband noise. We analyze the detection probability, and validate our findings with numerical simulations and experiments with off-the-shelf UWB transceivers in an indoor environment. © 2007-2012 IEEE.

An experimental study of UWB device-free person detection and ranging

Passive person detection and localization is an emerging area in UWB localization systems, whereby people are not required to carry any UWB ranging device. Based on experimental data, we propose a novel method to detect static persons in the absence of template waveforms, and to compute distances to these persons. Our method makes very little assumptions on the environment and can achieve ranging performances on the order of 50 cm, using off-the-shelf UWB devices. © 2013 IEEE.

Lacustrine flow (divers, side scan sonar, hydrogeology, water penetrating radar) used to understand the location of a drowned person

An unusual application of hydrological understanding to a police search is described. The lacustrine search for a missing person provided reports of bottom-water currents in the lake and contradictory indications from cadaver dogs. A hydrological model of the area was developed using pre-existing information from side scan sonar, a desktop hydrogeological study and deployment of water penetrating radar (WPR). These provided a hydrological theory for the initial search involving subaqueous groundwater flow, focused on an area of bedrock surrounded by sediment, on the lake floor. The work shows the value a hydrological explanation has to a police search operation (equally to search and rescue). With hindsight, the desktop study should have preceded the search, allowing better understanding of water conditions. The ultimate reason for lacustrine flow in this location is still not proven, but the hydrological model explained the problems encountered in the initial search.

First-order characteristics of the person-to-vehicle channel at 5.8 GHz

In this paper we investigate the first order characteristics of the radio channel between a moving vehicle and a stationary person positioned by the side of a road at 5.8 GHz. The experiments considered a transmitter positioned at different locations on both the body and receivers positioned on the vehicle. The transmitter was alternated between positions on the central chest region, back and the wrist (facing the roadside) of the body, with the receivers placed on the outside roof, the outside rear window and the inside dashboard of the vehicle. The Rice fading model was applied to the measurement data to assess its suitability for characterizing this emerging type of wireless channel. The Ricean K factors calculated from the data suggest that a significant dominant component existed in the majority of the channels considered in this study.