Building autonomous sensitive artificial listeners (Extended abstract)

This paper describes a substantial effort to build a real-time interactive multimodal dialogue system with a focus on emotional and non-verbal interaction capabilities. The work is motivated by the aim to provide technology with competences in perceiving and producing the emotional and non-verbal behaviours required to sustain a conversational dialogue. We present the Sensitive Artificial Listener (SAL) scenario as a setting which seems particularly suited for the study of emotional and non-verbal behaviour, since it requires only very limited verbal understanding on the part of the machine. This scenario allows us to concentrate on non-verbal capabilities without having to address at the same time the challenges of spoken language understanding, task modeling etc. We first summarise three prototype versions of the SAL scenario, in which the behaviour of the Sensitive Artificial Listener characters was determined by a human operator. These prototypes served the purpose of verifying the effectiveness of the SAL scenario and allowed us to collect data required for building system components for analysing and synthesising the respective behaviours. We then describe the fully autonomous integrated real-time system we created, which combines incremental analysis of user behaviour, dialogue management, and synthesis of speaker and listener behaviour of a SAL character displayed as a virtual agent. We discuss principles that should underlie the evaluation of SAL-type systems. Since the system is designed for modularity and reuse, and since it is publicly available, the SAL system has potential as a joint research tool in the affective computing research community.

Multimodal Education for Very Young Learners

Invited Plenary Speaker

Deep Head Pose: Gaze-Direction Estimation in Multimodal Video

In this paper we present a convolutional neuralnetwork (CNN)-based model for human head pose estimation inlow-resolution multi-modal RGB-D data. We pose the problemas one of classification of human gazing direction. We furtherfine-tune a regressor based on the learned deep classifier. Next wecombine the two models (classification and regression) to estimateapproximate regression confidence. We present state-of-the-artresults in datasets that span the range of high-resolution humanrobot interaction (close up faces plus depth information) data tochallenging low resolution outdoor surveillance data. We buildupon our robust head-pose estimation and further introduce anew visual attention model to recover interaction with theenvironment. Using this probabilistic model, we show thatmany higher level scene understanding like human-human/sceneinteraction detection can be achieved. Our solution runs inreal-time on commercial hardware

D4.3 Prototype Software (Documentation)

The agent-based social simulation component of the TELL ME project (WP4) developed prototype software to assist communications planners to understand the complex relationships between communication, personal protective behaviour and epidemic spread. Using the simulation, planners can enter different potential communications plans, and see their simulated effect on attitudes, behaviour and the consequent effect on an influenza epidemic.

The model and the software to run the model are both freely available (see section 2.2.1 for instructions on how to obtain the relevant files). This report provides the documentation for the prototype software. The major component is the user guide (Section 2). This provides instructions on how to set up the software, some training scenarios to become familiar with the model operation and use, and details about the model controls and output.

The model contains many parameters. Default values and their source are described at Section 3. These are unlikely to be suitable for all countries, and may also need to be changed as new research is conducted. Instructions for how to customise these values are also included (see section 3.5).

The final technical reference contains two parts. The first is a guide for advanced users who wish to run multiple simulations and analyse the results (section 4.1). The second is to orient programmers who wish to adapt or extend the simulation model (section 4.2). This material is not suitable for general users.