Assessment of Speech Dialog Systems using Multi-Modal Cognitive Load Analysis and Driving Performance Metrics

In this paper, cognitive load analysis via acoustic- and CAN-Bus-based driver performance metrics is employed to assess two different commercial speech dialog systems (SDS) during in-vehicle use. Several metrics are proposed to measure increases in stress, distraction and cognitive load and we compare these measures with statistical analysis of the speech recognition component of each SDS. It is found that care must be taken when designing an SDS as it may increase cognitive load which can be observed through increased speech response delay (SRD), changes in speech production due to negative emotion towards the SDS, and decreased driving performance on lateral control tasks. From this study, guidelines are presented for designing systems which are to be used in vehicular environments.

Multi-modal summarization of key events and top players in sports tournament videos

To detect and annotate the key events of live sports videos, we need to tackle the semantic gaps of audio-visual information. Previous work has successfully extracted semantic from the time-stamped web match reports, which are synchronized with the video contents. However, web and social media articles with no time-stamps have not been fully leveraged, despite they are increasingly used to complement the coverage of major sporting tournaments. This paper aims to address this limitation using a novel multimodal summarization framework that is based on sentiment analysis and players' popularity. It uses audiovisual contents, web articles, blogs, and commentators' speech to automatically annotate and visualize the key events and key players in a sports tournament coverage. The experimental results demonstrate that the automatically generated video summaries are aligned with the events identified from the official website match reports.

Pressure behaviour of self-assembled In0.55Al0.45As/Al0.5Ga0.5As quantum dots with multi-modal distribution in size

The pressure behaviour of In0.55Al0.45As/Al0.5Ga0.5As self-assembled quantum dots (QDs) has been studied at 15 K in the pressure range of 0-1.3 GPa. The atomic force microscopy image shows that the QDs have a multi-modal distribution in size. Three emission peaks were observed in the photoluminescence (PL) spectra, corresponding to the different QD families. The measured pressure coefficients are 82, 93 and 98 meV GPa(-1) for QDs with average lateral size of 26, 52 and 62 nm, respectively. The pressure coefficient of small QDs is about 17% smaller than that of bulk In0.55Al0.45As An envelope-function calculation was used to analyse the effect of pressure-induced change of barrier height, effective mass and dot size on the pressure coefficients of QDs. The Gamma-X state mixing was also included in the evaluation of the reduction of the pressure coefficients. The results indicate that both the pressure-induced increase of effective mass and Gamma-X mixing respond to the decrease of pressure coefficients, and the Gamma-X mixing is more important for small dots. The calculated Gamma-X interaction potentials are 15 and 10 meV for QDs with lateral size of 26 and 52 nm, respectively. A type-II alignment for the X conduction band is suggested according to the pressure dependence of the PL intensities. The valence-band offset was then estimated as 0.15 +/- 0.02.

Multi-level Input Fusion in multi-modal systems

Context-aware multimodal interactive systems aim to adapt to the needs and behavioural patterns of users and offer a way forward for enhancing the efficacy and quality of experience (QoE) in human-computer interaction. The various modalities that constribute to such systems each provide a specific uni-modal response that is integratively presented as a multi-modal interface capable of interpretation of multi-modal user input and appropriately responding to it through dynamically adapted multi-modal interactive flow management , This paper presents an initial background study in the context of the first phase of a PhD research programme in the area of optimisation of data fusion techniques to serve multimodal interactivite systems, their applications and requirements.

Architecture of a multi-modal dialogue system oriented to multilingual question-answering

In this paper, a proposal of a multi-modal dialogue system oriented to multilingual question-answering is presented. This system includes the following ways of access: voice, text, avatar, gestures and signs language. The proposal is oriented to the question-answering task as a user interaction mechanism. The proposal here presented is in the first stages of its development phase and the architecture is presented for the first time on the base of the experiences in question-answering and dialogues previously developed. The main objective of this research work is the development of a solid platform that will permit the modular integration of the proposed architecture.

Springboards into design : exploring multiple representations of interaction in a dental surgery

This paper demonstrates that in order to understand and design for interactions in complex work environments, a variety of representational artefacts must be developed and employed. A study was undertaken to explore the design of better interaction technologies to support patient record keeping in a dental surgery. The domain chosen is a challenging real context that exhibits problems that could potentially be solved by ubiquitous computing and multi-modal interaction technologies. Both transient and durable representations were used to develop design understandings. We describe the representations, the kinds of insights developed from the representations and the way that the multiple representations interact and carry forward in the design process.

Multi-edge decimation in multi-modal 3D collaborative applications

Multi-resolution modelling has become essential as modern 3D applications demand 3D objects with higher LODs (LOD). Multi-modal devices such as PDAs and UMPCs do not have sufficient resources to handle the original 3D objects. The increased usage of collaborative applications has created many challenges for remote manipulation working with 3D objects of different quality. This paper studies how we can improve multi-resolution techniques by performing multiedge decimation and using annotative commands. It also investigates how devices with poorer quality 3D object can participate in collaborative actions.

Multi-vehicles interaction graph model for cooperative collision warning system

Cooperative collision warning system for road vehicles, enabled by recent advances in positioning systems and wireless communication technologies, can potentially reduce traffic accident significantly. To improve the system, we propose a graph model to represent interactions between multiple road vehicles in a specific region and at a specific time. Given a list of vehicles in vicinity, we can generate the interaction graph using several rules that consider vehicle's properties such as position, speed, heading, etc. Safety applications can use the model to improve emergency warning accuracy and optimize wireless channel usage. The model allows us to develop some congestion control strategies for an efficient multi-hop broadcast protocol.

Multi-modal object tracking using dynamic performance metrics

Intelligent surveillance systems typically use a single visual spectrum modality for their input. These systems work well in controlled conditions, but often fail when lighting is poor, or environmental effects such as shadows, dust or smoke are present. Thermal spectrum imagery is not as susceptible to environmental effects, however thermal imaging sensors are more sensitive to noise and they are only gray scale, making distinguishing between objects difficult. Several approaches to combining the visual and thermal modalities have been proposed, however they are limited by assuming that both modalities are perfuming equally well. When one modality fails, existing approaches are unable to detect the drop in performance and disregard the under performing modality. In this paper, a novel middle fusion approach for combining visual and thermal spectrum images for object tracking is proposed. Motion and object detection is performed on each modality and the object detection results for each modality are fused base on the current performance of each modality. Modality performance is determined by comparing the number of objects tracked by the system with the number detected by each mode, with a small allowance made for objects entering and exiting the scene. The tracking performance of the proposed fusion scheme is compared with performance of the visual and thermal modes individually, and a baseline middle fusion scheme. Improvement in tracking performance using the proposed fusion approach is demonstrated. The proposed approach is also shown to be able to detect the failure of an individual modality and disregard its results, ensuring performance is not degraded in such situations.