Robust Multimodal Person Identification With Limited Training Data

This paper presents a novel method of audio-visual feature-level fusion for person identification where both the speech and facial modalities may be corrupted, and there is a lack of prior knowledge about the corruption. Furthermore, we assume there are limited amount of training data for each modality (e.g., a short training speech segment and a single training facial image for each person). A new multimodal feature representation and a modified cosine similarity are introduced to combine and compare bimodal features with limited training data, as well as vastly differing data rates and feature sizes. Optimal feature selection and multicondition training are used to reduce the mismatch between training and testing, thereby making the system robust to unknown bimodal corruption. Experiments have been carried out on a bimodal dataset created from the SPIDRE speaker recognition database and AR face recognition database with variable noise corruption of speech and occlusion in the face images. The system's speaker identification performance on the SPIDRE database, and facial identification performance on the AR database, is comparable with the literature. Combining both modalities using the new method of multimodal fusion leads to significantly improved accuracy over the unimodal systems, even when both modalities have been corrupted. The new method also shows improved identification accuracy compared with the bimodal systems based on multicondition model training or missing-feature decoding alone.

Multimodal functional and anatomical imaging identifies preclinical microvascular abnormalities in type 1 diabetes mellitus

Structural and functional change in the microcirculation in type 1 diabetes mellitus predicts future end-organ damage and macrovascular events. We explored the utility of novel signal processing techniques to detect and track change in ocular hemodynamics in patients with this disease. 24 patients with uncomplicated type 1 diabetes mellitus, and 18 age-and-sex matched control subjects were studied. Doppler ultrasound was used to interrogate the carotid and ophthalmic arteries and digital photography to image the retinal vasculature. Frequency analysis algorithms were applied to quantify velocity waveform structure and retinal photographic data at baseline and following inhalation of 100% oxygen. Frequency data was compared between groups. No significant differences were found in the resistive index between groups at baseline or following inhaled oxygen. Frequency analysis of the Doppler flow velocity waveforms identified significant differences in bands 3-7 between patients and controls in data captured from the ophthalmic artery (p<0.01 for each band). In response to inhaled oxygen, changes in the frequency band amplitudes were significantly greater in control subjects compared with patients (p<0.05). Only control subjects demonstrated a positive correlation (R=0.61) between change in retinal vessel diameter and frequency band amplitudes derived from ophthalmic artery waveform data. The use of multimodal signal processing techniques applied to Doppler flow velocity waveforms and retinal photographic data identified preclinical change in the ocular microcirculation in patients with uncomplicated diabetes mellitus. An impaired autoregulatory response of the retinal microvasculature may contribute to the future development of retinopathy in such patients.

Spectacles of military urbanism in online representations of the Elite Squad of the military police of Rio de Janeiro: a multimodal approach

This paper presents a multimodal analysis of online self-representations of the Elite Squad of the military police of Rio de Janeiro, the Special Police Operations Battalion BOPE. The analysis is placed within the wider context of a “new military urbanism”, which is evidenced in the ongoing “Pacification” of many of the city’s favelas, in which BOPE plays an active interventionist as well as a symbolic role, and is a kind of solution which clearly fails to address the root causes of violence which lie in poverty and social inequality. The paper first provides a sociocultural account of BOPE’s role in Rio’s public security and then looks at some of the mainly visual mediated discourses the Squad employs in constructing a public image of itself as a modern and efficient, yet at the same time “magical” police force.

Multimodal Critical Discourse Analysis (MCDA)

Critical Discourse Analysis (CDA) has probably made the most comprehensive attempt to develop a theory of the inter-connectedness of discourse, power and ideology and is specifically concerned with the role that discourse plays in main-taining and legitimizing inequality in society. While CDA’s general thrust has been towards the analysis of linguistic structures, some critical discourse analysts have begun to focus on multimodal discourses because of the increasingly impor-tant role these play in many social and political contexts. Still, a great deal of CDA analysis has remained largely monomodal. The principal aim of this chapter is therefore to address this situation and demonstrate in what ways CDA can be deployed to analyse the ways that ideological discourses can be communicated, naturalised and legitimated beyond the linguistic level. The chapter also offers a rationale for a multimodal approach based on Halliday’s Systemic Functional Linguistics (SFL), by which it is directly informed

Cascaded multimodal biometric recognition framework

A practically viable multi-biometric recognition system should not only be stable, robust and accurate but should also adhere to real-time processing speed and memory constraints. This study proposes a cascaded classifier-based framework for use in biometric recognition systems. The proposed framework utilises a set of weak classifiers to reduce the enrolled users' dataset to a small list of candidate users. This list is then used by a strong classifier set as the final stage of the cascade to formulate the decision. At each stage, the candidate list is generated by a Mahalanobis distance-based match score quality measure. One of the key features of the authors framework is that each classifier in the ensemble can be designed to use a different modality thus providing the advantages of a truly multimodal biometric recognition system. In addition, it is one of the first truly multimodal cascaded classifier-based approaches for biometric recognition. The performance of the proposed system is evaluated both for single and multimodalities to demonstrate the effectiveness of the approach.

Interactions between pyrazole derived enantiomers and Chiralcel OJ: Prediction of enantiomer absolute configurations and elution order by molecular dynamics simulations

The separation of enantiomers and confirmation of their absolute configurations is significant in the development of chiral drugs. The interactions between the enantiomers of chiral pyrazole derivative and polysaccharide-based chiral stationary phase cellulose tris(4-methylbenzoate) (Chiralcel OJ) in seven solvents and under different temperature were studied using molecular dynamics simulations. The results show that solvent effect has remarkable influence on the interactions. Structure analysis discloses that the different interactions between two isomers and chiral stationary phase are dependent on the nature of solvents, which may invert the elution order. The computational method in the present study can be used to predict the elution order and the absolute configurations of enantiomers in HPLC separations and therefore would be valuable in development of chiral drugs.

Multimodal Education for Very Young Learners

Invited Plenary Speaker

Deep Head Pose: Gaze-Direction Estimation in Multimodal Video

In this paper we present a convolutional neuralnetwork (CNN)-based model for human head pose estimation inlow-resolution multi-modal RGB-D data. We pose the problemas one of classification of human gazing direction. We furtherfine-tune a regressor based on the learned deep classifier. Next wecombine the two models (classification and regression) to estimateapproximate regression confidence. We present state-of-the-artresults in datasets that span the range of high-resolution humanrobot interaction (close up faces plus depth information) data tochallenging low resolution outdoor surveillance data. We buildupon our robust head-pose estimation and further introduce anew visual attention model to recover interaction with theenvironment. Using this probabilistic model, we show thatmany higher level scene understanding like human-human/sceneinteraction detection can be achieved. Our solution runs inreal-time on commercial hardware

Capillary Electrophoresis of Erythrocytes.

Capillary electrophoresis (CE) of erythrocytes from different sources under various conditions is reported in this paper. It was found that erythrocyte samples from sheep, duck, and human showed characteristic and reproducible elution peaks, and that the retention times of A-, B-, AB-, and O-type erythrocytes from human blood were distinctively different; even subtle differences, among individuals with the same blood type could be detected by CE. A strictly linear correlation was obtained between the peak area and the amount of human erythrocyte over a range of 4.8×102–1.9×104 cells (r=0.999), indicating that CE could be used for rapid and accurate quantification of erythrocytes. Using this CE protocol, the decrease of the surface electrical charge of erythrocyte during storage was confirmed. Therefore, this work demonstrated that CE could be a useful alternative for characterizing and quantifying erythrocytes or other cells.

Haptic information stabilizes and destabilizes coordination dynamics

Goal-directed, coordinated movements in humans emerge from a variety of constraints that range from 'high-level' cognitive strategies based oil perception of the task to 'low-level' neuromuscular-skeletal factors such as differential contributions to coordination from flexor and extensor muscles. There has been a tendency in the literature to dichotomize these sources of constraint, favouring one or the other rather than recognizing and understanding their mutual interplay. In this experiment, subjects were required to coordinate rhythmic flexion and extension movements with an auditory metronome, the rate of which was systematically increased. When subjects started in extension on the beat of the metronome, there was a small tendency to switch to flexion at higher rates, but not vice versa. When subjects: were asked to contact a physical stop, the location of which was either coincident with or counterphase to the auditor) stimulus, two effects occurred. When haptic contact was coincident with sound, coordination was stabilized for both flexion and extension. When haptic contact was counterphase to the metronome, coordination was actually destabilized, with transitions occurring from both extension to flexion on the beat and from flexion to extension on the beat. These results reveal the complementary nature of strategic and neuromuscular factors in sensorimotor coordination. They also suggest the presence of a multimodal neural integration process-which is parametrizable by rate and context - in which intentional movement, touch and sound are bound into a single, coherent unit.

Target Detection and Tracking With Heterogeneous Sensors

We present a multimodal detection and tracking algorithm for sensors composed of a camera mounted between two microphones. Target localization is performed on color-based change detection in the video modality and on time difference of arrival (TDOA) estimation between the two microphones in the audio modality. The TDOA is computed by multiband generalized cross correlation (GCC) analysis. The estimated directions of arrival are then postprocessed using a Riccati Kalman filter. The visual and audio estimates are finally integrated, at the likelihood level, into a particle filter (PF) that uses a zero-order motion model, and a weighted probabilistic data association (WPDA) scheme. We demonstrate that the Kalman filtering (KF) improves the accuracy of the audio source localization and that the WPDA helps to enhance the tracking performance of sensor fusion in reverberant scenarios. The combination of multiband GCC, KF, and WPDA within the particle filtering framework improves the performance of the algorithm in noisy scenarios. We also show how the proposed audiovisual tracker summarizes the observed scene by generating metadata that can be transmitted to other network nodes instead of transmitting the raw images and can be used for very low bit rate communication. Moreover, the generated metadata can also be used to detect and monitor events of interest.

Development and validation of a dried blood spot-LC-APCI-MS assay for estimation of canrenone in paediatric samples

A selective and sensitive liquid chromatography (LC)-atmospheric pressure chemical ionisation (APCI)-mass spectroscopic (MS) assay of canrenone has been developed and validated employing Dried Blood Spots (DBS) as the sample collection medium. DBS samples were prepared by applying 30 mu l of spiked whole blood onto Guthrie cards. A 6 mm disc was punched from the each DBS and extracted with 2 ml of methanolic solution of 17 alpha-methyltestosterone (Internal Standard). The methanolic extract was evaporated to dryness and reconstituted in acetonitrile:water (1:9, v/v). The reconstituted solution was further subjected to solid phase extraction using HLB cartridges. Chromatographic separation was achieved using Waters Sunfire C18 reversed-phase column using isocratic elution, followed by a high organic wash to clear late eluting/highly retained components. The mobile phase consisted of methanol:water (60:40, v/v) pumped at a flow rate of 0.3 ml/min. LC-APCI-MS detection was performed in the selected-ion monitoring (SIM) mode using target ions at m/z 341.1 and 303.3 for canrenone and internal standard respectively. The selectivity of the method was established by analysing DBS samples from 6 different sources (individuals). The calibration curve for canrenone was found to be linear over 25-1000 ng/ml (r >0.994). Accuracy (% RE) and precision (% CV) values for within and between day were