Comparison of Image Transform-Based Features for Visual Speech Recognition in Clean and Corrupted Videos

We present results of a study into the performance of a variety of different image transform-based feature types for speaker-independent visual speech recognition of isolated digits. This includes the first reported use of features extracted using a discrete curvelet transform. The study will show a comparison of some methods for selecting features of each feature type and show the relative benefits of both static and dynamic visual features. The performance of the features will be tested on both clean video data and also video data corrupted in a variety of ways to assess each feature type's robustness to potential real-world conditions. One of the test conditions involves a novel form of video corruption we call jitter which simulates camera and/or head movement during recording.

Hidden Conditional Random Fields for Visual Speech Recognition

In this paper we present the application of Hidden Conditional Random Fields (HCRFs) to modelling speech for visual speech recognition. HCRFs may be easily adapted to model long range dependencies across an observation sequence. As a result visual word recognition performance can be improved as the model is able to take more of a contextual approach to generating state sequences. Results are presented from a speaker-dependent, isolated digit, visual speech recognition task using comparisons with a baseline HMM system. We firstly illustrate that word recognition rates on clean video using HCRFs can be improved by increasing the number of past and future observations being taken into account by each state. Secondly we compare model performances using various levels of video compression on the test set. As far as we are aware this is the first attempted use of HCRFs for visual speech recognition.

Inter-Frame Contextual Modelling For Visual Speech Recognition

In this paper, we present a new approach to visual speech recognition which improves contextual modelling by combining Inter-Frame Dependent and Hidden Markov Models. This approach captures contextual information in visual speech that may be lost using a Hidden Markov Model alone. We apply contextual modelling to a large speaker independent isolated digit recognition task, and compare our approach to two commonly adopted feature based techniques for incorporating speech dynamics. Results are presented from baseline feature based systems and the combined modelling technique. We illustrate that both of these techniques achieve similar levels of performance when used independently. However significant improvements in performance can be achieved through a combination of the two. In particular we report an improvement in excess of 17% relative Word Error Rate in comparison to our best baseline system.

Music, Magic, and the Mythic: The Dynamics of Visual and Aural Discourse in Souleymane Cissé’s Yeelen

The 1980s saw a wave of African films that aimed to represent, on both local and international screens, a sophisticated pre-colonial Africa, thus debunking notions of the continent as primitive. Toward this aim the films inscribed the conventions of oral performance within their visual styles, denying spectator identification with the protagonists and emphasising the presence of the narrator. However, some critics argued that these films exoticised Africa, while their use of oral performance’s distancing effect echoed the ‘scientific’ distance structured by the ethnographic film, in which African societies were represented as ‘the other’. Souleymane Cissé’s Yeelen exemplifies this tension, transposing into cinematic form oral storytelling techniques in the depiction of a power struggle within the covert cult of the komo, a Bambara initiation society unfamiliar to most non-Bambara viewers. This paper demonstrates how the film negotiates this tension via music, which interpellates the international spectator by eliciting a greater identification with the protagonists than that determined at a visual level, while encoding a verisimilitude to rituals that may otherwise be read as the superstitious practices of ‘the other’. In this way, music and image in Yeelen operate as parallel, though often overlapping, discourses, bridging the gap between the film’s culturally specific narrative and formal components, and its international spectators.

Voyage(s) to Sarajevo: Jean-Luc Godard and the Image of War

This essay discusses Jean-Luc Godard’s artistic response to the Bosnian War (1992-95), and its representations in the Western mass media. For Godard, the reluctance of Europe’s advanced liberal democracies to intervene meaningfully in Bosnia – their insistence that 'humanitarianism' rather than protective intervention was the order of the day – was tantamount to supporting Serbian fascism, and – a fortiori – regressing to a policy of appeasement reminiscent of the days of the Munich Agreement. Although Godard's stance set him against some of his former compatriots on the left, speculating on his ideological motivations is beside the point. Rather, it is is in his filmmaking, in his vision of cinema, and how it relates to other histories of the image, that Godard’s sensibility can be most keenly felt and understood. As the essay points out, even his recent contribution to Jean-Michel Frodon's compilation film, Bridges of Sarajevo/Les ponts de Sarajevo (2014, 114 mn.), persists in posing questions about how the past continues to shape the present, and how Sarajevo and its contemporary history still delineates the identity of Europe. 

Automated image analysis for experimental investigations of salt water intrusion in coastal aquifers

A novel methodology has been developed to quantify important saltwater intrusion parameters in a sandbox style experiment using image analysis. Existing methods found in the literature are based mainly on visual observations, which are subjective, labour intensive and limits the temporal and spatial resolutions that can be analysed. A robust error analysis was undertaken to determine the optimum methodology to convert image light intensity to concentration. Results showed that defining a relationship on a pixel-wise basis provided the most accurate image to concentration conversion and allowed quantification of the width of mixing zone between the saltwater and freshwater. A large image sample rate was used to investigate the transient dynamics of saltwater intrusion, which rendered analysis by visual observation unsuitable. This paper presents the methodologies developed to minimise human input and promote autonomy, provide high resolution image to concentration conversion and allow the quantification of intrusion parameters under transient conditions.

Deep Salience: Visual Salience Modeling via Deep Belief Propagation

Visual salience is an intriguing phenomenon observed in biological neural systems. Numerous attempts have been made to model visual salience mathematically using various feature contrasts, either locally or globally. However, these algorithmic models tend to ignore the problem’s biological solutions, in which visual salience appears to arise during the propagation of visual stimuli along the visual cortex. In this paper, inspired by the conjecture that salience arises from deep propagation along the visual cortex, we present a Deep Salience model where a multi-layer model based on successive Markov random fields (sMRF) is proposed to analyze the input image successively through its deep belief propagation. As a result, the foreground object can be automatically separated from the background in a fully unsupervised way. Experimental evaluation on the benchmark dataset validated that our Deep Salience model can consistently outperform eleven state-of-the-art salience models, yielding the higher rates in the precision-recall tests and attaining the best F-measure and mean-square error in the experiments.