Audiovisual citation: a project update

Existing referencing systems frequently prove inadequate for the citation of moving image and sound media such as vidcasts, streaming television, sound files, un-catalogued archive footage, amateur content hosted online or
non-broadcast radio recordings. Back in 2009 and 2010 a British working group funded by Higher Education Funding Council for England (HEFCE) and co-ordinated by the British Universities Film and Video Council investigated this problem. This report documents the early stages of the project.

Target Detection and Tracking With Heterogeneous Sensors

We present a multimodal detection and tracking algorithm for sensors composed of a camera mounted between two microphones. Target localization is performed on color-based change detection in the video modality and on time difference of arrival (TDOA) estimation between the two microphones in the audio modality. The TDOA is computed by multiband generalized cross correlation (GCC) analysis. The estimated directions of arrival are then postprocessed using a Riccati Kalman filter. The visual and audio estimates are finally integrated, at the likelihood level, into a particle filter (PF) that uses a zero-order motion model, and a weighted probabilistic data association (WPDA) scheme. We demonstrate that the Kalman filtering (KF) improves the accuracy of the audio source localization and that the WPDA helps to enhance the tracking performance of sensor fusion in reverberant scenarios. The combination of multiband GCC, KF, and WPDA within the particle filtering framework improves the performance of the algorithm in noisy scenarios. We also show how the proposed audiovisual tracker summarizes the observed scene by generating metadata that can be transmitted to other network nodes instead of transmitting the raw images and can be used for very low bit rate communication. Moreover, the generated metadata can also be used to detect and monitor events of interest.

The Fraternity of Metaphors: Jean-Luc Godard's Film Socialisme

Since the 'completion' of Histoire(s) du cinema (1988-1998), Jean-Luc Godard's work has become increasingly mosaic-like in its forms and configurations, and markedly elegiac in its ruminations on history, cinema, art, and thought. While his associative aesthetic and citational method –including his choice of ‘actors’, and the fragmentariness of his ‘soundtracks’ – can combine to create a distinctive cinematic event, the films themselves refuse to cohere around a unifying concern, or yield to a thematic schema. Not surprisingly, Film Socialisme does not offer us the illusion of narrative or structural integrity anymore than it contributes to the quotidian rhetoric of political and moral argument. It is, however, a political film in the sense that it alters something more fundamental than opinions and points of view. It transforms a way of seeing and understanding reality and history, fiction and documentary, images, and images of images. If anything, it belongs to that dissident or ‘dissensual’ category of artwork capable of ‘emancipating the spectator’ by disturbing what Jacques Rancière terms ‘the distribution of the sensible’ in that it generates gaps, openings, and spaces, poses questions, invites associations without positing a fixed position, imposing an interpretation, or allowing itself to invest in the illusion of expressive objectivity and the stability of meaning. The myriad citations and fragments that comprise the film are never intended to culminate into anything cohesive, never mind conclusive. In one sense, they have no source and no context beyond their moment in the film itself, and what we make of that moment. This article studies the degree to which Godard allows these images and sounds to combine and collide, associate and dissolve in this film, arguing that Film Socialisme is both an important intervention in the history of contemporary cinema, and necessary point of reference in any serious discussion of the relations between that cinema and political reality.

EDEN EDEN EDEN

An overnight audiovisual composition for 13 musicians and film projections. The ensemble continuously constructs and deconstructs a 26-part chord, a 13-part natural harmonic series that always beats in near-unison with itself. A projector shows three superimposed films prepared as continuously extending loops. Members of the audience, who are invited to sleep during the performance, shift between waking and dreaming states. Concept and composition by Gascia Ouzounian. Film by Chloe Griffin.

The SEMAINE Database: Annotated Multimodal Records of Emotionally Colored Conversations between a Person and a Limited Agent

SEMAINE has created a large audiovisual database as a part of an iterative approach to building Sensitive Artificial Listener (SAL) agents that can engage a person in a sustained, emotionally colored conversation. Data used to build the agents came from interactions between users and an operator simulating a SAL agent, in different configurations: Solid SAL (designed so that operators displayed an appropriate nonverbal behavior) and Semi-automatic SAL (designed so that users' experience approximated interacting with a machine). We then recorded user interactions with the developed system, Automatic SAL, comparing the most communicatively competent version to versions with reduced nonverbal skills. High quality recording was provided by five high-resolution, high-framerate cameras, and four microphones, recorded synchronously. Recordings total 150 participants, for a total of 959 conversations with individual SAL characters, lasting approximately 5 minutes each. Solid SAL recordings are transcribed and extensively annotated: 6-8 raters per clip traced five affective dimensions and 27 associated categories. Other scenarios are labeled on the same pattern, but less fully. Additional information includes FACS annotation on selected extracts, identification of laughs, nods, and shakes, and measures of user engagement with the automatic system. The material is available through a web-accessible database. © 2010-2012 IEEE.

Optimal diet choice for large herbivores: an extended contingency model

1. A more general contingency model of optimal diet choice is developed, allowing for simultaneous searching and handling, which extends the theory to include grazing and browsing by large herbivores.</p><p>2. Foraging resolves into three modes: purely encounter-limited, purely handling-limited and mixed-process, in which either a handling-limited prey type is added to an encounter-limited diet, or the diet becomes handling-limited as it expands.</p><p>3. The purely encounter-limited diet is, in general, broader than that predicted by the conventional contingency model,</p><p>4. As the degree of simultaneity of searching and handling increases, the optimal diet expands to the point where it is handling-limited, at which point all inferior prey types are rejected,</p><p>5. Inclusion of a less profitable prey species is not necessarily independent of its encounter rate and the zero-one rule does not necessarily hold: some of the less profitable prey may be included in the optimal diet. This gives an optimal foraging explanation for herbivores' mixed diets.</p><p>6. Rules are shown for calculating the boundary between encounter-limited and handling-limited diets and for predicting the proportion of inferior prey to be included in a two-species diet,</p><p>7. The digestive rate model is modified to include simultaneous searching and handling, showing that the more they overlap, the more the predicted diet-breadth is likely to be reduced.</p>

AVEC 2011–The First International Audio/Visual Emotion Challenge

The Audio/Visual Emotion Challenge and Workshop (AVEC 2011) is the first competition event aimed at comparison of multimedia processing and machine learning methods for automatic audio, visual and audiovisual emotion analysis, with all participants competing under strictly the same conditions. This paper first describes the challenge participation conditions. Next follows the data used – the SEMAINE corpus – and its partitioning into train, development, and test partitions for the challenge with labelling in four dimensions, namely activity, expectation, power, and valence. Further, audio and video baseline features are introduced as well as baseline results that use these features for the three sub-challenges of audio, video, and audiovisual emotion recognition.

Demo: Have a Chat with Sensitive Artificial Listeners

Sensitive Artificial Listeners (SAL) are virtual dialogue partners based on audiovisual analysis and synthesis. Despite their very limited verbal understanding, they intend to engage the user in a conversation by paying attention to the user’s emotions and non- verbal expressions. The SAL characters have their own emotionally defined personality, and attempt to drag the user towards their dom- inant emotion through a combination of verbal and non-verbal ex- pression. The demonstrator shows the publicly available, fully au- tonomous SAL system.