Computational media aesthetics: Finding meaning beautiful

Innovative media management, annotation, delivery, and navigation services will enrich online shopping, help-desk services, and anytime-anywhere training over wireless devices. However, the semantic gap between the rich meaning that users want when they query and browse media and the shallowness of the content descriptions that one can actually compute is weakening today's automatic content-annotation systems. To address such problems, an approach that markedly departs from existing methods based on detecting and annotating low-level audio-visual features is advocated.

Application of computational media aesthetics methodology to extracting color semantics in film

Using film grammar as the underpinning, we study the extraction of structures in video based on color using a wide configuration of clustering methods combined with existing and new similarity measures. We study the visualisation of these structures, which we call Scene-Cluster Temporal Charts and show how it can bring out the interweaving of different themes and settings in a film. We also extract color events that filmmakers use to draw/force a viewer's attention to a shot/scene. This is done by first extracting a set of colors used rarely in film, and then building a probabilistic model for color event detection. We demonstrate with experimental results from ten movies that our algorithms are effective in the extraction of both scene-cluster temporal charts and color events.

Transmedia as ‘Unmixed Media’ aesthetics

What is ‘transmedia’? This chapter attempts to understand what this phenomenon is from a transhistorical perspective. The transhistorical nature of media combinations is interrogated in light of media specificity and in relation to the key phenomena of Intermedia. Transmedia is argued to be a transhistorical urge towards unifying that which has been artificially estranged.

IMCE : integrated media creation environment

We discuss the design goals for an integrated media creation environment (IMCE) aimed at enabling the average user to create media artifacts with professional qualities. The resulting requirements are implemented and we demonstrate the efficacy of the resulting system with the generation of two simple home movies. The significance for the average user seeking to create home movies lies in the flexible and automatic application of film principles to the task, removal of tedious low-level editing by means of well-formed media transformations in terms of high-level film constructs (e.g., tempo), and content repurposing powered by those same transformations added to the rich semantic information maintained at each phase of the process.

Finding the beat : an analysis of the rhythmic elements of motion pictures

This paper presents a new study on the application of the framework of Computational Media Aesthetics to the problem of automated understanding of film. Leveraging Film Grammar as the means to closing the "semantic gap" in media analysis, we examine film rhythm, a powerful narrative concept used to endow structure and form to the film compositionally and enhance its lyrical quality experientially. The novelty of this paper lies in the specification and investigation of the rhythmic elements that are present in two cinematic devices; namely motion and editing patterns, and their potential usefulness to automated content annotation and management systems. In our rhythm model, motion behavior is classified as being either nonexistent, fluid or staccato for a given shot. Shot neighborhoods in movies are then grouped by proportional makeup of these motion behavioral classes to yield seven high-level rhythmic arrangements that prove to be adept at indicating likely scene content (e.g. dialogue or chase sequence) in our experiments. The second part of our investigation presents a computational model to detect editing patterns as either metric, accelerated, decelerated or free. Details of the algorithm for the extraction of these classes are presented, along with experimental results on real movie data. We show with an investigation of combined rhythmic patterns that, while detailed content identification via rhythm types alone is not possible by virtue of the fact that film is not codified to this level in terms of rhythmic elements, analysis of the combined motion/editing rhythms can allow us to determine that the content has changed and hypothesize as to why this is so. We present three such categories of change and demonstrate their efficacy for capturing useful film elements (e.g. scene change precipitated by plot event), by providing data support from five motion pictures.

Scene extraction in motion pictures

This paper addresses the challenge of bridging the semantic gap between the rich meaning users desire when they query to locate and browse media and the shallowness of media descriptions that can be computed in today's content management systems. To facilitate high-level semantics-based content annotation and interpretation, we tackle the problem of automatic decomposition of motion pictures into meaningful story units, namely scenes. Since a scene is a complicated and subjective concept, we first propose guidelines from fill production to determine when a scene change occurs. We then investigate different rules and conventions followed as part of Fill Grammar that would guide and shape an algorithmic solution for determining a scene. Two different techniques using intershot analysis are proposed as solutions in this paper. In addition, we present different refinement mechanisms, such as film-punctuation detection founded on Film Grammar, to further improve the results. These refinement techniques demonstrate significant improvements in overall performance. Furthermore, we analyze errors in the context of film-production techniques, which offer useful insights into the limitations of our method.

Situated event bootstrapping and capture guidance for automated home movie authoring

This paper describes a novel interactive media authoring framework, MediaTE, that enables amateurs to create videos of higher narrative or aesthetic quality with a completely mobile lifecycle. A novel event bootstrapping dialog is used to derive shot suggestions that yield both targetted footage and annotation enabling an automatic Computational Media Aesthetics-aware editing phase, the manual performance of which is typically a barrier to the amateur. This facilitates a move away from requiring a prior-conception of the events or locale being filmed, in the form of a template, to at-capture bootstrapping of this information. Metadata gathered as part of the critical path of media creation also has implications for the longevity and reuse of captured media assets. Results of an evaluation performed on both the usability and delivered media aspects of the system are discussed, which highlight the tenability of the proposed framework and the quality of the produced media.

Weaving stories in digital media : when Spielberg makes home movies

In this paper we describe research aimed at enabling amateur video makers to improve both the technical quality and communicative capacity of their work. Motivated by the recognition that untold hours of home video are simply abandoned after capture, we have formulated the problem as one of defining the what and how of footage capture. We have implemented a framework that answers the first problem, the what, by means of the age-old communicative powers of Story; the second problem, the how, is addressed by means of well documented aesthetic principles that constitute the film profession, which impact both technical and cinematic considerations for a given project. We provide a brief overview of the process, beginning with the narrative template, embodying a chosen story, through the principal phases of generating a storyboard, directing, and editing, resulting in the finished product. We demonstrate the interplay of narrative, purpose for the production, and aesthetic agents, and their influence on the automatically generated storyboard with examples.

Distracted

Distracted is a luminous, interactive, computational media installation of sound, light and translucent sculptural materials. The work is inspired by scientific ice core samples taken in Antarctica. The sculpture is capable of displaying data taken from these ice core samples, and responding to the proximity of an audience. Rather than simply using the interface as a didactic display device, we have chosen a more poetic approach of generating visual effects from the data that are evocative of the ice, fluids and the notion of change. The data has also been used in the composition of an evolving soundscape. As well as data from ice core samples, such as the Vostok ice core, we have incorporated data from the Keeling Curve that shows the annual rise and fall of atmospheric carbon dioxide, following the pattern of the Northern Hemisphere winter. These effects combine with changes caused directly by audience members as they come within close proximity to the work.

Distracted : poetic interpretations of climate change

This paper introduces the creative work Distracted and discusses conceptual, aesthetic and technical aspects of the work. The work was conceived as a luminous, interactive, computational media installation informed by our interest in the Antarctica. Through the paper we focus on: how the work addresses the themes of climate change and sustainability; how we attempted to work with selected sets of scientific data to evoke the delicate yet extreme nature of the environment and the ways in which ice is a record of the earth’s geological history and recent human impacts; and how the process of making this artwork caused us to reconsider our practices and formulate strategies for redirecting our practice in a manner that addresses the challenges of sustainability.

Emergence and the art system ‘plus minus now’

Emergence is discussed in the context of a practice-based study of interactive art and a new taxonomy of emergence is proposed. The interactive art system ‘plus minus now’ is described and its relationship to emergence is discussed. ‘Plus minus now’ uses a novel method for instantiating emergent shapes. A preliminary investigation of this art system has been conducted and reveals the creation of temporal compositions by a participant. These temporal compositions and the emergent shapes are described using the taxonomy of emergence. Characteristics of emergent interactions and the implications of designing for them are discussed.

Ghost/Machine: cinema in a digital age

Cinema, with its passive cinematic apparatus and linear narrative is often characterised as a contrast to new media narrative strategies, yet from Vertov’s Man with a Movie Camera to Mike Figgis’ TimeCode and Wong Kar Wei’s 2046 cinema provides narrative strategies and spatial conceptualisations which prefigure or are contiguous with new media environments. Both our perception of what cyberspace constitutes and the technology that actualises those perceptions arise out of and are driven by fantasy and desire. This paper will explore the metaphors used to represent and understand new media aesthetics through cinematic representations of new media environments. Two key themes relevant to new media aesthetics emerge. Irigaray, Haraway, and Grosz are used to explore the de-essentialising haptic and penetrative potential of new technologies and their ability to collapse the boundary between the body and the machine. The second fantasy, of new media as a liminal space that expresses the memorialising function of technology and its relation to mourning, is analysed using Benjamin, Burgin and Rutsky. These altered spaces and perceptions of the body and memory of the post-cinematic subject are illustrated through an analysis of Gondry’s Eternal Sunshine of the Spotless Mind and Jonze’s Being John Malkovich. [From the Author]