Visuacy, imagination, and engagement

Australia’s National Review of Visual Education (DEEWR, 2009) asserts the primacy of visual language ability, or ‘visuacy” in problem-solving. This paper reports on a recent university/schools research project with ‘at risk’ middle school students in which visuacy was promoted as a primary medium for obtaining data relating to issues of immediate concern to the students. Using a students-as-researchers approach, the project investigated middle school students’ perspectives on school engagement and disengagement. In this project, novice researchers used a variety of data gathering methods including photography, video interviews and drawn images as well as more traditional verbal methods, such as interviews, and quantitative methods, such as questionnaires. Engaging student imagination was a key focus of the approach taken by the project, acknowledging that student participants may be reluctant to enter dialogue with teachers and researchers on matters to which they have previously had little input. Students who have previously been marginalized and prevented from contributing their voices to educational forums often have difficulty in adjusting to the novelty of collaborative research with adults (Rudduck, 2003) and may be uncertain of their own place in the relationship that defines teacher/student interactions. It is argued that the project’s promotion of visuacy, alongside more traditional literacies and numeracy in education research, helped to overcome these concerns, engaged the imaginations of the student researchers, and provided a medium for the expression of the voices of marginalised young people.

Dancing the Thesis : Potentials and Pitfalls in Practice-led Research

Practice-led or multi modal theses (describing examinable outcomes of postgraduate study which comprise the practice of dancing/choreography with an accompanying exegesis) are an emerging strength of dance scholarship; a form of enquiry that has been gaining momentum for over a decade, particularly in Australia and the United Kingdom. It has been strongly argued that, in this form of research, legitimate claims to new knowledge are embodied predominantly within the practice itself (Pakes, 2003) and that these findings are emergent, contingent and often interstitial, contained within both the material form of the practice and in the symbolic languages surrounding the form. In a recent study on ‘dancing’ theses Phillips, Stock, Vincs (2009) found that there was general agreement from academics and artists that ‘there could be more flexibility in matching written language with conceptual thought expressed in practice’. The authors discuss how the seemingly intangible nature of danced / embodied research, reliant on what Melrose (2003) terms ‘performance mastery’ by the ‘expert practitioner’ (2006, Point 4) involving ‘expert’ intuition (2006, Point 5), might be accessed, articulated and validated in terms of alternative ways of knowing through exploring an ongoing dialogue in which the danced practice develops emergent theory. They also propose ways in which the danced thesis can be ‘converted’ into the required ‘durable’ artefact which the ephemerality of live performance denies, drawing on the work of Rye’s ‘multi-view’ digital record (2003) and Stapleton’s ‘multi-voiced audio visual document’(2006, 82). Building on a two-year research project (2007-2008) Dancing Between Diversity and Consistency: Refining Assessment in Postgraduate Degrees in Dance, which examined such issues in relation to assessment in an Australian context, the three researchers have further explored issues around interdisciplinarity, cultural differences and documentation through engaging with the following questions:  How do we represent research in which understandings, meanings and findings are situated within the body of the dancer/choreographer?  Do these need a form of ‘translating’ into textual form in order to be accessed as research?  What kind of language structures can be developed to effect this translation: metaphor, allusion, symbol?  How important is contextualising the creative practice?  How do we incorporate differing cultural inflections and practices into our reading and evaluation?  What kind of layered documentation can assist in producing a ‘durable’ research artefact from a non-reproduce-able live event?

Towards a definition of the performing AudioVisualist

The evolution of the laptop computer as a musical instrument in the 1990s provided a tool for empowering the solo musician and divergent approaches to the application of this technology in performance remain consistently debated.  The increasing ubiquity of digital media combined with the power of current generation notebook technology has provided the perfect platform to realise integrated audio-visual toolsets that respond to musical controllers and provide mixed-media results.  Despite emerging practitioners increasingly availing themselves to the musical affordances of this technology, theoretical discussion in the field ignores the various approaches a solo musician might take in developing integrated media works for performance.   In an increasingly crowded niche there is a clear compulsion to consider expanded modes of performance, yet lacking any formal framework these integrations can easily alienate an audience, distract from performance and lead to criticisms of novelty for novelty's sake. 

Performing with grid music systems

Grid music systems provide discrete geometric methods for simplified music-making by providing spatialised input to construct patterned music on a 2D matrix layout. While they are conceptually simple, grid systems may be layered to enable complex and satisfying musical results. Grid music systems have been applied to a range of systems from small portable devices up to larger systems. In this paper we will discuss the use of grid music systems in general and present an overview of the HarmonyGrid system we have developed as a new interactive performance system. We discuss a range of issues related to the design and use of larger-scale grid- based interactive performance systems such as the HarmonyGrid.

Digital rights management

Digital rights management allows information owners to control the use and dissemination of electronic documents via a machine-readable licence. Documents are distributed in a protected form such that they may only be used with trusted environments, and only in accordance with terms and conditions stated in the licence. Digital rights management has found uses in protecting copyrighted audio-visual productions, private personal information, and companies' trade secrets and intellectual property. This chapter describes a general model of digital rights management together with the technologies used to implement each component of a digital rights management system, and desribes how digital rights management can be applied to secure the distribution of electronic information in a variety of contexts.

Visualizing digital media interactions : providing feedback on jam2jam AV performances

Instrumental music performance is a well-established case of real-time interaction with technology and, when extended to ensembles, of interaction with others. However, these interactions are fleeting and the opportunities to reflect on action is limited, even though audio and video recording has recently provided important opportunities in this regard. In this paper we report on research to further extend these reflective opportunities through the capture and visualization of gestural data collected during collaborative virtual performances; specifically using the digital media instrument Jam2jam AV and the specifically-developed visualization software Jam2jam AV Visualize. We discusses how such visualization may assist performance development and understanding. The discussion engages with issues of representation, authenticity of virtual experiences, intersubjectivity and wordless collaboration, and creativity support. Two usage scenarios are described showing that collaborative intent is evident in the data visualizations more clearly than in audio-visual recordings alone, indicating that the visualization of performance gestures can be an efficient way of identifying deliberate and co-operative performance behaviours.

Trans digital storytelling : everyday activism, mutable identity and the problem of visibility

Mainstream representations of trans people typically run the gamut from victim to mentally ill and are almost always articulated by non-trans voices. The era of user-generated digital content and participatory culture has heralded unprecedented opportunities for trans people who wish to speak their own stories in public spaces. Digital Storytelling, as an easy accessible autobiographic audio-visual form, offers scope to play with multi-dimensional and ambiguous representations of identity that contest mainstream assumptions of what it is to be ‘male’ or ‘female’. Also, unlike mainstream media forms, online and viral distribution of Digital Stories offer potential to reach a wide range of audiences, which is appealing to activist oriented storytellers who wish to confront social prejudices. However, with these newfound possibilities come concerns regarding visibility and privacy, especially for storytellers who are all too aware of the risks of being ‘out’ as trans. This paper explores these issues from the perspective of three trans storytellers, with reference to the Digital Stories they have created and shared online and on DVD. These examplars are contextualised with some popular and scholarly perspectives on trans representation, in particular embodied and performed identity. It is contended that trans Digital Stories, while appearing in some ways to be quite conventional, actually challenge common notions of gender identity in ways that are both radical and transformative.

Multi-modal summarization of key events and top players in sports tournament videos

To detect and annotate the key events of live sports videos, we need to tackle the semantic gaps of audio-visual information. Previous work has successfully extracted semantic from the time-stamped web match reports, which are synchronized with the video contents. However, web and social media articles with no time-stamps have not been fully leveraged, despite they are increasingly used to complement the coverage of major sporting tournaments. This paper aims to address this limitation using a novel multimodal summarization framework that is based on sentiment analysis and players' popularity. It uses audiovisual contents, web articles, blogs, and commentators' speech to automatically annotate and visualize the key events and key players in a sports tournament coverage. The experimental results demonstrate that the automatically generated video summaries are aligned with the events identified from the official website match reports.

Automatic generation of effective video summaries

As the popularity of video as an information medium rises, the amount of video content that we produce and archive keeps growing. This creates a demand for shorter representations of videos in order to assist the task of video retrieval. The traditional solution is to let humans watch these videos and write textual summaries based on what they saw. This summarisation process, however, is time-consuming. Moreover, a lot of useful audio-visual information contained in the original video can be lost. Video summarisation aims to turn a full-length video into a more concise version that preserves as much information as possible. The problem of video summarisation is to minimise the trade-off between how concise and how representative a summary is. There are also usability concerns that need to be addressed in a video summarisation scheme. To solve these problems, this research aims to create an automatic video summarisation framework that combines and improves on existing video summarisation techniques, with the focus on practicality and user satisfaction. We also investigate the need for different summarisation strategies in different kinds of videos, for example news, sports, or TV series. Finally, we develop a video summarisation system based on the framework, which is validated by subjective and objective evaluation. The evaluation results shows that the proposed framework is effective for creating video skims, producing high user satisfaction rate and having reasonably low computing requirement. We also demonstrate that the techniques presented in this research can be used for visualising video summaries in the form web pages showing various useful information, both from the video itself and from external sources.

Robust speaker verification via fusion of speech and lip modalities

This paper investigates the use of lip information, in conjunction with speech information, for robust speaker verification in the presence of background noise. It has been previously shown in our own work, and in the work of others, that features extracted from a speaker's moving lips hold speaker dependencies which are complementary with speech features. We demonstrate that the fusion of lip and speech information allows for a highly robust speaker verification system which outperforms the performance of either sub-system. We present a new technique for determining the weighting to be applied to each modality so as to optimize the performance of the fused system. Given a correct weighting, lip information is shown to be highly effective for reducing the false acceptance and false rejection error rates in the presence of background noise

The use of temporal speech and lip information for multi-modal speaker identification via multi-stream HMMs

Investigates the use of temporal lip information, in conjunction with speech information, for robust, text-dependent speaker identification. We propose that significant speaker-dependent information can be obtained from moving lips, enabling speaker recognition systems to be highly robust in the presence of noise. The fusion structure for the audio and visual information is based around the use of multi-stream hidden Markov models (MSHMM), with audio and visual features forming two independent data streams. Recent work with multi-modal MSHMMs has been performed successfully for the task of speech recognition. The use of temporal lip information for speaker identification has been performed previously (T.J. Wark et al., 1998), however this has been restricted to output fusion via single-stream HMMs. We present an extension to this previous work, and show that a MSHMM is a valid structure for multi-modal speaker identification

Content-based video indexing for sports applications using integrated multi-modal approach

To sustain an ongoing rapid growth of video information, there is an emerging demand for a sophisticated content-based video indexing system. However, current video indexing solutions are still immature and lack of any standard. This doctoral consists of a research work based on an integrated multi-modal approach for sports video indexing and retrieval. By combining specific features extractable from multiple audio-visual modalities, generic structure and specific events can be detected and classified. During browsing and retrieval, users will benefit from the integration of high-level semantic and some descriptive mid-level features such as whistle and close-up view of player(s).

Re-made in Asia : transformation across Asian markets and popular culture

The thesis is an examination of how Japanese popular culture products are remade (rimeiku). Adaptation of manga, anime and television drama, from one format to another, frequently occurs within Japan. The rights to these stories and texts are traded in South Korea and Taiwan. The ‘spin-off’ products form part of the Japanese content industry. When products are distributed and remade across geographical boundaries, they have a multi-dimensional aspect and potentially contribute to an evolving cultural re-engagement between Japan and East Asia. The case studies are the television dramas Akai Giwaku and Winter Sonata and two manga, Hana yori Dango and Janguru Taitei. Except for the television drama Winter Sonata these texts originated in Japan. Each study shows how remaking occurs across geographical borders. The study argues that Japan has been slow to recognise the value of its popular culture through regional and international media trade. Japan is now taking steps to remedy this strategic shortfall to enable the long-term viability of the Japanese content industry. The study includes an examination of how remaking raises legal issues in the appropriation of media content. Unauthorised copying and piracy contributes to loss of financial value. To place the three Japanese cultural products into a historical context, the thesis includes an overview of Japanese copying culture from its early origins through to the present day. The thesis also discusses the Meiji restoration and the post-World War II restructuring that resulted in Japan becoming a regional media powerhouse. The localisation of Japanese media content in South Korea and Taiwan also brings with it significant cultural influences, which may be regarded as contributing to a better understanding of East Asian society in line with the idea of regional ‘harmony’. The study argues that the commercial success of Japanese products beyond Japan is governed by perceptions of the quality of the story and by the cultural frames of the target audience. The thesis draws on audience research to illustrate the loss or reinforcement of national identity as a consequence of cross-cultural trade. The thesis also examines the contribution to Japanese ‘soft power’ (Nye, 2004, p. x). The study concludes with recommendations for the sustainability of the Japanese media industry.