Some Reflections on the Emergence of Multimodal Assemblages of Meaning

In this commentary I reflect upon the conceptualisation of human meaning-making, utilised in the two target articles, that relies heavily on speech as the main mode of semiosis and considers time only in its chronological form. Instead I argue that human existence is embodied and lived through multiple modalities, and involves not only sequential experience of time, but also experience of emergence. In order to move towards a conception of meaning-making that takes this into account, I introduce the social-semiotic theory of multimodality (Kress 2010) and discuss notions of ‘real duration’ (Bergson 1907/1998) and ‘lived time’ (Martin-Vallas 2009). I argue that dialogical (idiographic) researchers need to develop analytic and methodological tools that allow exploring the emergence of multimodal assemblages of meaning in addition to trying to avoid the monologisation of complex dynamic dialogical phenomena.

Historical timelines : analyzing multimodal text design

The recent focus on literacy in Social Studies has been on linguistic design, particularly that related to the grammar of written and spoken text. When students are expected to produce complex hybridized genres such as timelines, a focus on the teaching and learning of linguistic design is necessary but not sufficient to complete the task. Theorizations of new literacies identify five interrelated meaning making designs for text deconstruction and reproduction: linguistic, spatial, visual, gestural, and audio design. Honing in on the complexity of timelines, this paper casts a lens on the linguistic, visual, spatial, and gestural designs of three pairs of primary school aged Social Studies learners. Drawing on a functional metalanguage, we analyze the linguistic, visual, spatial, and gestural designs of their work. We also offer suggestions of their effect, and from there consider the importance of explicit instruction in text design choices for this Social Studies task. We conclude the analysis by suggesting the foci of explicit instruction for future lessons.

The kineikonic mode : a multimodal analysis of students’ claymation movies

Globalised communication in society today is characterised by multimodal forms of meaning making in the context of increased cultural and linguistic diversity. This research paper responds to these imperatives, applying Halliday's (1978, 1994) categories of systemic functional linguistics - representational or ideational, interactive or interpersonal, and compositional or textual meanings. Following the work of Kress (2000), van Leeuwen (Kress and van Leeuwen, 1996), and Jewitt (2006), multimodal semiotic analysis is applied to claymation movies that were collaboratively designed by Year 6 students. The significance of this analysis is the metalanguage for textual work in the kineikonic mode - moving images.

Multimodal viral communication: a social semiotic analysis of the popular Spanish "Ola ke ase" meme

Multimodality – the interdependence of semiotic resources in text – is an existential element of today’s media. The term multimodality attends systematically to the social interpretation of a wide range of communicational forms used in meaning making. A primary focus of social- semiotic multimodal analysis is on mapping how modal resources are used by people in a given social context. In November 2012 the “Ola ke ase” catchphrase, which is a play on “Hola ¿qué hace?”, appeared for the first time in Spain and immediately has been adopted as a Twitter hashtag and an image macro series. Its viral spread on social networks has been tremendous, being a trending topic in various Spanish-speaking countries. The objective of analysis is how language and image work together in the “Ola ke ase” meme. The interplay between text and image in one of the original memes and some of its variations is quantitatively analysed applying a social-semiotic approach. Results demonstrate how the “Ola ke ase” meme functions through its multimodal character and the non-standard orthography. The spread of uncountable variations of the meme shows the social process that goes on in the meaning making of the semiotic elements.

The orchestration of modes and EFL audio-visual comprehension: A multimodal discourse analysis of vodcasts

This thesis explores the role of multimodality in language learners’ comprehension, and more specifically, the effects on students’ audio-visual comprehension when different orchestrations of modes appear in the visualization of vodcasts. Firstly, I describe the state of the art of its three main areas of concern, namely the evolution of meaning-making, Information and Communication Technology (ICT), and audio-visual comprehension. One of the most important contributions in the theoretical overview is the suggested integrative model of audio-visual comprehension, which attempts to explain how students process information received from different inputs. Secondly, I present a study based on the following research questions: ‘Which modes are orchestrated throughout the vodcasts?’, ‘Are there any multimodal ensembles that are more beneficial for students’ audio-visual comprehension?’, and ‘What are the students’ attitudes towards audio-visual (e.g., vodcasts) compared to traditional audio (e.g., audio tracks) comprehension activities?’. Along with these research questions, I have formulated two hypotheses: Audio-visual comprehension improves when there is a greater number of orchestrated modes, and students have a more positive attitude towards vodcasts than traditional audios when carrying out comprehension activities. The study includes a multimodal discourse analysis, audio-visual comprehension tests, and students’ questionnaires. The multimodal discourse analysis of two British Council’s language learning vodcasts, entitled English is GREAT and Camden Fashion, using ELAN as the multimodal annotation tool, shows that there are a variety of multimodal ensembles of two, three and four modes. The audio-visual comprehension tests were given to 40 Spanish students, learning English as a foreign language, after the visualization of vodcasts. These comprehension tests contain questions related to specific orchestrations of modes appearing in the vodcasts. The statistical analysis of the test results, using repeated-measures ANOVA, reveal that students obtain better audio-visual comprehension results when the multimodal ensembles are constituted by a greater number of orchestrated modes. Finally, the data compiled from the questionnaires, conclude that students have a more positive attitude towards vodcasts in comparison to traditional audio listenings. Results from the audio-visual comprehension tests and questionnaires prove the two hypotheses of this study.

Making sense of violence: a study of narrative meaning

Dramatized violence has been a feature of entertainment in western civilization throughout history. The function of film violence is explored and compared to violence encountered in real life. The role of narrative in individuals' meaning-making processes is also investigated. Six adults were individually interviewed using a semi-structured schedule and narrative analysis was implemented. The findings revealed that real life violence is experientially distinct from film violence but narrative was found to be central to participants' quest for the meaning of violence in both contexts. The narrative framework of violence and whether it is justifiable were fundamental to participants' understanding. The function of violent film was found to be multifaceted: it can teach viewers about the consequences of violence; it allows them to speculate about their own and others' reactions to violence; and it provides an opportunity to experience something which is ordinarily outside of our experience in order to satisfy our human existential needs.

The Material Dynamics of a London-French Blog: A Multimodal Reading of Migrant Habitus

Through a fine-grained reading of a London-French blog, this article aims to shed light on the lived experience of the French community in London. The ethnosemiotic conceptual framework brings together ethnographic and semiotic schools of thought, focusing in particular on Pierre Bourdieu’s concept of habitus and Gunther Kress’s multimodal social semiotic analytical model. Habitus is broken down into its material manifestations of habitat, habit and habituation, all displayed in the blog and revealing of the blogger’s identity and positioning within the migration setting. As all modes are considered to be of equal semiotic potential, equivalent emphasis is placed on the multiple modes of meaning-making present in the blog, such as layout, colour, typography and language. By examining the dynamic relationships between blogger and audience, subjectivity and objectivity, on-line and on-land habitus, and intermodal dynamics themselves, through the prism of multimodality, hidden facets of the blogger’s cultural identity and sense of community belonging within the diasporic context begin to materialise.

here/there/then/now

here/there/then/now was a practice-led research project that brought together 10 independent artists in dance, music, theatre and visual/media arts to create a site-specific program within the walls of the Brisbane Powerhouse. The purpose was to explore how to best conceive flexible performance platforms, theatricalise site-specific work and engage new audiences through forms of promenade experience that could provide open choices on how and where to view it. The sold out season of 6 performances, which took place 14-19 May 2002, presented three discrete performance installations set in intimate parts of the building, each with their own aesthetic and communicative intention, culminating in a fourth in-theatre installation, where memories of the first three coalesced and were reinterrogated. Each site thereby investigated meaning-making via the moving body and its critical relationship with space and objects, in a dramatic re-contextualisation of traditional solo dance forms, now re-articulated through interdisciplinary practices. The benefit of this approach was the creation of a layered and multimodal experience that could be both shared and subsequently critiqued by performers and audience alike.

Mixing colours: An ICT tool based on a semiotic framework for mathematical meaning-makng about ratio and fractions

This paper reports on the research and development of an ICT tool to facilitate the learning of ratio and fractions by adult prisoners. The design of the ICT tool was informed by a semiotic framework for mathematical meaning-making. The ICT tool thus employed multiple semiotic resources including topological, typological, and social-actional resources. The results showed that individual semiotic resource could only represent part of the mathematical concept, while at the same time it might signify something else to create a misconception. When multiple semiotic resources were utilised the mathematical ideas could be better learnt.

Spatial literacies, design texts and emergent pedagogies in purposeful literacy curriculum

There has recently been an emphasis within literacy studies on both the spatial dimensions of social practices (Leander & Sheehy, 2004) and the importance of incorporating design and multiple modes of meaning-making into contemporary understandings of literacy (Cope & Kalantzis, 2000; New London Group, 1996). Kress (2003) in particular has outlined the potential implications of the cultural shift from the dominance of writing, based on a logic of time and sequence in time, to the dominance of the mode of the image, based on a logic of space. However, the widespread re-design of curriculum and pedagogy by classroom teachers to allow students to capitalise on the various affordances of different modes of meaning-making – including the spatial – remains in an emergent stage. We report on a project in which university researchers’ expertise in architecture, literacy and communications enabled two teachers in one school to expand the forms of literacy that primary school children engaged in. Starting from the school community’s concerns about an urban renewal project in their neighbourhood, we worked together to develop a curriculum of spatial literacies with real-world goals and outcomes.

The meaning of home for Karen and Chin refugees from Burma: An interpretative phenomenological approach

Loss of home is common to all people from a refugee background yet we have little understanding of the diversity of meaning associated with this important concept. A phenomenological approach was used to explore experiences of home amongst Karen and Chin refugees residing in Brisbane. In-depth, semi-structured interviews were conducted with nine participants from Karen and Chin backgrounds. The participants comprised five females and four males (mean age 40 years, median length of time in Australia 1.33 years). Participants described their migration stories, including pre- and post-migration history. Analysis was conducted using interpretative phenomenological analysis. Three superordinate themes, explicating the meaning of home for participants, were identified: home as the experience of a psychological space of safety and retreat; home as the socio-emotional space of relatedness to family; and home as geographical-emotional landscape. Loss of home was experienced as a multidimensional loss associated with emotional and physical disturbances. These findings, based upon a phenomenological paradigm, enhance understanding of the experience of being a refugee and of the suffering engendered by loss of home. They open up the possibility for conceptualizing refugee responses in terms of human suffering and meaning making.

Reshaping children’s knowledge through film making: What to film, how to film and why

How can teachers reinvigorate content area knowledge and representation through filmmaking? We give examples of what to film, how to film, and why, drawing on our visual ethnographic research with Year 5 students in a working class suburb of Logan, Queensland. The unit developed content knowledge of Indigenous places and practices through sensitising activities in nature. Valuing students’ funds of knowledge, we interpreted local places through epistemologies of different cultures. Through filmmaking workshops by a digital artist, students filmed community members in a local shopping mall about their perceptions of health and happiness in local places. Students were positioned as future community leaders, presenting their films at a national conference. To conclude, we map the dominant and marginalised, local and specialised, and print and visual forms of knowledge that were interwoven, reshaped, and shared through multimodal design.

Multiliteracies and a new metalanguage for the moving image

Globalised communication in society today is characterised by multimodal forms of meaning making in a context of increased cultural and linguistic diversity, calling for the teaching of multiliteracies. This transformation requires the development of a new metalanguage or language of description for the burgeoning and hybridised variety of text forms associated with information and multimedia technologies. To continue to teach to a narrow band of print-based genres, grammars, and skills is to ignore the reality of textual practices outside of schools. This paper draws from classroom research in a multiliteracies classroom to provide a multimodal analysis of a claymation movie. The significance of the paper is the synthesis of a multimodal metalanguage for teachers and students to describe the features of work in the kineikonic (moving image) mode.

Playing with Grammar in the Early Years : Learning about Language in the Australian Curriculum : English

As teacher/researchers interested in the pursuit of socially-just outcomes in early childhood education, the form and function of language occupies a special position in our work. We believe that mastering a range of literacy competences includes not only the technical skills for learning, but also the resources for viewing and constructing the world (Freire and Macdeo, 1987). Rather than seeing knowledge about language as the accumulation of technical skills alone, the viewpoint to which we subscribe treats knowledge about language as a dialectic that evolves from, is situated in, and contributes to a social arena (Halliday, 1978). We do not shy away from this position just because children are in the early years of schooling. In ‘Playing with Grammar’, we focus on the Foundation to Year 2 grouping, in line with the Australian Curriculum, Assessment and Reporting Authority’s (hereafter ACARA) advice on the ‘nature of learners’ (ACARA, 2013). With our focus on the early years of schooling comes our acknowledgement of the importance and complexity of play. At a time where accountability in education has moved many teachers to a sense of urgency to prove language and literacy achievement (Genishi and Dyson, 2009), we encourage space to revisit what we know about literature choices and learning experiences and bring these together to facilitate language learning. We can neither ignore, nor overemphasise, the importance of play for the development of language through: the opportunities presented for creative use and practice; social interactions for real purposes; and, identifying and solving problems in the lives of young children (Marsh and Hallet, 2008). We argue that by engaging young children in opportunities to play with language we are ultimately empowering them to be active in their language learning and in the process fostering a love of language and the intricacies it holds. Our goal in this publication is to provide a range of highly practical strategies for scaffolding young children through some of the Content Descriptions from the Australian Curriculum English Version 5.0, hereafter AC:E V5.0 (ACARA, 2013). This recently released curriculum offers a new theoretical approach to building children’s knowledge about language. The AC:E V5.0 uses selected traditional terms through an approach developed in systemic functional linguistics (see Halliday and Matthiessen, 2004) to highlight the dynamic forms and functions of multimodal language in texts. For example, the following statement, taken from the ‘Language: Knowing about the English language’ strand states: English uses standard grammatical terminology within a contextual framework, in which language choices are seen to vary according to the topics at hand, the nature and proximity of the relationships between the language users, and the modalities or channels of communication available (ACARA, 2013). Put simply, traditional grammar terms are used within a functional framework made up of field, tenor, and mode. An understanding of genre is noted with the reference to a ‘contextual framework’. The ‘topics at hand’ concern the field or subject matter of the text. The ‘relationships between the language users’ is a description of tenor. There is reference to ‘modalities’, such as spoken, written or visual text. We posit that this innovative approach is necessary for working with contemporary multimodal and cross-cultural texts (see Exley and Mills, 2012). We believe there is enormous power in using literature to expose children to the richness of language and in turn develop language and literacy skills. Taking time to look at language patterns within actual literature is a pathway to ‘…capture interest, stir the imagination and absorb the [child]’ into the world of language and literacy (Saxby, 1993, p. 55). In the following three sections, we have tried to remain faithful to our interpretation of the AC:E V5.0 Content Descriptions without giving an exhaustive explanation of the grammatical terms. Other excellent tomes, such as Derewianka (2011), Humphrey, Droga and Feez (2012), and Rossbridge and Rushton (2011) provide these more comprehensive explanations as does the AC:E V5.0 Glossary. We’ve reproduced some of the AC:E V5.0 glossary at the end of this publication. Our focus is on the structure and unfolding of the learning experiences. We outline strategies for working with children in Foundation, Year 1 and Year 2 by providing some demonstration learning experiences based on texts we’ve selected, but maintain that the affordances of these strategies will only be realised when teaching and learning is purposively tied to authentic projects in local contexts. We strongly encourage you not to use only the resource texts we’ve selected, but to capitalise upon your skill for identifying the language features in the texts you and the children are studying and adapt some of the strategies we have outlined. Each learning experience is connected to one of the Content Descriptions from the AC:E V5.0 and contains an experience specific purpose, a suggested resource text and a sequence for the experience that always commences with an orientation to text followed by an examination of a particular grammatical resource. We expect that each of these learning experiences will take a couple if not a few teaching episodes to work through, especially if children are meeting a concept for the first time. We hope you use as much, or as little, of each experience as is needed. Our plans allow for focused discussion, shared exploration and opportunities to revisit the same text for the purpose of enhancing meaning making. We do not want the teaching of grammar to slip into a crisis of irrelevance or to be seen as a series of worksheet drills with finite answers. Strategies for effective practice, however, have much portability. We are both very keen to hear from teachers who are adopting and adapting these learning experiences in their classrooms. Please email us on b.exley@qut.edu.au or lkervin@uow.edu.au. We’d love to continue the conversation with you over time.