Directing for the 360 degree frame : developing a directorial approach to performance capture

What happens when the traditional framing mechanisms of our performance environments are removed and we are forced as directors to work with actors in digital environments that capture performance in 360 degrees? As directors contend with the challenges of interactive performance, the emergence of the online audience and the powerful influence of the games industry, how can we approach the challenges of directing work that is performance captured and presented in real time using motion capture and associated 3D imaging software? The 360 degree real time capture of performance, while allowing for an unlimited amount of framing potential, demands a unique and uncompromisingly disciplined style of direction and performance that has thus far remained unstudied and unquantified. By a close analysis of the groundbreaking work of artists like Robert Zemeckis and the Wetta Digital studio it is possible to begin to quantify what the technical requirements and challenges of 360 degree direction might be, but little has been discovered about the challenges of communicating the unlimited potential of framing and focus to the actors who work with these directors within these systems. It cannot be argued that the potential of theatrical space has evolved beyond the physical and moved into a more accessible virtual and digitised form, so how then can we direct for this unlimited potential and where do we place the focus of our directed (and captured) performance?

Preparing the actor for performance capture

Notions of capture and manipulation infer the existence of an interface that combines performer with system, an interface that separates (or intervenes in) the space between performance, intention and history. It is precisely the effect and provided opportunity of the intermediary device on the practice and craft of the actor, that is particularly examined in this work. Defining the scope of current practice for the contemporary actor is a key construct of this challenge with the most appropriate definition revolving around the pursuit of providing a required mixture of performance and content for live, mediated, framed and variously captured formats that exist in the present day. One of these particular formats is Performance Capture which this paper interrogate in more detail.

Preparing for performance capture #1

This video article articulates two exercises that have been developed to respond to the need for preparedness in the growing field of Performance Capture. The first is called Walking Through (Delbridge 2013), where the actor navigates a series of objects that exist in screen space through a developed sense of the existing physical particularities of the studio and an interaction with a screen (or feedback loop). The second exercise is called The Donut (Delbridge 2013), where the performer continues to navigate screen space but this time does so through the literal stepping through of a Torus in the virtual – again with nothing but the studio infrastructure and the screen as a guide. Notions of Motion Captured performance infer the existence of an interface that combines performer with system, separating (or intervening in) the space between performance and the screen. It is precisely the effect and provided opportunity of the intermediary device on the practice, craft and preparedness of the actor as they navigate the connection between 3D screen space and the physical properties of the studio that is examined here. Defining the scope of current practice for the contemporary actor is a key construct of this challenge, with the most appropriate definition revolving around the provision of a required mixture of performance and content for live, mediated, framed and variously captured formats. One of these particular formats is Performance Capture. The exercises presented here are two from a series, developed over a three year study that contribute to our understanding of the potential for a training regimen to be developed for the rigors of Performance Capture.

The cooling steam of the Polar Express : historical origins, properties and implications of performance capture

As the first academically rigorous interrogation of the generation of performance within the global frame of the motion capture volume, this research presents a historical contextualisation and develops and tests a set of first principles through an original series of theoretically informed, practical exercises to guide those working in the emergent space of performance capture. It contributes a new understanding of the framing of performance in The Omniscient Frame, and initiates and positions performance capture as a new and distinct interdisciplinary discourse in the fields of theatre, animation, performance studies and film.

Effects of lunar cycle and fishing operations on longline-caught pelagic fish: fishing performance, capture time, and survival of fish

Commercial longline fishing data were analyzed and experiments were conducted with gear equipped with hook timers and timedepth recorders in the Réunion Island fishery (21°5ʹS lat., 53°28ʹE long.) to elucidate direct and indirect effects of the lunar cycle and other operational factors that affect catch rates, catch composition, fish behavior, capture time, and fish survival. Logbook data from 1998 through 2000, comprising 2009 sets, indicated that swordfish (Xiphias gladius) catch-per unit of effort (CPUE) increased during the first and last quarter of the lunar phase, whereas albacore (Thunnus alalunga) CPUE was highest during the full moon. Swordfish were caught rapidly after the longline was set and, like bigeye tuna (Thunnus obesus), they were caught during days characterized by a weak lunar illumination—mainly during low tide. We found a significant but very low influence of chemical lightsticks on CPUE and catch composition. At the time the longline was retrieved, six of the 11 species in the study had >40% survival. Hook timers indicated that only 8.4% of the swordfish were alive after 8 hours of capture, and two shark species (blue shark [Prionace glauca] and oceanic whitetip shark [Carcharhinus longimanus]) showed a greater resilience to capture: 29.3% and 23.5% were alive after 8 hours, respectively. Our results have implications for current fishing practices and we comment on the possibilities of modifying fishing strategies in order to reduce operational costs, bycatch, loss of target fish at sea, and detrimental impacts on the environment.

A brush with the real world : the future of inertial motion capture in live performance

3D Motion capture is a medium that plots motion, typically human motion, converting it into a form that can be represented digitally. It is a fast evolving field and recent inertial technology may provide new artistic possibilities for its use in live performance. Although not often used in this context, motion capture has a combination of attributes that can provide unique forms of collaboration with performance arts. The inertial motion capture suit used for this study has orientation sensors placed at strategic points on the body to map body motion. Its portability, real-time performance, ease of use, and its immunity from line-of-sight problems inherent in optical systems suggest it would work well as a live performance technology. Many animation techniques can be used in real-time. This research examines a broad cross-section of these techniques using four practice-led cases to assess the suitability of inertial motion capture to live performance. Although each case explores different visual possibilities, all make use of the performativity of the medium, using either an improvisational format or interactivity among stage, audience and screen that would be difficult to emulate any other way. A real-time environment is not capable of reproducing the depth and sophistication of animation people have come to expect through media. These environments take many hours to render. In time the combination of what can be produced in real-time and the tools available in a 3D environment will no doubt create their own tree of aesthetic directions in live performance. The case study looks at the potential of interactivity that this technology offers.

The silk road project : dance and embodiment in the age of motion capture

The Silk Road Project was a practice-based research project investigating the potential of motion capture technology to inform perceptions of embodiment in dance performance. The project created a multi-disciplinary collaborative performance event using dance performance and real-time motion capture at Deakin University’s Deakin Motion Lab. Several new technological advances in producing real-time motion capture performance were produced, along with a performance event that examined the aesthetic interplay between a dancer’s movement and the precise mappings of its trajectories created by motion capture and real-time motion graphic visualisations.

Reproduction, mediation, and experience : virtual reality, motion capture and early modern theatre

This article investigates virtual reality representations of performance in London’s late sixteenth-century Rose Theatre, a venue that, by means of current technology, can once again challenge perceptions of space, performance, and memory. The VR model of The Rose represents a virtual recreation of this venue in as much detail as possible and attempts to recover graphic demonstrations of the trace memories of the performance modes of the day. The VR model is based on accurate archeological and theatre historical records and is easy to navigate. The introduction of human figures onto The Rose’s stage via motion capture allows us to explore the relationships between space, actor and environment. The combination of venue and actors facilitates a new way of thinking about how the work of early modern playwrights can be stored and recalled. This virtual theatre is thus activated to intersect productively with contemporary studies in performance; as such, our paper provides a perspective on and embodiment of the relation between technology, memory and experience. It is, at its simplest, a useful archiving project for theatrical history, but it is directly relevant to contemporary performance practice as well. Further, it reflects upon how technology and ‘re-enactments’ of sorts mediate the way in which knowledge and experience are transferred, and even what may be considered ‘knowledge.’ Our work provides opportunities to begin addressing what such intermedial confrontations might produce for ‘remembering, experiencing, thinking and imagining.’ We contend that these confrontations will enhance live theatre performance rather than impeding or disrupting contemporary performance practice. Our ‘paper’ is in the form of a video which covers the intellectual contribution while also permitting a demonstration of the interventions we are discussing.

The ecological approach to visual perception and the actor performance captured in the gaming landscape

The use of Performance Capture techniques in the creation of games that involve Motion Capture is a relatively new phenomenon. To date there is no prescribed methodology that prepares actors for the rigors of this new industry and as such there are many questions to be answered around how actors navigate these environments successfully when all available training and theoretical material is focused on performance for theatre and film. This article proposes that through a deployment of an Ecological Approach to Visual Perception we may begin to chart this territory for actors and begin to contend with the demands of performing for the motion captured gaming scenario.

Motion capture and the future of virtual production

Motion capture continues to be adopted across a range of creative fields including, animation, games, visual effects, dance, live theatre and the visual arts. This panel will discuss and showcase the use of motion capture across these creative fields and consider the future of virtual production in the creative industries.

Experimental arts practice incorporating motion capture

My practice-led research explores and maps workflows for generating experimental creative work involving inertia based motion capture technology. Motion capture has often been used as a way to bridge animation and dance resulting in abstracted visuals outcomes. In early works this process was largely done by rotoscoping, reference footage and mechanical forms of motion capture. With the evolution of technology, optical and inertial forms of motion capture are now more accessible and able to accurately capture a larger range of complex movements. Made by Motion is a collaboration between digital artist Paul Van Opdenbosch and performer and choreographer Elise May; a series of studies on captured motion data used to generate experimental visual forms that reverberate in space and time. The project investigates the invisible forces generated by and influencing the movement of a dancer. Along with how the forces can be captured and applied to generating visual outcomes that surpass simple data visualisation, projecting the intent of the performer’s movements. The source or ‘seed’ comes from using an Xsens MVN – Inertial Motion Capture system to capture spontaneous dance movements, with the visual generation conducted through a customised dynamics simulation. In my presentation I will be displaying and discussing a selected creative works from the project along with the process and considerations behind the work.

Agency and animation: the performance of interactive game characters

As an animator and practice-based researcher with a background in games development, I am interested in technological change in the video game medium, with a focus on the tools and technologies that drive game character animation and interactive story. In particular, I am concerned with the issue of ‘user agency’, or the ability of the end user to affect story development—a key quality of the gaming experience and essential to the aesthetics of gaming, which is defined in large measure by its interactive elements. In this paper I consider the unique qualities of the video game1 as an artistic medium and the impact that these qualities have on the production of animated virtual character performances. I discuss the somewhat oppositional nature of animated character performances found in games from recent years, which range from inactive to active—in other words, low to high agency. Where procedural techniques (based on coded rules of movement) are used to model dynamic character performances, the user has the ability to interactively affect characters in real-time within the larger sphere of the game. This game play creates a high degree of user agency. However, it lacks the aesthetic nuances of the more crafted sections of games: the short cut-scenes, or narrative interludes where entire acted performances are mapped onto game characters (often via performance capture)2 and constructed into relatively cinematic representations. While visually spectacular, cut-scenes involve minimal interactivity, so user agency is low. Contemporary games typically float between these two distinct methods of animation, from a focus on user agency and dynamically responsive animation to a focus on animated character performance in sections where the user is a passive participant. We tend to think of the majority of action in games as taking place via playable figures: an avatar or central character that represents a player. However, there is another realm of characters that also partake in actions ranging from significant to incidental: non-playable characters, or NPCs, which populate action sequences where game play takes place as well as cut scenes that unfold without much or any interaction on the part of the player. NPCs are the equivalent to supporting roles, bit characters, or extras in the world of cinema. Minor NPCs may simply be background characters or enemies to defeat, but many NPCs are crucial to the overall game story. It is my argument that, thus far, no game has successfully utilized the full potential of these characters to contribute toward development of interactive, high performance action. In particular, a type of NPC that I have identified as ‘pivotal’3—those constituting the supporting cast of a video game—are essential to the telling of a game story, particularly in genres that focus on story and characters: adventure games, action games, and role-playing games. A game story can be defined as the entirety of the narrative, told through non-interactive cut-scenes as well a interactive sections of play, and development of more complex stories in games clearly impacts the animation of NPCs. I argue that NPCs in games must be capable of acting with emotion throughout a game—in the cutscenes, which are tightly controlled, but also in sections of game play, where player agency can potentially alter the story in real-time. When the animated performance of NPCs and user agency are not continuous throughout the game, the implication is that game stories may be primarily told through short movies within games, making it more difficult to define video games animation as a distinct artistic medium.

Animated Kolam Patterns

The brief for the creative work was to produce a digital backdrop that would be projected behind and enhance a dance performance. The animation needed to display a static kolam pattern that would then dissolve at a choreographed point in the performance. The dissolving mimics the fragmentation that occurs to physical kolam patterns throughout the day as people interact with the drawings. The final animated work was incorporated into Vanessa Mafe-Keane’s performance titled “Paired Back” performed at the Judith Wright Centre, Brisbane 2013 as part of “Dance. Indie Dance. Through the use of motion capture technology the process of dissolving the pattern is a direct result of the performer’s movements allowing visual and temporal connection between motion of performer and digital graphic to be observed. This creative work presented an opportunity to expand upon experiments conducted in the production of experimental visual forms undertaken at QUT using the Xsens MVN Inertial Motion Capture System. The project took on the form of an investigation into practice with a focus on the additional complexities of capturing, then applying multiple data sources into the production of animated visuals along with bringing to light the considerations taken into account when producing this type of generative art work for live performance. The reported outcomes from this investigation have contributed to a larger study on the use of motion capture in the generative arts, furthering the understanding of and generating theories on practice.

Made by motion, Series one

“Made by Motion” is a collaboration between digital artist Paul Van Opdenbosch and performer and choreographer Elise May; a series of studies on captured motion data used to generating experimental visual forms that reverberate in space and time. The project investigates the invisible forces generated by and influencing the movement of a dancer. Along with how the forces can be captured and applied to generating visual outcomes that surpass simple data visualisation, projecting the intent of the performer’s movements. The source or ‘seed’ comes from using an Xsens MVN - Inertial Motion Capture system to capture spontaneous dance movements, with the visual generation conducted through a customised dynamics simulation. In this first series the visual investigation focused on manipulating the movement date at the instance of capture, capture been the recording of three-dimensional movement as ‘seen’ by the hardware and ‘understood’ through the calibration of software. By repositioning the capture hardware on the body we can effectively change how the same sequence of movements is ‘seen’ by the motion capture system thus generating a different visual result from effetely identical movement. The outcomes from the experiments clearly demonstrates the effectiveness of using motion capture hardware as a creative tool to manipulate the perception of the capture subject, in this case been a sequence of dance movements. The creative work exhibited is a cross-section of the experiments conducted in practice with the first animated work (Movement A - Control) using the motion capture hardware in its default ‘normal’ configuration. Following this is the lower body moved to the upper body (Lb-Ub), right arm moved onto the left arm (Ra-La), right leg moved onto the left leg (Rl-Ll) and finally the left leg moved onto a object that is then held in the left hand (Ll-Pf (Lh)).

Adopting virtual production for animated filmaking

Virtual Production is a rapidly growing approach to filmmaking that utilises 3D software, virtual camera systems and motion capture technology to visually interact with a real-time virtual environment. The use of these technologies has continued to increase, however, little has been done to document the various approaches for incorporating this new film making technique into a production. This practice-led research project outlines the development of virtual production in the entertainment industry and explores possible strategies for adopting aspects of this new film making technique into the production of short animated films. The outcome is an improved understanding of possible strategies that could be utilised to assist producers and directors with the transition into this new film making technique. - See more at: http://dl4.globalstf.org/?wpsc-product=adopting-virtual-production-for-animated-filmaking#sthash.DLzRph4Z.dpuf