Measuring Online user engagement : the limitations of Web analytics

The ability to identify and assess user engagement with transmedia productions is vital to the success of individual projects and the sustainability of this mode of media production as a whole. It is essential that industry players have access to tools and methodologies that offer the most complete and accurate picture of how audiences/users engage with their productions and which assets generate the most valuable returns of investment. Drawing upon research conducted with Hoodlum Entertainment, a Brisbane-based transmedia producer, this chapter outlines an initial assessment of the way engagement tends to be understood, why standard web analytics tools are ill-suited to measuring it, how a customised tool could offer solutions, and why this question of measuring engagement is so vital to the future of transmedia as a sustainable industry.

The multi-platform user engagement and data visualisation tool

The ability to identify and assess user engagement with transmedia productions is vital to the success of individual projects and the sustainability of this mode of media production as a whole. It is essential that industry players have access to tools and methodologies that offer the most complete and accurate picture of how audiences/users engage with their productions and which assets generate the most valuable returns of investment. Drawing upon research conducted with Hoodlum Entertainment, a Brisbane-based transmedia producer, this project involved an initial assessment of the way engagement tends to be understood, why standard web analytics tools are ill-suited to measuring it, how a customised tool could offer solutions, and why this question of measuring engagement is so vital to the future of transmedia as a sustainable industry. Working with data provided by Hoodlum Entertainment and Foxtel Marketing, the outcome of the study was a prototype for a custom data visualisation tool that allowed access, manipulation and presentation of user engagement data, both historic and predictive. The prototyped interfaces demonstrate how the visualization tool would collect and organise data specific to multiplatform projects by aggregating data across a number of platform reporting tools. Such a tool is designed to encompass not only platforms developed by the transmedia producer but also sites developed by fans. This visualisation tool accounted for multiplatform experience projects whose top level is comprised of people, platforms and content. People include characters, actors, audience, distributors and creators. Platforms include television, Facebook and other relevant social networks, literature, cinema and other media that might be included in the multiplatform experience. Content refers to discreet media texts employed within the platform, such as tweet, a You Tube video, a Facebook post, an email, a television episode, etc. Core content is produced by the creators’ multiplatform experiences to advance the narrative, while complimentary content generated by audience members offers further contributions to the experience. Equally important is the timing with which the components of the experience are introduced and how they interact with and impact upon each other. Being able to combine, filter and sort these elements in multiple ways we can better understand the value of certain components of a project. It also offers insights into the relationship between the timing of the release of components and user activity associated with them, which further highlights the efficacy (or, indeed, failure) of assets as catalysts for engagement. In collaboration with Hoodlum we have developed a number of design scenarios experimenting with the ways in which data can be visualised and manipulated to tell a more refined story about the value of user engagement with certain project components and activities. This experimentation will serve as the basis for future research.

Introducing telemetrics

This paper introduces a new methodology for analyzing and measuring engagement with television content by users of Twitter. Drawing on factors such as the network, viewing audience, and date of broadcast to establish a baseline expectation for volume of tweets around a television show, and applying techniques from the field of sabermetrics to create neutral volume figures (‘weighted tweets’) which exclude these variables, our metrics provide new insights into television’s social media presence. The methodology provides a variety of new measures for analysing the social media strategies of individual television programs, channels and networks, for comparing users’ engagement with programs, channels or networks, and for predicting future volumes of tweets.

Curvilinear relationships between the job demands-resources (JD-R) model, and work engagement : a police service case study

The current study aims to investigate the non-linear relationship between the JD-R model and work engagement. Previous research has identified linear relationships between these constructs; however there are strong theoretical arguments for testing curvilinear relationships (e.g., Warr, 1987). Data were collected via a self-report online survey from officers of one Australian police service (N = 2,626). Results demonstrated a curvilinear relationship between job demands and job resources and engagement. Gender (as a control variable) was also found to be a significant predictor of work engagement. The results indicated that male police officers experienced significantly higher job demands and colleague support than female officers. However, female police officers reported significantly higher levels of work engagement than male officers. This study emphasises the need to test curvilinear relationships, as well as simple linear associations, when measuring psychological health.

Moving beyond transition pedagogy: Maturity models and student engagement

Australian higher education institutions (HEIs) have entered a new phase of regulation and accreditation which includes performance-based funding relating to the participation and retention of students from social and cultural groups previously underrepresented in higher education. However, in addressing these priorities, it is critical that HEIs do not further disadvantage students from certain groups by identifying them for attention because of their social or cultural backgrounds, circumstances which are largely beyond the control of students. In response, many HEIs are focusing effort on university-wide approaches to enhancing the student experience because such approaches will enhance the engagement, success and retention of all students, and in doing so, particularly benefit those students who come from underrepresented groups. Measuring and benchmarking student experiences and engagement that arise from these efforts is well supported by extensive collections of student experience survey data. However no comparable instrument exists that measures the capability of institutions to influence and/or enhance student experiences where capability is an indication of how well an organisational process does what it is designed to do (Rosemann & de Bruin, 2005). This paper proposes that the concept of a maturity model (Marshall, 2010; Paulk, 1999) may be useful as a way of assessing the capability of HEIs to provide and implement student engagement, success and retention activities. We will describe the Student Engagement, Success and Retention Maturity Model (SESR-MM), (Clarke, Nelson & Stoodley, 2012; Nelson, Clarke & Stoodley, 2012) we are currently investigating. We will discuss if our research may address the current gap by facilitating the development of an SESR-MM instrument that aims (i) to enable institutions to assess the capability of their current student engagement and retention programs and strategies to influence and respond to student experiences within the institution; and (ii) to provide institutions with the opportunity to understand various practices across the sector with a view to further improving programs and practices relevant to their context. The first aim of our research is to extend the generational approach which has been useful in considering the evolutionary nature of the first year experience (FYE) (Wilson, 2009). Three generations have been identified and explored: First generation approaches that focus on co-curricular strategies (e.g. orientation and peer programs); Second generation approaches that focus on curriculum (e.g. pedagogy, curriculum design, and learning and teaching practice); and third generation approaches—also referred to as transition pedagogy—that focus on the production of an institution-wide integrated holistic intentional blend of curricular and co-curricular activities (Kift, Nelson & Clarke, 2010). The second aim of this research is to move beyond assessments of students’ experiences to focus on assessing institutional processes and their capability to influence student engagement. In essence, we propose to develop and use the maturity model concept to produce an instrument that will indicate the capability of HEIs to manage and improve student engagement, success and retention programs and strategies. References Australian Council for Educational Research. (n.d.). Australasian Survey of Student Engagement. Retrieved from http://www.acer.edu.au/research/ausse/background Clarke, J., Nelson, K., & Stoodley, I. (2012, July). The Maturity Model concept as framework for assessing the capability of higher education institutions to address student engagement, success and retention: New horizon or false dawn? A Nuts & Bolts presentation at the 15th International Conference on the First Year in Higher Education, “New Horizons,” Brisbane, Australia. Kift, S., Nelson, K., & Clarke, J. (2010) Transition pedagogy - a third generation approach to FYE: A case study of policy and practice for the higher education sector. The International Journal of the First Year in Higher Education, 1(1), pp. 1-20. Department of Education, Employment and Workplace Relations. (n.d.). The University Experience Survey. Advancing quality in higher education information sheet. Retrieved from http://www.deewr.gov.au/HigherEducation/Policy/Documents/University_Experience_Survey.pdf Marshall, S. (2010). A quality framework for continuous improvement of e-Learning: The e-Learning Maturity Model. Journal of Distance Education, 24(1), 143-166. Nelson, K., Clarke, J., & Stoodley, I. (2012). An exploration of the Maturity Model concept as a vehicle for higher education institutions to assess their capability to address student engagement. A work in progress. Submitted for publication. Paulk, M. (1999). Using the Software CMM with good judgment, ASQ Software Quality Professional, 1(3), 19-29. Wilson, K. (2009, June–July). The impact of institutional, programmatic and personal interventions on an effective and sustainable first-year student experience. Keynote address presented at the 12th Pacific Rim First Year in Higher Education Conference, “Preparing for Tomorrow Today: The First Year as Foundation,” Townsville, Australia. Retrieved from http://www.fyhe.com.au/past_papers/papers09/ppts/Keithia_Wilson_paper.pdf

Verifying a Student Engagement, Success and Retention Maturity Model (SESR-MM): A reality check of the model using data derived from three universities

Australian higher education institutions (HEIs) have entered a new phase of regulation and accreditation which includes performance-based funding relating to the participation and retention of students from social and cultural groups previously underrepresented in higher education. However, in addressing these priorities, it is critical that HEIs do not further disadvantage students from certain groups by identifying them for attention because of their social or cultural backgrounds, circumstances which are largely beyond the control of students. In response, many HEIs are focusing effort on university-wide approaches to enhancing the student experience because such approaches will enhance the engagement, success and retention of all students, and in doing so, particularly benefit those students who come from underrepresented groups. Measuring and benchmarking student experiences and engagement that arise from these efforts is well supported by extensive collections of student experience survey data. However no comparable instrument exists that measures the capability of institutions to influence and/or enhance student experiences where capability is an indication of how well an organisational process does what it is designed to do (Rosemann & de Bruin, 2005). We have proposed that the concept of a maturity model (Marshall, 2010; Paulk, 1999) may be useful as a way of assessing the capability of HEIs to provide and implement student engagement, success and retention activities and we are currently articulating a Student Engagement, Success and Retention Maturity Model (SESR-MM), (Clarke, Nelson & Stoodley, 2012; Nelson, Clarke & Stoodley, 2012). Our research aims to address the current gap by facilitating the development of an SESR-MM instrument that aims (i) to enable institutions to assess the capability of their current student engagement and retention programs and strategies to influence and respond to student experiences within the institution; and (ii) to provide institutions with the opportunity to understand various practices across the sector with a view to further improving programs and practices relevant to their context. Our research extends the generational approach which has been useful in considering the evolutionary nature of the first year experience (FYE) (Wilson, 2009). Three generations have been identified and explored: First generation approaches that focus on co-curricular strategies (e.g. orientation and peer programs); Second generation approaches that focus on curriculum (e.g. pedagogy, curriculum design, and learning and teaching practice); and third generation approaches—also referred to as transition pedagogy—that focus on the production of an institution-wide integrated holistic intentional blend of curricular and co-curricular activities (Kift, Nelson & Clarke, 2010). Our research also moves beyond assessments of students’ experiences to focus on assessing institutional processes and their capability to influence student engagement. In essence, we propose to develop and use the maturity model concept to produce an instrument that will indicate the capability of HEIs to manage and improve student engagement, success and retention programs and strategies. The issues explored in this workshop are (i) whether the maturity model concept can be usefully applied to provide a measure of institutional capability for SESR; (ii) whether the SESR-MM can be used to assess the maturity of a particular set of institutional practices; and (iii) whether a collective assessment of an institution’s SESR capabilities can provide an indication of the maturity of the institution’s SESR activities. The workshop will be approached in three stages. Firstly, participants will be introduced to the key characteristics of maturity models, followed by a discussion of the SESR-MM and the processes involved in its development. Secondly, participants will be provided with resources to facilitate the development of a maturity model and an assessment instrument for a range of institutional processes and related practices. In the final stage of the workshop, participants will “assess” the capability of these practices to provide a collective assessment of the maturity of these processes. References Australian Council for Educational Research. (n.d.). Australasian Survey of Student Engagement. Retrieved from http://www.acer.edu.au/research/ausse/background Clarke, J., Nelson, K., & Stoodley, I. (2012, July). The Maturity Model concept as framework for assessing the capability of higher education institutions to address student engagement, success and retention: New horizon or false dawn? A Nuts & Bolts presentation at the 15th International Conference on the First Year in Higher Education, “New Horizons,” Brisbane, Australia. Department of Education, Employment and Workplace Relations. (n.d.). The University Experience Survey. Advancing quality in higher education information sheet. Retrieved from http://www.deewr.gov.au/HigherEducation/Policy/Documents/University_Experience_Survey.pdf Kift, S., Nelson, K., & Clarke, J. (2010) Transition pedagogy - a third generation approach to FYE: A case study of policy and practice for the higher education sector. The International Journal of the First Year in Higher Education, 1(1), pp. 1-20. Marshall, S. (2010). A quality framework for continuous improvement of e-Learning: The e-Learning Maturity Model. Journal of Distance Education, 24(1), 143-166. Nelson, K., Clarke, J., & Stoodley, I. (2012). An exploration of the Maturity Model concept as a vehicle for higher education institutions to assess their capability to address student engagement. A work in progress. Submitted for publication. Paulk, M. (1999). Using the Software CMM with good judgment, ASQ Software Quality Professional, 1(3), 19-29. Wilson, K. (2009, June–July). The impact of institutional, programmatic and personal interventions on an effective and sustainable first-year student experience. Keynote address presented at the 12th Pacific Rim First Year in Higher Education Conference, “Preparing for Tomorrow Today: The First Year as Foundation,” Townsville, Australia. Retrieved from http://www.fyhe.com.au/past_papers/papers09/ppts/Keithia_Wilson_paper.pdf

Concealing their communication : exploring psychosocial predictors of young drivers’ intentions and engagement in concealed texting

Making a conscious effort to hide the fact that you are texting while driving (i.e., concealed texting) is a deliberate and risky behaviour involving attention diverted away from the road. As the most frequent users of text messaging services and mobile phones while driving, young people appear at heightened risk of crashing from engaging in this behaviour. This study investigated the phenomenon of concealed texting while driving, and utilised an extended Theory of Planned Behaviour (TPB) including the additional predictors of moral norm, mobile phone involvement, and anticipated regret to predict young drivers’ intentions and subsequent behaviour. Participants (n = 171) were aged 17 to 25 years, owned a mobile phone, and had a current driver’s licence. Participants completed a questionnaire measuring their intention to conceal texting while driving, and a follow-up questionnaire a week later to report their behavioural engagement. The results of hierarchical multiple regression analyses showed overall support for the predictive utility of the TPB with the standard constructs accounting for 69% of variance in drivers’ intentions, and the extended predictors contributing an additional 6% of variance in intentions over and above the standard constructs. Attitude, subjective norm, PBC, moral norm, and mobile phone involvement emerged as significant predictors of intentions; and intention was the only significant predictor of drivers’ self-reported behaviour. These constructs can provide insight into key focal points for countermeasures including advertising and other public education strategies aimed at influencing young drivers to reconsider their engagement in this risky behaviour.

Keep It Tame: Promoting respect online Safe And Well Online Pilot Study: Evaluating the design, engagement and impact of a social marketing approach aimed at 12 to 18 year olds

This report describes the Year One Pilot Study processes, and articulates findings from the major project components designed to address these challenges noted above (See Figure 1). Specifically, the pilot study tested the campaign research and development process involving participatory design with young people and sector partners, and the efficacy and practicality of conducting a longitudinal, randomised control trial online with minors, including ways oflinking survey data to campaign data. Each sub-study comprehensively considered the ethical requirements of conducting online research with minors in school settings. The theoretical and methodological framework for measuring campaign engagement and efficacy (Sub-studies 3, 4 and 5) drew on the Model of Goal-Directed Behaviour (MGB) (Perugini & Bagozzi 2001) and Nudge Theory (Thaler & Sunstein, 2008).

Measuring impact: Positioning evaluation in the Australian arts and culture landscape

Arts culture organisations and funding authorities increasingly need to evaluate the impact of festivals, events and performances. Economic impacts are often privileged over 'soft data' about community experience and engagement. This new book offers a timely and scholarly demonstration of how cultural value and impact can be evaluated. It offers an innovative approach whereby the relationship developed between the researchers/evaluator and the commissioning arts and cultural producer provides an opportunity to rethink the traditional process of reporting back on value and impact through the singular entity of funds acquittal. Using three commissioned evaluations undertaken at an Australian university as an extended case study, the book investigates the two positions most often adopted by researchers/evaluators - embedded and collaborative, or external and distanced - and argues the merits and deficiencies of the two approaches. Offering an examination of how arts evaluation 'works' in theory and practice and more importantly, why it is needed now and in the future to demonstrate the reach and cultural gains from arts and cultural projects, this will be essential reading for students in arts management, professionals working in arts and cultural organisations, scholars working in association with creative industries and cultural development.

Clinical Accuracy of the MedGem Indirect Calorimeter for Measuring Resting Energy Expenditure in Cancer Patients

OBJECTIVE: To compare, in patients with cancer and in healthy subjects, measured resting energy expenditure (REE) from traditional indirect calorimetry to a new portable device (MedGem) and predicted REE. DESIGN: Cross-sectional clinical validation study. SETTING: Private radiation oncology centre, Brisbane, Australia. SUBJECTS: Cancer patients (n = 18) and healthy subjects (n = 17) aged 37-86 y, with body mass indices ranging from 18 to 42 kg/m(2). INTERVENTIONS: Oxygen consumption (VO(2)) and REE were measured by VMax229 (VM) and MedGem (MG) indirect calorimeters in random order after a 12-h fast and 30-min rest. REE was also calculated from the MG without adjustment for nitrogen excretion (MGN) and estimated from Harris-Benedict prediction equations. Data were analysed using the Bland and Altman approach, based on a clinically acceptable difference between methods of 5%. RESULTS: The mean bias (MGN-VM) was 10% and limits of agreement were -42 to 21% for cancer patients; mean bias -5% with limits of -45 to 35% for healthy subjects. Less than half of the cancer patients (n = 7, 46.7%) and only a third (n = 5, 33.3%) of healthy subjects had measured REE by MGN within clinically acceptable limits of VM. Predicted REE showed a mean bias (HB-VM) of -5% for cancer patients and 4% for healthy subjects, with limits of agreement of -30 to 20% and -27 to 34%, respectively. CONCLUSIONS: Limits of agreement for the MG and Harris Benedict equations compared to traditional indirect calorimetry were similar but wide, indicating poor clinical accuracy for determining the REE of individual cancer patients and healthy subjects.

Task profiling : A task-Based approach to measuring the integration of ICT in the classroom

The measurement of ICT (information and communication technology) integration is emerging as an area of research interest with such systems as Education Queensland including it in their recently released list of research priorities. Studies to trial differing integration measurement instruments have taken place within Australia in the last few years, particularly Western Australia (Trinidad, Clarkson, & Newhouse, 2004; Trinidad, Newhouse & Clarkson, 2005), Tasmania (Fitzallen 2005) and Queensland (Finger, Proctor, & Watson, 2005). This paper will add to these investigations by describing an alternate and original methodological approach which was trialled in a small-scale pilot study conducted jointly by Queensland Catholic Education Commission (QCEC) and the Centre of Learning Innovation, Queensland University of Technology (QUT) in late 2005. The methodology described is based on tasks which, through a process of profiling, can be seen to be artefacts which embody the internal and external factors enabling and constraining ICT integration.