Measuring Online user engagement : the limitations of Web analytics

The ability to identify and assess user engagement with transmedia productions is vital to the success of individual projects and the sustainability of this mode of media production as a whole. It is essential that industry players have access to tools and methodologies that offer the most complete and accurate picture of how audiences/users engage with their productions and which assets generate the most valuable returns of investment. Drawing upon research conducted with Hoodlum Entertainment, a Brisbane-based transmedia producer, this chapter outlines an initial assessment of the way engagement tends to be understood, why standard web analytics tools are ill-suited to measuring it, how a customised tool could offer solutions, and why this question of measuring engagement is so vital to the future of transmedia as a sustainable industry.

Conceptualising and measuring student engagement through the Australasian Survey of Student Engagement (AUSSE) : a critique

Student engagement has rapidly developed a central place in the quality agenda of Australian universities since the introduction of the Australasian Survey of Student Engagement (AUSSE). The AUSSE is based on one developed in the USA. The main arguments given for adopting this survey in Australia are that it provides a valid instrument for measuring engagement and that it enables international comparisons. However, the survey instrument and scales have been adopted with little scrutiny of these arguments. This paper examines these arguments by considering different perspectives of engagement, examining the importance of contextual differences and evaluating the AUSSE engagement scales in the light of both. The paper concludes that the AUSSE results should be used by universities and policy-makers with caution.

Measuring passers-by engagement with AmPost: a printed interactive audio poster

This paper presents AmPost, a prototype of an interactive audio poster, which integrates ink-jet printed sonic and tactile elements along with textual and graphical information. This enables users to directly interact and engage with the poster. To achieve a seamless paper-based interactive poster that includes printed interactive elements, we implement a printed speaker and audio feedback system directly into the paper-based medium, which plays a short tune when someone walks by. Engaging passers-by with this new interface becomes a challenge as posters, in general, are non-interactive. This paper also presents an initial empirical evaluation of AmPost that captures and analyses the frequency and length of user engagement, and evaluates whether user engagement improves with the integration of interactive features, in comparison to the traditional non-interactive poster. This study facilitates in building the generic approach to measure engagement that applies to any kind of posters.

The multi-platform user engagement and data visualisation tool

The ability to identify and assess user engagement with transmedia productions is vital to the success of individual projects and the sustainability of this mode of media production as a whole. It is essential that industry players have access to tools and methodologies that offer the most complete and accurate picture of how audiences/users engage with their productions and which assets generate the most valuable returns of investment. Drawing upon research conducted with Hoodlum Entertainment, a Brisbane-based transmedia producer, this project involved an initial assessment of the way engagement tends to be understood, why standard web analytics tools are ill-suited to measuring it, how a customised tool could offer solutions, and why this question of measuring engagement is so vital to the future of transmedia as a sustainable industry. Working with data provided by Hoodlum Entertainment and Foxtel Marketing, the outcome of the study was a prototype for a custom data visualisation tool that allowed access, manipulation and presentation of user engagement data, both historic and predictive. The prototyped interfaces demonstrate how the visualization tool would collect and organise data specific to multiplatform projects by aggregating data across a number of platform reporting tools. Such a tool is designed to encompass not only platforms developed by the transmedia producer but also sites developed by fans. This visualisation tool accounted for multiplatform experience projects whose top level is comprised of people, platforms and content. People include characters, actors, audience, distributors and creators. Platforms include television, Facebook and other relevant social networks, literature, cinema and other media that might be included in the multiplatform experience. Content refers to discreet media texts employed within the platform, such as tweet, a You Tube video, a Facebook post, an email, a television episode, etc. Core content is produced by the creators’ multiplatform experiences to advance the narrative, while complimentary content generated by audience members offers further contributions to the experience. Equally important is the timing with which the components of the experience are introduced and how they interact with and impact upon each other. Being able to combine, filter and sort these elements in multiple ways we can better understand the value of certain components of a project. It also offers insights into the relationship between the timing of the release of components and user activity associated with them, which further highlights the efficacy (or, indeed, failure) of assets as catalysts for engagement. In collaboration with Hoodlum we have developed a number of design scenarios experimenting with the ways in which data can be visualised and manipulated to tell a more refined story about the value of user engagement with certain project components and activities. This experimentation will serve as the basis for future research.

A importância da liderança para o engagement dos colaboradores nas empresas do Vale do Ave

Dissertação apresentada ao Instituto Politécnico do Porto para obtenção do Grau Mestre em Gestão das Organizações, Ramo de Gestão de Empresas Orientada pela Professora Doutora Maria Alexandra Pacheco Ribeiro da Costa

Congruencia entre los valores personales y organizacionales y su influencia con el engagement en el trabajo : estudio de caso en una organización del sector salud

La capacidad de gestión del personal se ha convertido en un imperativo para las organizaciones modernas. Por tanto se vienen introduciendo temas como la administración en valores y el engagement de los trabajadores. Sin embargo, la relación entre estos dos aún no ha sido estudiada. El presente estudio tiene como objetivo analizar el efecto que tiene el grado de articulación y reconocimiento de los valores organizacionales y personales y organizacionales sobre los niveles de engagement de los empleados. Para esta investigación se utilizó una muestra constituida por 54 trabajadores de una organización del sector salud de la ciudad de Bogotá a quienes les fueron aplicadas dos escalas: el Inventario para Medir la Articulación entre la Persona y la Organización (Inventario APO) y Utrech Work Engagement Scale (UWES). Como principal resultado se obtuvo que de las tres dimensiones consideradas predictoras del engagement, solamente el reconocimiento de los valores organizacionales tuvo un efecto estadísticamente significativo.

Introducing telemetrics

This paper introduces a new methodology for analyzing and measuring engagement with television content by users of Twitter. Drawing on factors such as the network, viewing audience, and date of broadcast to establish a baseline expectation for volume of tweets around a television show, and applying techniques from the field of sabermetrics to create neutral volume figures (‘weighted tweets’) which exclude these variables, our metrics provide new insights into television’s social media presence. The methodology provides a variety of new measures for analysing the social media strategies of individual television programs, channels and networks, for comparing users’ engagement with programs, channels or networks, and for predicting future volumes of tweets.

From engagement to outcomes: measuring the impact of researcher training using the rugby team impact framework

This study addresses our approach to the difficult task of measuring the impact of an eLearning service, the Graduate Virtual Research Environment (GVRE), provided to doctoral students at a UK University since October 2009. The GVRE provides research students with access to a training needs analysis tool which is linked to a repository of video learning resources created by academics and experienced research students. This paper explores the use of the Rugby Team Impact Framework as a guide to measuring impact and our use of a number of techniques to gather evidence about the changes resulting from use of the GVRE. The framework gives four levels of evidence, starting with simple measures of provision, through attendance, interest and to outcomes. As with other research, we found the former easy to assess but the outcomes harder to define. We conclude with a critical evaluation of our research process and outcomes.

Measuring student engagement : using 'flow' theory to guide question development

In December 2009 the Department of Education, Employment and Workplace Relations (DEEWR) issued a discussion paper, ‘An Indicator Framework for Higher Education Performance Funding’, with the purpose of contributing to the development of indicators for guiding ‘compact’ negotiations between Australian universities and the government. The discussion paper makes it clear that ‘student experience’ will be an important part of the indicators and discusses the possibility that the Government will develop a new
questionnaire to investigate the engagement and satisfaction of students in their first year of study. This paper contributes ideas to assist the development of such an instrument, particularly for the measurement of student engagement. We discuss problems with the predictive validity of existing instruments and argue that predictive validity may be improved by a focus on highly engaging ‘flow-type’ activities. Specifically, we
investigate flow activities in the context of online learning. Our findings suggest that some of the current questions from the Australasian Survey of Student Engagement (AUSSE) do appear to capture flow activities. We also identify gaps in the AUSSE. Finally, we make recommendations about the inclusion of additional items that capture highly engaging, flow activities for either the AUSSE or the Government’s proposed new instrument.

Measuring work-study engagement in built environment undergraduate education in Australia

The cost of accessing higher education is expensive causing students to juggle the demands of paid work with study responsibilities. Whilst some work can be beneficial to student leaning this research seeks a more accurate understanding of why students undertake paid experience work to the level that they do. This paper examines the extent of work and study during an undergraduate program in construction at RMIT University Australia. Students responded to a questionnaire on the duration and nature of their work and study times. The results indicate that students who were involved in paid work do in excess of 20 hours per week, whist also enrolled as full-time undergraduates. The results of the study show that students in the early years of the program seem to be more engaged with study and spent slightly less time at work. This is contrasted with students in the final two years of the course spend considerable more time in paid work and less time undertaking study.
The paper concludes by suggesting that the results are partly the result of the unstructured work-experience requirements that occur from about year 3 of the program. Students who were encouraged by the university to undertake paid work-experience appeared to be increasingly disinterested in connecting with the broader university experience.

eVALUate: an evaluation instrument for measuring students’ perceptions of their engagement and learning outcomes

In the current climate in Australian higher education, quality assurance in university teaching is a priority. In particular, the introduction of the Learning and Teaching Performance Fund (LTPF) has refocused attention on universities’ internal student evaluation survey instruments. This paper reports the development, validation and implementation of a new unit survey instrument which prompts students to reflect on what helps their achievement of unit learning outcomes, and to report their levels of motivation, engagement and overall satisfaction with a semester‐long course or unit of study. The instrument (eVALUate) was created from precepts reported in the research literature, current practices in evaluating teaching, and sound quality assurance practices appropriate to a university outcomes‐focused education paradigm.

Curvilinear relationships between the job demands-resources (JD-R) model, and work engagement : a police service case study

The current study aims to investigate the non-linear relationship between the JD-R model and work engagement. Previous research has identified linear relationships between these constructs; however there are strong theoretical arguments for testing curvilinear relationships (e.g., Warr, 1987). Data were collected via a self-report online survey from officers of one Australian police service (N = 2,626). Results demonstrated a curvilinear relationship between job demands and job resources and engagement. Gender (as a control variable) was also found to be a significant predictor of work engagement. The results indicated that male police officers experienced significantly higher job demands and colleague support than female officers. However, female police officers reported significantly higher levels of work engagement than male officers. This study emphasises the need to test curvilinear relationships, as well as simple linear associations, when measuring psychological health.

Moving beyond transition pedagogy: Maturity models and student engagement

Australian higher education institutions (HEIs) have entered a new phase of regulation and accreditation which includes performance-based funding relating to the participation and retention of students from social and cultural groups previously underrepresented in higher education. However, in addressing these priorities, it is critical that HEIs do not further disadvantage students from certain groups by identifying them for attention because of their social or cultural backgrounds, circumstances which are largely beyond the control of students. In response, many HEIs are focusing effort on university-wide approaches to enhancing the student experience because such approaches will enhance the engagement, success and retention of all students, and in doing so, particularly benefit those students who come from underrepresented groups. Measuring and benchmarking student experiences and engagement that arise from these efforts is well supported by extensive collections of student experience survey data. However no comparable instrument exists that measures the capability of institutions to influence and/or enhance student experiences where capability is an indication of how well an organisational process does what it is designed to do (Rosemann & de Bruin, 2005). This paper proposes that the concept of a maturity model (Marshall, 2010; Paulk, 1999) may be useful as a way of assessing the capability of HEIs to provide and implement student engagement, success and retention activities. We will describe the Student Engagement, Success and Retention Maturity Model (SESR-MM), (Clarke, Nelson & Stoodley, 2012; Nelson, Clarke & Stoodley, 2012) we are currently investigating. We will discuss if our research may address the current gap by facilitating the development of an SESR-MM instrument that aims (i) to enable institutions to assess the capability of their current student engagement and retention programs and strategies to influence and respond to student experiences within the institution; and (ii) to provide institutions with the opportunity to understand various practices across the sector with a view to further improving programs and practices relevant to their context. The first aim of our research is to extend the generational approach which has been useful in considering the evolutionary nature of the first year experience (FYE) (Wilson, 2009). Three generations have been identified and explored: First generation approaches that focus on co-curricular strategies (e.g. orientation and peer programs); Second generation approaches that focus on curriculum (e.g. pedagogy, curriculum design, and learning and teaching practice); and third generation approaches—also referred to as transition pedagogy—that focus on the production of an institution-wide integrated holistic intentional blend of curricular and co-curricular activities (Kift, Nelson & Clarke, 2010). The second aim of this research is to move beyond assessments of students’ experiences to focus on assessing institutional processes and their capability to influence student engagement. In essence, we propose to develop and use the maturity model concept to produce an instrument that will indicate the capability of HEIs to manage and improve student engagement, success and retention programs and strategies. References Australian Council for Educational Research. (n.d.). Australasian Survey of Student Engagement. Retrieved from http://www.acer.edu.au/research/ausse/background Clarke, J., Nelson, K., & Stoodley, I. (2012, July). The Maturity Model concept as framework for assessing the capability of higher education institutions to address student engagement, success and retention: New horizon or false dawn? A Nuts & Bolts presentation at the 15th International Conference on the First Year in Higher Education, “New Horizons,” Brisbane, Australia. Kift, S., Nelson, K., & Clarke, J. (2010) Transition pedagogy - a third generation approach to FYE: A case study of policy and practice for the higher education sector. The International Journal of the First Year in Higher Education, 1(1), pp. 1-20. Department of Education, Employment and Workplace Relations. (n.d.). The University Experience Survey. Advancing quality in higher education information sheet. Retrieved from http://www.deewr.gov.au/HigherEducation/Policy/Documents/University_Experience_Survey.pdf Marshall, S. (2010). A quality framework for continuous improvement of e-Learning: The e-Learning Maturity Model. Journal of Distance Education, 24(1), 143-166. Nelson, K., Clarke, J., & Stoodley, I. (2012). An exploration of the Maturity Model concept as a vehicle for higher education institutions to assess their capability to address student engagement. A work in progress. Submitted for publication. Paulk, M. (1999). Using the Software CMM with good judgment, ASQ Software Quality Professional, 1(3), 19-29. Wilson, K. (2009, June–July). The impact of institutional, programmatic and personal interventions on an effective and sustainable first-year student experience. Keynote address presented at the 12th Pacific Rim First Year in Higher Education Conference, “Preparing for Tomorrow Today: The First Year as Foundation,” Townsville, Australia. Retrieved from http://www.fyhe.com.au/past_papers/papers09/ppts/Keithia_Wilson_paper.pdf

Verifying a Student Engagement, Success and Retention Maturity Model (SESR-MM): A reality check of the model using data derived from three universities

Australian higher education institutions (HEIs) have entered a new phase of regulation and accreditation which includes performance-based funding relating to the participation and retention of students from social and cultural groups previously underrepresented in higher education. However, in addressing these priorities, it is critical that HEIs do not further disadvantage students from certain groups by identifying them for attention because of their social or cultural backgrounds, circumstances which are largely beyond the control of students. In response, many HEIs are focusing effort on university-wide approaches to enhancing the student experience because such approaches will enhance the engagement, success and retention of all students, and in doing so, particularly benefit those students who come from underrepresented groups. Measuring and benchmarking student experiences and engagement that arise from these efforts is well supported by extensive collections of student experience survey data. However no comparable instrument exists that measures the capability of institutions to influence and/or enhance student experiences where capability is an indication of how well an organisational process does what it is designed to do (Rosemann & de Bruin, 2005). We have proposed that the concept of a maturity model (Marshall, 2010; Paulk, 1999) may be useful as a way of assessing the capability of HEIs to provide and implement student engagement, success and retention activities and we are currently articulating a Student Engagement, Success and Retention Maturity Model (SESR-MM), (Clarke, Nelson & Stoodley, 2012; Nelson, Clarke & Stoodley, 2012). Our research aims to address the current gap by facilitating the development of an SESR-MM instrument that aims (i) to enable institutions to assess the capability of their current student engagement and retention programs and strategies to influence and respond to student experiences within the institution; and (ii) to provide institutions with the opportunity to understand various practices across the sector with a view to further improving programs and practices relevant to their context. Our research extends the generational approach which has been useful in considering the evolutionary nature of the first year experience (FYE) (Wilson, 2009). Three generations have been identified and explored: First generation approaches that focus on co-curricular strategies (e.g. orientation and peer programs); Second generation approaches that focus on curriculum (e.g. pedagogy, curriculum design, and learning and teaching practice); and third generation approaches—also referred to as transition pedagogy—that focus on the production of an institution-wide integrated holistic intentional blend of curricular and co-curricular activities (Kift, Nelson & Clarke, 2010). Our research also moves beyond assessments of students’ experiences to focus on assessing institutional processes and their capability to influence student engagement. In essence, we propose to develop and use the maturity model concept to produce an instrument that will indicate the capability of HEIs to manage and improve student engagement, success and retention programs and strategies. The issues explored in this workshop are (i) whether the maturity model concept can be usefully applied to provide a measure of institutional capability for SESR; (ii) whether the SESR-MM can be used to assess the maturity of a particular set of institutional practices; and (iii) whether a collective assessment of an institution’s SESR capabilities can provide an indication of the maturity of the institution’s SESR activities. The workshop will be approached in three stages. Firstly, participants will be introduced to the key characteristics of maturity models, followed by a discussion of the SESR-MM and the processes involved in its development. Secondly, participants will be provided with resources to facilitate the development of a maturity model and an assessment instrument for a range of institutional processes and related practices. In the final stage of the workshop, participants will “assess” the capability of these practices to provide a collective assessment of the maturity of these processes. References Australian Council for Educational Research. (n.d.). Australasian Survey of Student Engagement. Retrieved from http://www.acer.edu.au/research/ausse/background Clarke, J., Nelson, K., & Stoodley, I. (2012, July). The Maturity Model concept as framework for assessing the capability of higher education institutions to address student engagement, success and retention: New horizon or false dawn? A Nuts & Bolts presentation at the 15th International Conference on the First Year in Higher Education, “New Horizons,” Brisbane, Australia. Department of Education, Employment and Workplace Relations. (n.d.). The University Experience Survey. Advancing quality in higher education information sheet. Retrieved from http://www.deewr.gov.au/HigherEducation/Policy/Documents/University_Experience_Survey.pdf Kift, S., Nelson, K., & Clarke, J. (2010) Transition pedagogy - a third generation approach to FYE: A case study of policy and practice for the higher education sector. The International Journal of the First Year in Higher Education, 1(1), pp. 1-20. Marshall, S. (2010). A quality framework for continuous improvement of e-Learning: The e-Learning Maturity Model. Journal of Distance Education, 24(1), 143-166. Nelson, K., Clarke, J., & Stoodley, I. (2012). An exploration of the Maturity Model concept as a vehicle for higher education institutions to assess their capability to address student engagement. A work in progress. Submitted for publication. Paulk, M. (1999). Using the Software CMM with good judgment, ASQ Software Quality Professional, 1(3), 19-29. Wilson, K. (2009, June–July). The impact of institutional, programmatic and personal interventions on an effective and sustainable first-year student experience. Keynote address presented at the 12th Pacific Rim First Year in Higher Education Conference, “Preparing for Tomorrow Today: The First Year as Foundation,” Townsville, Australia. Retrieved from http://www.fyhe.com.au/past_papers/papers09/ppts/Keithia_Wilson_paper.pdf