Conceptualising and measuring student engagement through the Australasian Survey of Student Engagement (AUSSE) : a critique

Student engagement has rapidly developed a central place in the quality agenda of Australian universities since the introduction of the Australasian Survey of Student Engagement (AUSSE). The AUSSE is based on one developed in the USA. The main arguments given for adopting this survey in Australia are that it provides a valid instrument for measuring engagement and that it enables international comparisons. However, the survey instrument and scales have been adopted with little scrutiny of these arguments. This paper examines these arguments by considering different perspectives of engagement, examining the importance of contextual differences and evaluating the AUSSE engagement scales in the light of both. The paper concludes that the AUSSE results should be used by universities and policy-makers with caution.

Governmentality in higher education : a critical analysis of the National Survey of Student Engagement (NSSE)

In this dissertation, the National Survey of Student Engagement (NSSE) serves as a nodal point through which to examine the power relations shaping the direction and practices of higher education in the twenty-first century. Theoretically, my analysis is informed by Foucault’s concept of governmentality, briefly defined as a technology of power that influences or shapes behavior from a distance. This form of governance operates through apparatuses of security, which include higher education. Foucault identified three essential characteristics of an apparatus—the market, the milieu, and the processes of normalization—through which administrative mechanisms and practices operate and govern populations. In this project, my primary focus is on the governance of faculty and administrators, as a population, at residential colleges and universities. I argue that the existing milieu of accountability is one dominated by the neoliberal assumption that all activity—including higher education—works best when governed by market forces alone, reducing higher education to a market-mediated private good. Under these conditions, what many in the academy believe is an essential purpose of higher education—to educate students broadly, to contribute knowledge for the public good, and to serve as society’s critic and social conscience (Washburn 227)—is being eroded. Although NSSE emerged as a form of resistance to commercial college rankings, it did not challenge the forces that empowered the rankings in the first place. Indeed, NSSE data are now being used to make institutions even more responsive to market forces. Furthermore, NSSE’s use has a normalizing effect that tends to homogenize classroom practices and erode the autonomy of faculty in the educational process. It also positions students as part of the system of surveillance. In the end, if aspects of higher education that are essential to maintaining a civil society are left to be defined solely in market terms, the result may be a less vibrant and, ultimately, a less just society.

Verifying a Student Engagement, Success and Retention Maturity Model (SESR-MM): A reality check of the model using data derived from three universities

Australian higher education institutions (HEIs) have entered a new phase of regulation and accreditation which includes performance-based funding relating to the participation and retention of students from social and cultural groups previously underrepresented in higher education. However, in addressing these priorities, it is critical that HEIs do not further disadvantage students from certain groups by identifying them for attention because of their social or cultural backgrounds, circumstances which are largely beyond the control of students. In response, many HEIs are focusing effort on university-wide approaches to enhancing the student experience because such approaches will enhance the engagement, success and retention of all students, and in doing so, particularly benefit those students who come from underrepresented groups. Measuring and benchmarking student experiences and engagement that arise from these efforts is well supported by extensive collections of student experience survey data. However no comparable instrument exists that measures the capability of institutions to influence and/or enhance student experiences where capability is an indication of how well an organisational process does what it is designed to do (Rosemann & de Bruin, 2005). We have proposed that the concept of a maturity model (Marshall, 2010; Paulk, 1999) may be useful as a way of assessing the capability of HEIs to provide and implement student engagement, success and retention activities and we are currently articulating a Student Engagement, Success and Retention Maturity Model (SESR-MM), (Clarke, Nelson & Stoodley, 2012; Nelson, Clarke & Stoodley, 2012). Our research aims to address the current gap by facilitating the development of an SESR-MM instrument that aims (i) to enable institutions to assess the capability of their current student engagement and retention programs and strategies to influence and respond to student experiences within the institution; and (ii) to provide institutions with the opportunity to understand various practices across the sector with a view to further improving programs and practices relevant to their context. Our research extends the generational approach which has been useful in considering the evolutionary nature of the first year experience (FYE) (Wilson, 2009). Three generations have been identified and explored: First generation approaches that focus on co-curricular strategies (e.g. orientation and peer programs); Second generation approaches that focus on curriculum (e.g. pedagogy, curriculum design, and learning and teaching practice); and third generation approaches—also referred to as transition pedagogy—that focus on the production of an institution-wide integrated holistic intentional blend of curricular and co-curricular activities (Kift, Nelson & Clarke, 2010). Our research also moves beyond assessments of students’ experiences to focus on assessing institutional processes and their capability to influence student engagement. In essence, we propose to develop and use the maturity model concept to produce an instrument that will indicate the capability of HEIs to manage and improve student engagement, success and retention programs and strategies. The issues explored in this workshop are (i) whether the maturity model concept can be usefully applied to provide a measure of institutional capability for SESR; (ii) whether the SESR-MM can be used to assess the maturity of a particular set of institutional practices; and (iii) whether a collective assessment of an institution’s SESR capabilities can provide an indication of the maturity of the institution’s SESR activities. The workshop will be approached in three stages. Firstly, participants will be introduced to the key characteristics of maturity models, followed by a discussion of the SESR-MM and the processes involved in its development. Secondly, participants will be provided with resources to facilitate the development of a maturity model and an assessment instrument for a range of institutional processes and related practices. In the final stage of the workshop, participants will “assess” the capability of these practices to provide a collective assessment of the maturity of these processes. References Australian Council for Educational Research. (n.d.). Australasian Survey of Student Engagement. Retrieved from http://www.acer.edu.au/research/ausse/background Clarke, J., Nelson, K., & Stoodley, I. (2012, July). The Maturity Model concept as framework for assessing the capability of higher education institutions to address student engagement, success and retention: New horizon or false dawn? A Nuts & Bolts presentation at the 15th International Conference on the First Year in Higher Education, “New Horizons,” Brisbane, Australia. Department of Education, Employment and Workplace Relations. (n.d.). The University Experience Survey. Advancing quality in higher education information sheet. Retrieved from http://www.deewr.gov.au/HigherEducation/Policy/Documents/University_Experience_Survey.pdf Kift, S., Nelson, K., & Clarke, J. (2010) Transition pedagogy - a third generation approach to FYE: A case study of policy and practice for the higher education sector. The International Journal of the First Year in Higher Education, 1(1), pp. 1-20. Marshall, S. (2010). A quality framework for continuous improvement of e-Learning: The e-Learning Maturity Model. Journal of Distance Education, 24(1), 143-166. Nelson, K., Clarke, J., & Stoodley, I. (2012). An exploration of the Maturity Model concept as a vehicle for higher education institutions to assess their capability to address student engagement. A work in progress. Submitted for publication. Paulk, M. (1999). Using the Software CMM with good judgment, ASQ Software Quality Professional, 1(3), 19-29. Wilson, K. (2009, June–July). The impact of institutional, programmatic and personal interventions on an effective and sustainable first-year student experience. Keynote address presented at the 12th Pacific Rim First Year in Higher Education Conference, “Preparing for Tomorrow Today: The First Year as Foundation,” Townsville, Australia. Retrieved from http://www.fyhe.com.au/past_papers/papers09/ppts/Keithia_Wilson_paper.pdf

Moving beyond transition pedagogy: Maturity models and student engagement

Australian higher education institutions (HEIs) have entered a new phase of regulation and accreditation which includes performance-based funding relating to the participation and retention of students from social and cultural groups previously underrepresented in higher education. However, in addressing these priorities, it is critical that HEIs do not further disadvantage students from certain groups by identifying them for attention because of their social or cultural backgrounds, circumstances which are largely beyond the control of students. In response, many HEIs are focusing effort on university-wide approaches to enhancing the student experience because such approaches will enhance the engagement, success and retention of all students, and in doing so, particularly benefit those students who come from underrepresented groups. Measuring and benchmarking student experiences and engagement that arise from these efforts is well supported by extensive collections of student experience survey data. However no comparable instrument exists that measures the capability of institutions to influence and/or enhance student experiences where capability is an indication of how well an organisational process does what it is designed to do (Rosemann & de Bruin, 2005). This paper proposes that the concept of a maturity model (Marshall, 2010; Paulk, 1999) may be useful as a way of assessing the capability of HEIs to provide and implement student engagement, success and retention activities. We will describe the Student Engagement, Success and Retention Maturity Model (SESR-MM), (Clarke, Nelson & Stoodley, 2012; Nelson, Clarke & Stoodley, 2012) we are currently investigating. We will discuss if our research may address the current gap by facilitating the development of an SESR-MM instrument that aims (i) to enable institutions to assess the capability of their current student engagement and retention programs and strategies to influence and respond to student experiences within the institution; and (ii) to provide institutions with the opportunity to understand various practices across the sector with a view to further improving programs and practices relevant to their context. The first aim of our research is to extend the generational approach which has been useful in considering the evolutionary nature of the first year experience (FYE) (Wilson, 2009). Three generations have been identified and explored: First generation approaches that focus on co-curricular strategies (e.g. orientation and peer programs); Second generation approaches that focus on curriculum (e.g. pedagogy, curriculum design, and learning and teaching practice); and third generation approaches—also referred to as transition pedagogy—that focus on the production of an institution-wide integrated holistic intentional blend of curricular and co-curricular activities (Kift, Nelson & Clarke, 2010). The second aim of this research is to move beyond assessments of students’ experiences to focus on assessing institutional processes and their capability to influence student engagement. In essence, we propose to develop and use the maturity model concept to produce an instrument that will indicate the capability of HEIs to manage and improve student engagement, success and retention programs and strategies. References Australian Council for Educational Research. (n.d.). Australasian Survey of Student Engagement. Retrieved from http://www.acer.edu.au/research/ausse/background Clarke, J., Nelson, K., & Stoodley, I. (2012, July). The Maturity Model concept as framework for assessing the capability of higher education institutions to address student engagement, success and retention: New horizon or false dawn? A Nuts & Bolts presentation at the 15th International Conference on the First Year in Higher Education, “New Horizons,” Brisbane, Australia. Kift, S., Nelson, K., & Clarke, J. (2010) Transition pedagogy - a third generation approach to FYE: A case study of policy and practice for the higher education sector. The International Journal of the First Year in Higher Education, 1(1), pp. 1-20. Department of Education, Employment and Workplace Relations. (n.d.). The University Experience Survey. Advancing quality in higher education information sheet. Retrieved from http://www.deewr.gov.au/HigherEducation/Policy/Documents/University_Experience_Survey.pdf Marshall, S. (2010). A quality framework for continuous improvement of e-Learning: The e-Learning Maturity Model. Journal of Distance Education, 24(1), 143-166. Nelson, K., Clarke, J., & Stoodley, I. (2012). An exploration of the Maturity Model concept as a vehicle for higher education institutions to assess their capability to address student engagement. A work in progress. Submitted for publication. Paulk, M. (1999). Using the Software CMM with good judgment, ASQ Software Quality Professional, 1(3), 19-29. Wilson, K. (2009, June–July). The impact of institutional, programmatic and personal interventions on an effective and sustainable first-year student experience. Keynote address presented at the 12th Pacific Rim First Year in Higher Education Conference, “Preparing for Tomorrow Today: The First Year as Foundation,” Townsville, Australia. Retrieved from http://www.fyhe.com.au/past_papers/papers09/ppts/Keithia_Wilson_paper.pdf

Measuring student engagement : using 'flow' theory to guide question development

In December 2009 the Department of Education, Employment and Workplace Relations (DEEWR) issued a discussion paper, ‘An Indicator Framework for Higher Education Performance Funding’, with the purpose of contributing to the development of indicators for guiding ‘compact’ negotiations between Australian universities and the government. The discussion paper makes it clear that ‘student experience’ will be an important part of the indicators and discusses the possibility that the Government will develop a new
questionnaire to investigate the engagement and satisfaction of students in their first year of study. This paper contributes ideas to assist the development of such an instrument, particularly for the measurement of student engagement. We discuss problems with the predictive validity of existing instruments and argue that predictive validity may be improved by a focus on highly engaging ‘flow-type’ activities. Specifically, we
investigate flow activities in the context of online learning. Our findings suggest that some of the current questions from the Australasian Survey of Student Engagement (AUSSE) do appear to capture flow activities. We also identify gaps in the AUSSE. Finally, we make recommendations about the inclusion of additional items that capture highly engaging, flow activities for either the AUSSE or the Government’s proposed new instrument.

Some Social-Relational Correlates of Student Engagement in Portugal

Este E-Book reúne um conjunto de investigações apresentadas no “I Congresso Internacional Envolvimento dos Alunos na Escola: Perspetivas da Psicologia e Educação” (ICIEAE), organizado no âmbito do “Projeto PTDC/CPE-CED/114362/2009 - Envolvimento dos Alunos na Escola: Diferenciação e Promoção” (EAE-DP), financiado pela Fundação para a Ciência e a Tecnologia (FCT).

Cultural universality and specificity of student engagement in school: The results of an international study from 12 countries

The Hong Kong subproject was supported by the Quality Education Fund of the Education Bureau in Hong Kong, whereas the Portuguese subproject was supported by the Portuguese Foundation for Science and Technology and by the Institute of Education of the University of Lisbon. The data of this paper were part of the data collected in a multinational project initiated by the International School Psychology Association.

Forms of Student Engagement: development and application of the Southampton post-course research (UKSE) iSurvey for a traditionally taught module

Forms of Student Engagement: development and application of the Southampton post-course research (UKSE) iSurvey for a traditionally taught module

An analysis of student engagement when taught in classes of different sizes in rural, regional and metropolitan campuses of an Australian university

This study investigates the impact of class size on student engagement and student performance. It is based on an analysis of student university enter scores, student grades and student evaluations in metropolitan, regional and rural campuses of an Australian universityduring trimester 1 of years 2008, 9 & 10. Past literature appears to support the predominant influence of the class size effect on learning, though some findings are mixed and inconclusive. Contrary to the accepted view that higher entry level scores result in higher grades and, conversely, lower entry level scores result in lower grades, the findings suggest that factorsother than entry level scores, contribute to student outcomes and student engagement. The study reveals that student satisfaction of teaching quality is higher in the rural and regional campuses where the cohorts are smaller than at the metropolitan campus. This may be an indication that class size seems to have a predominant influence on student engagement and learning outcomes.

Discovering the value that the library can contribute to quality teaching and learning through student evaluation data

Large surveys of library user service quality perception are common. However, student evaluation of teaching (SET) data often show a disparity between ratings of library service quality and library resource quality. In this situation, perhaps SET data can also provide insights into what contributes to the perception of library resource quality, and hence identify leverage points for quality improvement interventions. This paper documents an analysis of available Deakin University SET data relating to student interaction with, and evaluation of, library resources. It highlights significant correlations associated with library-related SET items, and from them infers actions that the library could undertake to improve the value and perception of the quality of library resources. The following results were observed. High ratings for library resources were likely to be associated with high general ratings of teaching and unit quality. Postgraduate coursework students rated library resources significantly higher than students in the first three years of undergraduate programs. Students in one faculty (Health) rated library resources significantly higher than students in all other faculties. There was a strong correlation observed in Australasian Survey of Student Engagement data for both 2009 and 2010 between the two items “Used library resources on campus or online” and “Worked on an essay or assignment that required integrating ideas or information from various sources”. These findings suggest the following conclusions. Well-planned learning environments are likely to integrate meaningful student interaction with the library. Initiatives to improve the value and perception of the quality of library resources should be focussed on the specific characteristics and needs of particular student cohorts to have maximum impact. More sophisticated assessment tasks that require students to interact with the library have the potential to result in higher student ratings of the value of library resources.

Active learning: the importance of developing a comprehensive measure

This article reports on an investigation into the validity of a widely used scale for measuring the extent to which higher education students employ active learning strategies. The scale is the active learning scale in the Australasian Survey of Student Engagement. This scale is based on the Active and Collaborative Learning scale of the National Survey of Student Engagement. The particular focus of the study was to investigate effects resulting from the addition of a small number of items to the active learning scale designed to capture some highly engaging, mostly online, activities. The items were developed in response to concerns that students studying in distance mode often report lower average scores on active learning scales than their on-campus counterparts. The additional items relate to activities such as working online with other students and faculty. The findings show that average scores on the Australasian Survey of Student Engagement/National Survey of Student Engagement scale increase significantly when the new items are included and that some differences between on-campus and distance education students narrow significantly. These findings have implications for the development of more robust and comprehensive instruments to measure active learning.

Assessment of student engagement among junior high school students and associations with self-esteem, burnout, and academic achievement

The aim of the study was to investigate the structure of affective and cognitive engagement using the Student Engagement Instrument (SEI; Appleton, Christenson, Kim, & Reschly, 2006) and to examine the associations to behavioral engagement, as well as student-reported self-esteem, burnout, and academic achievement among Finnish junior high school students. The analyses were carried out in the main sample of 2,485 students, as well as in an independent sample of 821 students. The results showed that the original five-factor structure of the SEI construed along three affective and two cognitive engagement factors fit the current data relatively well. Affective and cognitive student engagement correlated positively with an independent measure of behavioral engagement. Furthermore, affective and cognitive engagement were positively associated with student-reported self-esteem and academic achievement, and negatively with school burnout. The findings provided corroborating evidence for the psychometric properties and utilization of the SEI instrument for assessing the engagement of junior high school students. (DIPF/Orig.)

Using AUSSE data for enhancement

The Australasian Survey of Student Engagement (AUSSE) provides data for each institution on student engagement that is both generalisable and sensitive to institutional context. The data can help universities monitor and enhance the quality of education.