Kaleidoscope : Reframing learning and teaching evaluation through a shared stakeholder lens

Sector wide interest in Reframe: QUT’s Evaluation Framework continues with a number of institutions requesting finer details as QUT embeds the new approach to evaluation across the university in 2013. This interest, both nationally and internationally has warranted QUT’s collegial response to draw upon its experiences from developing Reframe into distilling and offering Kaleidoscope back to the sector. The word Reframe is a relevant reference for QUT’s specific re-evaluation, reframing and adoption of a new approach to evaluation; whereas Kaleidoscope reflects the unique lens through which any other institution will need to view their own cultural specificity and local context through an extensive user-led stakeholder engagement approach when introducing new approaches to learning and teaching evaluation. Kaleidoscope’s objectives are for QUT to develop its research-based stakeholder approach to distil the successful experience exhibited in the Reframe Project into a transferable set of guidelines for use by other tertiary institutions across the sector. These guidelines will assist others to design, develop, and deploy, their own culturally specific widespread organisational change informed by stakeholder engagement and organisational buy-in. It is intended that these guidelines will promote, support and enable other tertiary institutions to embark on their own evaluation projects and maximise impact. Kaleidoscope offers an institutional case study of widespread organisational change underpinned by Reframe’s (i) evidence-based methodology; (ii) research including published environmental scan, literature review (Alderman, et al., 2012), development of a conceptual model (Alderman, et al., in press 2013), project management principles (Alderman & Melanie, 2012) and national conference peer reviews; and (iii) year-long strategic project with national outreach to collaboratively engage the development of a draft set of National Guidelines. Kaleidoscope’s aims are to inform Higher Education evaluation policy development through national stakeholder engagement, the finalisation of proposed National Guidelines. In correlation with the conference paper, the authors will present a Draft Guidelines and Framework ready for external peer review by evaluation practitioners from the Higher Education sector, as part of Kaleidoscope’s dissemination strategy (Hinton & Gannaway, 2011) applying illuminative evaluation theory (Parlett & Hamilton, 1976), through conference workshops and ongoing discussions (Shapiro, et al., 1983; Jacobs, 2000). The initial National Guidelines will be distilled from the Reframe: QUT’s Evaluation Framework’s Policy, Protocols, and incorporated Business Rules. It is intended that the outcomes of Kaleidoscope are owned by and reflect sectoral engagement, including iterative evaluation through multiple avenues of dissemination and collaboration including the Higher Education sector. The dissemination strategy with the inclusion of Illuminative Evaluation methodology provides an inclusive opportunity for other institutions and stakeholders across the Higher Education sector to give voice through the information-gathering component of evaluating the draft Guidelines, providing a comprehensive understanding of the complex realities experienced across the Higher Education sector, and thereby ‘illuminating’ both the shared and unique lenses and contexts. This process will enable any final guidelines developed to have broader applicability, greater acceptance, enhanced sustainability and additional relevance benefiting the Higher Education sector, and the adoption and adaption by any single institution for their local contexts.

Kaleidoscope : illuminating learning and teaching evaluation for a shared stakeholder lens

Within the Australian higher education sector, institutions are required to evaluate teaching, units and courses to assure the quality of the student learning experience, however with hardly any regulatory parameters guiding institutions, and with disparate practices, there are few opportunities to benchmakr across institutions or the sector. QUT has received interest and requests from national and international universities on accessing Reframe: QUT's Evaluation Framework.

Troubled Times in Australian Teacher Education : 2012-2013

The 2012 OLT National Teaching Fellowship described in this report has mapped and analysed the complex and competing internal and external agencies impacting on the whole-of-curriculum design in contemporary higher education in Australia, particularly on degrees in Education with an emphasis on initial teacher education. The Fellowship was conducted at a time of both heightened public and political scrutiny of teacher education and the imposition of new nationally-consistent accreditation processes. This scrutiny culminated in a call by the previous Federal Government for TEQSA (Tertiary Education Quality Standards Agency) to conduct a comprehensive review of teacher education beginning in 2014 and the incoming Government announcing it will establish a short term ministerial advisory group to report on the “priority issue of improving teacher quality” (Pyne, 2013).

Making prevention matter : a survey of child sexual abuse prevention programs in Australia

Throughout Australia, there is considerable concern about the prevalence of child sexual abuse. Sexual abuse is experienced by over 3,500 Australian children each year, causing long-term psychological, health and social problems to children, their families and communities, and significant economic costs to society as a whole. In many countries, the provision of school-based programs has been a core strategy in efforts to prevent child sexual abuse. However, little is known about the range of programs in use in Australia, the numbers of children who have received programs, and the contents and methods used in program delivery. This presentation reports on a detailed National survey of child sexual abuse prevention programs currently used in Australian primary schools. An online survey was conducted over a six-month period from November 2011 to April 2012 yielding detailed data from 38 programs. The presentation will provide an overview of: the scope and reach of programs; program content, teaching strategies and resources; barriers and facilitators to program adoption by schools; and program evaluation strategies.

Teacher Evidence Matrix : an evaluation using multiple lines of evidence

Welcome to the Teacher evidence matrix. This matrix is designed for highly qualified discipline experts to evaluate their teaching in a systematic manner. The primary purpose of the Teacher evidence matrix is to provide a tool that an academic staff member at university can annually review their teaching. The annual review will result in you being ready for performance, planning and review; promotion; awards; or employment application. This tool is designed for individual use and will lead to an action plan for implementation.

Building indicators in media assistance; four ways evaluation reports present “change”

The international aid and development community has supported programs that aim to build the capacity of media professionals or contribute to an enabling environment throughout the past 20 years. However, two decades on from the first modern media assistance programs, the sector is still struggling to identify, measure and understand the changes effected by their programs. There are questions raised as to whether it is even feasible to identify impacts on society and governance. This paper draws on some preliminary findings from a comparative thematic analysis of 47 evaluation documents of media assistance programs. The aim of this analysis is to identify trends in impact evaluation practice in the media assistance field, as well as the strengths and weaknesses of different evaluation approaches. This paper presents four types of social change claims commonly presented in reports; hypothetical changes, introduction of new opportunities, concrete examples of immediate impacts, and analysis of ongoing social and political changes. Although these types may appear as a spectrum from weak to strong, the interactions are perhaps more accurately understood using metaphors such as building blocks. This paper explores these types in more detail and suggests that a robust set of impacts-types could be useful in developing more grounded theories of change and indicators.

Innovation through action research in environmental education : from project to praxis

This thesis is a work-in-progress that articulates my research journey based on the development of a curriculum innovation in environmental education. This journey had two distinct, but intertwined phases: action research based fieldwork, conducted collaboratively, to create a whole school approach to environmental education curriculum planning; and a phase of analysis and reflection based on the emerging findings, as I sought to create personal "living educational theory" about change and innovation. A key stimulus for the study was the perceived theory-practice gap in environmental education, which is often presented in the literature as a criticism of teachers for failing to achieve the values and action objectives of critical environmental education. Hence, many programs and projects are considered to be superficial and inconsequential in terms of their ability to seriously address environmental issues. The intention of this study was to work with teachers in a project that would be an exemplar of critical environmental education. This would be in the form of a whole school "learnscaping" curriculum in a primary school whereby the schoolgrounds would be utilised for interdisciplinary critical environment education. Parallel with the three cycles of action research in this project, my research objectives were to identify and comment upon the factors that influence the generation of successful educational innovation. It was anticipated that the project would be a collaboration involving me, as researcher-facilitator, and many of the teachers in the school as active participants. As the project proceeded through its action cycles, however, it became obvious that the goal of developing a critical environmental education curriculum, and the use of highly participatory processes, were unrealistic. Institutional and organisational rigidities in education generally, teachers' day-to-day work demands, and the constant juggle of work, family and other responsibilities for all participants acted as significant constraints. Consequently, it became apparent that the learnscaping curriculum would not be the hoped-for exemplar. Progress was slow and, at times, the project was in danger of stalling permanently. While the curriculum had some elements of critical environmental education, these were minor and not well spread throughout the school. Overall, the outcome seemed best described as a "small win"; perhaps just another example of the theory-practice gap that I had hoped this project would bridge. Towards the project's end, however, my continuing reflection led to an exploration of chaos/complexity theory which gave new meaning to the concept of a "small win". According to this theory, change is not the product of linear processes applied methodically in purposeful and diligent ways, but emerges from serendipitous events that cannot be planned for, or forecast in advance. When this perspective of change is applied to human organisations - in this study, a busy school - the context for change is recognised not as a stable, predictable environment, but as a highly complex system where change happens all the time, cannot be controlled, and no one can be really sure where the impacts might lead. This so-called "butterfly effect" is a central idea of this theory where small changes or modifications are created - the effects of which are difficult to know, let alone determine - and which can have large-scale impacts. Allied with this effect is the belief that long term developments in an organisation that takes complexity into account, emerge by spontaneous self-organising evolution, requiring political interaction and learning in groups, rather than systematic progress towards predetermined goals or "visions". Hence, because change itself and the contexts of change are recognised as complex, chaos/complexity theory suggests that change is more likely to be slow and evolutionary - cultural change - rather than fast and revolutionary where the old is quickly ushered out by radical reforms and replaced by new structures and processes. Slow, small-scale changes are "normal", from a complexity viewpoint, while rapid, wholesale change is both unlikely and unrealistic. Therefore, the frustratingly slow, small-scale, imperfect educational changes that teachers create - including environmental education initiatives - should be seen for what they really are. They should be recognised as successful changes, the impacts of which cannot be known, but which have the potential to magnify into large-scale changes into the future. Rather than being regarded as failures for not meeting critical education criteria, "small wins" should be cause for celebration and support. The intertwined phases of collaborative action research and individual researcher reflection are mirrored in the thesis structure. The first three chapters, respectively, provide the thesis overview, the literature underpinning the study's central concern, and the research methodology. Chapters 4, 5, and 6 report on each of the three action research cycles of the study, namely Laying the Groundwork, Down to Work!, and The Never-ending Story. Each of these chapters presents a narrative of events, a literature review specific to developments in the cycle, and analysis and critique of the events, processes and outcomes of each cycle. Chapter 7 provides a synthesis of the whole of the study, outlining my interim propositions about facilitating curriculum change in schools through action research, and the implications of these for environmental education.

Innovation for Equity in Rural Education : Proceedings of the International Symposium for Innovation in Rural Education

PREFACE The concept of an international symposium on rural education arose from a meeting between members of the SiMERR National Centre, Australia and the NURI Teacher Education Innovation Centre (NURI-TEIC) at Kongju National University, Korea in 2007. Despite the very different national contexts, the teams were struck by the similarities of the challenges facing rural schools in the two countries, and curious about the degree to which these challenges were shared by other countries. At a subsequent meeting in Australia in December 2007, the two centre Directors - Professor John Pegg (SiMERR) and Professor Youn-Kee Im (NURI-TEIC) - agreed on a framework for the first International Symposium for Innovation in Rural Education (ISFIRE). This volume consists of the keynotes and refereed papers presented at ISFIRE 2009. The papers provide insights into rural education in Australia, Bhutan, Canada, Korea, Norway, South Africa and the United States along the following themes: 1. Promoting rural policy initiatives; 2. Nurturing the rural teacher experience; 3. Enhancing rural student experience and growth; 4. Optimising the curriculum; 5. Improving resources in rural schools; and 6. Addressing special issues in rural education. The authors and titles of a further 23 presentations based on refereed abstracts only are listed at the end of this volume. The abstracts for these presentations can be found in the symposium Program. The academics and practitioners who came together for this symposium are passionate about rural education and have dedicated their time and capacities to working for the benefit of rural teachers, students and communities. The papers in this volume offer direction not only for Australia and South Korea, who jointly hosted this symposium, but for all countries in which rural education is contested ground. The keynotes in particular contribute rich perspectives on rural education trends, policies and practices. We hope that this volume generates a greater appreciation of the advantages of rural education and the many innovative approaches being implemented to meet its challenges.

Metacognitive instruction : global and local shifts in considering listening input

A key shift of thinking for effective learning and teaching of listening input has been seen and organized in education locally and globally. This study has probed whether metacognitive instruction through a pedagogical cycle shifts high-intermediate students' English language learning and English as a second language (ESL) teacher's teaching focus on listening input. Twenty male Iranian students with an age range of 18 to 24 received a guided methodology including metacognitive strategies (planning, monitoring, and evaluation) for a period of three months. This study has used the strategies and probed the importance of metacognitive instruction through interviewing both the teacher and the students. The results have shown that metacognitive instruction helped both the ESL teacher's and the students' shift of thinking about teaching and learning listening input. This key shift of thinking has implications globally and locally for classroom practices of listening input.

Critical reflections on the use of participatory methodologies to build evaluation capacities in international development organisations

"In this chapter the authors present a critique of Participatory Evaluation as worked in development projects, in this case, in Nepal. The article works between established claims that Participatory Evaluation builds capacity at programmatic and organisational levels, and the specific experiences of these claims in the authors’ current work. They highlight the need to address key difficulties such as high turn-over of staff and resulting loss of capacity to engage in Participatory Evaluation, and the difficulty of communication between academic as compared with local practical wisdoms. A key issue is the challenge of addressing the inevitable issues of power inequities that such approaches encounter. While Participatory Evaluation has been around for some time, it has only enjoyed more widespread recognition of its value in comparatively recent times, with its uptake in international development environments. To this extent, the practice is still in its early stages of development, and Jo, June and Michael’s work contributes to strengthening and more comprehensively understanding it. With regard to the meta-theme of this publication, this chapter is an example of how context not only influences the methodology to be used and the praxis of how it is to be used, but contributes to early explication of the core nature of an emerging methodology."

Promoting resilience in children with intellectual disability : a randomized controlled trial in Australian schools

Children with intellectual disability are more vulnerable to adverse developmental outcomes because of the lifelong risks associated with cognitive impairment. Difficulties with learning and adaptive behaviour inevitably produce considerable personal, social and economic disadvantage. Of concern is consistent evidence that psychiatric disorders affect a substantial proportion of people with intellectual disability. The estimated prevalence rate of between 35 and 49 % is three times that found in the general population (Wallander, Dekker, & Koot, 2006). Until recently, mental illness has been relatively neglected for people with intellectual disability, especially in relation to prevention or early detection (Kolaitis, 2008) and most research to date has been descriptive rather than focused on intervention (Bouras, 2013). Yet a considerable body of evidence demonstrates that efficacious interventions do exist for preventing psychopathology and enhancing resilience in typically developing children and adolescents (see Mallin, Walker, & Levin, 2013 for a review). In order to prevent the high comorbidity of intellectual disability and psychopathology, there is a compelling need for evidence-based practices that promote the resilience of individuals with intellectual disability (Matson, Terlonge, & Minshawi, 2008). In this chapter, we describe a randomized controlled trial of an intervention that was designed to enhance the resilience of a group of children with mild intellectual disability as they prepared to make the transition to high school. We report results from our evaluation of this intervention, and reflect on the difficulties of providing successful interventions for children whose lives are complicated not only by intellectual disability, but also by a range of contextual disadvantages.

Emotional climate and high quality learning experiences in science teacher education

The role of emotion during learning encounters in science teacher education is under-researched and under-theorized. In this case study we explore the emotional climates, that is, the collective states of emotional arousal, of a preservice secondary science education class to illuminate practice for producing and reproducing high quality learning experiences for preservice science teachers. Theories related to the sociology of emotions informed our analyses from data sources such as preservice teachers’ perceptions of the emotional climate of their class, emotional facial expressions, classroom conversations, and cogenerative dialogue. The major outcome from our analyses was that even though preservice teachers reported high positive emotional climate during the professor’s science demonstrations, they also valued the professor’s in the moment reflections on her teaching that were associated with low emotional climate ratings. We co-relate emotional climate data and preservice teachers’ comments during cogenerative dialogue to expand our understanding of high quality experiences and emotional climate in science teacher education. Our study also contributes refinements to research perspectives on emotional climate.

How do preservice teacher education students move from novice to expert assessors?

Despite the acknowledged importance of assessment in education, there has been minimal research into the preparation of preservice teachers for the important role of involving preservice teachers in marking, grading, moderating and providing feedback on student work. This article reports on a pilot project in which preservice teachers participated in an ongoing peer assessment and social moderation process in a dedicated course on assessment. The purpose of the project was to investigate specific ways in which key assessment processes can be effectively taught to preservice teachers. The research involved 96 preservice teachers who completed a Likert scale survey and free text responses to set questions. The results indicated that while preservice teachers valued the process, continual opportunities to learn the nature and purpose of essential assessment practices related to marking, grading, moderating and providing feedback are necessary to graduate competent and work-ready assessors.

Building a community of learners within the STEM higher education classroom

Building rich and authentic learning experiences in the STEM classroom, is a challenge for many educators within Higher Education. While many Higher Education Institutions have embraced the need to transform current teaching and learning practices and include a range of online tools, this has often been met with some resistance and approaches that do not always recognise the academic who are a critical component to the success of the transformational process. Over the last decade the Internet has evolved from being a tool used by a few dedicated educators to one that is being used by the majority of educators. However, what is important is how this great resource is used in teaching and learning to allow students to build knowledge. The ability for students to construct knowledge and engage in higher order thinking skills is at the heart of educational practices, and building a community of learners has the potential to support these practices, especially within STEM education. This paper explores the relationship between students and an academic teaching in a technology rich STEM learning environment and their adoption of social community and shared tools. In particular the paper reports on the critical components that make a successful community of learners and the educational tools and approaches that were successfully used to enhance the student learning experience in a STEM classroom.