Kaleidoscope : Reframing learning and teaching evaluation through a shared stakeholder lens

Sector wide interest in Reframe: QUT’s Evaluation Framework continues with a number of institutions requesting finer details as QUT embeds the new approach to evaluation across the university in 2013. This interest, both nationally and internationally has warranted QUT’s collegial response to draw upon its experiences from developing Reframe into distilling and offering Kaleidoscope back to the sector. The word Reframe is a relevant reference for QUT’s specific re-evaluation, reframing and adoption of a new approach to evaluation; whereas Kaleidoscope reflects the unique lens through which any other institution will need to view their own cultural specificity and local context through an extensive user-led stakeholder engagement approach when introducing new approaches to learning and teaching evaluation. Kaleidoscope’s objectives are for QUT to develop its research-based stakeholder approach to distil the successful experience exhibited in the Reframe Project into a transferable set of guidelines for use by other tertiary institutions across the sector. These guidelines will assist others to design, develop, and deploy, their own culturally specific widespread organisational change informed by stakeholder engagement and organisational buy-in. It is intended that these guidelines will promote, support and enable other tertiary institutions to embark on their own evaluation projects and maximise impact. Kaleidoscope offers an institutional case study of widespread organisational change underpinned by Reframe’s (i) evidence-based methodology; (ii) research including published environmental scan, literature review (Alderman, et al., 2012), development of a conceptual model (Alderman, et al., in press 2013), project management principles (Alderman & Melanie, 2012) and national conference peer reviews; and (iii) year-long strategic project with national outreach to collaboratively engage the development of a draft set of National Guidelines. Kaleidoscope’s aims are to inform Higher Education evaluation policy development through national stakeholder engagement, the finalisation of proposed National Guidelines. In correlation with the conference paper, the authors will present a Draft Guidelines and Framework ready for external peer review by evaluation practitioners from the Higher Education sector, as part of Kaleidoscope’s dissemination strategy (Hinton & Gannaway, 2011) applying illuminative evaluation theory (Parlett & Hamilton, 1976), through conference workshops and ongoing discussions (Shapiro, et al., 1983; Jacobs, 2000). The initial National Guidelines will be distilled from the Reframe: QUT’s Evaluation Framework’s Policy, Protocols, and incorporated Business Rules. It is intended that the outcomes of Kaleidoscope are owned by and reflect sectoral engagement, including iterative evaluation through multiple avenues of dissemination and collaboration including the Higher Education sector. The dissemination strategy with the inclusion of Illuminative Evaluation methodology provides an inclusive opportunity for other institutions and stakeholders across the Higher Education sector to give voice through the information-gathering component of evaluating the draft Guidelines, providing a comprehensive understanding of the complex realities experienced across the Higher Education sector, and thereby ‘illuminating’ both the shared and unique lenses and contexts. This process will enable any final guidelines developed to have broader applicability, greater acceptance, enhanced sustainability and additional relevance benefiting the Higher Education sector, and the adoption and adaption by any single institution for their local contexts.

Kaleidoscope : illuminating learning and teaching evaluation for a shared stakeholder lens

Within the Australian higher education sector, institutions are required to evaluate teaching, units and courses to assure the quality of the student learning experience, however with hardly any regulatory parameters guiding institutions, and with disparate practices, there are few opportunities to benchmakr across institutions or the sector. QUT has received interest and requests from national and international universities on accessing Reframe: QUT's Evaluation Framework.

Evaluation of the quality of diverse teaching practice in higher education research methods

The purpose of this paper is to report on a methods research project investigating the evaluation of diverse teaching practice in Higher Education. The research method is a single site case study of an Australian university with data collected through published documents, surveys, interviews and focus groups. This project provides evidence of the wide variety of evaluation practice and diverse teaching practice across the university. This breadth identifies the need for greater flexibility of evaluation processes, tools and support to assist teaching staff to evaluate their diverse teaching practice. The employment opportunities for academics benchmark the university nationally and position the case study in the field. Finally this reaffirms the institutional responsibility for services to support teaching staff in an ongoing manner.

Building and sustaining a culture to support peer review of teaching

1. Background/context This presentation will report on emerging results from a two phase project funded by the Australian Learning and Teaching Council (ALTC). The project was designed in partnership with five universities and aimed to embed peer review within the local teaching and learning culture by using a distributive leadership framework. 2. The initiative/practice The presentation will highlight research outcomes that bring together both the fundamentals of peer review of teaching with the broader contextual elements of Integration, Leadership and Development. It will be demonstrated that peer review of teaching can be implemented and have advantages for academic staff, teaching evaluation and an organisation if attention is given to strategies that influence the contexts and cultures of teaching. Peer review as a strategy to develop excellence in teaching is considered from a holistic perspective that by necessity encompasses all elements of an educational environment. Results demonstrate achievements that can be obtained through working to foster conditions needed for sustainable leadership and change. The work has implications for policy, research, teaching development and student outcomes and has potential application world-wide. 3. Method(s) of evaluative data collection and analysis The 2 phase project collected focus group and questionnaire data to inform research results that were analysed using a thematic qualitative approach and statistical exploration. 4. Evidence of effectiveness The presentation will demonstrate the effectiveness of distributive leadership and strategic approaches to working for cultural change through the presentation of project findings.

Developing a culture to support peer review of teaching in higher education

This presentation addresses issues related to leadership, academic development and scholarship of teaching and learning, and highlights research funded by the Australian Office of Learning and Teaching (OLT) designed to embed and sustain peer review of teaching within the culture of 5 Australian universities: Queensland University of Technology, University of Technology, Sydney, University of Adelaide, Curtin University, and Charles Darwin University. Peer review of teaching in higher education will be emphasised as a professional process for providing feedback on teaching and learning practice, which if sustained, can become an effective ongoing strategy for academic development (Barnard et al, 2011; Bell, 2005; Bolt and Atkinson, 2010; McGill & Beaty 2001, 1992; Kemmis & McTaggart, 2000). The research affirms that using developmental peer review models (Barnard et al, 2011; D'Andrea, 2002; Hammersley-Fletcher & Orsmond, 2004) can bring about successful implementation, especially when implemented within a distributive leadership framework (Spillane & Healey, 2010). The project’s aims and objectives were to develop leadership capacity and integrate peer review as a cultural practice in higher education. The research design was a two stage inquiry process over 2 years. The project began in July 2011 and encompassed a development and pilot phase followed by a cascade phase with questionnaire and focus group evaluation processes to support ongoing improvement and measures of outcome. Leadership development activities included locally delivered workshops complemented by the identification and support of champions. To optimise long term sustainability, the project was implemented through existing learning and teaching structures and processes within the respective partner universities. Research outcomes highlight the fundamentals of peer review of teaching and the broader contextual elements of integration, leadership and development, expressed as a conceptual model for embedding peer review of teaching within higher education. The research opens a communicative space about introduction of peer review that goes further than simply espousing its worth and introduction. The conceptual model highlights the importance of development of distributive leadership capacity, integration of policies and processes, and understanding the values, beliefs, assumptions and behaviors embedded in an organizational culture. The presentation overviews empirical findings that demonstrate progress to advance peer review requires an ‘across-the-board’ commitment to embed change, and inherently demands a process that co-creates connection across colleagues, discipline groups, and the university sector. Progress toward peer review of teaching as a cultural phenomenon can be achieved and has advantages for academic staff, scholarship, teaching evaluation and an organisation, if attention is given to strategies that influence the contexts and cultures of teaching practice. Peer review as a strategy to develop excellence in teaching is considered from a holistic perspective that by necessity encompasses all elements of an educational environment and has a focus on scholarship of teaching. The work is ongoing and has implication for policy, research, teaching development and student outcomes, and has potential application world-wide.

Framing student evaluations of university learning and teaching : discursive strategies and textual outcomes

Evaluation in higher education is an evolving social practice, that is, it involves what people, institutions and broader systems do and say, how they do and say it, what they value, the effects of these practices and values, and how meanings are ascribed. The textual products (verbal, written, visual, gestural) that inform and are produced by, for and through evaluative practices are important as they promulgate particular kinds of meanings and values in specific contexts. This paper reports on an exploratory study that sought to investigate, using discourse analysis, the types of evaluative practices that were ascribed value, and the student responses that ensued, in different evaluative instruments. Findings indicate that when a reflective approach is taken to evaluation, students’ responses are more considered, they interrogate their own engagement in the learning context and they are more likely to demonstrate reconstructive thought. These findings have implications for reframing evaluation as reflective learning.

Evaluation of motivation methods used to obtain appropriate oral hygiene levels in schoolchildren.

The aim of the present study is to assess the behaviour of different motivation methods on levels of oral hygiene among schoolchildren aged from 7 to 9 years in Araraquara, SP, Brazil. The methods tested were: indirect instruction using 'The Smiling Robot' (group I), indirect instruction through class presentation (group II) and direct instruction with macromodels (group III). A control group was also constituted, which received no kind of motivation (group IV). The O'Leary Plaque Index was used as the evaluation method, applied before the instruction and 30 days after application of the different methods. It was noted that the plaque index had not decreased in group IV only. In conclusion, all the motivation methods promoted significant decrease of plaque index and among these methods, the 'The Smiling Robot' was the one that provided the best results.

An evaluation of English language instruction in universities/colleges in Taiwan

The purpose of this study is about to survey the effectiveness of English teaching in universities/colleges in Taiwan and provide suggestions to the educational authorities for upgrading or improving the English language teaching techniques or methods currently in use. On the one hand, the suggestions of English teachers in universities/colleges are the main concerns in this study. On the other hand, equal attention is given to the opinions and learning attitudes of the students, including the interaction between the teachers and the students.^ A social investigation methodology, consisting of questionnaire and data analysis, is devoted to this evaluation and analysis of teaching effectiveness. Among the significant findings reported in this study is the fact that both students and teachers consider evaluation of teaching effectiveness as a very important aspect of English Language programs at the university/college level. Another significant finding is that students are less concerned about the grades earned or awarded in English Language courses and more concerned about how much they learned in these classes.^ The analysis and suggestions based on the results for English teaching evaluation of universities/colleges in this dissertation can help teachers modify their teaching techniques and methods, the students to adjust their learning attitudes, and the authorities to schematize the English teaching plan that can facilitate the teaching goals to be realized. ^

Longitudinal Student Evaluations of a Postgraduate Unit Using Importance-Performance Analysis

The Australian tourism tertiary education sector operates in a competitive and dynamic environment, which necessitates a market orientation to be successful. Academic staff and management in the sector must regularly assess the perceptions of prospective and current students, and monitor the satisfaction levels of current students. This study is concerned with the setting and monitoring of satisfaction levels of current students, reporting the results of three longitudinal investigations of student satisfaction in a postgraduate unit. The study also addresses a limitation of a university’s generic teaching evaluation instrument. Importance-performance analysis (IPA) has been recommended as a simple but effective tool for overcoming the deficiencies of many student evaluation studies, which have generally measured only attribute importance or importance at the end of a semester. IPA was used to compare student expectations of the unit at the beginning of semester with their perceptions of performance ten weeks later. The first stage documented key benchmarks for which amendments to the unit based on student feedback could be evaluated during subsequent teaching periods.

Levelling expectations across different years in an undergraduate degree to support work-related learning : evidence for a multi-pronged approach.

One of the ways in which university departments and faculties can enhance the quality of learning and assessment is to develop a ‘well thought out criterion‐referenced assessment system’ (Biggs, 2003, p. 271). In designing undergraduate degrees (courses) this entails making decisions about the levelling of expectations across different years through devising objectives and their corresponding criteria and standards: a process of alignment analogous to what happens in unit (subject) design. These decisions about levelling have important repercussions in terms of supporting students’ work‐related learning, especially in relation to their ability to cope with the increasing cognitive and skill demands made on them as they progress through their studies. They also affect the accountability of teacher judgments of students’ responses to assessment tasks, achievement of unit objectives and, ultimately, whether students are awarded their degrees and are sufficiently prepared for the world of work. Research reveals that this decision‐making process is rarely underpinned by an explicit educational rationale (Morgan et al, 2002). The decision to implement criterion referenced assessment in an undergraduate microbiology degree was the impetus for developing such a rationale because of the implications for alignment, and therefore ‘levelling’ of expectations across different years of the degree. This paper provides supporting evidence for a multi‐pronged approach to levelling, through backward mapping of two revised units (foundation and exit year). This approach adheres to the principles of alignment while combining a work‐related approach (via industry input) with the blended disciplinary and learner‐centred approaches proposed by Morgan et al. (2002). It is suggested that this multi‐pronged approach has the potential for making expectations, especially work‐related ones across different year levels of degrees, more explicit to students and future employers.

Longitudinal evaluations of student satisfaction with a postgraduate unit using importance-performance analysis

The Australian tourism tertiary education sector operates in a competitive and dynamic environment, which necessitates a market orientation to be successful. Academic staff and management in the sector must regularly assess the perceptions of prospective and current students and monitor the satisfaction levels of current students. This study is concerned with the setting and monitoring of satisfaction levels of current students, reporting the results of three longitudinal investigations of student satisfaction in a postgraduate unit. The study also addresses a limitation of a university’s generic teaching evaluation instrument. Importance-Performance Analysis (IPA) has been recommended as a simple but effective tool for overcoming the deficiencies of many student evaluation studies, which have generally measured only attribute performance at the end of a semester. IPA was used to compare student expectations of the unit at the beginning of a semester with their perceptions of performance 10 weeks later. The first stage documented key benchmarks for which amendments to the unit based on student feedback could be evaluated during subsequent teaching periods.

Fresh frozen cadavers as a tool to educate student paramedics in procedural skills

Objectives – To describe the development of an educational workshop to develop procedural skills in undergraduate Paramedic students using fresh frozen cadavers and to report the student’s assessment of the program. Methods – A six-hour anatomy based workshop was developed using fresh frozen cadavers to teach a range of airway and invasive procedural skills to second year undergraduate paramedic students. Embedded QUAN (qual) methodology will be utilised to evaluate the student’s satisfaction, perception and quality of teaching as compared to other existing clinical teaching techniques such as high fidelity simulation. Students will be asked to complete an anonymous validated survey (10 questions formulated on a 5 point Likert scale) and provide a qualitative feedback pre and post the six-hour workshop. Results – This is a prospective study planned for September 2013. Low-risk human research ethics are being sought. Teaching evaluation results from the inaugural 2012 workshop (undergraduate and postgraduate Paramedic students) and interim results for 2013 will be presented. Conclusions – Clinical teaching using fresh frozen cadavers thus far has predominately been used in the education of medical and surgical trainees. A number of studies have found them to be effective and in some cases superior to traditional high fidelity simulation teaching strategies. Fresh frozen cadavers are said to provide perfect anatomy, normal tissue consistency and a realistic operative training experience (Lloyd, Maxwell-Armstrong et al. 2011). The authors believe that this study will show that the use of fresh frozen cadavers offers a safe and effective mode to teach procedural skills to student paramedics that will help bridge the skills gap and increase confidence prior to students undertaking such interventions on living patients. A modified training program may be formulated for general practitioners undertaking Emergency Medicine Advanced Rural Skills.

Kaleidoscope : shaping a better future through a stakeholder approach to widespread organisational change

The main focus of ‘Kaleidoscope: Reframing evaluation through a stakeholder approach to sustainable, cultural change in Higher Education’ is to develop a set of principles to guide user-led engagement in widespread organisational change and maximise its impact. The word kaleidoscope represents the unique lens through which each institution will need to view their cultural specificity and local context through an extensive process of collaboration and engagement, followed by communication and dissemination. Kaleidoscope has particular relevance when new approaches to learning and teaching evaluation are introduced by tertiary institutions. Building on the Reframe Project, which involved three years of user-led consultation and was designed to meet stakeholders’ needs, QUT successfully introduced a new evaluation framework in 2013 across the university. Reframe was evidence based, involved scholarly reflection and was founded on a strong theoretical framework. The evolution of the evaluation framework included analysis of scholarly literature and environmental scans across the higher education sector (Alderman, et al., 2012), researched development of conceptual theory (Alderman, et al., in press 2013), incorporated the stakeholder voice and framed within project management principles (Alderman & Melanie, 2012). Kaleidoscope’s objectives are for QUT to develop its research-based stakeholder approach to distil the successful experience exhibited in the Reframe Project into a transferable set of guidelines for use by other tertiary institutions across the sectors. These guidelines will assist others to design, develop, and deploy, their own culturally specific widespread organisational change informed by stakeholder engagement and organisational buy-in. It is intended that these guidelines will promote, support and enable other tertiary institutions to embark on their own projects and maximise the impact. In correlation with a our conference paper, this round table presents the Draft Guidelines and Framework ready for external peer review by evaluation practitioners, as part of Kaleidoscope’s dissemination (Hinton & Gannaway, 2011) applying illuminative evaluation theory (Parlett & Hamilton, 1976), through conference workshops and linked round table discussions (Shapiro, et al., 1983; Jacobs, 2000).

Avaliação em contextos de blended learning no ensino superior

The increasing complexity of the Information Society and the Bologna Declaration in the European context has led Higher Education (HE) institutions to revise their curricula courses, as far as the adoption of new strategies for teaching and learning as well as evaluation are concerned. It can also be emphasized that there has been a growing use of eLearning and of blended learning (bLearning) in HE, since these modes of training seem to be a very convenient option for lifelong learning. In this context, quality is taken as an essential goal in the development of the European Space for Education and Training, where HE institutions compete among themselves, and where evaluation is determinant as a promoter of this quality. Considering the problems summarized above, the research developed, based on four published scientific papers, intended to answer a set of research questions related to evaluation of bLearning contexts in HE. The study used diverse techniques and instruments (questionnaires, document analysis, and observation mediated technologies) spanning two methodological approaches: i) study of descriptive and exploratory nature and ii) case studies of bLearning modules. In the first approach an evaluation model for bLearning courses was developed, where we collected and analyzed, at a national level, the opinions of teachers with bLearning experience about the model dimensions. The case studies presented are post graduation curricular units, where bLearning teaching, learning and evaluation strategies were explored and evaluated, namely peer assessment. The main contributions of the first approach are: the process of questioning around the evaluation of bLearning courses, namely the quality assurance criteria for bLearning, as well as the model developed, providing a framework of theoretical, methodological and empirical elements that can be adapted in similar contexts. From the case studies emerged: the developed evaluation guidelines and the data collection instruments, in order to disseminate evaluation “best practices” that may be useful for other units in similar contexts. Regarding the recommendations about the evaluation of teaching of bLearning courses we emphasize: the use of versatile evaluation objects; the evaluation throughout the process and not just at the end; and the involvement of multiple evaluators, including students (whose feedback is essential to monitor the quality of teaching and learning). From the case studies we highlight: the need for discussion of evaluation frameworks to explore, and consequent increase in the transparency of the evaluation process; the increased interaction between groups; and the peer assessment as a strategy to promote active and autonomous learning. In addition to the contributions and recommendations for practice and research in the area of evaluation in bLearning contexts in HE, listed above, it also emerged from this study useful guidelines regarding educational evaluation in bLearning contexts, in order to improve the quality of teaching, learning and evaluation in such contexts.