Kaleidoscope : Reframing learning and teaching evaluation through a shared stakeholder lens

Sector wide interest in Reframe: QUT’s Evaluation Framework continues with a number of institutions requesting finer details as QUT embeds the new approach to evaluation across the university in 2013. This interest, both nationally and internationally has warranted QUT’s collegial response to draw upon its experiences from developing Reframe into distilling and offering Kaleidoscope back to the sector. The word Reframe is a relevant reference for QUT’s specific re-evaluation, reframing and adoption of a new approach to evaluation; whereas Kaleidoscope reflects the unique lens through which any other institution will need to view their own cultural specificity and local context through an extensive user-led stakeholder engagement approach when introducing new approaches to learning and teaching evaluation. Kaleidoscope’s objectives are for QUT to develop its research-based stakeholder approach to distil the successful experience exhibited in the Reframe Project into a transferable set of guidelines for use by other tertiary institutions across the sector. These guidelines will assist others to design, develop, and deploy, their own culturally specific widespread organisational change informed by stakeholder engagement and organisational buy-in. It is intended that these guidelines will promote, support and enable other tertiary institutions to embark on their own evaluation projects and maximise impact. Kaleidoscope offers an institutional case study of widespread organisational change underpinned by Reframe’s (i) evidence-based methodology; (ii) research including published environmental scan, literature review (Alderman, et al., 2012), development of a conceptual model (Alderman, et al., in press 2013), project management principles (Alderman & Melanie, 2012) and national conference peer reviews; and (iii) year-long strategic project with national outreach to collaboratively engage the development of a draft set of National Guidelines. Kaleidoscope’s aims are to inform Higher Education evaluation policy development through national stakeholder engagement, the finalisation of proposed National Guidelines. In correlation with the conference paper, the authors will present a Draft Guidelines and Framework ready for external peer review by evaluation practitioners from the Higher Education sector, as part of Kaleidoscope’s dissemination strategy (Hinton & Gannaway, 2011) applying illuminative evaluation theory (Parlett & Hamilton, 1976), through conference workshops and ongoing discussions (Shapiro, et al., 1983; Jacobs, 2000). The initial National Guidelines will be distilled from the Reframe: QUT’s Evaluation Framework’s Policy, Protocols, and incorporated Business Rules. It is intended that the outcomes of Kaleidoscope are owned by and reflect sectoral engagement, including iterative evaluation through multiple avenues of dissemination and collaboration including the Higher Education sector. The dissemination strategy with the inclusion of Illuminative Evaluation methodology provides an inclusive opportunity for other institutions and stakeholders across the Higher Education sector to give voice through the information-gathering component of evaluating the draft Guidelines, providing a comprehensive understanding of the complex realities experienced across the Higher Education sector, and thereby ‘illuminating’ both the shared and unique lenses and contexts. This process will enable any final guidelines developed to have broader applicability, greater acceptance, enhanced sustainability and additional relevance benefiting the Higher Education sector, and the adoption and adaption by any single institution for their local contexts.

Kaleidoscope : illuminating learning and teaching evaluation for a shared stakeholder lens

Within the Australian higher education sector, institutions are required to evaluate teaching, units and courses to assure the quality of the student learning experience, however with hardly any regulatory parameters guiding institutions, and with disparate practices, there are few opportunities to benchmakr across institutions or the sector. QUT has received interest and requests from national and international universities on accessing Reframe: QUT's Evaluation Framework.

Research and/or learning and teaching : a study of Australian professors' priorities, beliefs and behaviours

This paper presents findings from an empirical study of key aspects of the teaching and research priorities, beliefs and behaviours of 72 professorial and associate professorial academics in Science, Information Technology and Engineering across four faculties in three Australian universities. The academics ranked 16 research activities and 16 matched learning and teaching (L&T) activities from three perspectives: job satisfaction, role model behaviour and perceptions of professional importance. The findings were unequivocally in favour of research in all three areas and remarkably consistent across the universities. The only L&T activity that was ranked consistently well was 'improving student satisfaction ratings for teaching', an area in which academics are increasingly held accountable. Respondents also indicated that their seniors encourage research efforts more than L&T efforts. Recommendations include that higher education rewards for quality L&T are maintained or improved and that recognition of L&T research domains is further strengthened.

Objectifying the Subjective : Assessment and Feedback in Creative Arts Studio Learning and Teaching

This paper examines the Assessment and Feedback aspects of Studio Teaching as Creative Arts pedagogy. Prompted by USQ’s newly offered Bachelor of Creative Arts (BCA), the author has developed an Assessment Matrix specifically designed to satisfy a number of imperatives, including: • ‘objectifying’ the subjective aspects of creative practice as assessable coursework/research • providing the means by which accurate, detailed, personalised and confidential feedback may be provided to students individually • providing consistent, accurate, meaningful assessment records for student, lecturer, and institution • ensuring consistency, continuity, and transparency of assessment processes and records to satisfy quality audits • minimising marking and assessment time, whilst maximising assessment integrity and depth • requiring only basic level skills and knowledge of a computer application already in common use (Microsoft Excel) • adaptability to a range of creative courses ‐ across disciplines This Assessment Matrix has been in development (and trialled) since January 2009.

A state-wide systems approach to embedding the learning and teaching of sustainability in teacher education

Education systems have a key role to play in preparing future citizens to engage in sustainable living practices and help create a more sustainable world. Many schools throughout Australia have begun to develop whole-school approaches to sustainability education that are supported by national and state policies and curriculum frameworks. Preservice teacher education, however, lags behind in building the capacity of new teachers to initiate and implement such approaches (ARIES, 2010). This proposed project seeks to develop a state-wide systems approach to embedding Education for Sustainability (EfS) in teacher education that is aligned with the Australian National Curriculum and the aspirations for EfS in the Melbourne Declaration and other national documents. Representatives from all teacher education institutions and other agents of change in the Queensland education system will be engaged in a multilevel systems approach, involving collaboration at the state, institutional and course levels, to develop curriculum practices that reflect a shared vision of EfS.

Learning and teaching epidemiology with audience response technology

Background The learning and teaching of epidemiology is core to many public health programs. Many students find the content of epidemiology, and specifically risk of bias assessment, challenging to learn. Howbeit, learning is enhanced when knowledge is able to be acquired from an active-learning, hands-on experience. Methods The innovative use of wireless audience response technology “clickers” was incorporated into the lectures of the university’s post-graduate epidemiology units and the tailored epidemiological modules delivered for professional disciplines (e.g. optometry). Clickers were used to apply several pedagogical approaches of active learning including peer-instruction and real-world simulation. Students were also assessed for their gain in knowledge within the lecture (pre-post) and their perceptions of how the use of clickers helped them learn. The routine university-wide end of semester Insight Survey provided further information of the student’s satisfaction with the approach. Results The technology was useful in identifying deficits of knowledge of key concepts either before or after instruction. Where key concepts were re-tested post-lecture, as expected, knowledge increased significantly and provided immediate feed-back to students. Across the lecture series, typically 85% of students identified the technology helped them learn, increased their opportunity to interact with the lecturer, and recommend their use for future classes. The Insight Survey report identified 93% of respondents identified the unit in which clickers were consistently used provided good learning opportunities. Numerous student comments supported the teaching method. Conclusions Epidemiological subject matter lends itself to incorporation of audience response technology. The use of the technology to facilitate interactive voting provides an instant response and participation of everyone to enhance the classroom experience. The pedagogical approach increases students’ knowledge and increases their satisfaction with the unit.

From policy borrowing to implementation : an illuminative evaluation of learning and teaching in higher education in Australia (2002 to 2008)

This study documents and theorises the consequences of the 2003 Australian Government Reform Package focussed on learning and teaching in Higher Education during the period 2002 to 2008. This is achieved through the perspective of program evaluation and the methodology of illuminative evaluation. The findings suggest that the three national initiatives of that time, Learning and Teaching Performance Fund (LTPF), Australian Learning and Teaching Council (ALTC), and Australian Universities Quality Agency (AUQA), were successful in repositioning learning and teaching as a core activity in universities. However, there were unintended consequences brought about by international policy borrowing, when the short-lived nature of LTPF suggests a legacy of quality compliance rather than one of quality enrichment.

Framing student evaluations of university learning and teaching : discursive strategies and textual outcomes

Evaluation in higher education is an evolving social practice, that is, it involves what people, institutions and broader systems do and say, how they do and say it, what they value, the effects of these practices and values, and how meanings are ascribed. The textual products (verbal, written, visual, gestural) that inform and are produced by, for and through evaluative practices are important as they promulgate particular kinds of meanings and values in specific contexts. This paper reports on an exploratory study that sought to investigate, using discourse analysis, the types of evaluative practices that were ascribed value, and the student responses that ensued, in different evaluative instruments. Findings indicate that when a reflective approach is taken to evaluation, students’ responses are more considered, they interrogate their own engagement in the learning context and they are more likely to demonstrate reconstructive thought. These findings have implications for reframing evaluation as reflective learning.

The WIL (Work Integrated Learning) Report : A national scoping study [Australian Learning and Teaching Council (ALTC) Final report]

This report provides an account of the first large-scale scoping study of work integrated learning (WIL) in contemporary Australian higher education. The explicit aim of the project was to identify issues and map a broad and growing picture of WIL across Australia and to identify ways of improving the student learning experience in relation to WIL. The project was undertaken in response to high levels of interest in WIL, which is seen by universities both as a valid pedagogy and as a means to respond to demands by employers for work-ready graduates, and demands by students for employable knowledge and skills. Over a period of eight months of rapid data collection, 35 universities and almost 600 participants contributed to the project. Participants consistently reported the positive benefits of WIL and provided evidence of commitment and innovative practice in relation to enhancing student learning experiences. Participants provided evidence of strong partnerships between stakeholders and highlighted the importance of these relationships in facilitating effective learning outcomes for students. They also identified a range of issues and challenges that face the sector in growing WIL opportunities; these issues and challenges will shape the quality of WIL experiences. While the majority of comments focused on issues involved in ensuring quality placements, it was recognised that placements are just one way to ensure the integration of work with learning. Also, the WIL experience is highly contextualised and impacted by the expectations of students, employers, the professions, the university and government policy.

Caught in the headlights: Designing for creative learning and teaching in higher education

This study was an exploration of the different ways that educators conceptualise and approach creative learning and teaching. The research revealed theoretical and practice-based insights, demonstrating that exemplary teachers adopt an ecological approach to designing for student creativity; this approach acknowledges and works with the complexity of the higher education environment and the dynamic relationships between students, peers and teachers. The inquiry confirmed the value of using learning design patterns to uncover hidden creative processes.

The paradox of creative uncertainty in learning and teaching

In an ever-changing and globalised world there is a need for higher education to adapt and evolve its models of learning and teaching. The old industrial model has lost traction, and new patterns of creative engagement are required. These new models potentially increase relevancy and better equip students for the future. Although creativity is recognised as an attribute that can contribute much to the development of these pedagogies, and creativity is valued by universities as a graduate capability, some educators understandably struggle to translate this vision into practice. This paper reports on selected survey findings from a mixed methods research project which aimed to shed light on how creativity can be designed for in higher education learning and teaching settings. A social constructivist epistemology underpinned the research and data was gathered using survey and case study methods. Descriptive statistical methods and informed grounded theory were employed for the analysis reported here. The findings confirm that creativity is valued for its contribution to the development of students’ academic work, employment opportunities and life in general; however, tensions arise between individual educator’s creative pedagogical goals and the provision of institutional support for implementation of those objectives. Designing for creativity becomes, paradoxically, a matter of navigating and limiting complexity and uncertainty, while simultaneously designing for those same states or qualities.

Grant application: QUT Learning and Teaching scheme 2012 Cracking the Code - A checklist to complement CRAs for first year Justice students.

Widening participation brings with it increasing diversity, increased variation in the level of academic preparedness (Clarke, 2011; Nelson, Clarke, & Kift 2010). Cultural capital coupled with negotiating the academic culture creates an environment based on many assumptions about academic writing and university culture. Variations in staff and student expectations relating to the teaching and learning experience is captured in a range of national and institutional data (AUSSE, CEQ, LEX). Nationally, AUSSE data (2009) indicates that communication, writing, speaking and analytic skills, staff expectations are quite a bit higher than students. The research team noted a recognisable shift in the changing cohort of students and their understanding and engagement with feedback and CRAs, as well as variations in teaching staff and student expectations. The current reality of tutor and student roles is that: - Students self select when/how they access lectures and tutorials. - Shorter tutorial times result in reduced opportunity to develop rapport with students. - CRAs are not always used consistently by staff (different marking styles and levels of feedback). - Marking is not always undertaken by the student’s tutor/lecturer. - Student support services might be recommended to students once a poor grade has been given. Students can perceive this as remedial and a further sense of failure. - CRA sheet has a mark /grade attached to it. Stigma attached to low mark. Hard to focus on the CRA feedback with a poor mark etched next to it. - Limited opportunities for sessionals to access professional development to assist with engaging students and feedback. - FYE resources exist, however academic time is a factor in exploring and embedding these resources. Feedback is another area with differing expectations and understandings. Sadler (2009) contends that students are not equipped to decode the statements properly. For students to be able to apply feedback, they need to understand the meaning of the feedback statement. They also need to identify, the particular aspects of their work that need attention. The proposed Checklist/guide would be one page and submitted with each assessment piece thereby providing an interface to engage students and tutors in managing first year understandings and expectations around CRAs, feedback, and academic practice.

How Jung's concept of the wounded healer can guide learning and teaching in social work and human services

The concept of the “wounded healer” has been used to explain why those with adverse childhood histories often enter helping professions such as social work and human services (SWHS). Psychotherapist Carl Jung (1875–1961) believed wounded healers developed insight and resilience from their own experiences, enabling transformative interventions to occur with clients. Concerns exist that students with adverse childhood histories in SWHS may display unresolved emotional issues. This journal article explores how Jung’s interpretation of the wounded healer can be critically applied to understanding the learning needs of SWHS students with histories of abuse, neglect or other childhood adversity. The relevance of the wounded healer to SWHS education is explored in three key areas: - 1) the increased possibility of the occurrence of countertransference; - 2) the potential for vicarious traumatisation and burnout, and; - 3) personal and professional resilience displayed by SWHS students with a history of childhood adversity. The wounded healer metaphor allows for a more nuanced understanding of SWHS students with these histories. It also provides insight into the pedagogical considerations associated with teaching this student cohort.