Evaluation of the quality of diverse teaching practice in higher education research methods

The purpose of this paper is to report on a methods research project investigating the evaluation of diverse teaching practice in Higher Education. The research method is a single site case study of an Australian university with data collected through published documents, surveys, interviews and focus groups. This project provides evidence of the wide variety of evaluation practice and diverse teaching practice across the university. This breadth identifies the need for greater flexibility of evaluation processes, tools and support to assist teaching staff to evaluate their diverse teaching practice. The employment opportunities for academics benchmark the university nationally and position the case study in the field. Finally this reaffirms the institutional responsibility for services to support teaching staff in an ongoing manner.

Looking Yonder : Longitudinal Research and Evaluation of the Yonder Participative Arts-based Learning Model Report 2010-2012

Since 2007, KITE Arts Education Program @ QPAC has been engaged in a series of arts and drama-based experiences for students in selected primary schools on the edges of Brisbane and in regional Queensland. The in-school workshop experiences of the program have culminated in a performance by the children for their school community, parents and carers at the Queensland Performing Arts Centre or a regional cultural venue. In conducting an analysis of the Yonder project, the researcher aimed to provide evidence of outcomes brought about through participation by schools, school staff, students and their communities in the Yonder project. To develop longitudinal data project initiators, participants were interviewed at six-monthly intervals to establish patterns of engagement and participation. The report analyses arts-based workshops conducted by the teacher artist in edge-city Brisbane and a regional centre; interviews with teachers and school administrators from the participating schools; interviews with teacher artist and professional artists; interviews with community partners; teacher professional development workshops; community-based workshops; performance outcomes that were the culminating events of the workshop program; student work samples and student reflections on the program. This document covers data and project outputs from February 2010 to July 2012. There have been five iterations of the Yonder project since its commencement in mid-2009 — three in regional Queensland (February–April 2010; February–May 2011; February–May 2012) and two in edge-city1 Brisbane (July–September 2010; August–October 2011). This report is a result of a research partnership between Queensland Performing Arts Centre and Queensland University of Technology (QUT) Creative Industries Faculty(Drama).

Community development with survivors of torture and trauma : an evaluation framework

Despite an increase in community development initiatives in refugee contexts, there is a lack of evaluation frameworks to assess the effectiveness of interventions in the recovery of refugee communities. In response to this gap, the Forum of Australian Services for Survivors of Torture and Trauma has developed an evaluation framework in consultation with refugee client groups and agencies' staff members. This paper contextualizes the goals, principles and strategies of services implementing community development initiatives with torture and trauma survivors and describes the process of developing the framework within a participatory action approach. Both outcome evaluation and process evaluation are discussed, and examples of the framework are presented. Community development agencies and professionals working with survivors of torture and trauma can play a significant role by fostering community empowerment through evaluation.

Skill development and social inclusion via peer assisted learning in first-year higher education : an evaluation of student experiences across two law units

Because professions seek graduates who can 'collaborate, share skills and knowledge, and communicate' (Kruck and Reif, 2001, p 37), it is important that university graduates are not equipped solely with the content knowledge of their discipline, but also with prospective employment skills. Furthermore, when students 'interact more in positive ways with their teachers and peers, they gain more in terms of essential skills and competencies, such as critical thinking, problem~solving [and] effective communication' (NSSE, 2000, p 2)./n this way, peer assisted fellowing has the potential to enhance students' professional development, and provide the social inclusion and engagement necessary for effective learning. This session describes two peer assisted learning models embedded within first year QUT Faculty of Law units. Through a partnership between teaching staff, student mentors and mentees, the models aim to facilitate student socialisation whilst supplementing understanding of substantive law with the development of academic and work·related skills. Mentor and mentee perceptions, and program implications, are considered.

An evaluation of the Australian Capital Territory Sexual Assault Reform Program (SARP): Final Report

In 2005 the Australian Capital Territory (ACT) Office of the Director of Public Prosecutions (DPP) and the Australian Federal Police (AFP) produced a report, Responding to sexual assault: The challenge of change (DPP & AFP 2005), which made 105 recommendations for reforming the way sexual offence cases are handled by the ACT’s criminal justice system. The Sexual Assault Reform Program (SARP) is one key initiative developed in response to these recommendations. Managed by the ACT Justice and Community Safety Directorate (JACS), SARP’s main objective is to improve aspects of the criminal justice system relating to: processes and support for victims of sexual offences as they progress through the system; attrition in sexual offence matters in the criminal justice system; and coordination and collaboration among the agencies involved. In November 2007 the ACT Attorney-General announced $4 million of funding for several SARP reforms. This funding provided for additional victim support staff; a dedicated additional police officer, prosecutor and legal policy officer; and an upgrade of equipment for the Supreme Court and Magistrates Court, including improvements in technology to assist witnesses in giving evidence, and the establishment of an off-site facility to allow witnesses to give evidence from a location outside of the court. In addition, the reform agenda included a number of legislative amendments that changed how evidence can be given by victims of sexual and family violence offences, children and other vulnerable witnesses. The primary objectives of these legislative changes are to provide an unintimidating, safe environment for vulnerable witnesses (including sexual offence complainants) to give evidence and to obtain prompt statements from witnesses to improve the quality of evidence captured (DPP 2009: 13). The current evaluation The funding for SARP reforms also provided for a preliminary evaluation of the reforms; this report outlines findings from the evaluation. The evaluation sought to address whether the program has met its key objectives: better support for victims, lower attrition rates and improved coordination and collaboration among agencies involved in administering SARP. The evaluation was conducted in two stages and involved a mixed-methods approach. During stage 1 key indicators for the evaluation were developed with stakeholders. During stage 2 quantitative data were collected by stakeholders and provided to the AIC for analysis. Qualitative interviews were also conducted with service delivery providers, and with a small number (n=5) of victim/survivors of sexual offences whose cases had recently been resolved in the ACT criminal justice system. The current evaluation is preliminary in nature. As the SARP reforms will take time to become entrenched within the ACT’s criminal justice system, some of the impacts of the reforms may not yet be evident. Nonetheless, this evaluation provides an insight into how well the SARP reforms have been implemented to date, as well as key areas that could be addressed in the future. Key findings from the preliminary evaluation are outlined briefly below.

Quality Assessment Matrix : A Collaborative Evaluation Using Multiple Lines of Evidence

Welcome to the Quality assessment matrix. This matrix is designed for highly qualified discipline experts to evaluate their course, major or unit in a systematic manner. The primary purpose of the Quality assessment matrix is to provide a tool that a group of academic staff at universities can collaboratively review the assessment within a course, major or unit annually. The annual review will result in you being read for an external curricula review at any point in time. This tool is designed for use in a workshop format with one, two or more academic staff, and will lead to an action plan for implementation.

Exploring empathy as a variable in the evaluation of professional development programmes for palliative care nurses

Research indicates that empathy, a quality regarded as fundamentally important to nursing practice, is a teachable skill. Because empathic nurse-patient relationships are particularly important in the care of the terminally ill, this has direct relevance to the professional development of palliative care nurses. This article discusses the place of empathy as a criterion variable in the evaluation of a professional development program for palliative care nurses introduced at the Centre for Mental Health Nursing Research at Queensland University of Technology, Brisbane, Australia. A modified version of the Staff-Patient Interaction Response Scale (SPIRS) was used as a pre- and postintervention measure to assess the expressed empathy of the participating nurses. The modifications to SPIR and its coding system to make it suitable for palliative care nursing, and the mechanisms for improving and evaluating the reliability of this instrument will be discussed. The full description of this particular modification of SPIRS for palliative care research is provided as an example of how this instrument could be used in projects for which nurses undertake the difficult task of providing compassionate care to the terminally ill.

Teacher Evidence Matrix : an evaluation using multiple lines of evidence

Welcome to the Teacher evidence matrix. This matrix is designed for highly qualified discipline experts to evaluate their teaching in a systematic manner. The primary purpose of the Teacher evidence matrix is to provide a tool that an academic staff member at university can annually review their teaching. The annual review will result in you being ready for performance, planning and review; promotion; awards; or employment application. This tool is designed for individual use and will lead to an action plan for implementation.

Systematic evaluation of GoSoapBox in tertiary education : a student response system for improving learning experiences and outcomes

There is currently a wide range of research into the recent introduction of student response systems in higher education and tertiary settings (Banks 2006; Kay and Le Sange, 2009; Beatty and Gerace 2009; Lantz 2010; Sprague and Dahl 2009). However, most of this pedagogical literature has generated ‘how to’ approaches regarding the use of ‘clickers’, keypads, and similar response technologies. There are currently no systematic reviews on the effectiveness of ‘GoSoapBox’ – a more recent, and increasingly popular student response system – for its capacity to enhance critical thinking, and achieve sustained learning outcomes. With rapid developments in teaching and learning technologies across all undergraduate disciplines, there is a need to obtain comprehensive, evidence-based advice on these types of technologies, their uses, and overall efficacy. This paper addresses this current gap in knowledge. Our teaching team, in an undergraduate Sociology and Public Health unit at the Queensland University of Technology (QUT), introduced GoSoapBox as a mechanism for discussing controversial topics, such as sexuality, gender, economics, religion, and politics during lectures, and to take opinion polls on social and cultural issues affecting human health. We also used this new teaching technology to allow students to interact with each other during class – both on both social and academic topics – and to generate discussions and debates during lectures. The paper reports on a data-driven study into how this interactive online tool worked to improve engagement and the quality of academic work produced by students. This paper will firstly, cover the recent literature reviewing student response systems in tertiary settings. Secondly, it will outline the theoretical framework used to generate this pedagogical research. In keeping with the social and collaborative features of Web 2.0 technologies, Bandura’s Social Learning Theory (SLT) will be applied here to investigate the effectiveness of GoSoapBox as an online tool for improving learning experiences and the quality of academic output by students. Bandura has emphasised the Internet as a tool for ‘self-controlled learning’ (Bandura 2001), as it provides the education sector with an opportunity to reconceptualise the relationship between learning and thinking (Glassman & Kang 2011). Thirdly, we describe the methods used to implement the use of GoSoapBox in our lectures and tutorials, and which aspects of the technology we drew on for learning purposes, as well as the methods for obtaining feedback from the students about the effectiveness or otherwise of this tool. Fourthly, we report cover findings from an examination of all student/staff activity on GoSoapBox as well as reports from students about the benefits and limitations of it as a learning aid. We then display a theoretical model that is produced via an iterative analytical process between SLT and our data analysis for use by academics and teachers across the undergraduate curriculum. The model has implications for all teachers considering the use of student response systems to improve the learning experiences of their students. Finally, we consider some of the negative aspects of GoSoapBox as a learning aid.

Critical reflections on the use of participatory methodologies to build evaluation capacities in international development organisations

There has been a greater focus on strengthening evaluation capacity building (ECB) within development organisations in recent years. This can be attributed in part to the growing appreciation of the value of participatory and collaborative forms of evaluation. Evaluation is increasingly seen as an ongoing learning process and an important means of strengthening capacity and improving organisational performance (Horton et al., 2003:7). While there are many benefits of using participatory methodologies in ECB projects, our experiences and a review of the literature in this area highlight the many challenges, issues and contradictions that can affect the success of such ECB efforts. We discuss these issues, drawing on our learnings from the ongoing participatory action research (PAR) project 'Assessing Communication for Social Change’ (AC4SC). This four year project, which began in 2007, is a collaboration between communication and development academics and evaluation specialists from two Australian universities and communication for development practitioners and monitoring and evaluation (M&E) staff in the NGO Equal Access Nepal (EAN). The aim is to develop, implement, and evaluate a participatory methodology for assessing the social change impacts of community radio programs produced by EAN. It builds on previous projects that used ethnographic action research (EAR) methodology (Tacchi et al., 2007).

Critical reflections on the use of participatory methodologies to build evaluation capacities in international development organisations

"In this chapter the authors present a critique of Participatory Evaluation as worked in development projects, in this case, in Nepal. The article works between established claims that Participatory Evaluation builds capacity at programmatic and organisational levels, and the specific experiences of these claims in the authors’ current work. They highlight the need to address key difficulties such as high turn-over of staff and resulting loss of capacity to engage in Participatory Evaluation, and the difficulty of communication between academic as compared with local practical wisdoms. A key issue is the challenge of addressing the inevitable issues of power inequities that such approaches encounter. While Participatory Evaluation has been around for some time, it has only enjoyed more widespread recognition of its value in comparatively recent times, with its uptake in international development environments. To this extent, the practice is still in its early stages of development, and Jo, June and Michael’s work contributes to strengthening and more comprehensively understanding it. With regard to the meta-theme of this publication, this chapter is an example of how context not only influences the methodology to be used and the praxis of how it is to be used, but contributes to early explication of the core nature of an emerging methodology."

Protocols : Reframe : QUT's evaluation framework

Reframe is changing our approach to the evaluation of courses, units, teaching and student experience at QUT. We are moving away from a single survey tool to a richer, more holistic and customisable approach. These protocols allows academic staff and administrators access to the ways in which the policy is enacted through process.

Evaluation of the Australian Performing Arts Market 2014-2018 - Year One Report

The Australia Council awarded the tender of APAMs 2014, 2016 and 2018 to the Brisbane Powerhouse. The Australia Council, in awarding the contract for the presentation of APAM by Brisbane Powerhouse, stipulated that a formal evaluation of the three iterations of APAM and activity in the intervening years be undertaken. Queensland University of Technology, Creative Industries Faculty, under the leadership of Associate Professor Sandra Gattenhof, were contracted to undertake the formal evaluation. This is the first year report on the Brisbane iteration of the Market. This report has drawn from data collected across a range of sources, drawing on the scoping study undertaken by Justin Macdonnell addressing the Market from 1994–2010; the tender document submitted by the Brisbane Powerhouse; in-person interviews with APAM staff, APAM Stakeholders, Vox Pops from delegates in response to individual sessions, producer company/artist case studies and, most significantly, responses from a detailed online survey sent to all delegates. The main body of the report is organised around three key research aims, as outlined in the Brisbane Powerhouse Tender document (2011). These have been articulated as: Evaluation of international market development outcomes through showcasing work to targeted international presenters and agents Evaluation of national market development outcomes through showcasing work to national presenters and producers Evaluation of the exchange ideas, dialogue, skill 
development, partnerships, collaborations and co- productions and networks with local and international peers. The culmination of the data analysis has been articulated through five key recommendations, which may assist the APAM delivery team for the next version, in 2016. In summary, the recommendations are described as: 1. Indigenous focus to remain central to the conception and delivery of APAM 2. Re-framing APAM’s function and its delivery 3. Logistics and communications in a multi-venue approach, including communications and housekeeping, volunteers, catering, re-calibrating the employment of Brisbane Powerhouse protocols and processes for APAM 4. Presentation and promotion for presenters 5. Strategic targeting of Asian producers.

Designing for collaborative learning spaces : an academic staff development pilot program

To prepare for the delivery of new Bachelor of Science units in collaborative learning spaces, academic and professional staff at Queensland University of Technology piloted an academic development program over the period of a semester. The program was informed by Rogers’ theory of innovation and diffusion (2003) and structured according to Wilson’s framework for faculty development (2007). Through a series of workshops and group mentoring activities, the program modelled inquiry-based learning in a collaborative learning space, and the participants designed and practiced the delivery of teaching activities. This paper provides a preliminary evaluation of the effectiveness of the pilot based on survey responses from participants, notes from the development team who coordinated the program and audience feedback from the final showcase session. The design and structure of the program is discussed as well as possible future directions.