Future directions in course quality assurance

The Course Quality Assurance System at Queensland University of Technology (QUT) has as its centrepiece an exemplar of data visualisation known as the Individual Course Report (ICR). This report provides every course coordinator with an annual snapshot of their performance data evaluated against QUT and national benchmarks. In this article, the impact of the ICR is explored through the case study of one undergraduate course identified as underperforming. The case study features an innovative, ethnographic approach to working with course teams and highlights the importance of context, collaboration and appropriate support in creating evidence-based action plans for course improvement.

“Filling out the forms was a nightmare” : project evaluation and the reflective practitioner in community theatre in contemporary Northern Ireland

Since the Good Friday Agreement of 1998, large sums have been invested in community theatre projects in Northern Ireland, in the interests of conflict transformation and peace building. While this injection of funds has resulted in an unprecedented level of applied theatre activity, opportunities to maximise learning from this activity are being missed. It is generally assumed that project evaluation is undertaken at least partly to assess the degree of success of projects against important social objectives, with a view to learning what works, what does not, and what might work in the future. However, three ethnographic case studies of organisations delivering applied theatre projects in Northern Ireland indicate that current processes used to evaluate such projects are both flawed and inadequate for this purpose. Practitioners report that the administrative work involved in applying for and justifying funding is onerous, burdensome, and occurs at the expense of artistic activity. This is a very real concern when the time and effort devoted to ‘filling out the forms’ does not ultimately result in useful evaluative information. There are strong disincentives for organisations to report honestly on their experiences of difficulties, or undesirable impacts of projects, and this problem is not transcended by the use of external evaluators. Current evaluation processes provide little opportunity to capture unexpected benefits of projects, and small but significant successes which occur in the context of over-ambitious objectives. Little or no attempt is made to assess long-term impacts of projects on communities. Finally, official evaluation mechanisms fail to capture the reflective practice and dialogic analysis of practitioners, which would richly inform future projects. The authors argue that there is a need for clearer lines of communication, and more opportunities for mutual learning, among stakeholders involved in community development. In particular, greater involvement of the higher education sector in partnership with government and non-government agencies could yield significant benefits in terms of optimizing learning from applied theatre project evaluations.

Critical reflections on the use of participatory methodologies to build evaluation capacities in international development organisations

There has been a greater focus on strengthening evaluation capacity building (ECB) within development organisations in recent years. This can be attributed in part to the growing appreciation of the value of participatory and collaborative forms of evaluation. Evaluation is increasingly seen as an ongoing learning process and an important means of strengthening capacity and improving organisational performance (Horton et al., 2003:7). While there are many benefits of using participatory methodologies in ECB projects, our experiences and a review of the literature in this area highlight the many challenges, issues and contradictions that can affect the success of such ECB efforts. We discuss these issues, drawing on our learnings from the ongoing participatory action research (PAR) project 'Assessing Communication for Social Change’ (AC4SC). This four year project, which began in 2007, is a collaboration between communication and development academics and evaluation specialists from two Australian universities and communication for development practitioners and monitoring and evaluation (M&E) staff in the NGO Equal Access Nepal (EAN). The aim is to develop, implement, and evaluate a participatory methodology for assessing the social change impacts of community radio programs produced by EAN. It builds on previous projects that used ethnographic action research (EAR) methodology (Tacchi et al., 2007).