Navigating theory, negotiating difference and beating the system : a critical how-to of media assistance evaluation

The wave of democratisation across Europe, Africa, Asia and Latin America in the early 1990s triggered an increase in donor funding to media assistance initiatives, primarily within good governance policy frameworks. However, few media assistance projects have managed to effectively evaluate the impacts of their work. This thesis explores how the impacts of Australian media assistance on social change and governance can be most effectively evaluated and understood. The findings of this research suggest the importance of early investment in participatory planning of evaluation designs, which are then periodically revisited. These evaluation designs should be based on a theoretically sound link between models of change, evaluative questions and methods.

Building indicators in media assistance; four ways evaluation reports present “change”

The international aid and development community has supported programs that aim to build the capacity of media professionals or contribute to an enabling environment throughout the past 20 years. However, two decades on from the first modern media assistance programs, the sector is still struggling to identify, measure and understand the changes effected by their programs. There are questions raised as to whether it is even feasible to identify impacts on society and governance. This paper draws on some preliminary findings from a comparative thematic analysis of 47 evaluation documents of media assistance programs. The aim of this analysis is to identify trends in impact evaluation practice in the media assistance field, as well as the strengths and weaknesses of different evaluation approaches. This paper presents four types of social change claims commonly presented in reports; hypothetical changes, introduction of new opportunities, concrete examples of immediate impacts, and analysis of ongoing social and political changes. Although these types may appear as a spectrum from weak to strong, the interactions are perhaps more accurately understood using metaphors such as building blocks. This paper explores these types in more detail and suggests that a robust set of impacts-types could be useful in developing more grounded theories of change and indicators.

From Civil Strife to Civil Society: civil and military responsibilities in disrupted states

"The 1990s saw the United Nations, the militaries of key member states, and NGOs increasingly entangled in the complex affairs of disrupted states. Whether as deliverers of humanitarian assistance or as agents of political, social, and civic reconstruction, whether in Somalia, Bosnia, Kosovo, or East Timor, these actors have had to learn ways of interacting with each other in order to optimize the benefits for the populations they seek to assist. Yet the challenges have proved daunting. Civil and military actors have different organizational cultures and standard operating procedures and are confronted with the need to work together to perform tasks to which different actors may attach quite different priorities."--BOOK JACKET.

Introduction

The 1990s saw the United Nations, the militaries of key member states, and NGOs increasingly entangled in the complex affairs of disrupted states. Whether as deliverers of humanitarian assistance or as agents of political, social, and civic reconstruction, whether in Somalia, Bosnia, Kosovo, or East Timor, these actors have had to learn ways of interacting with each other in order to optimize the benefits for the populations they seek to assist. Yet the challenges have proved daunting. Civil and military actors have different organizational cultures and standard operating procedures and are confronted with the need to work together to perform tasks to which different actors may attach quite different priorities. From Civil Strife to Civil Society explores the nature of these challenges, blending the experience of scholars and practitioners. It is underpinned by an understanding that recovery from disruption is a laborious process that can easily be de-railed. The first part of the book offers a rigorous examination of the dimensions of state disruption and the roles of the international community in responding to it; the second part looks at military doctrine for dealing with disorder and humanitarian emergencies; the third part examines mechanisms for ending violence and delivering justice in post-conflict times; the fourth part investigates the problems of rebuilding trust and promoting democracy; the fifth part deals with the reconstitution of the rule of law; while the sixth and seventh parts address the reestablishment of social and civil order.

An evaluation framework for educational reform projects for teacher quality improvements in developing countries : a case study of Egyptian education reform

The role of the evaluation for Official Development Assistance (ODA) enterprises including educational development has become critical after increasing “aid fatigue” experienced by the international community in the 1990s. To date, however, monitoring and evaluating outcomes of the projects has been limited to the project life. Consequently these have been mainly through the international aid agencies. Furthermore, the monitoring and evaluation led by international aid agencies have paid little attention to aspects of the sustainability of technical cooperation in educational development. To sustain the impact of technical cooperation, the reinforcement of evaluation has drawn increasing attention in light of the emerging modalities in international development. Therefore this research was inspired to investigate alternative evaluation frameworks for an educational reform project for teacher quality improvement that may increase possibilities for long term sustainability. Importantly, the new modalities in international development and educational issues provide new options. In addition, the research reviewed theoretical and practical issues surrounding evaluation in general, and highlighted the evaluation of education reform projects. The research reported explored via case studies, the evaluation processes employed by the Egyptian education reform projects implemented by the Japan International Cooperation Agency (JICA) and the United Nations Children’s Fund (UNICEF). The case studies used three data sources (archival and relevant documents, a survey questionnaire and interviews) to illuminate the contextually-embedded evaluation processes. The research found that process evaluation is a potential alternative method since it is likely to be locally institutionalised, which may yield long-term sustainability of the projects.

Comprehensive evaluation of driver gaze pattern using fuzzy rules for driver training

The over representation of novice drivers in crashes is alarming. Research indicates that one in five drivers’ crashes within their first year of driving. Driver training is one of the interventions aimed at decreasing the number of crashes that involve young drivers. Currently, there is a need to develop comprehensive driver evaluation system that benefits from the advances in Driver Assistance Systems. Since driving is dependent on fuzzy inputs from the driver (i.e. approximate distance calculation from the other vehicles, approximate assumption of the other vehicle speed), it is necessary that the evaluation system is based on criteria and rules that handles uncertain and fuzzy characteristics of the drive. This paper presents a system that evaluates the data stream acquired from multiple in-vehicle sensors (acquired from Driver Vehicle Environment-DVE) using fuzzy rules and classifies the driving manoeuvres (i.e. overtake, lane change and turn) as low risk or high risk. The fuzzy rules use parameters such as following distance, frequency of mirror checks, gaze depth and scan area, distance with respect to lanes and excessive acceleration or braking during the manoeuvre to assess risk. The fuzzy rules to estimate risk are designed after analysing the selected driving manoeuvres performed by driver trainers. This paper focuses mainly on the difference in gaze pattern for experienced and novice drivers during the selected manoeuvres. Using this system, trainers of novice drivers would be able to empirically evaluate and give feedback to the novice drivers regarding their driving behaviour.

An assessment system for evaluation of driving competencies

Automobiles have deeply impacted the way in which we travel but they have also contributed to many deaths and injury due to crashes. A number of reasons for these crashes have been pointed out by researchers. Inexperience has been identified as a contributing factor to road crashes. Driver’s driving abilities also play a vital role in judging the road environment and reacting in-time to avoid any possible collision. Therefore driver’s perceptual and motor skills remain the key factors impacting on road safety. Our failure to understand what is really important for learners, in terms of competent driving, is one of the many challenges for building better training programs. Driver training is one of the interventions aimed at decreasing the number of crashes that involve young drivers. Currently, there is a need to develop comprehensive driver evaluation system that benefits from the advances in Driver Assistance Systems. A multidisciplinary approach is necessary to explain how driving abilities evolves with on-road driving experience. To our knowledge, driver assistance systems have never been comprehensively used in a driver training context to assess the safety aspect of driving. The aim and novelty of this thesis is to develop and evaluate an Intelligent Driver Training System (IDTS) as an automated assessment tool that will help drivers and their trainers to comprehensively view complex driving manoeuvres and potentially provide effective feedback by post processing the data recorded during driving. This system is designed to help driver trainers to accurately evaluate driver performance and has the potential to provide valuable feedback to the drivers. Since driving is dependent on fuzzy inputs from the driver (i.e. approximate distance calculation from the other vehicles, approximate assumption of the other vehicle speed), it is necessary that the evaluation system is based on criteria and rules that handles uncertain and fuzzy characteristics of the driving tasks. Therefore, the proposed IDTS utilizes fuzzy set theory for the assessment of driver performance. The proposed research program focuses on integrating the multi-sensory information acquired from the vehicle, driver and environment to assess driving competencies. After information acquisition, the current research focuses on automated segmentation of the selected manoeuvres from the driving scenario. This leads to the creation of a model that determines a “competency” criterion through the driving performance protocol used by driver trainers (i.e. expert knowledge) to assess drivers. This is achieved by comprehensively evaluating and assessing the data stream acquired from multiple in-vehicle sensors using fuzzy rules and classifying the driving manoeuvres (i.e. overtake, lane change, T-crossing and turn) between low and high competency. The fuzzy rules use parameters such as following distance, gaze depth and scan area, distance with respect to lanes and excessive acceleration or braking during the manoeuvres to assess competency. These rules that identify driving competency were initially designed with the help of expert’s knowledge (i.e. driver trainers). In-order to fine tune these rules and the parameters that define these rules, a driving experiment was conducted to identify the empirical differences between novice and experienced drivers. The results from the driving experiment indicated that significant differences existed between novice and experienced driver, in terms of their gaze pattern and duration, speed, stop time at the T-crossing, lane keeping and the time spent in lanes while performing the selected manoeuvres. These differences were used to refine the fuzzy membership functions and rules that govern the assessments of the driving tasks. Next, this research focused on providing an integrated visual assessment interface to both driver trainers and their trainees. By providing a rich set of interactive graphical interfaces, displaying information about the driving tasks, Intelligent Driver Training System (IDTS) visualisation module has the potential to give empirical feedback to its users. Lastly, the validation of the IDTS system’s assessment was conducted by comparing IDTS objective assessments, for the driving experiment, with the subjective assessments of the driver trainers for particular manoeuvres. Results show that not only IDTS was able to match the subjective assessments made by driver trainers during the driving experiment but also identified some additional driving manoeuvres performed in low competency that were not identified by the driver trainers due to increased mental workload of trainers when assessing multiple variables that constitute driving. The validation of IDTS emphasized the need for an automated assessment tool that can segment the manoeuvres from the driving scenario, further investigate the variables within that manoeuvre to determine the manoeuvre’s competency and provide integrated visualisation regarding the manoeuvre to its users (i.e. trainers and trainees). Through analysis and validation it was shown that IDTS is a useful assistance tool for driver trainers to empirically assess and potentially provide feedback regarding the manoeuvres undertaken by the drivers.

Framing local stories for community consumption facilitation and evaluation in narrative driven arts-based projects

Cyclone Yasi struck the Cassowary Coast of Queensland in the early hours of Feb 3, 2011, destroying many homes sand property, including the destruction of the Cardwell and district historical society’s premises. With their own homes flattened, many were forced to live in mobile accommodation, with extended family, or leave altogether. The historical society members however were more devastated by their flattened foreshore museum and loss of their collection material. A call for assistance was made through the OHAA Qld branch, who along with QUT sponsored a trip to somehow plan how they could start to pick up the pieces to start again. This presentation highlights the need for communities to gather, preserve and present their own stories, in a way that is sustainable and meaningful to them, but that good advice and support along the way is important. Two 2 day workshops were held in March and then September, augmented by plenty of email correspondence and phone calls in between. Participants learnt that if they could conduct quality oral history interviews, they could later use these in many exhibitable ways including: documentary pieces; digital stories; photographic collections; creative short stories; audio segments –while also drawing closely together a suffering community. This story is not only about the people who were interviewed about the night Yasi struck, but the amazing women (all over 50) of the historical society who were willing to try and leap the digital divide that faces older Australians, especially those in rural Australia, so that their older local stories would not be lost and so that new stories could also be remembered.

Facilitating local stories in post-disaster regional communities : evaluation in narrative-driven oral history projects

Cyclone Yasi struck the Cassowary Coast of Northern Queensland, Australia, in the early hours of February 3, 2011, destroying many homes and property, including the destruction of the Cardwell and district historical society’s premises. With their own homes flattened, many residents were forced to live in mobile accommodation, with extended family, or leave the area altogether. The historical society members seemed, however, particularly devastated by their flattened foreshore museum and loss of their precious collection of material. A call for assistance was made through the Oral History Association of Australia’s Queensland branch (OHAA Qld), which along with a Queensland University of Technology (QUT) research team sponsored a trip to best plan how they could start to pick up the pieces to rebuild the museum. This chapter highlights the need for communities to gather, preserve and present their own stories, in a way that is sustainable and meaningful to them – whether that be because of a disaster, or as they go about life in their contemporary communities – the key being that good advice, professional support and embedded evaluation practices at crucial moments along the way can be critically important.

Evaluating NEXUS: A suicide prevention program for young people from refugee backgrounds (2007 – 2009)

Although suicide deaths in Australia continue to decline since the peak of 2,720 suicide deaths in 1997, youth suicide and self-harm are a major health issue. In 2006, in the 20 to 24 year age group, suicide accounted for approximately 21% of all male deaths and 14% of all female deaths. There is a lack of solid data on the rates of suicide and self-harm among young people from refugee backgrounds. However, this population faces a significant number of post-resettlement stressors that may add to their vulnerability and increase their risk of suicide and self-harm. The NEXUS program is an innovative strategy developed by the Queensland Program of Assistance to Survivors of Torture and Trauma (QPASTT) that aims to reduce risk factors for suicide and self-harm and to promote protective factors among youth from refugee backgrounds living in Brisbane and Toowoomba. QPASTT is a non-government organisation that provides culturally appropriate support services to refugee and humanitarian entrants to Australia. QPASTT’s primary function is to provide counselling, advocacy support and community development activities for survivors of torture and trauma at an individual, family and community level. Since 2002 the NEXUS program has been developed and implemented by QPASTT. Since then, this multi-component program has been funded by the Commonwealth Department of Health and Ageing through the National Suicide Prevention Strategy (NSPS). NSPS funding of local community suicide prevention activities will contribute to the outcomes specified in the strategic framework: Living is for Everyone (LIFE): A framework for prevention of suicide and self-harm in Australia. The focus of this report is the evaluation of the NEXUS program conducted by QPASTT between August 2007 and May 2009.

Spotlight on animating spaces: Evaluation of Artslink Queensland's Animating Spaces 2013 & 2014

Animating Spaces 2013-14 Interim Report Artslink Queensland’s Animating Spaces initiative is a three-year statewide project (2013-2015) that aims to revitalise and celebrate non-traditional, significant, and unusual spaces within fifteen regional Queensland communities. After the completion of two years of Animating Spaces this is the first public interim report highlighting outcomes to date. A final evaluation report will be collated in early 2016 that will capture the Animating Spaces project in its entirety. Developed by Queensland University of Technology evaluation team, Professor Helen Klaebe and Dr. Elizabeth Ellison with the extraordinary assistance of Artslink Queensland staff, in particular Kerryanne Farrer.