Meta-evaluation to improve learning, evaluation capacity development and sustainability : Findings from a participatory evaluation project in Nepal

Recent studies suggest that meta-evaluation can be valuable in developing new approaches to evaluation, building evaluation capacities, and enhancing organizational learning. These new extensions of the concept of meta-evaluation are significant, given the growing emphasis on improving the quality and effectiveness of evaluation practices in the South Asian region. Following a review of the literature, this paper presents a case study of the use of concurrent meta-evaluation in the four-year project Assessing Communication for Social Change which developed and trialled a participatory impact assessment methodology in collaboration with a development communication Non-government organization (NGO) in Nepal. Key objectives of the meta-evaluation included to: continuously develop, adapt and improve the impact assessment methodology, Monitoring and Evaluation (M&E) systems and process and other project activities; identify impacts of the project; and build capacities in critical reflection and review. Our analysis indicates that this meta-evaluation was essential to understanding various constraints related to the organizational context that affected the success of the project and the development of improved M&E systems and capacities within the NGO. We identified several limitations of our meta-evaluation methods, which were balanced by the strengths of other methods. Our case study suggests that as well as assessing the quality, credibility and value of evaluation practices, meta-evaluations need to focus on important contextual issues that can have significant impacts on the outcomes of participatory evaluation projects. They include hierarchical organizational cultures, communication barriers, power/knowledge relations, and the time and resources available. Meta-evaluations also need to consider wider issues such as the sustainability of evaluation systems and approaches.

From 'what' to 'how' : capacity building in health promotion for HIV/AIDS prevention in the Solomon Islands

This paper describes a capacity building process undertaken within the HIV/AIDS prevention project of the Adventist Development and Relief Agency (ADRA) in the Solomon Islands. ADRA HIV/AIDS has recently reoriented its project structure, moving beyond its awareness raising approach to incorporate health promotion frameworks, theories, strategies and assumptions. These have been used to inform project practice in project planning, delivery and evaluation. This paper shares what has worked and not worked in the capacity building process, including a project evaluation of the initial HIV/AIDS awareness raising project and the application of a number of capacity building strategies, including utilising a volunteer Australian Youth Ambassador for Development (AYAD) funded by the Australian Agency for International Development (AusAID). Existing and new projects are outlined. The underlying theme is that any capacity building exercise must include structural support (e.g. management, national frameworks) to ensure the incorporation of new initiatives and approaches. With time this enables ownership by counterparts and external partnerships to develop. The presence of an AYAD volunteer has been an effective strategy to achieve this. Reflections from the evaluators, the AYAD volunteer and the HIV/AIDS team are included.

Road safety knowledge exchange and capacity building between China and Australia

Road traffic injuries are one of the major public health burdens worldwide. The United Nations Decade of Action for Road Safety (2011-2020) implores all nations to work to reduce this burden. This decade represents a unique and historic period of time in the field of road safety. Information exchange and co-operation between nations is an important step in achieving the goal. The burden of road crashes, fatalities and injuries is not equally distributed. We know that low and middle-income countries experience the majority of the road trauma burden. Therefore it is imperative that these countries learn from the successes of others that have developed and implemented road safety laws, public education campaigns and countermeasures over many years and have achieved significant road trauma reductions as a result. China is one of the countries experiencing a large road trauma burden. Vulnerable road users such as pedestrians and cyclists make up a large proportion of fatalities and injuries in China. Speeding, impaired/drug driving, distracted driving, vehicle overloading, inadequate road infrastructure, limited use of safety restraints and helmets, and limited road safety training have all been identified as contributing to the problem. Some important steps have been taken to strengthen China’s approach, including increased penalties for drunk driving in May 2011 and increased attention to school bus safety in 2011/12. However, there is still a large amount of work needed to improve the current road safety position in China. This paper provides details of a program to assist with road safety knowledge exchange between China and Australia that was funded by the Australian Government which was undertaken in the latter part of 2012. The four month program provided the opportunity for the first author to work closely with key agencies in Australia that are responsible for policy development and implementation of a broad range of road safety initiatives. In doing so, an in-depth understanding was gained about key road safety strategies in Australia and processes for developing and implementing them. Insights were also gained into the mechanisms used for road safety policy development, implementation and evaluation in several Australian jurisdictions. Road traffic law and enforcement issues were explored with the relevant jurisdictional transport and police agencies to provide a greater understanding of how Chinese laws and practices could be enhanced. Working with agencies responsible for public education and awareness campaigns about road safety in Australia also provided relevant information about how to promote road safety at the broader community level in China. Finally, the program provided opportunities to work closely with several world-renowned Australian research centres and key expert researchers to enhance opportunities for ongoing road safety research in China. The overall program provided the opportunity for the first author to develop knowledge in key areas of road safety strategy development, implementation and management which are directly relevant to the current situation in China. This paper describes some main observations and findings from participation in the program.

Learning, evaluation, action and reflection for new technologies, empowerment and rural sustainability (The LEARNERS project)

-

Research monograph : emergence of smart urban ecosystems : application of ICT for capacity building in environmental decision making

Since the industrial revolution, our world has experienced rapid and unplanned industrialization and urbanization. As a result, we have had to cope with serious environmental challenges. In this context, explanation of how smart urban ecosystems can emerge, gains a crucial importance. Capacity building and community involvement have always been the key issues in achieving sustainable development and enhancing urban ecosystems. By considering these, this paper looks at new approaches to increase public awareness of environmental decision making. This paper will discuss the role of Information and Communication Technologies (ICT), particularly Web-based Geographic Information Systems (Web-based GIS) as spatial decision support systems to aid public participatory environmental decision making. The paper also explores the potential and constraints of these web-based tools for collaborative decision making.

Planning for smart urban ecosystems : information technology applications for capacity building in environmental decision making

Since the industrial revolution, our world has experienced rapid and unplanned industrialization and urbanization. As a result, we have had to cope with serious environmental challenges. In this context, an explanation of how smart urban ecosystems can emerge, gains a crucial importance. Capacity building and community involvement have always been key issues in achieving sustainable development and enhancing urban ecosystems. By considering these, this paper looks at new approaches to increase public awareness of environmental decision making. This paper will discuss the role of Information and Communication Technologies (ICT), particularly Webbased Geographic Information Systems (Web-based GIS) as spatial decision support systems to aid public participatory environmental decision making. The paper also explores the potential and constraints of these webbased tools for collaborative decision making.

A discussion of the research performance evaluation in building and construction management discipline : the journal impact factors and SCI-index journals

The trend of diminished funding, demands for greater efficiency and higher public accountability have led to a rapid expansion of interest in the bibliometric assessment of research performance of universities. A pilot research is conducted to provide a preliminary overview of the research performance of the building and construction schools or departments through the analysis of bibliometric indicators including the journal impact factor (JIF) published by Institute for Scientific Information (ISI). The suitability of bibliometric evaluation approaches as a measure of research quality in building and construction management research field is discussed.

Fostering cultural awareness in pre-service teacher education : an experiment in Asia literacy capacity building in an Australian university

This paper presents the findings of a small pilot study conducted with a group of final year pre-service teachers studying a secondary social science curriculum method unit in an Australian university. One of the study’s research objectives aimed at identifying how students responded to efforts to embed intercultural understanding through Studies of Asia in their final curriculum method unit. The unit was designed and taught by the researcher on the assumption that beginning social science teachers need to be empowered with pedagogical skills and new dispositions to deal with value laden emerging regional and global concerns in their Australian secondary school classrooms. This pilot study’s research methodology was located within the qualitative framework of a participatory action research model whereby the lecturer who designed, coordinated and taught the unit was also the researcher. Its scope was limited to one semester with volunteer students. The pilot study sought to investigate responses to several issues, and this paper reports on pre-service teacher reflections on the content, pedagogy and learning they experienced in their weekly sessions with specific reference to cultural understanding, Studies of Asia and the development of Asia literacy. It also reports on pre-service teacher reflections about their own evolving capacity as beginning teachers. The findings indicate that pre-service teachers valued the opportunity to engage with learning experiences which enhanced their conceptual understandings about culture whilst also extending their pedagogical and content knowledge.

Report on capacity-building workshops for road safety in Indonesia

In addition to the established problem of road safety in developing countries such as Indonesia, the agencies responsible for road safety often lack personnel with professional training in road safety. In Indonesia this is compounded by a need for more effective collaboration between agencies. In 2009, CARRS-Q was commissioned under the Indonesia Transport Safety Assistance Package to provide professional training in road safety for middle level officers in Jakarta, the province of Jawa Barat, and the cities of Bandung, Bogor and Sukabumi, aimed at developing action plans and fostering collaboration between agencies. This was achieved through a workshop, which followed up by a second workshop with the same participants. The course was very well received, action plans were successfully prepared during the first workshop, and most had progressed well by the time of the second workshop. Good cooperation between agencies was also evident. There would be considerable benefits in extending modified workshops more widely in Indonesia.

Capacity-building workshops on the responsibility to protect (R2P) and the protection of civilians (POC) in armed conflict - Summary Report

In June 2011, a research project team from the Institute for Ethics, Governance and Law (IEGL), Queensland University of Technology, the United Nations University, and the Australian Government’s Asia Pacific Civil-Military Centre of Excellence (APCMCOE) held three Capacity-Building Workshops (the Workshops) on the Responsibility to Protect (R2P) and the Protection of Civilians (POC) in Armed Conflict in Manila, Kuala Lumpur, and Jakarta. The research project is funded by the Australian Responsibility to Protect Fund, with support from APCMCOE. Developments in Libya and Cote d’Ivoire and the actions of the United Nations Security Council have given new significance to the relationship between R2P and POC, providing impetus to the relevance and application of the POC principle recognised in numerous Security Council resolutions, and the R2P principle, which was recognised by the United Nations General Assembly in 2005 and, now, by the Security Council. The Workshops considered the relationship between R2P and POC. The project team presented the preliminary findings of their study and sought contributions and feedback from Workshop participants. Prior to the Workshops, members of the project team undertook interviews with UN offices and agencies, international organisations (IOs) and non-government organisations (NGOs) in Geneva and New York as part of the process of mapping the relationship between R2P and POC. Initial findings were considered at an Academic-Practitioner Workshop held at the University of Sydney in November 2010. In addition to an extensive literature review and a series of academic publications, the project team is preparing a practical guidance text (the Guide) on the relationship between R2P and POC to assist the United Nations, governments, regional bodies, IOs and NGOs in considering and applying appropriate protection strategies. It is intended that the Guide be presented to the United Nations Secretariat in New York in early 2012. The primary aim of the Workshops was to test the project’s initial findings among an audience of diplomats, military, police, civilian policy-makers, practitioners, researchers and experts from within the region. Through dialogue and discussion, the project team gathered feedback – comments, questions, critique and suggestions – to help shape the development of practical guidance about when, how and by whom R2P and POC might be implemented.

Factors that influence Chinese TEFL academics' research capacity building : an institutional case study

Research capacity building has become a prominent theme in higher education institutions across the world. To build research capacity, it is necessary to identify areas of challenges academics face within the academia. This case study focuses on Chinese teaching English as a foreign language (TEFL) academics with the purpose of identifying factors that influence their research capacity building. Six TEFL academics from a Chinese national university were interviewed and institutional research documents were analysed. Findings showed that obstacles and difficulties in conducting research were more related to departmental factors than individual characteristics. The institution was keen on developing a research culture, and encouraged research and publications. Departmental support for research was improving, but it seems that it was more generic than tailored to individual needs. The findings of this study provide implications for research administrators in further supporting TEFL academics’ research capacity building.

Academic legal research and academic research capacity enhancement working topic : "purposes and objectives of law schools beyond educating students (research, capacity building,community service and outreach)"

One of the main objectives of law schools beyond educating students is to produce viable legal research. The comments in this paper are basically confined to the Australian context, and to examine this topic effectively, it is necessary to briefly review the current tertiary research agenda in Australia. This paper argues that there is a need for recognition and support for an expanded legal research framework along with additional research training for legal academics. There also needs to be more effective methods of measuring and recognising quality in legal research. This method needs to be one that can engender respect in an interdisciplinary context.

Critical reflections on the use of participatory methodologies to build evaluation capacities in international development organisations

There has been a greater focus on strengthening evaluation capacity building (ECB) within development organisations in recent years. This can be attributed in part to the growing appreciation of the value of participatory and collaborative forms of evaluation. Evaluation is increasingly seen as an ongoing learning process and an important means of strengthening capacity and improving organisational performance (Horton et al., 2003:7). While there are many benefits of using participatory methodologies in ECB projects, our experiences and a review of the literature in this area highlight the many challenges, issues and contradictions that can affect the success of such ECB efforts. We discuss these issues, drawing on our learnings from the ongoing participatory action research (PAR) project 'Assessing Communication for Social Change’ (AC4SC). This four year project, which began in 2007, is a collaboration between communication and development academics and evaluation specialists from two Australian universities and communication for development practitioners and monitoring and evaluation (M&E) staff in the NGO Equal Access Nepal (EAN). The aim is to develop, implement, and evaluate a participatory methodology for assessing the social change impacts of community radio programs produced by EAN. It builds on previous projects that used ethnographic action research (EAR) methodology (Tacchi et al., 2007).