Meta-evaluation to improve learning, evaluation capacity development and sustainability : Findings from a participatory evaluation project in Nepal

Recent studies suggest that meta-evaluation can be valuable in developing new approaches to evaluation, building evaluation capacities, and enhancing organizational learning. These new extensions of the concept of meta-evaluation are significant, given the growing emphasis on improving the quality and effectiveness of evaluation practices in the South Asian region. Following a review of the literature, this paper presents a case study of the use of concurrent meta-evaluation in the four-year project Assessing Communication for Social Change which developed and trialled a participatory impact assessment methodology in collaboration with a development communication Non-government organization (NGO) in Nepal. Key objectives of the meta-evaluation included to: continuously develop, adapt and improve the impact assessment methodology, Monitoring and Evaluation (M&E) systems and process and other project activities; identify impacts of the project; and build capacities in critical reflection and review. Our analysis indicates that this meta-evaluation was essential to understanding various constraints related to the organizational context that affected the success of the project and the development of improved M&E systems and capacities within the NGO. We identified several limitations of our meta-evaluation methods, which were balanced by the strengths of other methods. Our case study suggests that as well as assessing the quality, credibility and value of evaluation practices, meta-evaluations need to focus on important contextual issues that can have significant impacts on the outcomes of participatory evaluation projects. They include hierarchical organizational cultures, communication barriers, power/knowledge relations, and the time and resources available. Meta-evaluations also need to consider wider issues such as the sustainability of evaluation systems and approaches.

Large area noise evaluation

A software tool (DRONE) has been developed to evaluate road traffic noise in a large area with the consideration of network dynamic traffic flow and the buildings. For more precise estimation of noise in urban network where vehicles are mainly in stop and go running conditions, vehicle sound power level (for acceleration/deceleration cruising and ideal vehicle) is incorporated in DRONE. The calculation performance of DRONE is increased by evaluating the noise in two steps of first estimating the unit noise database and then integrating it with traffic simulation. Details of the process from traffic simulation to contour maps are discussed in the paper and the implementation of DRONE on Tsukuba city is presented.

Near infrared spectroscopy for non-destructive evaluation of articular cartilage

There is a need for an accurate real-time quantitative system that would enhance decision-making in the treatment of osteoarthritis. To achieve this objective, significant research is required that will enable articular cartilage properties to be measured and categorized for health and functionality without the need for laboratory tests involving biopsies for pathological evaluation. Such a system would provide the capability of access to the internal condition of the cartilage matrix and thus extend the vision-based arthroscopy that is currently used beyond the subjective evaluation of surgeons. The system required must be able to non-destructively probe the entire thickness of the cartilage and its immediate subchondral bone layer. In this thesis, near infrared spectroscopy is investigated for the purpose mentioned above. The aim is to relate it to the structure and load bearing properties of the cartilage matrix to the near infrared absorption spectrum and establish functional relationships that will provide objective, quantitative and repeatable categorization of cartilage condition outside the area of visible degradation in a joint. Based on results from traditional mechanical testing, their innovative interpretation and relationship with spectroscopic data, new parameters were developed. These were then evaluated for their consistency in discriminating between healthy viable and degraded cartilage. The mechanical and physico-chemical properties were related to specific regions of the near infrared absorption spectrum that were identified as part of the research conducted for this thesis. The relationships between the tissue's near infrared spectral response and the new parameters were modeled using multivariate statistical techniques based on partial least squares regression (PLSR). With significantly high levels of statistical correlation, the modeled relationships were demonstrated to possess considerable potential in predicting the properties of unknown tissue samples in a quick and non-destructive manner. In order to adapt near infrared spectroscopy for clinical applications, a balance between probe diameter and the number of active transmit-receive optic fibres must be optimized. This was achieved in the course of this research, resulting in an optimal probe configuration that could be adapted for joint tissue evaluation. Furthermore, as a proof-of-concept, a protocol for obtaining the new parameters from the near infrared absorption spectra of cartilage was developed and implemented in a graphical user interface (GUI)-based software, and used to assess cartilage-on-bone samples in vitro. This conceptual implementation has been demonstrated, in part by the individual parametric relationship with the near infrared absorption spectrum, the capacity of the proposed system to facilitate real-time, non-destructive evaluation of cartilage matrix integrity. In summary, the potential of the optical near infrared spectroscopy for evaluating articular cartilage and bone laminate has been demonstrated in this thesis. The approach could have a spin-off for other soft tissues and organs of the body. It builds on the earlier work of the group at QUT, enhancing the near infrared component of the ongoing research on developing a tool for cartilage evaluation that goes beyond visual and subjective methods.

Evaluation of precision time synchronisation methods for substation applications

Many substation applications require accurate time-stamping. The performance of systems such as Network Time Protocol (NTP), IRIG-B and one pulse per second (1-PPS) have been sufficient to date. However, new applications, including IEC 61850-9-2 process bus and phasor measurement, require accuracy of one microsecond or better. Furthermore, process bus applications are taking time synchronisation out into high voltage switchyards where cable lengths may have an impact on timing accuracy. IEEE Std 1588, Precision Time Protocol (PTP), is the means preferred by the smart grid standardisation roadmaps (from both the IEC and US National Institute of Standards and Technology) of achieving this higher level of performance, and integrates well into Ethernet based substation automation systems. Significant benefits of PTP include automatic path length compensation, support for redundant time sources and the cabling efficiency of a shared network. This paper benchmarks the performance of established IRIG-B and 1-PPS synchronisation methods over a range of path lengths representative of a transmission substation. The performance of PTP using the same distribution system is then evaluated and compared to the existing methods to determine if the performance justifies the additional complexity. Experimental results show that a PTP timing system maintains the synchronising performance of 1-PPS and IRIG-B timing systems, when using the same fibre optic cables, and further meets the needs of process buses in large substations.

Buy, boycott or blog : exploring online consumer power to share, discuss and distribute controversial advertising messages

The availability of new media as a universal communication tool has an impact on the power of the general public to comment on a variety of issues. This paper examines this increase in consumer power with respect to bloggers. The research context is controversial advertising, and specifically Tourism Australia’s “Where the bloody hell are you?” campaign. By utilising Denegri-Knott’s (2006) four on-line power strategies, a content analysis of weblogs reveals that consumers are distributing information, opinion and even banned advertising material, thereby forming power hubs of like-minded people, with the potential to become online pressure groups. The consequences and implications of this augmented power on regulators, advertisers and bloggers are explored. The findings contribute to the understanding of blogs as a new communication platform and bloggers as a new demographic of activists in the process of advertising.

Document clustering evaluation : Divergence from a random baseline

Divergence from a random baseline is a technique for the evaluation of document clustering. It ensures cluster quality measures are performing work that prevents ineffective clusterings from giving high scores to clusterings that provide no useful result. These concepts are defined and analysed using intrinsic and extrinsic approaches to the evaluation of document cluster quality. This includes the classical clusters to categories approach and a novel approach that uses ad hoc information retrieval. The divergence from a random baseline approach is able to differentiate ineffective clusterings encountered in the INEX XML Mining track. It also appears to perform a normalisation similar to the Normalised Mutual Information (NMI) measure but it can be applied to any measure of cluster quality. When it is applied to the intrinsic measure of distortion as measured by RMSE, subtraction from a random baseline provides a clear optimum that is not apparent otherwise. This approach can be applied to any clustering evaluation. This paper describes its use in the context of document clustering evaluation.

A 3D collision avoidance strategy for UAVs in a non-cooperative environment

This paper presents a feasible 3D collision avoidance approach for fixed-wing unmanned aerial vehicles (UAVs). The proposed strategy aims to achieve the desired relative bearing in the horizontal plane and relative elevation in the vertical plane so that the host aircraft is able to avoid collision with the intruder aircraft in 3D. The host aircraft will follow a desired trajectory in the collision avoidance course and resume the pre-arranged trajectory after collision is avoided. The approaching stopping condition is determined for the host aircraft to trigger an evasion maneuver to avoid collision in terms of measured heading. A switching controller is designed to achieve the spatial collision avoidance strategy. Simulation results demonstrate that the proposed approach can effectively avoid spatial collision, making it suitable for integration into flight control systems of UAVs.

Evaluation of coronary angiographic projections to balance the clinical yield with the radiation risk

Objective: Radiation safety principles dictate that imaging procedures should minimise the radiation risks involved, without compromising diagnostic performance. This study aims to define a core set of views that maximises clinical information yield for minimum radiation risk. Angiographers would supplement these views as clinically indicated. Methods: An algorithm was developed to combine published data detailing the quality of information derived for the major coronary artery segments through the use of a common set of views in angiography with data relating to the dose–area product and scatter radiation associated with these views. Results: The optimum view set for the left coronary system comprised four views: left anterior oblique (LAO) with cranial (Cr) tilt, shallow right anterior oblique (AP-RAO) with caudal (Ca) tilt, RAO with Ca tilt and AP-RAO with Cr tilt. For the right coronary system three views were identified: LAO with Cr tilt, RAO and AP-RAO with Cr tilt. An alternative left coronary view set including a left lateral achieved minimally superior efficiency (,5%), but with an ,8% higher radiation dose to the patient and 40% higher cardiologist dose. Conclusion: This algorithm identifies a core set of angiographic views that optimises the information yield and minimises radiation risk. This basic data set would be supplemented by additional clinically determined views selected by the angiographer for each case. The decision to use additional views for diagnostic angiography and interventions would be assisted by referencing a table of relative radiation doses for the views being considered.

When non-significance may be significant : lessons learned from a study of the development, implementation and evaluation of a fleet risk assessment tool

This paper reports on the development and implementation of a self-report risk assessment tool that was developed in an attempt to increase the efficacy of crash prediction within Australian fleet settings. This study forms a part of a broader program of research into work related road safety and identification of driving risk. The first phase of the study involved a series of focus groups being conducted with 217 professional drivers which revealed that the following factors were proposed to influence driving performance: Fatigue, Knowledge of risk, Mood, Impatience and frustration, Speed limits, Experience, Other road users, Passengers, Health, and Culture. The second phase of the study involved piloting the newly developed 38 item Driving Risk Assessment Scale - Work Version (DRAS-WV) with 546 professional drivers. Factor analytic techniques identified a 9 factor solution that was comprised of speeding, aggression, time pressure, distraction, casualness, awareness, maintenance, fatigue and minor damage. Speeding and aggressive driving manoeuvres were identified to be the most frequent aberrant driving behaviours engaged in by the sample. However, a series of logistic regression analyses undertaken to determine the DRAS-WV scale’s ability to predict self-reported crashes revealed limited predictive efficacy e.g., 10% of crashes. This paper outlines proposed reasons for this limited predictive ability of the DRAS-WV as well as provides suggestions regarding the future of research that aims to develop methods to identify “at risk” drivers.

Queensland Emergency Medicine Research Foundation: Special report

The development of any new profession is dependent on the development of a special body of knowledge which is the domain of the profession and key to this is the conduct of research. In 2007, as part of the settlement of an Enterprise Bargaining Agreement and following sustained lobbying by Emergency Physicians, the Queensland Government agreed to establish an Emergency Medicine Research Fund to foster the development of research activities in Emergency Medicine in Queensland. That fund is now managed by the Queensland Emergency Medicine Research Foundation. The aims of this article are to describe the strategic approaches taken by the Foundation and its first three years of experience, to describe the application of research funds and to foreshadow an evaluative framework for determining the strategic value of this community investment. The Foundation has developed a range of personnel and project support funding programs and competition for funding has increased. Ongoing evaluation will seek to determine the effectiveness of this funding strategy on improving the effectiveness of research performance and the clinical and organisational outcomes that may derive from that initiative.

Producing solidarity : a settlement strategy of the Karen refugee community in Brisbane, Australia

This paper explores the link between experience and context. It places the lived experiences of Karen refugees during settlement in Brisbane, Australia within the socio-political context of Burma, or particularly the historical context of persecution. Two key events – the Wrist-tying Ceremony and the Karen New Year – provide a link between experience and context. The findings of this study show a community strategically at work in a new and ongoing settlement process. This process pays respect to the complexities of cultural integrity whilst also engaging with the challenges of integration. The complexities are local (in terms of cultural, linguistic and religious diversity), national (maintaining a broader sense of community that includes linkages across Australia, as well as an engagement with the Australian socio-political context), and transnational (participating in a global Karen community). This transnational community encompasses Karen settling elsewhere in the world, Karen in refugee camps neighbouring Burma, and Karen living inside Burma. This paper argues that substantial “identity work” is involved in Karen settlement. The two key community events are useful vignettes of this identity work. Both events demonstrate how Karen cultural practices can meaningfully negotiate deeply historical ideas of Karen identity with contemporary challenges of settlement. In addition, they set out a version of settlement that departs from traditional settlement constructs; they show how the lived experience of settlement is messy, complex and dynamic, and not reflective of the neat, idealistic models that immigration policy and settlement theory project.

Environmental and economic evaluation of N fertilizer rates in a maize crop in Italy : a spatial and temporal analysis using crop models

Crop simulation models have the potential to assess the risk associated with the selection of a specific N fertilizer rate, by integrating the effects of soil-crop interactions on crop growth under different pedo-climatic and management conditions. The objective of this study was to simulate the environmental and economic impact (nitrate leaching and N2O emissions) of a spatially variable N fertilizer application in an irrigated maize field in Italy. The validated SALUS model was run with 5 nitrogen rates scenarios, 50, 100, 150, 200, and 250 kg N ha−1, with the latter being the N fertilization adopted by the farmer. The long-term (25 years) simulations were performed on two previously identified spatially and temporally stable zones, a high yielding and low yielding zone. The simulation results showed that N fertilizer rate can be reduced without affecting yield and net return. The marginal net return was on average higher for the high yield zone, with values ranging from 1550 to 2650 € ha−1 for the 200 N and 1485 to 2875 € ha−1 for the 250 N. N leaching varied between 16.4 and 19.3 kg N ha−1 for the 200 N and the 250 N in the high yield zone. In the low yield zone, the 250 N had a significantly higher N leaching. N2O emissions varied between 0.28 kg N2O ha−1 for the 50 kg N ha−1 rate to a maximum of 1.41 kg N2O ha−1 for the 250 kg N ha−1 rate.

Impact of a government triple zero awareness campaign on emergency department patient characteristics

Objective: To evaluate the impact of a government triple zero community awareness campaign on the characteristics of patients attending an ED. Methods: A study using Emergency Department Information System data was conducted in an adult metropolitan tertiary-referral teaching hospital in Brisbane. The three outcomes measured in the 3 month post-campaign period were arrival mode, Australasian Triage Scale and departure status. These measures reflect ambulance usage, clinical urgency and illness severity, respectively. They were compared with those in the 3 month pre-campaign period. Multivariate logistic regression models were used to investigate the impacts of the campaign on each of the three outcome measures after controlling for age, sex, day and time of arrival, and daily minimum temperature. Results: There were 17 920 visits in the pre- and 17 793 visits in the post-campaign period. After the campaign, fewer patients arrived at the ED by road ambulance (odds ratio [OR] 0.90, 95% confidence interval [CI] 0.80–1.00), although the impact of the campaign on the arrival mode was only close to statistical significance (Wald χ2-test, P= 0.055); and patients were significantly less likely to have higher clinical urgency (OR 0.86, 95% CI 0.79–0.94), while more likely to be admitted (OR 1.68, 95% CI 1.38–2.05) or complete treatment in the ED (OR 1.46, 95% CI 1.23–1.73) instead of leaving without waiting to be seen. Conclusions: The campaign had no significant impact on the arrival mode of the patients. After the campaign, the illness acuity of the patients decreased, whereas the illness severity of the patients increased.

The committee on public information : a transmedia war propaganda campaign

This thesis develops an understanding of how propaganda entered the realm of journalism and popular culture in the United States during World War I through an examination of materials created by the Committee on Public Information (CPI). The CPI was a US governmental propaganda organisation created during World War I to persuade the nation to mobilise for war. Three of its divisions were chosen for this study: the Division of News (DoN), the Division of Four Minute Men (FMM) and the Division of Pictorial Publicity (DPP). Chapter 1 provides a general context for the thesis, outlines the research questions and details previous research on the CPI. Chapter 2 outlines the methods of analysis for interpreting the case study chapters and provides contextual information. The case studies are presented in Chapters 3, 4 and 5. These chapters are structured in the order of context, medium and content, and contain historical contextual information about each particular division, medialogical aspects of its propagated form and thematic groupings created from close reading of CPI materials. A semiotic analysis in the Peircian tradition is also performed on visual forms of propaganda in Chapter 5. Chapter 6 discusses how the expectations of persuasion, truth and amusement relate to each other when mediated in culture, using Lotman’s concept of the semiosphere. This further develops an understanding of propaganda as a cultural system in relation to other cultural systems – in this case, journalism and popular culture. Chapter 7 provides conclusions about the study, outlines relative strengths and weaknesses regarding the selection and deployment of methods, makes recommendations for future research, and summarises the key contributions of the thesis.

From 'what' to 'how' : capacity building in health promotion for HIV/AIDS prevention in the Solomon Islands

This paper describes a capacity building process undertaken within the HIV/AIDS prevention project of the Adventist Development and Relief Agency (ADRA) in the Solomon Islands. ADRA HIV/AIDS has recently reoriented its project structure, moving beyond its awareness raising approach to incorporate health promotion frameworks, theories, strategies and assumptions. These have been used to inform project practice in project planning, delivery and evaluation. This paper shares what has worked and not worked in the capacity building process, including a project evaluation of the initial HIV/AIDS awareness raising project and the application of a number of capacity building strategies, including utilising a volunteer Australian Youth Ambassador for Development (AYAD) funded by the Australian Agency for International Development (AusAID). Existing and new projects are outlined. The underlying theme is that any capacity building exercise must include structural support (e.g. management, national frameworks) to ensure the incorporation of new initiatives and approaches. With time this enables ownership by counterparts and external partnerships to develop. The presence of an AYAD volunteer has been an effective strategy to achieve this. Reflections from the evaluators, the AYAD volunteer and the HIV/AIDS team are included.