New evaluation methodologies for public art programs in cultural instiutions

I put my abstract in for this conference back in March, based on some evaluation work I had been doing in 2010 with my colleague Professor Greg Hearn for the 3C Regional Writing NeoGreography Project. I had been swapping notes with a colleague from the Smithsonian’s Centre for Folklife and Cultural Heritage about their evaluation work, and stuck inside during the rains of January, I decided to apply for a Qld Smithsonian fellowship based on the quandary of evaluation-particular in public histories (oral histories) and digital storytelling. In July I was awarded the fellowship, so I have tweaked my presentation to talk about what we hope to do with this collaboration, to propel the importance placed on evaluation in public arts programs in Qld and beyond.

A review of project evaluation methodologies to address net impacts and risks of toll road projects to the community

Project evaluation is a process of measuring costs, benefits, risks and uncertainties for the purpose of decision-making by estimating and assessing impacts of the project to the community. The effects of impacts of toll roads are similar but different from the general non-tolled roads. Project evaluation methodologies are extensively studied and applied to various transport infrastructure projects. However, there is no definitive methodology to evaluate toll roads. This review discusses the impacts of toll roads then reviews the limitations of existing project evaluation methodologies when evaluating toll road impacts. The review identified gaps of knowledge of toll evaluations. First, the treatment of toll in project evaluation, particularly in Cost-Benefit Analysis requires further study to explore the appropriate methodology. Secondly, the project evaluation methodology needs to place strong emphasis on empirically based risk and uncertainty assessment. Addressing the limitations of the existing project evaluation methodologies leads to improvements of the methodology in practical level as well as fills the gap of knowledge of project evaluation for toll roads with respect to net impacts to the community.

A review of project evaluation methodologies for toll road projects

Project evaluation is a process of measuring costs, benefits, risks and uncertainties for the purpose of decision-making by estimating and assessing impacts of the project to the community. The effects of impacts of toll roads are similar but different from the general non-tolled roads. Project evaluation methodologies are extensively studied and applied to various transport infrastructure projects. However, there is no definitive methodology to evaluate toll roads. This review discusses the impacts of toll roads then reviews the limitations of existing project evaluation methodologies when evaluating toll road impacts. The review identified gaps of knowledge of toll evaluations. First, the treatment of toll in project evaluation, particularly in Cost-Benefit Analysis requires further study to explore the appropriate methodology. Secondly, the project evaluation methodology needs to place strong emphasis on empirically based risk and uncertainty assessment. Addressing the limitations of the existing project evaluation methodologies leads to improvements of the methodology in practical level as well as fills the gap of knowledge of project evaluation for toll roads with respect to net impacts to the community.

Evaluation in health outcomes research : linking theories, methodologies and practice in health promotion.

Health outcomes research has developed as a means of evaluating the effectiveness of health care interventions and as an approach to informing resource allocation. The use of a health outcomes approach in health promotion has made increasing demands on evaluation methodologies to demonstrate program effectiveness. However, criticism of the contribution of health promotion to outcomes research has made several assumptions about the use of qualitative methodologies and the content of program objectives largely derived from a biomedical approach. In contrast to the measurement of biomedical interventions in clinical health care, health promotion practice involves social phenomena, wide-reaching cultural, psychological, political and ideological problems and issues. The integration of methodologies of health promotion evaluation will inform further conceptualisation of the health outcomes approach with the differentiation of three types of outcomes: health development outcomes; social health outcomes; and biomedical health outcomes. It is concluded that this differentiation moves away from dualist concepts that advocate the replacement of goals and targets with regional and locally based approaches. Rather, the future direction for health promotion evaluation needs to employ a framework that elaborates multiple methodologies and approaches necessary for establishing what relationships exist between morbidity, mortality, health advancement and equity.

Nation & internation practices in decision support tools in road asset management

This document provides a review of international and national practices in investment decision support tools in road asset management. Efforts were concentrated on identifying analytic frameworks, evaluation methodologies and criteria adopted by current tools. Emphasis was also given to how current approaches support Triple Bottom Line decision-making. Benefit Cost Analysis and Multiple Criteria Analysis are principle methodologies in supporting decision-making in Road Asset Management. The complexity of the applications shows significant differences in international practices. There is continuing discussion amongst practitioners and researchers regarding to which one is more appropriate in supporting decision-making. It is suggested that the two approaches should be regarded as complementary instead of competitive means. Multiple Criteria Analysis may be particularly helpful in early stages of project development, say strategic planning. Benefit Cost Analysis is used most widely for project prioritisation and selecting the final project from amongst a set of alternatives. Benefit Cost Analysis approach is useful tool for investment decision-making from an economic perspective. An extension of the approach, which includes social and environmental externalities, is currently used in supporting Triple Bottom Line decision-making in the road sector. However, efforts should be given to several issues in the applications. First of all, there is a need to reach a degree of commonality on considering social and environmental externalities, which may be achieved by aggregating the best practices. At different decision-making level, the detail of consideration of the externalities should be different. It is intended to develop a generic framework to coordinate the range of existing practices. The standard framework will also be helpful in reducing double counting, which appears in some current practices. Cautions should also be given to the methods of determining the value of social and environmental externalities. A number of methods, such as market price, resource costs and Willingness to Pay, are found in the review. The use of unreasonable monetisation methods in some cases has discredited Benefit Cost Analysis in the eyes of decision makers and the public. Some social externalities, such as employment and regional economic impacts, are generally omitted in current practices. This is due to the lack of information and credible models. It may be appropriate to consider these externalities in qualitative forms in a Multiple Criteria Analysis. Consensus has been reached in considering noise and air pollution in international practices. However, Australia practices generally omitted these externalities. Equity is an important consideration in Road Asset Management. The considerations are either between regions, or social groups, such as income, age, gender, disable, etc. In current practice, there is not a well developed quantitative measure for equity issues. More research is needed to target this issue. Although Multiple Criteria Analysis has been used for decades, there is not a generally accepted framework in the choice of modelling methods and various externalities. The result is that different analysts are unlikely to reach consistent conclusions about a policy measure. In current practices, some favour using methods which are able to prioritise alternatives, such as Goal Programming, Goal Achievement Matrix, Analytic Hierarchy Process. The others just present various impacts to decision-makers to characterise the projects. Weighting and scoring system are critical in most Multiple Criteria Analysis. However, the processes of assessing weights and scores were criticised as highly arbitrary and subjective. It is essential that the process should be as transparent as possible. Obtaining weights and scores by consulting local communities is a common practice, but is likely to result in bias towards local interests. Interactive approach has the advantage in helping decision-makers elaborating their preferences. However, computation burden may result in lose of interests of decision-makers during the solution process of a large-scale problem, say a large state road network. Current practices tend to use cardinal or ordinal scales in measure in non-monetised externalities. Distorted valuations can occur where variables measured in physical units, are converted to scales. For example, decibels of noise converts to a scale of -4 to +4 with a linear transformation, the difference between 3 and 4 represents a far greater increase in discomfort to people than the increase from 0 to 1. It is suggested to assign different weights to individual score. Due to overlapped goals, the problem of double counting also appears in some of Multiple Criteria Analysis. The situation can be improved by carefully selecting and defining investment goals and criteria. Other issues, such as the treatment of time effect, incorporating risk and uncertainty, have been given scant attention in current practices. This report suggested establishing a common analytic framework to deal with these issues.

The impact of red light cameras on safety in Arizona

Red light cameras (RLCs) have been used in a number of US cities to yield a demonstrable reduction in red light violations; however, evaluating their impact on safety (crashes) has been relatively more difficult. Accurately estimating the safety impacts of RLCs is challenging for several reasons. First, many safety related factors are uncontrolled and/or confounded during the periods of observation. Second, “spillover” effects caused by drivers reacting to non-RLC equipped intersections and approaches can make the selection of comparison sites difficult. Third, sites selected for RLC installation may not be selected randomly, and as a result may suffer from the regression to the mean bias. Finally, crash severity and resulting costs need to be considered in order to fully understand the safety impacts of RLCs. Recognizing these challenges, a study was conducted to estimate the safety impacts of RLCs on traffic crashes at signalized intersections in the cities of Phoenix and Scottsdale, Arizona. Twenty-four RLC equipped intersections in both cities are examined in detail and conclusions are drawn. Four different evaluation methodologies were employed to cope with the technical challenges described in this paper and to assess the sensitivity of results based on analytical assumptions. The evaluation results indicated that both Phoenix and Scottsdale are operating cost-effective installations of RLCs: however, the variability in RLC effectiveness within jurisdictions is larger in Phoenix. Consistent with findings in other regions, angle and left-turn crashes are reduced in general, while rear-end crashes tend to increase as a result of RLCs.

The hydrodynamic and capture/retention performance of a gross pollutant trap

This research shows that gross pollutant traps (GPTs) continue to play an important role in preventing visible street waste—gross pollutants—from contaminating the environment. The demand for these GPTs calls for stringent quality control and this research provides a foundation to rigorously examine the devices. A novel and comprehensive testing approach to examine a dry sump GPT was developed. The GPT is designed with internal screens to capture gross pollutants—organic matter and anthropogenic litter. This device has not been previously investigated. Apart from the review of GPTs and gross pollutant data, the testing approach includes four additional aspects to this research, which are: field work and an historical overview of street waste/stormwater pollution, calibration of equipment, hydrodynamic studies and gross pollutant capture/retention investigations. This work is the first comprehensive investigation of its kind and provides valuable practical information for the current research and any future work pertaining to the operations of GPTs and management of street waste in the urban environment. Gross pollutant traps—including patented and registered designs developed by industry—have specific internal configurations and hydrodynamic separation characteristics which demand individual testing and performance assessments. Stormwater devices are usually evaluated by environmental protection agencies (EPAs), professional bodies and water research centres. In the USA, the American Society of Civil Engineers (ASCE) and the Environmental Water Resource Institute (EWRI) are examples of professional and research organisations actively involved in these evaluation/verification programs. These programs largely rely on field evaluations alone that are limited in scope, mainly for cost and logistical reasons. In Australia, evaluation/verification programs of new devices in the stormwater industry are not well established. The current limitations in the evaluation methodologies of GPTs have been addressed in this research by establishing a new testing approach. This approach uses a combination of physical and theoretical models to examine in detail the hydrodynamic and capture/retention characteristics of the GPT. The physical model consisted of a 50% scale model GPT rig with screen blockages varying from 0 to 100%. This rig was placed in a 20 m flume and various inlet and outflow operating conditions were modelled on observations made during the field monitoring of GPTs. Due to infrequent cleaning, the retaining screens inside the GPTs were often observed to be blocked with organic matter. Blocked screens can radically change the hydrodynamic and gross pollutant capture/retention characteristics of a GPT as shown from this research. This research involved the use of equipment, such as acoustic Doppler velocimeters (ADVs) and dye concentration (Komori) probes, which were deployed for the first time in a dry sump GPT. Hence, it was necessary to rigorously evaluate the capability and performance of these devices, particularly in the case of the custom made Komori probes, about which little was known. The evaluation revealed that the Komori probes have a frequency response of up to 100 Hz —which is dependent upon fluid velocities—and this was adequate to measure the relevant fluctuations of dye introduced into the GPT flow domain. The outcome of this evaluation resulted in establishing methodologies for the hydrodynamic measurements and gross pollutant capture/retention experiments. The hydrodynamic measurements consisted of point-based acoustic Doppler velocimeter (ADV) measurements, flow field particle image velocimetry (PIV) capture, head loss experiments and computational fluid dynamics (CFD) simulation. The gross pollutant capture/retention experiments included the use of anthropogenic litter components, tracer dye and custom modified artificial gross pollutants. Anthropogenic litter was limited to tin cans, bottle caps and plastic bags, while the artificial pollutants consisted of 40 mm spheres with a range of four buoyancies. The hydrodynamic results led to the definition of global and local flow features. The gross pollutant capture/retention results showed that when the internal retaining screens are fully blocked, the capture/retention performance of the GPT rapidly deteriorates. The overall results showed that the GPT will operate efficiently until at least 70% of the screens are blocked, particularly at high flow rates. This important finding indicates that cleaning operations could be more effectively planned when the GPT capture/retention performance deteriorates. At lower flow rates, the capture/retention performance trends were reversed. There is little difference in the poor capture/retention performance between a fully blocked GPT and a partially filled or empty GPT with 100% screen blockages. The results also revealed that the GPT is designed with an efficient high flow bypass system to avoid upstream blockages. The capture/retention performance of the GPT at medium to high inlet flow rates is close to maximum efficiency (100%). With regard to the design appraisal of the GPT, a raised inlet offers a better capture/retention performance, particularly at lower flow rates. Further design appraisals of the GPT are recommended.

Development of school connectedness as a component of an injury prevention program for early adolescents

The current program of research addresses the need for multi-level programs to target the major increase in injury rates that occurs throughout adolescence. Specifically, it involves the investigation of school connectedness as a protective factor for adolescent injury, and the development of school connectedness as a component of an injury prevention program. To date, school-based risk taking and injury prevention has frequently been limited to addressing adolescents' knowledge and attitudes to risk behaviours, and has largely overlooked the importance of the wider school social context as a protective factor in adolescent development. Additionally, school connectedness has been primarily studied in terms of its impact on student achievement, wellbeing and risk taking behaviour, and research has not yet addressed possible links with injury. Further, school connectedness intervention programs have targeted risk taking behaviours without evaluating their potential impact on injury outcomes. This is the first reported research to develop strategies to increase school connectedness as part of a school-based injury prevention program. The research program was conceptualised as three distinct stages. The development of these research stages was informed by a comprehensive review of the literature on adolescent risk taking, injury and school-based prevention, as well as on school connectedness and its importance in adolescence. A review of the school connectedness literature indicated that students' connectedness is largely influenced by relationships within the school context including with teachers and other school staff, and is therefore a potentially modifiable factor that may be targeted in school-based programs. Overall, the literature shows school connectedness to be a key protective factor in adolescent development. This review established a foundation from which the current program of research was designed. The first stage of the research involved an empirical investigation of the relationship between adolescent risk taking-related injuries and school connectedness. Stage one incorporated two studies. The first involved the development of a measure of adolescent injury, the Extended Adolescent Injury Checklist (E-AIC), for use in the current research as well as in future school-based studies and program evaluation. The results of this study also highlighted the extent of the problem of risk-related injury in adolescence. The second study in Stage one examined the relationship between students' reports of school connectedness, risk taking behaviour and risk taking-related injuries on the E-AIC. The results of this study showed significant relationships between increased school connectedness and reduced reported engagement in transport and violence risk taking, and fewer associated injuries. This study therefore suggested the potential for school-based injury prevention programs to incorporate strategies targeting increased adolescent connectedness to school. The second stage of this research involved the compilation of an evidence base to inform the design of a school connectedness intervention. Stage two also incorporated two studies. The first study in Stage two involved a systematic review of programs that have targeted school connectedness for reduced risk taking and injury. The results of this study revealed that interventions targeting school connectedness can be effective in reducing adolescent risk taking behaviour, and also provided an evidence base for the design of the current school connectedness intervention. The second study in Stage two examined teachers' understanding and perceptions of school connectedness. This qualitative study indicated that teachers consider students' connectedness to be an important factor that relates to their risk taking behaviour; and also provided directions and content for the intervention design stage. The third stage of this research built upon the findings of each of the previous studies, and involved the design, implementation and evaluation of a school connectedness intervention as a component of an adolescent injury prevention program, Skills for Preventing Injury in Youth (SPIY). This connectedness intervention was designed as a professional development workshop for teachers of 13 to 14 year old adolescents, and was developed as a complementary component to the curriculum-based SPIY program. The SPIY connectedness component was implemented and evaluated using process and six-month impact evaluation methodologies. The results of this study revealed that teachers saw value in the program and made use of the strategies presented, and that program school students' self-reported violence risk behaviour was reduced at six-month follow-up. Despite these promising findings, the results of this study did not demonstrate a significant impact of the program on change in students' connectedness to school, relative to comparison schools. The positive impact on self-reported violence risk behaviour was however replicated in additional analyses comparing students participating in the connectedness version of SPIY with students participating in an earlier curriculumonly version of the program. This finding indicated that the connectedness component has additional benefits relating to reduction in violence risks, over and above a curriculum-only version of the program. This research was the first reported to address the relationship between school connectedness and adolescent injury outcomes, and to develop school connectedness as a component of an adolescent injury prevention program. Overall, the results of this program of research have demonstrated the importance of incorporating strategies targeting the wider school social context, including school connectedness, in adolescent injury prevention programs. This research has important implications for future research and practice in adolescent injury prevention.

Curriculum evaluation : approaches and methodologies

Curriculum evaluation, as a field of study, is dynamic. For over the years it has been responsive to the developments in the conceptualization of curriculum and the associated processes of curriculum change. The concept of curriculum is integral to curriculum evaluation and can be defined in terms of what can and shall be taught to whom, when, where, how, and why. Much of the decision making relates to what knowledge is to be selected for inclusion in the curriculum.

Critical reflections on the use of participatory methodologies to build evaluation capacities in international development organisations

There has been a greater focus on strengthening evaluation capacity building (ECB) within development organisations in recent years. This can be attributed in part to the growing appreciation of the value of participatory and collaborative forms of evaluation. Evaluation is increasingly seen as an ongoing learning process and an important means of strengthening capacity and improving organisational performance (Horton et al., 2003:7). While there are many benefits of using participatory methodologies in ECB projects, our experiences and a review of the literature in this area highlight the many challenges, issues and contradictions that can affect the success of such ECB efforts. We discuss these issues, drawing on our learnings from the ongoing participatory action research (PAR) project 'Assessing Communication for Social Change’ (AC4SC). This four year project, which began in 2007, is a collaboration between communication and development academics and evaluation specialists from two Australian universities and communication for development practitioners and monitoring and evaluation (M&E) staff in the NGO Equal Access Nepal (EAN). The aim is to develop, implement, and evaluate a participatory methodology for assessing the social change impacts of community radio programs produced by EAN. It builds on previous projects that used ethnographic action research (EAR) methodology (Tacchi et al., 2007).

Critical reflections on the use of participatory methodologies to build evaluation capacities in international development organisations

"In this chapter the authors present a critique of Participatory Evaluation as worked in development projects, in this case, in Nepal. The article works between established claims that Participatory Evaluation builds capacity at programmatic and organisational levels, and the specific experiences of these claims in the authors’ current work. They highlight the need to address key difficulties such as high turn-over of staff and resulting loss of capacity to engage in Participatory Evaluation, and the difficulty of communication between academic as compared with local practical wisdoms. A key issue is the challenge of addressing the inevitable issues of power inequities that such approaches encounter. While Participatory Evaluation has been around for some time, it has only enjoyed more widespread recognition of its value in comparatively recent times, with its uptake in international development environments. To this extent, the practice is still in its early stages of development, and Jo, June and Michael’s work contributes to strengthening and more comprehensively understanding it. With regard to the meta-theme of this publication, this chapter is an example of how context not only influences the methodology to be used and the praxis of how it is to be used, but contributes to early explication of the core nature of an emerging methodology."

Empirical evaluation of a service analysis and design methodology

Service-orientation has gained widespread acceptance and is increasingly being employed as a paradigm for structuring both business and IT architectures. An earlier study of extant service analysis and design methodologies discovered a need for holistic approaches that equally account for both business and software services, which motivated the design of a new, consolidated service analysis and design methodology. A challenge in design-oriented research is to evaluate the utility of the newly created artefacts (here: the methodology), as they are often intended to become part of complex socio-technical systems. Therefore, after presenting a brief overview of the consolidated methodology, the paper discusses possible approaches for the “evaluate” phase of this design-science research process and presents the results of an empirical evaluation conducted in an Action Research study at one of Australia’s largest financial services providers.

Evaluation of accessibility for knowledge-based cities

Purpose: The purpose of the paper is to develop a framework for evaluation of accessibility for knowledge based cities. ----- ----- Design/methodology/approach: This approach notifies common mistakes and problems in accessibility assessment for knowledge cities. ----- ----- Originality/value: Accessibility plays a key role in transport sustainability and recognizes the crucial links between transport and sustainable goals like air quality, environmental resource consumption & social equity. In knowledge cities, accessibility has significant effects on quality of life and social equity by improving the mobility of people and goods. Accessibility also influences patterns of growth and economic health by providing access to land. Accessibility is not only one of the components of knowledge cities but also affects other elements of knowledge cities directly or indirectly. ----- ----- Practical implications: The outcomes of the application will be helpful for developing particular methodologies for evaluating knowledge cities. On other words, this methodology attempts to develop an assessment procedure for examining accessibility of knowledge-based cities.

The evaluation of qualitative methods selection in the field of design and emotion

The aim of this paper is to aid researchers in selecting appropriate qualitative methods in order to develop and improve future studies in the field of emotional design. These include observations, think-aloud protocols, questionnaires, diaries and interviews. Based on the authors’ experiences, it is proposed that the methods under review can be successfully used for collecting data on emotional responses to evaluate user product relationships. This paper reviews the methods; discusses the suitability, advantages and challenges in relation to design and emotion studies. Furthermore, the paper outlines the potential impact of technology on the application of these methods, discusses the implications of these methods for emotion research and concludes with recommendations for future work in this area.

Evaluation of green public road procurement in Australia : current practices and gaps to fill

Public road authorities have a key responsibility in driving initiatives for reducing greenhouse gas (GHG) emissions in the road construction project lifecycle. A coherent and efficient chain of procurement processes and methods is needed to convert green policies into tangible actions that capture the potential for GHG reduction. Yet, many infrastructure clients lack developed methodologies regarding green procurement practices. Designing more efficient solutions for green procurement requires an evaluation of the current initiatives and stages of development. A mapping of the current GHG reduction initiatives in Australian public road procurement is presented in this paper. The study includes the five largest Australian state road authorities, which cover 94% of the total 817,089 km of Australian main roads (not local) and account for 96% of the total A$13 billion annual major road construction and maintenance expenditure. The state road authorities’ green procurement processes and tools are evaluated based on interviews and a review of documents. Altogether 12 people, comprising 1-3 people of each organisation, participated in the interviews and provided documents. An evaluation matrix was developed for mapping the findings across the lifecycle of road construction project delivery. The results show how Australian state road authorities drive decisions with an impact on GHG emissions on the strategic planning phase, project development phase, and project implementation phase. The road authorities demonstrate varying levels of advancement in their green procurement methodologies. Six major gaps in the current green procurement processes are identified and, respectively, six recommendations for future research and development are suggested. The greatest gaps remain in the project development phase, which has a critical role in fixing the project (GHG reduction) goals, identifying risks and opportunities, and selecting the contractor to deliver the project. Specifically, the role of mass-haul optimisation as a part of GHG minimisation was reviewed, and mass-haul management was found to be an underutilised element with GHG reduction potential.