New evaluation methodologies for public art programs in cultural instiutions

I put my abstract in for this conference back in March, based on some evaluation work I had been doing in 2010 with my colleague Professor Greg Hearn for the 3C Regional Writing NeoGreography Project. I had been swapping notes with a colleague from the Smithsonian’s Centre for Folklife and Cultural Heritage about their evaluation work, and stuck inside during the rains of January, I decided to apply for a Qld Smithsonian fellowship based on the quandary of evaluation-particular in public histories (oral histories) and digital storytelling. In July I was awarded the fellowship, so I have tweaked my presentation to talk about what we hope to do with this collaboration, to propel the importance placed on evaluation in public arts programs in Qld and beyond.

A review of project evaluation methodologies to address net impacts and risks of toll road projects to the community

Project evaluation is a process of measuring costs, benefits, risks and uncertainties for the purpose of decision-making by estimating and assessing impacts of the project to the community. The effects of impacts of toll roads are similar but different from the general non-tolled roads. Project evaluation methodologies are extensively studied and applied to various transport infrastructure projects. However, there is no definitive methodology to evaluate toll roads. This review discusses the impacts of toll roads then reviews the limitations of existing project evaluation methodologies when evaluating toll road impacts. The review identified gaps of knowledge of toll evaluations. First, the treatment of toll in project evaluation, particularly in Cost-Benefit Analysis requires further study to explore the appropriate methodology. Secondly, the project evaluation methodology needs to place strong emphasis on empirically based risk and uncertainty assessment. Addressing the limitations of the existing project evaluation methodologies leads to improvements of the methodology in practical level as well as fills the gap of knowledge of project evaluation for toll roads with respect to net impacts to the community.

A review of project evaluation methodologies for toll road projects

Project evaluation is a process of measuring costs, benefits, risks and uncertainties for the purpose of decision-making by estimating and assessing impacts of the project to the community. The effects of impacts of toll roads are similar but different from the general non-tolled roads. Project evaluation methodologies are extensively studied and applied to various transport infrastructure projects. However, there is no definitive methodology to evaluate toll roads. This review discusses the impacts of toll roads then reviews the limitations of existing project evaluation methodologies when evaluating toll road impacts. The review identified gaps of knowledge of toll evaluations. First, the treatment of toll in project evaluation, particularly in Cost-Benefit Analysis requires further study to explore the appropriate methodology. Secondly, the project evaluation methodology needs to place strong emphasis on empirically based risk and uncertainty assessment. Addressing the limitations of the existing project evaluation methodologies leads to improvements of the methodology in practical level as well as fills the gap of knowledge of project evaluation for toll roads with respect to net impacts to the community.

Evaluation methodologies for four federal motor vehicle safety standards. FMVSS 214: side door strength. FMVSS 301: fuel system integrity. FMVSS 215: exterior protection. FMVSS 208: occupant crash protection. Final report.

National Highway Traffic Safety Administration, Washington, D.C.

Evaluation methodologies for nine Federal Motor Vehicle Safety Standards: FMVSS 105, 108, 122, 202, 207, 213, 220, 221, 222. Final report.

National Highway Traffic Safety Administration, Washington, D.C.

Evaluation in health outcomes research : linking theories, methodologies and practice in health promotion.

Health outcomes research has developed as a means of evaluating the effectiveness of health care interventions and as an approach to informing resource allocation. The use of a health outcomes approach in health promotion has made increasing demands on evaluation methodologies to demonstrate program effectiveness. However, criticism of the contribution of health promotion to outcomes research has made several assumptions about the use of qualitative methodologies and the content of program objectives largely derived from a biomedical approach. In contrast to the measurement of biomedical interventions in clinical health care, health promotion practice involves social phenomena, wide-reaching cultural, psychological, political and ideological problems and issues. The integration of methodologies of health promotion evaluation will inform further conceptualisation of the health outcomes approach with the differentiation of three types of outcomes: health development outcomes; social health outcomes; and biomedical health outcomes. It is concluded that this differentiation moves away from dualist concepts that advocate the replacement of goals and targets with regional and locally based approaches. Rather, the future direction for health promotion evaluation needs to employ a framework that elaborates multiple methodologies and approaches necessary for establishing what relationships exist between morbidity, mortality, health advancement and equity.

Performance Evaluation of Distance-Hop Proportionality on Geometric Graph Models of Dense Sensor Networks

Wireless sensor networks can often be viewed in terms of a uniform deployment of a large number of nodes on a region in Euclidean space, e.g., the unit square. After deployment, the nodes self-organise into a mesh topology. In a dense, homogeneous deployment, a frequently used approximation is to take the hop distance between nodes to be proportional to the Euclidean distance between them. In this paper, we analyse the performance of this approximation. We show that nodes with a certain hop distance from a fixed anchor node lie within a certain annulus with probability approach- ing unity as the number of nodes n → ∞. We take a uniform, i.i.d. deployment of n nodes on a unit square, and consider the geometric graph on these nodes with radius r(n) = c q ln n n . We show that, for a given hop distance h of a node from a fixed anchor on the unit square,the Euclidean distance lies within [(1−ǫ)(h−1)r(n), hr(n)],for ǫ > 0, with probability approaching unity as n → ∞.This result shows that it is more likely to expect a node, with hop distance h from the anchor, to lie within this an- nulus centred at the anchor location, and of width roughly r(n), rather than close to a circle whose radius is exactly proportional to h. We show that if the radius r of the ge- ometric graph is fixed, the convergence of the probability is exponentially fast. Similar results hold for a randomised lattice deployment. We provide simulation results that il- lustrate the theory, and serve to show how large n needs to be for the asymptotics to be useful.

Evaluation of Earth system models using modern and palaeo-observations: the state-of-the-art

Earth system models are increasing in complexity and incorporating more processes than their predecessors, making them important tools for studying the global carbon cycle. However, their coupled behaviour has only recently been examined in any detail, and has yielded a very wide range of outcomes, with coupled climate-carbon cycle models that represent land-use change simulating total land carbon stores by 2100 that vary by as much as 600 Pg C given the same emissions scenario. This large uncertainty is associated with differences in how key processes are simulated in different models, and illustrates the necessity of determining which models are most realistic using rigorous model evaluation methodologies. Here we assess the state-of-the-art with respect to evaluation of Earth system models, with a particular emphasis on the simulation of the carbon cycle and associated biospheric processes. We examine some of the new advances and remaining uncertainties relating to (i) modern and palaeo data and (ii) metrics for evaluation, and discuss a range of strategies, such as the inclusion of pre-calibration, combined process- and system-level evaluation, and the use of emergent constraints, that can contribute towards the development of more robust evaluation schemes. An increasingly data-rich environment offers more opportunities for model evaluation, but it is also a challenge, as more knowledge about data uncertainties is required in order to determine robust evaluation methodologies that move the field of ESM evaluation from "beauty contest" toward the development of useful constraints on model behaviour.

Quality of experience evaluation of voice communication: an affect-based approach

Voice communication systems such as Voice-over IP (VoIP), Public Switched Telephone Networks, and Mobile Telephone Networks, are an integral means of human tele-interaction. These systems pose distinctive challenges due to their unique characteristics such as low volume, burstiness and stringent delay/loss requirements across heterogeneous underlying network technologies. Effective quality evaluation methodologies are important for system development and refinement, particularly by adopting user feedback based measurement. Presently, most of the evaluation models are system-centric (Quality of Service or QoS-based), which questioned us to explore a user-centric (Quality of Experience or QoE-based) approach as a step towards the human-centric paradigm of system design. We research an affect-based QoE evaluation framework which attempts to capture users' perception while they are engaged in voice communication. Our modular approach consists of feature extraction from multiple information sources including various affective cues and different classification procedures such as Support Vector Machines (SVM) and k-Nearest Neighbor (kNN). The experimental study is illustrated in depth with detailed analysis of results. The evidences collected provide the potential feasibility of our approach for QoE evaluation and suggest the consideration of human affective attributes in modeling user experience.

Nation & internation practices in decision support tools in road asset management

This document provides a review of international and national practices in investment decision support tools in road asset management. Efforts were concentrated on identifying analytic frameworks, evaluation methodologies and criteria adopted by current tools. Emphasis was also given to how current approaches support Triple Bottom Line decision-making. Benefit Cost Analysis and Multiple Criteria Analysis are principle methodologies in supporting decision-making in Road Asset Management. The complexity of the applications shows significant differences in international practices. There is continuing discussion amongst practitioners and researchers regarding to which one is more appropriate in supporting decision-making. It is suggested that the two approaches should be regarded as complementary instead of competitive means. Multiple Criteria Analysis may be particularly helpful in early stages of project development, say strategic planning. Benefit Cost Analysis is used most widely for project prioritisation and selecting the final project from amongst a set of alternatives. Benefit Cost Analysis approach is useful tool for investment decision-making from an economic perspective. An extension of the approach, which includes social and environmental externalities, is currently used in supporting Triple Bottom Line decision-making in the road sector. However, efforts should be given to several issues in the applications. First of all, there is a need to reach a degree of commonality on considering social and environmental externalities, which may be achieved by aggregating the best practices. At different decision-making level, the detail of consideration of the externalities should be different. It is intended to develop a generic framework to coordinate the range of existing practices. The standard framework will also be helpful in reducing double counting, which appears in some current practices. Cautions should also be given to the methods of determining the value of social and environmental externalities. A number of methods, such as market price, resource costs and Willingness to Pay, are found in the review. The use of unreasonable monetisation methods in some cases has discredited Benefit Cost Analysis in the eyes of decision makers and the public. Some social externalities, such as employment and regional economic impacts, are generally omitted in current practices. This is due to the lack of information and credible models. It may be appropriate to consider these externalities in qualitative forms in a Multiple Criteria Analysis. Consensus has been reached in considering noise and air pollution in international practices. However, Australia practices generally omitted these externalities. Equity is an important consideration in Road Asset Management. The considerations are either between regions, or social groups, such as income, age, gender, disable, etc. In current practice, there is not a well developed quantitative measure for equity issues. More research is needed to target this issue. Although Multiple Criteria Analysis has been used for decades, there is not a generally accepted framework in the choice of modelling methods and various externalities. The result is that different analysts are unlikely to reach consistent conclusions about a policy measure. In current practices, some favour using methods which are able to prioritise alternatives, such as Goal Programming, Goal Achievement Matrix, Analytic Hierarchy Process. The others just present various impacts to decision-makers to characterise the projects. Weighting and scoring system are critical in most Multiple Criteria Analysis. However, the processes of assessing weights and scores were criticised as highly arbitrary and subjective. It is essential that the process should be as transparent as possible. Obtaining weights and scores by consulting local communities is a common practice, but is likely to result in bias towards local interests. Interactive approach has the advantage in helping decision-makers elaborating their preferences. However, computation burden may result in lose of interests of decision-makers during the solution process of a large-scale problem, say a large state road network. Current practices tend to use cardinal or ordinal scales in measure in non-monetised externalities. Distorted valuations can occur where variables measured in physical units, are converted to scales. For example, decibels of noise converts to a scale of -4 to +4 with a linear transformation, the difference between 3 and 4 represents a far greater increase in discomfort to people than the increase from 0 to 1. It is suggested to assign different weights to individual score. Due to overlapped goals, the problem of double counting also appears in some of Multiple Criteria Analysis. The situation can be improved by carefully selecting and defining investment goals and criteria. Other issues, such as the treatment of time effect, incorporating risk and uncertainty, have been given scant attention in current practices. This report suggested establishing a common analytic framework to deal with these issues.

The impact of red light cameras on safety in Arizona

Red light cameras (RLCs) have been used in a number of US cities to yield a demonstrable reduction in red light violations; however, evaluating their impact on safety (crashes) has been relatively more difficult. Accurately estimating the safety impacts of RLCs is challenging for several reasons. First, many safety related factors are uncontrolled and/or confounded during the periods of observation. Second, “spillover” effects caused by drivers reacting to non-RLC equipped intersections and approaches can make the selection of comparison sites difficult. Third, sites selected for RLC installation may not be selected randomly, and as a result may suffer from the regression to the mean bias. Finally, crash severity and resulting costs need to be considered in order to fully understand the safety impacts of RLCs. Recognizing these challenges, a study was conducted to estimate the safety impacts of RLCs on traffic crashes at signalized intersections in the cities of Phoenix and Scottsdale, Arizona. Twenty-four RLC equipped intersections in both cities are examined in detail and conclusions are drawn. Four different evaluation methodologies were employed to cope with the technical challenges described in this paper and to assess the sensitivity of results based on analytical assumptions. The evaluation results indicated that both Phoenix and Scottsdale are operating cost-effective installations of RLCs: however, the variability in RLC effectiveness within jurisdictions is larger in Phoenix. Consistent with findings in other regions, angle and left-turn crashes are reduced in general, while rear-end crashes tend to increase as a result of RLCs.

The hydrodynamic and capture/retention performance of a gross pollutant trap

This research shows that gross pollutant traps (GPTs) continue to play an important role in preventing visible street waste—gross pollutants—from contaminating the environment. The demand for these GPTs calls for stringent quality control and this research provides a foundation to rigorously examine the devices. A novel and comprehensive testing approach to examine a dry sump GPT was developed. The GPT is designed with internal screens to capture gross pollutants—organic matter and anthropogenic litter. This device has not been previously investigated. Apart from the review of GPTs and gross pollutant data, the testing approach includes four additional aspects to this research, which are: field work and an historical overview of street waste/stormwater pollution, calibration of equipment, hydrodynamic studies and gross pollutant capture/retention investigations. This work is the first comprehensive investigation of its kind and provides valuable practical information for the current research and any future work pertaining to the operations of GPTs and management of street waste in the urban environment. Gross pollutant traps—including patented and registered designs developed by industry—have specific internal configurations and hydrodynamic separation characteristics which demand individual testing and performance assessments. Stormwater devices are usually evaluated by environmental protection agencies (EPAs), professional bodies and water research centres. In the USA, the American Society of Civil Engineers (ASCE) and the Environmental Water Resource Institute (EWRI) are examples of professional and research organisations actively involved in these evaluation/verification programs. These programs largely rely on field evaluations alone that are limited in scope, mainly for cost and logistical reasons. In Australia, evaluation/verification programs of new devices in the stormwater industry are not well established. The current limitations in the evaluation methodologies of GPTs have been addressed in this research by establishing a new testing approach. This approach uses a combination of physical and theoretical models to examine in detail the hydrodynamic and capture/retention characteristics of the GPT. The physical model consisted of a 50% scale model GPT rig with screen blockages varying from 0 to 100%. This rig was placed in a 20 m flume and various inlet and outflow operating conditions were modelled on observations made during the field monitoring of GPTs. Due to infrequent cleaning, the retaining screens inside the GPTs were often observed to be blocked with organic matter. Blocked screens can radically change the hydrodynamic and gross pollutant capture/retention characteristics of a GPT as shown from this research. This research involved the use of equipment, such as acoustic Doppler velocimeters (ADVs) and dye concentration (Komori) probes, which were deployed for the first time in a dry sump GPT. Hence, it was necessary to rigorously evaluate the capability and performance of these devices, particularly in the case of the custom made Komori probes, about which little was known. The evaluation revealed that the Komori probes have a frequency response of up to 100 Hz —which is dependent upon fluid velocities—and this was adequate to measure the relevant fluctuations of dye introduced into the GPT flow domain. The outcome of this evaluation resulted in establishing methodologies for the hydrodynamic measurements and gross pollutant capture/retention experiments. The hydrodynamic measurements consisted of point-based acoustic Doppler velocimeter (ADV) measurements, flow field particle image velocimetry (PIV) capture, head loss experiments and computational fluid dynamics (CFD) simulation. The gross pollutant capture/retention experiments included the use of anthropogenic litter components, tracer dye and custom modified artificial gross pollutants. Anthropogenic litter was limited to tin cans, bottle caps and plastic bags, while the artificial pollutants consisted of 40 mm spheres with a range of four buoyancies. The hydrodynamic results led to the definition of global and local flow features. The gross pollutant capture/retention results showed that when the internal retaining screens are fully blocked, the capture/retention performance of the GPT rapidly deteriorates. The overall results showed that the GPT will operate efficiently until at least 70% of the screens are blocked, particularly at high flow rates. This important finding indicates that cleaning operations could be more effectively planned when the GPT capture/retention performance deteriorates. At lower flow rates, the capture/retention performance trends were reversed. There is little difference in the poor capture/retention performance between a fully blocked GPT and a partially filled or empty GPT with 100% screen blockages. The results also revealed that the GPT is designed with an efficient high flow bypass system to avoid upstream blockages. The capture/retention performance of the GPT at medium to high inlet flow rates is close to maximum efficiency (100%). With regard to the design appraisal of the GPT, a raised inlet offers a better capture/retention performance, particularly at lower flow rates. Further design appraisals of the GPT are recommended.

Development of school connectedness as a component of an injury prevention program for early adolescents

The current program of research addresses the need for multi-level programs to target the major increase in injury rates that occurs throughout adolescence. Specifically, it involves the investigation of school connectedness as a protective factor for adolescent injury, and the development of school connectedness as a component of an injury prevention program. To date, school-based risk taking and injury prevention has frequently been limited to addressing adolescents' knowledge and attitudes to risk behaviours, and has largely overlooked the importance of the wider school social context as a protective factor in adolescent development. Additionally, school connectedness has been primarily studied in terms of its impact on student achievement, wellbeing and risk taking behaviour, and research has not yet addressed possible links with injury. Further, school connectedness intervention programs have targeted risk taking behaviours without evaluating their potential impact on injury outcomes. This is the first reported research to develop strategies to increase school connectedness as part of a school-based injury prevention program. The research program was conceptualised as three distinct stages. The development of these research stages was informed by a comprehensive review of the literature on adolescent risk taking, injury and school-based prevention, as well as on school connectedness and its importance in adolescence. A review of the school connectedness literature indicated that students' connectedness is largely influenced by relationships within the school context including with teachers and other school staff, and is therefore a potentially modifiable factor that may be targeted in school-based programs. Overall, the literature shows school connectedness to be a key protective factor in adolescent development. This review established a foundation from which the current program of research was designed. The first stage of the research involved an empirical investigation of the relationship between adolescent risk taking-related injuries and school connectedness. Stage one incorporated two studies. The first involved the development of a measure of adolescent injury, the Extended Adolescent Injury Checklist (E-AIC), for use in the current research as well as in future school-based studies and program evaluation. The results of this study also highlighted the extent of the problem of risk-related injury in adolescence. The second study in Stage one examined the relationship between students' reports of school connectedness, risk taking behaviour and risk taking-related injuries on the E-AIC. The results of this study showed significant relationships between increased school connectedness and reduced reported engagement in transport and violence risk taking, and fewer associated injuries. This study therefore suggested the potential for school-based injury prevention programs to incorporate strategies targeting increased adolescent connectedness to school. The second stage of this research involved the compilation of an evidence base to inform the design of a school connectedness intervention. Stage two also incorporated two studies. The first study in Stage two involved a systematic review of programs that have targeted school connectedness for reduced risk taking and injury. The results of this study revealed that interventions targeting school connectedness can be effective in reducing adolescent risk taking behaviour, and also provided an evidence base for the design of the current school connectedness intervention. The second study in Stage two examined teachers' understanding and perceptions of school connectedness. This qualitative study indicated that teachers consider students' connectedness to be an important factor that relates to their risk taking behaviour; and also provided directions and content for the intervention design stage. The third stage of this research built upon the findings of each of the previous studies, and involved the design, implementation and evaluation of a school connectedness intervention as a component of an adolescent injury prevention program, Skills for Preventing Injury in Youth (SPIY). This connectedness intervention was designed as a professional development workshop for teachers of 13 to 14 year old adolescents, and was developed as a complementary component to the curriculum-based SPIY program. The SPIY connectedness component was implemented and evaluated using process and six-month impact evaluation methodologies. The results of this study revealed that teachers saw value in the program and made use of the strategies presented, and that program school students' self-reported violence risk behaviour was reduced at six-month follow-up. Despite these promising findings, the results of this study did not demonstrate a significant impact of the program on change in students' connectedness to school, relative to comparison schools. The positive impact on self-reported violence risk behaviour was however replicated in additional analyses comparing students participating in the connectedness version of SPIY with students participating in an earlier curriculumonly version of the program. This finding indicated that the connectedness component has additional benefits relating to reduction in violence risks, over and above a curriculum-only version of the program. This research was the first reported to address the relationship between school connectedness and adolescent injury outcomes, and to develop school connectedness as a component of an adolescent injury prevention program. Overall, the results of this program of research have demonstrated the importance of incorporating strategies targeting the wider school social context, including school connectedness, in adolescent injury prevention programs. This research has important implications for future research and practice in adolescent injury prevention.

A simplified procedure for sizing solar thermal systems based on national assessment methods in the UK and Ireland.

Solar heating systems have the potential to be an efficient renewable energy technology, provided they are sized correctly. Sizing a solar thermal system for domestic applications does not warrant the cost of a simulation. As a result simplified sizing procedures are required. The size of a system depends on a number of variables including the efficiency of the collector itself, the hot water demand and the solar radiation at a given location. Domestic Hot Water (DHW) demand varies with time and is assessed using a multi-parameter detailed model. Secondly, the national energy evaluation methodologies are evaluated from the perspective of solar thermal system sizing. Based on the assessment of the standards, limitations in the evaluation method for solar thermal systems are outlined and an adapted method, specific to the sizing of solar thermal systems, is proposed. The methodology is presented for two common dwelling scenarios. Results from this showed that it is difficult to achieve a high solar fraction given practical sizes of system infrastructure (storage tanks) for standard domestic properties. However, solar thermal systems can significantly offset energy loads due associated DHW consumption, particularly when sized appropriately. The presented methodology is valuable for simple solar system design and also for the quick comparison of salient criteria.