Methods for exploring implementation variation and local context within a cluster tandomised community intervention trial

Insignificant or modest findings in intervention trials may be attributable to poorly designed or theorised interventions, poorly implemented interventions, or inadequate evaluation methods. The pre-existing context may also account for the effects observed. A combination of qualitative and quantitative methods is outlined that will permit the determination of how context level factors might modify intervention effectiveness, within a cluster randomised community intervention trial to promote the health of mothers with new babies. The methods include written and oral narratives, key informant interviews, impact logs, and inter-organisational network analyses. Context level factors, which may affect intervention uptake, success, and sustainability are the density of inter-organisational ties within communities at the start of the intervention, the centrality of the primary care agencies expected to take a lead with the intervention, the extent of context-level adaptation of the intervention, and the amount of local resources contributed by the participating agencies. Investigation of how intervention effects are modified by context is a new methodological frontier in community intervention trial research.

A proposed hybrid evaluation methodology for intelligent transport system deployment

The rapid deployment of intelligent transport systems in practice requires serious evaluation of new construction projects. Traditional transportation evaluation methods are insufficient to capture the new features of intelligent transport systems, particularly the applications of recently developed information technologies. This research aims to develop a new conceptual framework to evaluate the social, economic and environmental impacts of intelligent transport systems deployment, with explicit consideration of impacts of information technologies on variant travel behaviours. The research would provide a systematic evaluation methodology to enable decision makers and publics better evaluating and understanding the social, economic and environmental benefits of implementation of intelligent transport systems.

Assessing cost-effectiveness in obesity (ACE-Obesity) : an overview of the ACE approach, economic methods and cost results

Background : The aim of the ACE-Obesity study was to determine the economic credentials of interventions which aim to prevent unhealthy weight gain in children and adolescents. We have reported elsewhere on the modelled effectiveness of 13 obesity prevention interventions in children. In this paper, we report on the cost results and associated methods together with the innovative approach to priority setting that underpins the ACE-Obesity study.

Methods : The Assessing Cost-Effectiveness (ACE) approach combines technical rigour with 'due process' to facilitate evidence-based policy analysis. Technical rigour was achieved through use of standardised evaluation methods, a research team that assembles best available evidence and extensive uncertainty analysis. Cost estimates were based on pathway analysis, with resource usage estimated for the interventions and their 'current practice' comparator, as well as associated cost offsets. Due process was achieved through involvement of stakeholders, consensus decisions informed by briefing papers and 2nd stage filter analysis that captures broader factors that influence policy judgements in addition to cost-effectiveness results. The 2nd stage filters agreed by stakeholders were 'equity', 'strength of the evidence', 'feasibility of implementation', 'acceptability to stakeholders', 'sustainability' and 'potential for side-effects'.

Results : The intervention costs varied considerably, both in absolute terms (from cost saving [6 interventions] to in excess of AUD50m per annum) and when expressed as a 'cost per child' estimate (from <AUD1.0 [reduction of TV advertising of high fat foods/high sugar drinks] to >AUD31,000 [laparoscopic adjustable gastric banding for morbidly obese adolescents]). High costs per child reflected cost structure, target population and/or under-utilisation.

Conclusions : The use of consistent methods enables valid comparison of potential intervention costs and cost-offsets for each of the interventions. ACE-Obesity informs policy-makers about cost-effectiveness, health impact, affordability and 2nd stage filters for important options for preventing unhealthy weight gain in children. In related articles cost-effectiveness results and second stage filter considerations for each intervention assessed will be presented and analysed.

Toward the standardization of venture capital investment evaluation : decision criteria for rating investee business plans

This study examined the criteria used by venture capitalists to evaluate business plans in order to make investment decisions. A literature survey revealed two competing theories: “espoused criteria” where evaluation decisions are based on what venture capitalists say are the decisive factors; versus the use of “known attributes” that successful ventures actually possess. Brunswik’s Lens Model from Social Judgment Theory guided an empirical investigation of several different evaluation methods based on information contained in 129 business plans submitted for venture capital over a 3 year period. Data evaluation culminated in the comparison of the percentage of correct decisions (“hit-rate”) for each method. We found that decisions based on the known attributes of successful ventures have significantly better hit-rates than decisions made using espoused criteria. Discussion centred on the goal of achieving consistency in the conduct of venture analysis. Process standardization can aid in the achievement of consistency. Future research will both deepen and broaden insights.

Methods of usability evaluations of mobile devices

The usability of applications designed and developed for mobile devices integral user experience, particularly for current touch-screen mobile devices. This conceptual paper provides a profile of the ways in which mobile device usability evaluation methods are being adapted to better reflect technological. With an increasing capacity for greater user interactivity through a range of tasks and in a variety of contexts, the importance and impact of mobile device context-of-use components is of particular interest to researchers. This requires a need for a shift in evaluation method paradigms with a particular focus on methods involving the user, as part of the overall development process. Having a better understanding of the context-of-use may contribute to improvements in mobile device usability and the overall user experience.

Applying systems theory to the evaluation of a whole school approach to violence prevention

Issue addressed: Our Watch led a complex 12-month evaluation of a whole school approach to Respectful Relationships Education (RRE) implemented in 19 schools. RRE is an emerging field aimed at preventing gender-based violence. This paper will illustrate how from an implementation science perspective, the evaluation was a critical element in the change process at both a school and policy level. Methods: Using several conceptual approaches from systems science, the evaluation sought to examine how the multiple systems layers – student, teacher, school, community and government – interacted and influenced each other. A distinguishing feature of the evaluation included ‘feedback loops’; that is, evaluation data was provided to participants as it became available. Evaluation tools included a combination of standardised surveys (with pre- and post-intervention data provided to schools via individualised reports), reflection tools, regular reflection interviews and summative focus groups. Results: Data was shared during implementation with project staff, department staff and schools to support continuous improvement at these multiple systems levels. In complex settings, implementation can vary according to context; and the impact of evaluation processes, tools and findings differed across the schools. Interviews and focus groups conducted at the end of the project illustrated which of these methods were instrumental in motivating change and engaging stakeholders at both a school and departmental level and why. Conclusion: The evaluation methods were a critical component of the pilot’s approach, helping to shape implementation through data feedback loops and reflective practice for ongoing, responsive and continuous improvement. Future health promotion research on complex interventions needs to examine how the evaluation itself is influencing implementation. So what? The pilot has demonstrated that the evaluation, including feedback loops to inform project activity, were an asset to implementation. This has implications for other health promotion activities, where evaluation tools could be utilised to enhance, rather than simply measure, an intervention. The findings are relevant to a range of health promotion research activities because they demonstrate the importance of meta-evaluation techniques that seek to understand how the evaluation itself was influencing implementation and outcomes.

Trial application of a model of resource utiliztion, costs, and outcomes for stroke (MORUCOS) to assist priority setting in stroke.

Background and Purpose— Cost-effectiveness data for stroke interventions are limited, and comparisons between studies are confounded by methodological inconsistencies. The aim of this study was to trial the use of the intervention module of the economic model, a Model of Resource Utilization, Costs, and Outcomes for Stroke (MORUCOS) to facilitate evaluation and ranking of the options.

Methods— The approach involves using an economic model together with added secondary considerations. A consistent approach was taken using standard economic evaluation methods. Data from the North East Melbourne Stroke Incidence Study (NEMESIS) were used to model "current practice" (base case), against which 2 interventions were compared. A 2-stage process was used to measure benefit: health gains (expressed in disability-adjusted life years [DALYs]) and filter analysis. Incremental cost-effectiveness ratios (ICERs) were calculated, and probabilistic uncertainty analysis was undertaken.

Results— Aspirin, a low-cost intervention applicable to a large number of stroke patients (9153 first-ever cases), resulted in modest health benefits (946 DALYs saved) and a mean ICER (based on incidence costs) of US $1421 per DALY saved. Although the health gains from recombinant tissue-type plasminogen activator (rtPA) were less (155 DALYs saved), these results were impressive given the small number of persons (256) eligible for treatment. rtPA dominates current practice because it is more effective and cost-saving.

Conclusions— If used to assess interventions across the stroke care continuum, MORUCOS offers enormous capacity to support decision-making in the prioritising of stroke services.

Measurement of the economic impact of visual impairment from age-related macular degeneration in Australia

The purpose of this report was to: (i) outline the potential value of health economic studies into age-related macular degeneration (AMD); (ii) provide an overview of health economic studies pertinent to AMD; and (iii) outline the basic frame work of cost-of-illness studies (a useful first step in applying economic methods). The detection and management of sensory loss in the elderly plays a key role in the Australian Government's Healthy Ageing Strategy. Age-related macular degeneration is currently the leading cause of blindness in elderly Australians. Although a large proportion of AMD cases remain untreatable, the introduction of photo­dynamic therapy provides a relatively expensive and possibly cost-effective innovation for others. Antioxidant therapy has also been proven effective in reducing progression of early to late disease. The discipline of economics can contribute to an understanding of AMD prevention and treatment through: (i) describing the current burden of disease; (ii) predicting the changes in the burden of disease over time, and (iii) evaluating the efficiency of different interventions. Cost-of-illness studies have been performed in many fields of medicine. Little work, however, has been done on describing the economic impact from AMD. A number of different economic evaluation methods can be used in judging the efficiency of possible interventions to reduce the disease burden of AMD. Although complementary in nature, each has specific uses and limitations. Studies of the economic impact of eye diseases are both feasible and necessary for informed health care decision-making.

Capacity building for integrated health care at the local level

Aims & Rationale/Objectives
Taking a capacity building approach to research and evaluation within the context of a federally funded national program challenges the traditional paradigms of both research and evaluation. The objective of this approach was to foster attitudes and behaviours of reflection, critical inquiry and collaborative action amongst participants responsible for health care integration activities.

Methods
A series of workshops focusing on different elements of health care integration was conducted. Each workshop offered skill development in research and evaluation methods relevant to the participants' clinical practise. The workshops were multidisciplinary and cross-sectoral in order to promote discussion about shared patient care issues.

Principal Findings
Participatory action research facilitated by external agents can build the capacity of participants to identify and make changes that improve health care integration at local levels. A capacity building approach to research and evaluation can mediate tensions between top-down initiatives and on-the-ground practitioners.

Discussion
A capacity building approach was crucial to the success of this project particularly as the project proposal was developed at the corporate level. The workshops played an important role in engaging the participants and fostering the development of solutions for locally identified clinical issues. The opportunities for discussion with other health care service providers were both readily embraced and appreciated by the participants. The networks formed during the workshops are likely to be vital in sustaining integration efforts.

Implications
Education sessions such as the workshops held within this project ensure that health care integration remains on the agenda of the relevant organisations. These workshops fostered a continuous quality improvement approach whilst focusing on the skills required and the systemic barriers to achieving health care integration. The success of these workshops is evidence that the need and desire for shared education opportunities exists and the interdisciplinary focus is a powerful tool for developing an appreciation of the cultures within disciplines as well as linkages.

Caprice versus standardization in venture capital decision making

This study examined the criteria used by venture capitalists to evaluate business plans in order to make investment decisions. A literature survey revealed two competing theories: 'espoused criteria' where evaluation decisions are based on what venture capitalists say are the decisive factors, versus the use of 'known attributes' that successful ventures actually possess. Brunswik's Lens Model from Social Judgment Theory guided an empirical investigation of several different evaluation methods based on information contained in 129 business plans submitted for venture capital over a three-year period. Data evaluation culminated in the comparison of the percentage of correct decisions ('hit rate') for each method. We found that decisions based on the known attributes of successful ventures have significantly better hit rates than decisions made using espoused criteria. Discussion centered on the goal of achieving consistency in the conduct of venture analysis. Process standardization can aid in the achievement of consistency. Future research will both deepen and broaden insights.

Seasonal performance rating of heat pump water heaters

Seasonal performance evaluation methods for water heaters are reviewed and an experimental method for rating air-source heat pump water heaters is presented. The rating method is based on measured heat pump performance during heat-up operation of particular products rather than a generic simulation model of heat pump performance. The measured performance is used in a correlation model of the heat pump unit in an annual load-cycle system performance simulation based on the TRNSYS simulation package.

Using input-output data in life cycle inventory analysis

The impacts on the environment from human activities are of increasing concern. The need to consider the reduction in energy consumption is of particular interest, especially in the construction and operation of buildings, which accounts for between 30 and 40% of Australia's national energy consumption. Much past and more recent emphasis has been placed on methods for reducing the energy consumed in the operation of buildings. With the energy embodied in these buildings having been shown to account for an equally large proportion of a building's life cycle energy consumption, there is a need to look at ways of reducing the embodied energy of buildings and related products. Life cycle assessment (LCA) is considered to be the most appropriate tool for assessing the life cycle energy consumption of buildings and their products. The life cycle inventory analysis (LCIA) step of a LCA, where an inventory of material and energy inputs is gathered, may currently suffer from several limitations, mainly concerned with the use of incomplete and unreliable data sources and LCIA methods. These traditional methods of LCIA include process-based and input-output-based LCIA. Process-based LCIA uses process specific data, whilst input-output-based LCIA uses data produced from an analysis of the flow of goods and services between sectors of the Australian economy, also known as input-output data. With the incompleteness and unreliability of these two respective methods in mind, hybrid LCIA methods have been developed to minimise the errors associated with traditional LCIA methods, combining both process and input-output data. Hybrid LCIA methods based on process data have shown to be incomplete. Hybrid LCIA methods based on input-output data involve substituting available process data into the input-output model minimising the errors associated with process-based hybrid LCIA methods. However, until now, this LCIA method had not been tested for its level of completeness and reliability. The aim of this study was to assess the reliability and completeness of hybrid life cycle inventory analysis, as applied to the Australian construction industry. A range of case studies were selected in order to apply the input-output-based hybrid LCIA method and evaluate the subsequent results as obtained from each case study. These case studies included buildings: two commercial office buildings, two residential buildings, a recreational building; and building related products: a solar hot water system, a building integrated photovoltaic system and a washing machine. The range of building types and products selected assisted in testing the input-output-based hybrid LCIA method for its applicability across a wide range of product types. The input-output-based hybrid LCIA method was applied to each of the selected case studies in order to obtain their respective embodied energy results. These results were then evaluated with the use of a number of evaluation methods. These evaluation methods included an analysis of the difference between the process-based and input-output-based hybrid LCIA results as an evaluation of the completeness of the process-based LCIA method. The second method of evaluation used was a comparison between equivalent process and input-output values used in the input-output-based hybrid LCIA method as a measure of reliability. It was found that the results from a typical process-based LCIA and process-based hybrid LCIA have a large gap when compared to input-output-based hybrid LCIA results (up to 80%). This gap has shown that the currently available quantity of process data in Australia is insufficient. The comparison between equivalent process-based and input-output-based LCIA values showed that the input-output data does not provide a reliable representation of the equivalent process values, for material energy intensities, material inputs and whole products. Therefore, the use of input-output data to account for inadequate or missing process data is not reliable. However, as there is currently no other method for filling the gaps in traditional process-based LCIA, and as input-output data is considered to be more complete than process data, and the errors may be somewhat lower, using input-output data to fill the gaps in traditional process-based LCIA appears to be better than not using any data at all. The input-output-based hybrid LCIA method evaluated in this study has shown to be the most sophisticated and complete currently available LCIA method for assessing the environmental impacts associated with buildings and building related products. This finding is significant as the construction and operation of buildings accounts for a large proportion of national energy consumption. The use of the input-output-based hybrid LCIA method for products other than those related to the Australian construction industry may be appropriate, especially if the material inputs of the product being assessed are similar to those typically used in the construction industry. The input-output-based hybrid LCIA method has been used to correct some of the errors and limitations associated with previous LCIA methods, without the introduction of any new errors. Improvements in current input-output models are also needed, particularly to account for the inclusion of capital equipment inputs (i.e. the energy required to manufacture the machinery and other equipment used in the production of building materials, products etc.). Although further improvements in the quantity of currently available process data are also needed, this study has shown that with the current available embodied energy data for LCIA, the input-output-based hybrid LCIA appears to provide the most reliable and complete method for use in assessing the environmental impacts of the Australian construction industry.

User participation in the performance assessment of school buildings : a study of six primary schools in the Bellarine Peninsula region

Young children spend a significant portion of their lives at primary school. This research traces the development of school facilities in Victoria and examines the performance of six primary schools from the users' perspective. Performance assessments were carried out using participatory evaluation methods that included Touring Interviews with small groups of students aged from six to twelve years. The study found that participatory evaluation methods with both student and staff users generate significant information to improve school facilities. User comments were analysed with respect to 14 aspects of building quality and serviceability including character, thermal environment, privacy and flexibility. The study concludes that school buildings do not meet a number of key user requirements. Children expressed dissatisfaction with furniture and equipment in their classrooms and the playground, and to a lesser extent with student toilets, security of their school bags and personal privacy. Staff were dissatisfied with the provision of withdrawal areas and specialist spaces. Department of School Education facilities guidelines do not address the concerns of school users or meet user needs.

MUSTe method for quantifying virtual environment training system efficacy

In the current era increased attention and interest of utilizing advanced computer technologies for training and education at all managerial levels and functional areas is apparent. One of such technologies, virtual environment (VE), is perceived to be effective in enhancing human abilities to learn abstract concept and complex procedural tasks. Despite its adaptation for training and fast-paced technological advancements, ways in which to evaluate efficacy of such technology are unclear. We have approached this problem by developed a new evaluation method focus on cognitive, affective and skillbased learning dimensions, based on traditional usability evaluation methods but tailored to specifically suit for the quantification of 3D VE system. We first describe the construct of the new method and then report a study utilizing the method in the context of quantifying a VE efficacy in an object assembly task. At last, we discuss the implications of such a method.

A generalised data analysis approach for baggage handling systems simulation

Airport baggage handling systems are a critical infrastructure component within major airports, and essential to ensure smooth luggage transfer while preventing dangerous material being loaded onto aircraft. This paper proposes a standard set of measures to assess the expected performance of a baggage handling system through discrete event simulation. These evaluation methods also have application in the study of general network systems. Results from the application of these methods reveal operational characteristics of the studied BHS, in terms of metrics such as peak throughput, in-system time and system recovery time.