Planning and evaluation

- identify the importance of planning and evaluation in public health practice - recognise the links between planning and evaluation through the presentation of relevant models - identify the core concepts of needs assessment in public health - describe the evaluation cycle and the importance of an evaluation plan - understand evaluation designs and their application in practice.

Navigating theory, negotiating difference and beating the system : a critical how-to of media assistance evaluation

The wave of democratisation across Europe, Africa, Asia and Latin America in the early 1990s triggered an increase in donor funding to media assistance initiatives, primarily within good governance policy frameworks. However, few media assistance projects have managed to effectively evaluate the impacts of their work. This thesis explores how the impacts of Australian media assistance on social change and governance can be most effectively evaluated and understood. The findings of this research suggest the importance of early investment in participatory planning of evaluation designs, which are then periodically revisited. These evaluation designs should be based on a theoretically sound link between models of change, evaluative questions and methods.

The use of visual analogue scales to assess motivation to eat in human subjects: a review of their reliability and validity with an evaluation of new hand-held computerized stsytems for temporal tracking of appetite ratings.

This present paper reviews the reliability and validity of visual analogue scales (VAS) in terms of (1) their ability to predict feeding behaviour, (2) their sensitivity to experimental manipulations, and (3) their reproducibility. VAS correlate with, but do not reliably predict, energy intake to the extent that they could be used as a proxy of energy intake. They do predict meal initiation in subjects eating their normal diets in their normal environment. Under laboratory conditions, subjectively rated motivation to eat using VAS is sensitive to experimental manipulations and has been found to be reproducible in relation to those experimental regimens. Other work has found them not to be reproducible in relation to repeated protocols. On balance, it would appear, in as much as it is possible to quantify, that VAS exhibit a good degree of within-subject reliability and validity in that they predict with reasonable certainty, meal initiation and amount eaten, and are sensitive to experimental manipulations. This reliability and validity appears more pronounced under the controlled (but more arti®cial) conditions of the laboratory where the signal : noise ratio in experiments appears to be elevated relative to real life. It appears that VAS are best used in within-subject, repeated-measures designs where the effect of different treatments can be compared under similar circumstances. They are best used in conjunction with other measures (e.g. feeding behaviour, changes in plasma metabolites) rather than as proxies for these variables. New hand-held electronic appetite rating systems (EARS) have been developed to increase reliability of data capture and decrease investigator workload. Recent studies have compared these with traditional pen and paper (P&P) VAS. The EARS have been found to be sensitive to experimental manipulations and reproducible relative to P&P. However, subjects appear to exhibit a signi®cantly more constrained use of the scale when using the EARS relative to the P&P. For this reason it is recommended that the two techniques are not used interchangeably

Information dependencies between architects and services engineers for early design evaluation : a framework for an energy design tool for architects

Effective strategies for the design of effi cient and environmentally sensitive buildings require a close collaboration between architects and engineers in the design of the building shell and environmental control systems at the outset of projects. However, it is often not practical for engineers to be involved early on in the design process. It is therefore essential that architects be able to perform preliminary energy analyses to evaluate their proposed designs prior to the major building characteristics becoming fi xed. Subsequently, a need exists for a simplifi ed energy design tool for architects. This paper discusses the limitations of existing analysis software in supporting early design explorations and proposes a framework for the development of a tool that provides decision support by permitting architects to quickly assess the performance of design alternatives.

Development of optimal designs of insulated rail joints

Proper functioning of Insulated Rail Joints (IRJs) is essential for the safe operation of the railway signalling systems and broken rail identification circuitries. The Conventional IRJ (CIRJ) resembles structural butt joints consisting of two pieces of rails connected together through two joint bars on either side of their web and the assembly is held together through pre-tensioned bolts. As the IRJs should maintain electrical insulation between the two rails, a gap between the rail ends must be retained at all times and all metal contacting surfaces should be electrically isolated from each other using non-conductive material. At the gap, the rail ends lose longitudinal continuity and hence the vertical sections of the rail ends are often severely damaged, especially at the railhead, due to the passage of wheels compared to other continuously welded rail sections. Fundamentally, the reason for the severe damage can be related to the singularities of the wheel-rail contact pressure and the railhead stress. No new generation designs that have emerged in the market to date have focussed on this fundamental; they only have provided attention to either the higher strength materials or the thickness of the sections of various components of the IRJs. In this thesis a novel method of shape optimisation of the railhead is developed to eliminate the pressure and stress singularities through changes to the original sharp corner shaped railhead into an arc profile in the longitudinal direction. The optimal shape of the longitudinal railhead profile has been determined using three nongradient methods in search of accuracy and efficiency: (1) Grid Search Method; (2) Genetic Algorithm Method and (3) Hybrid Genetic Algorithm Method. All these methods have been coupled with a parametric finite element formulation for the evaluation of the objective function for each iteration or generation depending on the search algorithm employed. The optimal shape derived from these optimisation methods is termed as Stress Minimised Railhead (SMRH) in this thesis. This optimal SMRH design has exhibited significantly reduced stress concentration that remains well below the yield strength of the head hardened rail steels and has shifted the stress concentration location away from the critical zone of the railhead end. The reduction in the magnitude and the relocation of the stress concentration in the SMRH design has been validated through a full scale wheel – railhead interaction test rig; Railhead strains under the loaded wheels have been recorded using a non-contact digital image correlation method. Experimental study has confirmed the accuracy of the numerical predications. Although the SMRH shaped IRJs eliminate stress singularities, they can still fail due to joint bar or bolt hole cracking; therefore, another conceptual design, termed as Embedded IRJ (EIRJ) in this thesis, with no joint bars and pre-tensioned bolts has been developed using a multi-objective optimisation formulation based on the coupled genetic algorithm – parametric finite element method. To achieve the required structural stiffness for the safe passage of the loaded wheels, the rails were embedded into the concrete of the post tensioned sleepers; the optimal solutions for the design of the EIRJ is shown to simplify the design through the elimination of the complex interactions and failure modes of the various structural components of the CIRJ. The practical applicability of the optimal shapes SMRH and EIRJ is demonstrated through two illustrative examples, termed as improved designs (IMD1 & IMD2) in this thesis; IMD1 is a combination of the CIRJ and the SMRH designs, whilst IMD2 is a combination of the EIRJ and SMRH designs. These two improved designs have been simulated for two key operating (speed and wagon load) and design (wheel diameter) parameters that affect the wheel-rail contact; the effect of these parameters has been found to be negligible to the performance of the two improved designs and the improved designs are in turn found far superior to the current designs of the CIRJs in terms of stress singularities and deformation under the passage of the loaded wheels. Therefore, these improved designs are expected to provide longer service life in relation to the CIRJs.

Evaluation of robust autonomy and implications on UAS certification and design

As the number of potential applications of Unmanned Aircraft Systems (UAS) grows in civilian operations and national security, National Airworthiness Authorities are under increasing pressure to provide a path for certification and allow UAS integration into the national airspace. The success of this integration depends on developments in improved UAS reliability and safety, regulations for certification, and technologies for operational performance and safety assessment. This paper focusses on the latter and describes the use of a framework for evaluating robust autonomy of UAS, namely, the autonomous system’s ability to either continue operation in the presence of faults or safely shut down. The paper draws parallels between the proposed evaluation framework and the evaluation of pilots during the licensing process. It also discusses how the data from the proposed evaluation can be uses as an aid for decision making in certification and UAS designs.

Educational interventions for preventing vascular catheter bloodstream infections in critical care: Evidence map, systematic review and economic evaluation

Background Bloodstream infections resulting from intravascular catheters (catheter-BSI) in critical care increase patients' length of stay, morbidity and mortality, and the management of these infections and their complications has been estimated to cost the NHS annually £19.1–36.2M. Catheter-BSI are thought to be largely preventable using educational interventions, but guidance as to which types of intervention might be most clinically effective is lacking. Objective To assess the effectiveness and cost-effectiveness of educational interventions for preventing catheter-BSI in critical care units in England. Data sources Sixteen electronic bibliographic databases – including MEDLINE, MEDLINE In-Process & Other Non-Indexed Citations, Cumulative Index to Nursing and Allied Health Literature (CINAHL), NHS Economic Evaluation Database (NHS EED), EMBASE and The Cochrane Library databases – were searched from database inception to February 2011, with searches updated in March 2012. Bibliographies of systematic reviews and related papers were screened and experts contacted to identify any additional references. Review methods References were screened independently by two reviewers using a priori selection criteria. A descriptive map was created to summarise the characteristics of relevant studies. Further selection criteria developed in consultation with the project Advisory Group were used to prioritise a subset of studies relevant to NHS practice and policy for systematic review. A decision-analytic economic model was developed to investigate the cost-effectiveness of educational interventions for preventing catheter-BSI. Results Seventy-four studies were included in the descriptive map, of which 24 were prioritised for systematic review. Studies have predominantly been conducted in the USA, using single-cohort before-and-after study designs. Diverse types of educational intervention appear effective at reducing the incidence density of catheter-BSI (risk ratios statistically significantly < 1.0), but single lectures were not effective. The economic model showed that implementing an educational intervention in critical care units in England would be cost-effective and potentially cost-saving, with incremental cost-effectiveness ratios under worst-case sensitivity analyses of < £5000/quality-adjusted life-year. Limitations Low-quality primary studies cannot definitively prove that the planned interventions were responsible for observed changes in catheter-BSI incidence. Poor reporting gave unclear estimates of risk of bias. Some model parameters were sourced from other locations owing to a lack of UK data. Conclusions Our results suggest that it would be cost-effective and may be cost-saving for the NHS to implement educational interventions in critical care units. However, more robust primary studies are needed to exclude the possible influence of secular trends on observed reductions in catheter-BSI.

Evaluation of a workplace alcohol and other drug policy

BACKGROUND The current impetus for developing alcohol and/or other drugs (AODs) workplace policies in Australia is to reduce workplace AOD impairment, improve safety, and prevent AOD-related injury in the workplace. For these policies to be effective, they need to be informed by scientific evidence. Evidence to inform the development and implementation of effective workplace AOD policies is currently lacking. There does not currently appear to be conclusive evidence for the effectiveness of workplace AOD policies in reducing impairment and preventing AOD-related injury. There is also no apparent evidence regarding which factors facilitate or impede the success of an AOD policy, or whether, for example, unsuccessful policy outcomes were due to poor policy or merely poor implementation of the policy. It was the aim of this research to undertake a process, impact, and outcome evaluation of a workplace AOD policy, and to contribute to the body of knowledge on the development and implementation of effective workplace AOD policies. METHODS The research setting was a state-based power-generating industry in Australia between May 2008 and May 2010. Participants for the process evaluation study were individuals who were integral to either the development or the implementation of the workplace AOD policy, or both of these processes (key informants), and comprised the majority of individuals who were involved in the process of developing and/or implementing the workplace AOD policy. The sample represented the two main groups of interest—management and union delegates/employee representatives—from all three of the participating organisations. For the impact and outcome evaluation studies, the population included all employees from the three participating organisations, and participants were all employees who consented to participate in the study and who completed both the pre-and post-policy implementation questionnaires. Qualitative methods in the form of interviews with key stakeholders were used to evaluate the process of developing and implementing the workplace AOD policy. In order to evaluate the impact of the policy with regard to the risk factors for workplace AOD impairment, and the outcome of the policy in terms of reducing workplace AOD impairment, quantitative methods in the form of a non-randomised single group pre- and post-test design were used. Changes from Time 1 (pre) to Time 2 (post) in the risk factors for workplace AOD impairment, and changes in the behaviour of interest—(self-reported) workplace AOD impairment—were measured. An integration of the findings from the process, impact, and outcome evaluation studies was undertaken using a combination of qualitative and quantitative methods. RESULTS For the process evaluation study Study respondents indicated that their policy was developed in the context of comparable industries across Australia developing workplace AOD policies, and that this was mainly out of concern for the deleterious health and safety impacts of workplace AOD impairment. Results from the process evaluation study also indicated that in developing and implementing the workplace AOD policy, there were mainly ‗winners', in terms of health and safety in the workplace. While there were some components of the development and implementation of the policy that were better done than others, and the process was expensive and took a long time, there were, overall, few unanticipated consequences to implementing the policy and it was reported to be thorough and of a high standard. Findings also indicated that overall the policy was developed and implemented according to best-practice in that: consultation during the policy development phase (with all the main stakeholders) was extensive; the policy was comprehensive; there was universal application of the policy to all employees; changes in the workplace (with regard to the policy) were gradual; and, the policy was publicised appropriately. Furthermore, study participants' responses indicated that the role of an independent external expert, who was trusted by all stakeholders, was integral to the success of the policy. For the impact and outcome evaluation studies Notwithstanding the limitations of pre- and post-test study designs with regard to attributing cause to the intervention, the findings from the impact evaluation study indicated that following policy implementation, statistically significant positive changes with regard to workplace AOD impairment were recorded for the following variables (risk factors for workplace AOD impairment): Knowledge; Attitudes; Perceived Behavioural Control; Perceptions of the Certainty of being punished for coming to work impaired by AODs; Perceptions of the Swiftness of punishment for coming to work impaired by AODs; and Direct and Indirect Experience with Punishment Avoidance for workplace AOD impairment. There were, however, no statistically significant positive changes following policy implementation for Behavioural Intentions, Subjective Norms, and Perceptions of the Severity of punishment for workplace AOD impairment. With regard to the outcome evaluation, there was a statistically significant reduction in self-reported workplace AOD impairment following the implementation of the policy. As with the impact evaluation, these findings need to be interpreted in light of the limitations of the study design in being able to attribute cause to the intervention alone. The findings from the outcome evaluation study also showed that while a positive change in self-reported workplace AOD impairment following implementation of the policy did not appear to be related to gender, age group, or employment type, it did appear to be related to levels of employee general alcohol use, cannabis use, site type, and employment role. Integration of the process, impact, and outcome evaluation studies There appeared to be qualitative support for the relationship between the process of developing and implementing the policy, and the impact of the policy in changing the risk factors for workplace AOD impairment. That is, overall the workplace AOD policy was developed and implemented well and, following its implementation, there were positive changes in the majority of measured risk factors for workplace AOD impairment. Quantitative findings lend further support for a relationship between the process and impact of the policy, in that there was a statistically significant association between employee perceived fidelity of the policy (related to the process of the policy) and positive changes in some risk factors for workplace AOD impairment (representing the impact of the policy). Findings also indicated support for the relationship between the impact of the policy in changing the risk factors for workplace AOD impairment and the outcome of the policy in reducing workplace AOD impairment: positive changes in the risk factors for workplace AOD impairment (impact) were related to positive changes in self reported workplace AOD impairment (representing the main goal and outcome of the policy). CONCLUSIONS The findings from the research indicate support for the conclusion that the policy was appropriately implemented and that it achieved its objectives and main goal. The Doctoral research findings also addressed a number of gaps in the literature on workplace AOD impairment, namely: the likely effectiveness of AOD policies for reducing AOD impairment in the workplace, which factors in the development and implementation of a workplace AOD policy are likely to facilitate or impede the effectiveness of the policy to reduce workplace AOD impairment, and which employee groups are less likely to respond well to policies of this type. The findings from this research not only represent an example of translational, applied research—through the evaluation of the study industry's policy—but also add to the body of knowledge on workplace AOD policies and provide policy-makers with evidence which may be useful in the development and implementation of effective workplace AOD policies. Importantly, the findings espouse the importance of scientific evidence in the development, implementation, and evaluation of workplace AOD policies.