Highway safety management process - evaluation manual.

Mode of access: Internet.

Parent involvement on school advisory councils: A process evaluation at the middle school level

Increasing parental involvement was made an important goal for all Florida schools in educational reform legislation in the 1990's. A forum for this input was established and became known as the School Advisory Council (SAC). To demonstrate the importance of process and inclusion, a south Florida school district and its local teacher's union agreed on the following five goals for SACs: (a) to foster an environment of professional collaboration among all stakeholders, (b) to assist in the preparation and evaluation of the school improvement plan, (c) to address all state and district goals, (d) to serve as the avenue for authentic and representative input from all stakeholders, and (e) to ensure the continued existence of the consensus-building process on all issues related to the school's instructional program. ^ The purpose of this study was to determine to what extent and in what ways the parent members of one south Florida middle school's SAC achieved the five district goals during its first three years of implementation. The primary participants were 16 parents who served as members of the SAC, while 16 non-parent members provided perspective on parent involvement as “outside sources.” Being qualitative by design, factors such as school climate, leadership styles, and the quality of parental input were described from data collected from four sources: parent interviews, a questionnaire of non-parents, researcher observations, and relevant documents. A cross-case analysis of all data informed a process evaluation that described the similarities and differences of intended and observed outcomes of parent involvement from each source using Stake's descriptive matrix model. A formative evaluation of the process compared the observed outcomes with standards set for successful SACs, such as the district's five goals. ^ The findings indicated that parents elected to the SACs did not meet the intended goals set by the state and district. The school leadership did not foster an environment of professional collaboration and authentic decision-making for parents and other stakeholders. The overall process did not include consensus-building, and there was little if any input by parents on school improvement and other important issues relating to the instructional program. Only two parents gave the SAC a successful rating for involving parents in the decision-making process. Although compliance was met in many of the procedural transactions of the SAC, the reactions of parents to their perceived role and influence often reflected feelings of powerlessness and frustration with a process that many thought lacked meaningfulness and productivity. Two conclusions made from this study are as follows: (a) that the role of the principal in the collaborative process is pivotal, and (b) that the normative-re-educative approach to change would be most appropriate for SACs. ^

Parent involvement on school advisory councils: a process evaluation at the middle school level

Increasing parental involvement was made an important goal for all Florida schools in educational reform legislation in the 1990's. A forum for this input was established and became known as the School Advisory Council (SAC). To demonstrate the importance of process and inclusion, a south Florida school district and its local teacher's union agreed on the following five goals for SACs: (a) to foster an environment of professional collaboration among all stakeholders, (b) to assist in the preparation and evaluation of the school improvement plan, (c) to address all state and district goals, (d) to serve as the avenue for authentic and representative input from all stakeholders, and (e) to ensure the continued existence of the consensus-building process on all issues related to the school's instructional program. The purpose of this study was to determine to what extent and in what ways the parent members of one south Florida middle school's SAC achieved the five district goals during its first three years of implementation. The primary participants were 16 parents who served as members of the SAC, while 16 non-parent members provided perspective on parent involvement as "outside sources." Being qualitative by design, factors such as school climate, leadership styles, and the quality of parental input were described from data collected from four sources: parent interviews, a questionnaire of non-parents, researcher observations, and relevant documents. A cross-case analysis of all data informed a process evaluation that described the similarities and differences of intended and observed outcomes of parent involvement from each source using Stake's descriptive matrix model. A formative evaluation of the process compared the observed outcomes with standards set for successful SACs, such as the district's five goals. The findings indicated that parents elected to the SACs did not meet the intended goals set by the state and district. The school leadership did not foster an environment of professional collaboration and authentic decision-making for parents and other stakeholders. The overall process did not include consensus-building, and there was little if any input by parents on school improvement and other important issues relating to the instructional program. Only two parents gave the SAC a successful rating for involving parents in the decision-making process. Although compliance was met in many of the procedural transactions of the SAC, the reactions of parents to their perceived role and influence often reflected feelings of powerlessness and frustration with a process that many thought lacked meaningfulness and productivity. Two conclusions made from this study are as follows: (a) that the role of the principal in the collaborative process is pivotal, and (b) that the normative-re-educative approach to change would be most appropriate for SACs.

A process evaluation of a cluster randomised trial to reduce potentially inappropriate prescribing in older people in primary care (OPTI-SCRIPT study)

Background
The OPTI-SCRIPT cluster randomised controlled trial (RCT) found that a three-phase multifaceted intervention including academic detailing with a pharmacist, GP-led medicines reviews, supported by web-based pharmaceutical treatment algorithms, and tailored patient information leaflets, was effective in reducing potentially inappropriate prescribing (PIP) in Irish primary care. We report a process evaluation exploring the implementation of the intervention, the experiences of those participating in the study and lessons for future implementation.

Methods
The OPTI-SCRIPT trial included 21 GP practices and 196 patients. The process evaluation used mixed methods. Quantitative data were collected from all GP practices and semi-structured interviews were conducted with GPs from intervention and control groups, and a purposive sample of patients from the intervention group. All interviews were transcribed verbatim and analysed using a thematic analysis.

Results
Despite receiving a standardised academic detailing session, intervention delivery varied among GP practices. Just over 70 % of practices completed medicines review as recommended with the patient present. Only single-handed practices conducted reviews without patients present, highlighting the influence of practice characteristics and resources on variation. Medications were more likely to be completely stopped or switched to another more appropriate medication when reviews were conducted with patients present. The patient information leaflets were not used by any of the intervention practices. Both GP (32 %) and patient (40 %) recruitment rates were modest. For those who did participate, overall, the experience was positively viewed, with GPs and patients referring to the value of medication reviews to improve prescribing and reduce unnecessary medications. Lack of time in busy GP practices and remuneration were identified as organisational barriers to future implementation.

Conclusions
The OPTI-SCRIPT intervention was positively viewed by both GPs and patients, both of whom valued the study’s objectives. Patient information leaflets were not a successful component of the intervention. Academic detailing and medication reviews are important components in changing PIP, and having patients present during the review process seems to be a more effective approach for decreasing PIP.

Developing leadership in managers to facilitate the implementation of national guideline recommendations : a process evaluation of feasibility and usefulness

Background: Previous research supports the claim that managers are vital players in the implementation of clinical practice guidelines (CPGs), yet little is known about interventions aiming to develop managers’ leadership in facilitating implementation. In this pilot study, process evaluation was employed to study the feasibility and usefulness of a leadership intervention by exploring the intervention’s potential to support managers in the implementation of national guideline recommendations for stroke care in outpatient rehabilitation. Methods: Eleven senior and frontline managers from five outpatient stroke rehabilitation centers participated in a four-month leadership intervention that included workshops, seminars, and teleconferences. The focus was on developing knowledge and skills to enhance the implementation of CPG recommendations, with a particular focus on leadership behaviors. Each dyad of managers was assigned to develop a leadership plan with specific goals and leadership behaviors for implementing three rehabilitation recommendations. Feasibility and usefulness were explored through observations and interviews with the managers and staff members prior to the intervention, and then one month and one year after the intervention. Results: Managers considered the intervention beneficial, particularly the participation of both senior and frontline managers and the focus on leadership knowledge and skills for implementing CPG recommendations. All the managers developed a leadership plan, but only two units identified goals specific to implementing the three stroke rehabilitation recommendations. Of these, only one identified leadership behaviors that support implementation. Conclusion: Managers found that the intervention was delivered in a feasible way and appreciated the focus on leadership to facilitate implementation. However, the intervention appeared to have limited impact on managers’ behaviors or clinical practice at the units. Future interventions directed towards managers should have a stronger focus on developing leadership skills and behaviors to tailor implementation plans and support implementation of CPG recommendations.

Newham’s Every Child a Sports Person (NECaSP): A Summative Process Evaluation of a School-and-Community Based Intervention in East London, UK

Background: The NECaSP intervention aspires to increase sport and physical activity (PA) participation amongst young people in the UK. The aims of this paper are to report on a summative process evaluation of the NECaSP and make recommendations for future interventions. Methods: Seventeen schools provided data by students aged 11-13 (n=1,226), parents (n=192) and teachers (n= 14) via direct observation and questionnaires. Means, standard deviations and percentages were calculated for socio-demographic data. Qualitative data was analysed via directed content analysis and main themes identified. Results: Findings indicate further administrative, educational and financial support will help facilitate the success of the programme in improving PA outcomes for young people, and of other similar intervention programmes globally. Data highlighted the need to engage parents to increase likelihood of intervention success. Conclusions: One main strength of this study is the mixed-methods nature of the process evaluation. It is recommended that future school based interventions that bridge sports clubs and formal curriculum provision, should consider a more broad approach to the delivery of programmes throughout the academic year, school week and school day. Finally, changes in the school curriculum can be successful once all parties are involved (community, school, families).

Screen Time Weight-loss Intervention Targeting Children at Home (SWITCH): process evaluation of a randomised controlled trial intervention.

BACKGROUND: The Screen Time Weight-loss Intervention Targeting Children at Home (SWITCH) trial tested a family intervention to reduce screen-based sedentary behaviour in overweight children. The trial found no significant effect of the intervention on children's screen-based sedentary behaviour. To explore these null findings, we conducted a pre-planned process evaluation, focussing on intervention delivery and uptake. METHODS: SWITCH was a randomised controlled trial of a 6-month family intervention to reduce screen time in overweight children aged 9-12 years (n = 251). Community workers met with each child's primary caregiver to deliver the intervention content. Community workers underwent standard training and were monitored once by a member of the research team to assess intervention delivery. The primary caregiver implemented the intervention with their child, and self-reported intervention use at 3 and 6 months. An exploratory analysis determined whether child outcomes at 6 months varied by primary caregiver use of the intervention. RESULTS: Monitoring indicated that community workers delivered all core intervention components to primary caregivers. However, two thirds of primary caregivers reported using any intervention component "sometimes" or less frequently at both time points, suggesting that intervention uptake was poor. Additionally, analyses indicated no effect of primary caregiver intervention use on child outcomes at 6 months, suggesting the intervention itself lacked efficacy. CONCLUSIONS: Poor uptake, and the efficacy of the intervention itself, may have played a role in the null findings of the SWITCH trial on health behaviour and body composition. TRIAL REGISTRATION: The trial was registered in the Australian and New Zealand Clinical Trials Registry (no. ACTRN12611000164998 ); registration date: 10/02/2011.

A Process and Outcome Evaluation of the Under the Limit (UTL) Therapeutic Drink Driving Program for Recidivist and High Range Offenders

Drink driving is a major public health issue and this report examines the experiences of convicted offenders who participated in an established drink driving rehabilitation program Under the Limit (UTL). Course completers were surveyed at least three months after they had finished the 11-week UTL course. The aim of this study was to examine whether the UTL program reduced the level of alcohol consumption either directly as a result of participation in the UTL drink driving program or through increased use of community alcohol program by participants. The research involved a self-report outcome evaluation to determine whether the self-reported levels of alcohol use after the course had changed from the initial alcohol use reported by offenders. The findings are based on the responses of 30 drink-driving offenders who had completed the UTL program (response rate: 20%). While a process evaluation was proposed in the initial application, the low response rate meant that this follow up research was not feasible. The response rate was low for two reasons, it was difficult to: recruit participants who consented to follow up, and subsequently locate and survey those who had consented to involvement.

An evaluation framework for educational reform projects for teacher quality improvements in developing countries : a case study of Egyptian education reform

The role of the evaluation for Official Development Assistance (ODA) enterprises including educational development has become critical after increasing “aid fatigue” experienced by the international community in the 1990s. To date, however, monitoring and evaluating outcomes of the projects has been limited to the project life. Consequently these have been mainly through the international aid agencies. Furthermore, the monitoring and evaluation led by international aid agencies have paid little attention to aspects of the sustainability of technical cooperation in educational development. To sustain the impact of technical cooperation, the reinforcement of evaluation has drawn increasing attention in light of the emerging modalities in international development. Therefore this research was inspired to investigate alternative evaluation frameworks for an educational reform project for teacher quality improvement that may increase possibilities for long term sustainability. Importantly, the new modalities in international development and educational issues provide new options. In addition, the research reviewed theoretical and practical issues surrounding evaluation in general, and highlighted the evaluation of education reform projects. The research reported explored via case studies, the evaluation processes employed by the Egyptian education reform projects implemented by the Japan International Cooperation Agency (JICA) and the United Nations Children’s Fund (UNICEF). The case studies used three data sources (archival and relevant documents, a survey questionnaire and interviews) to illuminate the contextually-embedded evaluation processes. The research found that process evaluation is a potential alternative method since it is likely to be locally institutionalised, which may yield long-term sustainability of the projects.

An evaluation framework for sustaining the impact of educational development

Notwithstanding significant efforts by international aid agencies, aid ineffectiveness became apparent in 1990s as the impact of continued development intervention did not endure the expected outcomes. Conventional monitoring and evaluation by those agencies is critiqued for focusing on measuring project outcomes and giving little attention to aspects of sustainability. As a result, devising a rigorous evaluation framework for educational development has been sought in light of recent paradigm shifts in international development. This paper reports on a case study of an Egyptian educational development project highlighting the importance of transforming the evaluation procedures to process evaluation so as to enhance project impact and longevity. This requires building evaluation capacity of the aid recipient country.

Community development with survivors of torture and trauma : an evaluation framework

Despite an increase in community development initiatives in refugee contexts, there is a lack of evaluation frameworks to assess the effectiveness of interventions in the recovery of refugee communities. In response to this gap, the Forum of Australian Services for Survivors of Torture and Trauma has developed an evaluation framework in consultation with refugee client groups and agencies' staff members. This paper contextualizes the goals, principles and strategies of services implementing community development initiatives with torture and trauma survivors and describes the process of developing the framework within a participatory action approach. Both outcome evaluation and process evaluation are discussed, and examples of the framework are presented. Community development agencies and professionals working with survivors of torture and trauma can play a significant role by fostering community empowerment through evaluation.

Pilot evaluation of an adolescent risk and injury prevention programme incorporating curriculum and school connectedness components

School connectedness is an important protective factor for adolescent risk-taking behaviour. This study examined a pilot version of the Skills for Preventing Injury in Youth (SPIY) programme, combining teacher professional development for increasing school connectedness (connectedness component) with a risk and injury prevention curriculum for early adolescents (curriculum component). A process evaluation was conducted on the connectedness component, involving assessments of programme reach, participant receptiveness and initial use, and a preliminary impact evaluation was conducted on the combined connectedness and curriculum programme. The connectedness component was well received by teacher participants, who saw benefits for both themselves and their students. Classroom observation also showed that teachers who received professional development made use of the programme strategies. Grade 8 students who participated in the SPIY programme were less likely to report violent behaviour at six-month follow-up than were control students, and trends also suggested reduced transport injuries. The results of this research support the use of the combined SPIY connectedness and curriculum components in a large-scale effectiveness trial to assess the impact of the programme on students’ connectedness, risk-taking and associated injuries.