Formative evaluation for improving collaborative planning : a case study at the regional scale

In an earlier paper (Cameron & Johnson 2004) we introduced the idea of formative evaluation (or evaluation for development), the purpose of which is to provide information for improving planning programs and activities. This type of evaluation differs from the two other types: outcome evaluation which aims to judge the success or otherwise of a program; and evaluation for knowledge which seeks to contribute to theoretical work on planning processes and activities. In the earlier paper we also outlined the first stage of formative evaluation in the SEQ 2021 regional planning exercise showing how the process of planning for community engagement was modified in light of the evaluation findings. This current paper details the second stage of formative evaluation in which the collaborative planning component of SEQ 2021 was evaluated, as such it further demonstrates how formative evaluation can be used to improve planning programs. The evaluation findings also provide insights into strategies for more effective collaborative planning. We begin with an overview of collaborative approaches to regional planning, including the SEQ 2021 regional planning program. We then outline formal and informal evaluations of various collaborative regional planning exercises, including the predecessor of SEQ 2021 - SEQ 2001. This sets the scene for discussion of the approach used to evaluate the collaborative component of SEQ 2021. After outlining the main findings from the evaluation and the ways these findings were used to refine the collaborative planning process we conclude with a series of recommendations, relevant not only to SEQ 2021 but to other collaborative planning exercises

Get real : the development, implementation and evaluation of an intervention for primary aged children with High Functioning Autism

Children with Autism Spectrum Disorder experience difficulty in communication and in understanding the social world which can have negative consequences for their relationships, in managing emotions, and generally dealing with the challenges of everyday life. This thesis examines the effectiveness of the Active and Reflective components of the Get REAL program through the assessment of the detailed coding of video-recorded observations and longitudinal quantitative analysis. The aim of Get REAL is to increase the social, emotional, and cognitive learning of children with High Functioning Autism (HFA). Get REAL is a group program designed specifically for use in inclusive primary school settings. The Get REAL program was designed in response to the mixed success of generalisation of learning to new contexts of existing social skills programs. The theoretical foundation of Get REAL is based upon pedagogical theory and learning theory to facilitate transfer of learning, combined with experiential, individualised, evaluative and organisational approaches. This thesis is by publication and consists of four refereed journal papers; 1 accepted for publication and 3 that are under review. Paper 1 describes the development and theoretical basis of the Get REAL program and provides detail of the program structure and learning cycle. The focus of Paper 1 reflects the first question of interest in the thesis which is about the extent to which learning derived from participation in the program can be generalised to other contexts. Participants are 16 children with HFA ranging in age from 8-13 years. Results provided support for the generalisability of learning from Get REAL to home and school evidenced by parent and teacher data collected pre and post participation in Get REAL. Following establishment of the generalisation of learning from Get REAL, Papers 2 and 3 focus on the Active and Reflective components of the program in order to examine how individual and group learning takes place. Participants (N = 12) in the program are video-taped during the Active and Reflective Sessions. Using identical coding protocols of video data, improvements in prosocial behaviour and diminishing of inappropriate behaviours were apparent with the exception of perspective taking. Data also revealed that 2 of the participants had atypical trajectories. An in-depth case study analysis was then conducted with these 2 participants in Paper 4. Data included reports from health care and education professionals within the school and externally (e.g., paediatrician) and identified the multi-faceted nature of care needed for children with comorbid diagnoses and extremely challenging family circumstances as a complex task to effect change. Results of this research support the effectiveness of the Get REAL program in promoting pro social behaviours such as improvements in engaging with others and emotional regulation, and in diminishing unwanted behaviours such as conduct problems. Further, the gains made by the participating children were found to be generalisable beyond Get REAL to home and other school settings. The research contained in the thesis adds to current knowledge about how learning can take place for children with HFA. Results show that an experiential learning framework with a focus on social cognition, together with explicit teaching, scaffolded with video feedback, are key ingredients for the generalisation of social learning to broader contexts.

Sustainability of a successful health and nutrition program in a remote Aboriginal community

OBJECTIVE: To assess the long term effect of a nutrition program in a remote Aboriginal community (Minjilang). DESIGN: Evaluation of nutritional outcomes over the three years before and the three years after a health and nutrition program that ran from June 1989 to June 1990. Turnover of food items at the community store was used as a measure of dietary intake at Minjilang and a comparison community. SETTING: A community of about 150 Aboriginal people live at Minjilang on Croker Island, 240 km north-east of Darwin. A similar community of about 300 people on another island was used as the comparison. RESULTS: The program produced lasting improvements in dietary intake of most target foods (including fruit, vegetables and wholegrain bread) and nutrients (including folate, ascorbic acid and thiamine). Sugar intake fell in both communities before the program, but the additional decrease in sugar consumption during the program at Minjilang "rebounded" in the next year. Dietary improvements in the comparison community were delayed and smaller than at Minjilang. CONCLUSIONS: The success of the program at Minjilang was linked to an ongoing process of social change, which in turn provided a stimulus for dietary improvement in the comparison community. When Aboriginal people themselves control and maintain ownership of community-based intervention programs, nutritional improvements can be initiated and sustained.

Satisfaction survey for a medication management program : satisfaction guaranteed?

BACKGROUND Providing clinical pharmacy services to patients in their homes after discharge from hospital has been reported to reduce health care costs and improve outcomes. The Medication Management Program of the Fraser Health Authority involves pharmacists making home visits to provide clinical pharmacy services to elderly patients who have recently been discharged from hospital and others considered to be at high risk for adverse drug events. Although clinical and economic outcomes of this program have been evaluated, humanistic outcomes such as satisfaction have not been assessed. Moreover, very little evaluation of patient satisfaction with home pharmacy services has been reported in the literature. OBJECTIVE To evaluate patient satisfaction with the Medication Management Program. METHODS A telephone survey instrument, consisting of 7 Likert-scale items and 2 open-ended questions, was developed and administered to patients who received a home pharmacist visit between September 1 and November 23, 2011. In addition to the survey responses, demographic and clinical data for both respondents and nonrespondents were collected. RESULTS Of the 175 patients invited to participate in the survey, 103 (58.9%) agreed to participate. The majority of respondents agreed or strongly agreed with all of the survey items, indicating satisfaction with the program. For example, 97 (94%) agreed or strongly agreed that they would recommend the pharmacist home visit program continue to be available, and all 103 (100%) agreed or strongly agreed that they were satisfied with the pharmacist home visit. Respondents provided some suggestions for program improvement. CONCLUSIONS The survey findings demonstrate that patients were satisfied with the home clinical pharmacy services offered through the Fraser Health Medication Management Program.

Blast response and safety evaluation of a composite column for use as key element in structural systems

This paper presents the blast response, damage mechanism and evaluation of residual load capacity of a concrete–steel composite (CSC) column using dynamic computer simulation techniques. This study is an integral part of a comprehensive research program which investigated the vulnerability of structural framing systems to catastrophic and progressive collapse under blast loading and is intended to provide design information on blast mitigation and safety evaluation of load bearing vulnerable columns that are key elements in a building. The performance of the CSC column is compared with that of a reinforced concrete (RC) column with the same dimensions and steel ratio. Results demonstrate the superior performance of the CSC column, compared to the RC column in terms of residual load carrying capacity, and its potential for use as a key element in structural systems. The procedure and results presented herein can be used in the design and safety evaluation of key elements of multi-storey buildings for mitigating the impact of blast loads.

Calibration and evaluation of an objective measure of physical activity in pre-school children

Background The purposes of this study were 1) to establish accelerometer count cutoffs to categorize activity intensity of 3 to 5-y old-children and 2) to evaluate the accelerometer as a measure of children’s physical activity in preschool settings. Methods While wearing an ActiGraph accelerometer, 16 preschool children performed five, 3-min structured activities. Receiver Operating Characteristic (ROC) curve analyses identified count cutoffs for four physical activity intensities. In 9 preschools, 281 children wore an ActiGraph during observations performed by three trained observers (interobserver reli-ability = 0.91 to 0.98). Results Separate count cutoffs for 3, 4, and 5-y olds were established. Sensitivity and specificity for the count cutoffs ranged from 86.7% to 100.0% and 66.7% to 100.0%, respectively. ActiGraph counts/15 s were different among all activities (P < 0.05) except the two sitting activities. Correlations between observed and ActiGraph intensity categorizations at the preschools ranged from 0.46 to 0.70 (P < 0.001). Conclusions The ActiGraph count cutoffs established and validated in this study can be used to objectively categorize the time that preschool-age children spend in different physical activity intensity levels.

Evaluation of a community-based intervention to promote physical activity in youth : lessons from active winners

Purpose To test the effects of a community-based physical activity intervention designed to increase physical activity and to conduct an extensive process evaluation of the intervention. Design Quasi-experimental. Setting Two rural communities in South Carolina. One community received the intervention, and the other served as the comparison. Subjects Public school students who were in fifth grade at the start of the study (558 at baseline) were eligible to participate. A total of 436 students participated over the course of the study. Intervention The intervention included after-school and summer physical activity programs and home, school, and community components designed to increase physical activity in youth. The intervention took place over an 18-month period. Measures. Students reported after-school physical activity at three data collection points (prior to, during, and following the intervention) using the Previous Day Physical Activity Recall (PDPAR). They also completed a questionnaire designed to measure hypothesized psychosocial and environmental determinants of physical activity behavior The process evaluation used meeting records, documentation of program activities, interviews, focus groups, and heart rate monitoring to evaluate the planning and implementation of the intervention. Results There were no significant differences in the physical activity variables and few significant differences in the psychosocial variables between the intervention and comparison groups. The process evaluation indicated that the after-school and summer physical activity component of the intervention was implemented as planned, but because of resource and time limitations, the home, school, and community components were not implemented as planned. Conclusions The intervention did not have a significant effect on physical activity in the target population of children in the intervention community. This outcome is similar to that reported in other studies of community-based physical activity intervention.

A qualitative investigation of adolescents’ perceived mechanisms of change from a universal school-based depression prevention program

A recent meta-analysis provides evidence supporting the universal application of school-based prevention programs for adolescent depression. The mechanisms underlying such successful interventions, however, are largely unknown. We report on a qualitative analysis of 109 Grade 9 students’ beliefs about what they gained from an evidence-based depression prevention intervention, the Resourceful Adolescent Program (RAP-A). Fifty-four percent of interviewees articulated at least one specific example of program benefit. A thematic analysis of responses revealed two major themes, improved interpersonal relationships and improved self-regulation, both stronger than originally assumed. A more minor theme also emerged—more helpful cognitions. It is postulated that both improved interpersonal relationships and improved self-regulation are likely to enhance one another, and more helpful cognitions may express its contribution through enhanced self-regulation. These findings broaden our understanding of the impact of depression prevention programs, beginning to illuminate how such programs benefit participants.

Evaluation of emerging intelligent transport systems to improve safety on level crossings : an overview

Safety at railway level crossings (RLX) is one part of a wider picture of safety within the whole transport system. Governments, the rail industry and road organisations have used a variety of countermeasures for many years to improve RLX safety. New types of interventions are required in order to reduce the number of crashes and associated social costs at railway crossings. This paper presents the results of a large research program which aimed to assess the effectiveness of emerging Intelligent Transport Systems (ITS) interventions, both on-road and in-vehicle based, to improve the safety of car drivers at RLXs in Australia. The three most promising technologies selected from the literature review and focus groups were tested in an advanced driving simulator to provide a detailed assessment of their effects on driver behaviour. The three interventions were: (i) in-vehicle visual warning using a GPS/smartphone navigation-like system, (ii) in-vehicle audio warning and; (iii) on-road intervention known as valet system (warning lights on the road surface activated as a train approaches). The effects of these technologies on 57 participants were assessed in a systematic approach focusing on the safety of the intervention, effects on the road traffic around the crossings and driver’s acceptance of the technology. Given that the ITS interventions were likely to provide a benefit by improving the driver’s awareness of the crossing status in low visibility conditions, such conditions were investigated through curves in the track before arriving at the crossing. ITS interventions were also expected to improve driver behaviour at crossings with high traffic (blocking back issue), which were also investigated at active crossings. The key findings are: (i) interventions at passive crossings are likely to provide safety benefits; (ii) the benefits of ITS interventions on driver behaviour at active crossings are limited; (iii) the trialled ITS interventions did not show any issues in terms of driver distraction, driver acceptance or traffic delays; (iv) these interventions are easy to use, do not increase driver workload substantially; (v) participants’ intention to use the technology is high and; (vi) participants saw most value in succinct messages about approaching trains as opposed to knowing the RLX locations or the imminence of a collision with a train.

Breathe easier online : evaluation of a randomized controlled pilot trial of an internet-based intervention to improve well-being in children and adolescents with a chronic respiratory condition

Background Chronic respiratory illnesses are the most common group of childhood chronic health conditions and are overrepresented in socially isolated groups. Objective To conduct a randomized controlled pilot trial to evaluate the efficacy of Breathe Easier Online (BEO), an Internet-based problem-solving program with minimal facilitator involvement to improve psychosocial well-being in children and adolescents with a chronic respiratory condition. Methods We randomly assigned 42 socially isolated children and adolescents (18 males), aged between 10 and 17 years to either a BEO (final n = 19) or a wait-list control (final n = 20) condition. In total, 3 participants (2 from BEO and 1 from control) did not complete the intervention. Psychosocial well-being was operationalized through self-reported scores on depression symptoms and social problem solving. Secondary outcome measures included self-reported attitudes toward their illness and spirometry results. Paper-and-pencil questionnaires were completed at the hospital when participants attended a briefing session at baseline (time 1) and in their homes after the intervention for the BEO group or a matched 9-week time period for the wait-list group (time 2). Results The two groups were comparable at baseline across all demographic measures (all F < 1). For the primary outcome measures, there were no significant group differences on depression (P = .17) or social problem solving (P = .61). However, following the online intervention, those in the BEO group reported significantly lower depression (P = .04), less impulsive/careless problem solving (P = .01), and an improvement in positive attitude toward their illness (P = .04) compared with baseline. The wait-list group did not show these differences. Children in the BEO group and their parents rated the online modules very favorably. Conclusions Although there were no significant group differences on primary outcome measures, our pilot data provide tentative support for the feasibility (acceptability and user satisfaction) and initial efficacy of an Internet-based intervention for improving well-being in children and adolescents with a chronic respiratory condition. Trial registration Australian New Zealand Clinical Trials Registry number: ACTRN12610000214033;

Evaluation of the potential impact of the global economic crisis on HIV epidemics in Southeast Asia

Economic conditions around the world are likely to deteriorate in the short to medium term. The potential impact of this crisis on the spread of HIV is not clear. Government revenues and aid flows from international donors may face constraints, possibly leading to reductions in funding for HIV programs. Economic conditions (leading to increases in unemployment, for example) may also have an indirect impact on HIV epidemics by affecting the behaviour of individual people. Some behavioural changes may influence the rate of HIV transmission. This report presents findings from a study that investigates the potential impact of the economic crisis on HIV epidemics through the use of mathematical modelling. The potential epidemiological impacts of changes in the economy are explored for two distinctly characterised HIV epidemics: (i) a well-defined, established, and generalised HIV epidemic (specifically Cambodia, where incidence is declining); (ii) an HIV epidemic in its early expansion phase (specifically Papua New Guinea, where incidence has not yet peaked). Country-specific data are used for both settings and the models calibrated to accurately reflect the unique HIV epidemics in each population in terms of both incidence and prevalence. Models calibrated to describe the past and present epidemics are then used to forecast epidemic trajectories over the next few years under assumptions that behavioural or program conditions may change due to economic conditions. It should be noted that there are very limited solid data on how HIV/AIDS program funds may decrease or how social determinants related to HIV risk may change due to the economic crisis. Potential changes in key relevant factors were explored, along with sensitivity ranges around these assumptions, based on extensive discussions with in-country and international experts and stakeholders. As with all mathematical models, assumptions should be reviewed critically and results interpreted cautiously.

Youth off the Streets "Our Place, Walgett Youth and Young Families Project" Project Evaluation.

The Project: • YOTS is a major youth specific agency established in 1991. It is a non-denominational, non-discriminatory and not-for-profit organisation, providing a wide range of services and offering a full continuum of care. It seeks to build on the strengths and positive aspects of marginalised young people and communities. • The 'Our Place, Walgett Youth and Young Families Project' further develops an existing YOTS capacity to provide services to Aboriginal young people. • The project adopted an action-research and community development model in which YOTS worked in partnership with the Youth Sub-committee of the Walgett Interagency. • Specific goals/objectives of the program were to: Coordinate youth and young family activities in partnership with local services and the community to build self-esteem, pride, resilience, motivation and skills; Contribute to the prevention and reduction of homelessness, unstable and unsafe housing and disruptive mobility (Walgett/Redfern) in youth and young families; Increase and improve collaborative engagement between youth and family focused services; and, research, adapt and implement Australian and international best-practice homelessness prevention/reduction initiatives to contribute to new models of practice relevant to rural and regional areas. • The project centred around an out-reach model that focused on providing a safe space with relevant structured activities coordinated by YOTS youth and family workers. Through community and service provider consultation, it was proposed that local services could coordinate strategies and activities and run them, where possible, from the centre, providing ease of access in a safe and supportive context. • Specific activities included: Implementing regular meetings with the stakeholders and community representatives; Developing a Terms of Reference for YOTS presence in the Walgett community; Undertaking a community consultation prior to finalising program activities; Implementing a range of recreational activities (sports, music, arts and crafts) early on in the activity; Implementing young family support initiatives; implementing a volunteering program, including volunteer support to young families through intergenerational volunteering; running a series of Culture and Healing Camps in partnership with local Elders and other services; Running a series of Music Camps; Providing alternative education support and referrals in partnership with local schools; Researching, identifying and adapting other best-practice models.

Development of the OnTrack Diabetes program

Background: Type 2 diabetes affects an estimated 347 million people worldwide and often leads to serious complications including blindness, kidney disease, and limb amputation. Comorbid dysphoria is common and is an independent risk factor for poor glycaemic control. Professional support for diabetes self-management and dysphoria has limited availability and involves high costs, especially after regular hours, and in rural and remote areas. Web-based cognitive behavior therapy offers highly accessible, acceptable, and cost-effective support for people with diabetes. This paper describes the development of OnTrack Diabetes, a self-guided, Web-based program to promote improved physical and emotional self-management in people with Type 2 diabetes. Objective: The objective of the study is to describe the development of the OnTrack Diabetes program, which is a self-guided, Web-based program aimed to promote euthymia and improved disease self-management in people with Type 2 diabetes. Methods: Semistructured interviews with 12 general practitioners and 13 patients with Type 2 diabetes identified enablers of and barriers to effective diabetes self-management, requirements for additional support, and potential program elements. Existing resources and research data informed the development of content, and consultants from relevant disciplines provided feedback on draft segments and reviewed the program before release. Using a self-guided delivery format contained costs, in addition to adapting program features and modules from an existing OnTrack program. Results: A separate paper describes the protocol for a randomized controlled trial to provide this required evaluation. Conclusions: Development of the OnTrack Diabetes program demonstrates strategies that help ensure that a program is acceptable to users. The next stages involve testing users’ experiences and examining the program’s effectiveness and cost-effectiveness in randomized controlled trials.