A formative evaluation of the Stronger Smarter Learning Communities Project. 2011 report.

This report is a formative evaluation of the operations of the DEEWR funded Stronger Smarter Learning Community (SSLC) project from September 2009 to July 2011. It is undertaken by an independent team of researchers from Queensland University of Technology, the University of Newcastle and Harvard University. It reports on findings from: documentary analysis; qualitative case studies of SSLC Hub schools; descriptive, multivariate and multilevel analysis of survey data from school leaders and teachers from SSLC Hub and Affiliate schools and from a control group of non-SSLC schools; and multilevel analysis of school-level data on SSLC Hubs, Affiliates and ACARA like-schools. Key findings from this work are that: • SSLC school leaders and teachers are reporting progress in changing school ethos around issues of: recognition of Indigenous identity, Indigenous leadership, innovative approaches to staffing and school models, Indigenous community engagement and high expectations leadership; • Many Stronger Smarter messages are reportedly having better uptake in schools with high percentages of Indigenous students; • There are no major or consistent patterns of differences between SSLC and non-SSLC schools in teacher and school leader self-reports of curriculum and pedagogy practices; and • There is no evidence to date that SSLC Hubs and Affiliates have increased attendance or increased achievement gains compared to ACARA like-schools. Twenty-one months is relatively early in this school reform project. Hence the major focus of subsequent reports will be on the documentation of comparative longitudinal gains in achievement tests and improved attendance. The 2011 and 2012 research also will model the relationships between change in school ethos/climate, changed Indigenous community relations, improved curriculum/pedagogy, and gains in Indigenous student achievement, attendance and outcomes. The key challenge for SSLC and the Stronger Smarter approach will be whether it can systematically generate change and reform in curriculum and pedagogy practices that can be empirically linked to improved student outcomes.

Formative evaluation for improving collaborative planning : a case study at the regional scale

In an earlier paper (Cameron & Johnson 2004) we introduced the idea of formative evaluation (or evaluation for development), the purpose of which is to provide information for improving planning programs and activities. This type of evaluation differs from the two other types: outcome evaluation which aims to judge the success or otherwise of a program; and evaluation for knowledge which seeks to contribute to theoretical work on planning processes and activities. In the earlier paper we also outlined the first stage of formative evaluation in the SEQ 2021 regional planning exercise showing how the process of planning for community engagement was modified in light of the evaluation findings. This current paper details the second stage of formative evaluation in which the collaborative planning component of SEQ 2021 was evaluated, as such it further demonstrates how formative evaluation can be used to improve planning programs. The evaluation findings also provide insights into strategies for more effective collaborative planning. We begin with an overview of collaborative approaches to regional planning, including the SEQ 2021 regional planning program. We then outline formal and informal evaluations of various collaborative regional planning exercises, including the predecessor of SEQ 2021 - SEQ 2001. This sets the scene for discussion of the approach used to evaluate the collaborative component of SEQ 2021. After outlining the main findings from the evaluation and the ways these findings were used to refine the collaborative planning process we conclude with a series of recommendations, relevant not only to SEQ 2021 but to other collaborative planning exercises

Development and formative evaluation of a motorcycle rider training intervention to address risk taking

The need to address on-road motorcycle safety in Australia is important due to the disproportionately high percentage of riders and pillions killed and injured each year. One approach to preventing motorcycle-related injury is through training and education. However, motorcycle rider training lacks empirical support as an effective road safety countermeasure to reduce crash involvement. Previous reviews have highlighted that risk-taking is a contributing factor in many motorcycle crashes, rather than merely a lack of vehicle-control skills (Haworth & Mulvihill, 2005; Jonah, Dawson & Bragg, 1982; Watson et al, 1996). Hence, though the basic vehicle-handling skills and knowledge of road rules that are taught in most traditional motorcycle licence training programs may be seen as an essential condition of safe riding, they do not appear to be sufficient in terms of crash reduction. With this in mind there is considerable scope for the improvement of program focus and content for rider training and education. This program of research examined an existing traditional pre-licence motorcycle rider training program and formatively evaluated the addition of a new classroom-based module to address risky riding; the Three Steps to Safer Riding program. The pilot program was delivered in the real world context of the Q-Ride motorcycle licensing system in the state of Queensland, Australia. Three studies were conducted as part of the program of research: Study 1, a qualitative investigation of delivery practices and student learning needs in an existing rider training course; Study 2, an investigation of the extent to which an existing motorcycle rider training course addressed risky riding attitudes and motives; and Study 3, a formative evaluation of the new program. A literature review as well as the investigation of learning needs for motorcyclists in Study 1 aimed to inform the initial planning and development of the Three Steps to Safer Riding program. Findings from Study 1 suggested that the training delivery protocols used by the industry partner training organisation were consistent with a learner-centred approach and largely met the learning needs of trainee riders. However, it also found that information from the course needs to be reinforced by on-road experiences for some riders once licensed and that personal meaning for training information was not fully gained until some riding experience had been obtained. While this research informed the planning and development of the new program, a project team of academics and industry experts were responsible for the formulation of the final program. Study 2 and Study 3 were conducted for the purpose of formative evaluation and program refinement. Study 2 served primarily as a trial to test research protocols and data collection methods with the industry partner organisation and, importantly, also served to gather comparison data for the pilot program which was implemented with the same rider training organisation. Findings from Study 2 suggested that the existing training program of the partner organisation generally had a positive (albeit small) effect on safety in terms of influencing attitudes to risk taking, the propensity for thrill seeking, and intentions to engage in future risky riding. However, maintenance of these effects over time and the effects on riding behaviour remain unclear due to a low response rate upon follow-up 24 months after licensing. Study 3 was a formative evaluation of the new pilot program to establish program effects and possible areas for improvement. Study 3a examined the short term effects of the intervention pilot on psychosocial factors underpinning risky riding compared to the effects of the standard traditional training program (examined in Study 2). It showed that the course which included the Three Steps to Safer Riding program elicited significantly greater positive attitude change towards road safety than the existing standard licensing course. This effect was found immediately following training, and mean scores for attitudes towards safety were also maintained at the 12 month follow-up. The pilot program also had an immediate effect on other key variables such as risky riding intentions and the propensity for thrill seeking, although not significantly greater than the traditional standard training. A low response rate at the 12 month follow-up unfortunately prevented any firm conclusions being drawn regarding the impact of the pilot program on self-reported risky riding once licensed. Study 3a further showed that the use of intermediate outcomes such as self-reported attitudes and intentions for evaluation purposes provides insights into the mechanisms underpinning risky riding that can be changed by education and training. A multifaceted process evaluation conducted in Study 3b confirmed that the intervention pilot was largely delivered as designed, with course participants also rating most aspects of training delivery highly. The complete program of research contributed to the overall body of knowledge relating to motorcycle rider training, with some potential implications for policy in the area of motorcycle rider licensing. A key finding of the research was that psychosocial influences on risky riding can be shaped by structured education that focuses on awareness raising at a personal level and provides strategies to manage future riding situations. However, the formative evaluation was mainly designed to identify areas of improvement for the Three Steps to Safer Riding program and found several areas of potential refinement to improve future efficacy of the program. This included aspects of program content, program delivery, resource development, and measurement tools. The planned future follow-up of program participants' official crash and traffic offence records over time may lend further support for the application of the program within licensing systems. The findings reported in this thesis offer an initial indication that the Three Steps to Safer Riding is a useful resource to accompany skills-based training programs.

Formative evaluation to benchmark and improve climate-based decision support for graziers in Western Queelsland

Researchers developing climate-based forecasts, workshops, software tools and information to aid grazier decisions undertook an evaluation study to enhance planning and benchmark impact. One hundred graziers in Western Queensland were randomly selected from 7 shires and surveyed by mail and telephone (43 respondents) to explore levels of knowledge and use of climate information, practices and information needs. We found 36% of respondents apply the Southern Oscillation Index to property decisions but 92% were unaware El Niño Southern Oscillation’s predictive signal in the region is greater for pasture growth than rainfall, suggesting they may not recognise the potential of pasture growth forecasts. Almost 75% of graziers consider they are conservative or risk averse in their attitude to managing their enterprise. Mail respondents (n= 20) if given a 68%, on average, probability of exceeding median rainfall forecast may change a decision; almost two-thirds vary stocking rate based on forage available, last year’s pasture growth or the Southern Oscillation Index; the balance maintain a constant stocking rate strategy; 90% have access to a computer; 75% to the internet and 95% have a fax. This paper presents findings of the study and draws comparisons with a similar study of 174 irrigators in the Northern Murray-Darling Basin (Aust. J. Exp. Ag. 44, 247-257). New insights and information gained are helping the team better understand client needs and plan, design and extend tools and information tailored to grazier knowledge, practice, information needs and preferences. Results have also provided a benchmark against which to measure project impact and have influenced the team to make important changes to their project planning, activities and methods for transferring technology tailored to grazier preferences.

Design challenges for electronic medication administration record systems in residential aged care facilities: A formative evaluation

Introduction Electronic medication administration record (eMAR) systems are promoted as a potential intervention to enhance medication safety in residential aged care facilities (RACFs). The purpose of this study was to conduct an in-practice evaluation of an eMAR being piloted in one Australian RACF before its roll out, and to provide recommendations for system improvements. Methods A multidisciplinary team conducted direct observations of workflow (n=34 hours) in the RACF site and the community pharmacy. Semi-structured interviews (n=5) with RACF staff and the community pharmacist were conducted to investigate their views of the eMAR system. Data were analysed using a grounded theory approach to identify challenges associated with the design of the eMAR system. Results The current eMAR system does not offer an end-to-end solution for medication management. Many steps, including prescribing by doctors and communication with the community pharmacist, are still performed manually using paper charts and fax machines. Five major challenges associated with the design of eMAR system were identified: limited interactivity; inadequate flexibility; problems related to information layout and semantics; the lack of relevant decision support; and system maintenance issues.We suggest recommendations to improve the design of the eMAR system and to optimize existing workflows. Discussion Immediate value can be achieved by improving the system interactivity, reducing inconsistencies in data entry design and offering dedicated organisational support to minimise connectivity issues. Longer-term benefits can be achieved by adding decision support features and establishing system interoperability requirements with stakeholder groups (e.g. community pharmacies) prior to system roll out. In-practice evaluations of technologies like eMAR system have great value in identifying design weaknesses which inhibit optimal system use.

One or the other? Do we have to choose? : summative and-or formative evaluation. '¿Una u otra? ¿Tenemos que elegir? : evaluación formativa y-o sumativa'.

Se definen los conceptos de evaluación sumativa y evaluación formativa con el ánimo de ver si son tan diferentes y excluyentes. Se proponen caminos para combinar ambas y para incluir a los estudiantes en el proceso de evaluación.

Design and formative evaluation of an information kiosk on cattle health for landless cattle owners

This paper describes and analyses the experience of designing, installing and evaluating a farmer-usable touch screen information kiosk on cattle health in a veterinary institution in Pondicherry. The contents of the kiosk were prepared based on identified demands for information on cattle health, arrived at through various stakeholders meetings. Information on these cattle diseases and conditions affecting the livelihoods of the poor was provided through graphics, text and audio back-up, keeping in mind the needs of landless and illiterate poor cattle owners. A methodology for kiosk evaluation based on the feedback obtained from kiosk facilitator, critical group reflection and individual users was formulated. The formative evaluation reveals the potential strength this ICT has in transferring information to the cattle owners in a service delivery centre. Such information is vital in preventing diseases and helps cattle owners to present and treat their animals at an early stage of disease condition. This in turn helps prevent direct and indirect losses to the cattle owners. The study reveals how an information kiosk installed at a government institution as a freely accessible source of information to all farmers irrespective of their class and caste can help in transfer of information among poor cattle owners, provided periodic updating, interactivity and communication variability are taken care of. Being in the veterinary centre, the kiosk helps stimulate dialogue, and facilitates demand of services based on the information provided by the kiosk screens.

Workplace learning coordinators program : formative evaluation : final report

The workplace learning coordinators program was established to increase the number of young people undertaking workplace learning placements, especially in industries that provide strong vocational outcomes. Additional attention has been paid to the potential benefits of the program for Koorie young people and other students who experience disadvantage. The program also seeks to improve the alignment between VCAL and VET provision and local industry needs, with a focus on indentifying new workplace learning opportunities with each service area’s employers. This formative evaluation sought to monitor the initial implementation of the WLC program during its first twelve months. Over this period the evaluators recorded the early teething challenges faced by WLCs during their first few months, and the eventual consolidation of their practices during the second half of the program’s first year. In reviewing all of the data collected for this evaluation, and the progressive responses of WLCs and stakeholders over the program’s first year, the evaluators consider that the model of service delivery is appropriate for what the program aims to achieve and recommend its continuance to 2013.

Steps in the design, development and formative evaluation of obesity prevention-related behavior change trials

Obesity prevention interventions through dietary and physical activity change have generally not been effective. Limitations on possible program effectiveness are herein identified at every step in the mediating variable model, a generic conceptual framework for understanding how interventions may promote behavior change. To minimize these problems, and thereby enhance likely intervention effectiveness, four sequential types of formative studies are proposed: targeted behavior validation, targeted mediator validation, intervention procedure validation, and pilot feasibility intervention. Implementing these studies would establish the relationships at each step in the mediating variable model, thereby maximizing the likelihood that an intervention would work and its effects would be detected. Building consensus among researchers, funding agencies, and journal editors on distinct intervention development studies should avoid identified limitations and move the field forward.

Formative evaluation of baby makes 3

Baby Makes 3 was originally developed by Whitehorse Community Health Service and the City of Whitehorse as a program which seeks to prevent violence by promoting respect and equality between couples who have recently become parents for the first time. This is a significant but often stressful event in the life of families and it is not uncommon for couples to adopt ways of relating which reflect gendered norms and foster inequality.
By being offered as a program to all first-time parents by maternal and child health staff, the program in Whitehorse managed to engage with couples at this critical time, many of whom found the program enabled them to adopt greater equality in their relationships. As such, the initial Baby Makes 3 program has been found to be an effective and cost-efficient violence prevention strategy (Flynn, 2011).
The introduction of Baby Makes 3 to the Great South Coast Region is the first time this program has been implemented in a non-metropolitan setting. In addition to the original program aims, a number of additional aims and activities have been added, with the new project being known as Baby Makes 3 Plus.
Programs developed in urban contexts often face different challenges when implemented in rural and regional settings. Likewise, the factors for facilitating success may vary from those in an urban setting (Maidment and Bay, 2012). Hence, this formative evaluation aims to provide feedback on the initial implementation of the original Baby Makes 3 program in the Great South Coast Region with the expectation this will enable problems with implementation and/or opportunities for further enhancement of outcomes to be identified and appropriate action taken.