Understanding evaluation of learning support in mathematics and statistics

With rapid and continuing growth of learning support initiatives in mathematics and statistics found in many parts of the world, and with the likelihood that this trend will continue, there is a need to ensure that robust and coherent measures are in place to evaluate the effectiveness of these initiatives. The nature of learning support brings challenges for measurement and analysis of its effects. After briefly reviewing the purpose, rationale for, and extent of current provision, this article provides a framework for those working in learning support to think about how their efforts can be evaluated. It provides references and specific examples of how workers in this field are collecting, analysing and reporting their findings. The framework is used to structure evaluation in terms of usage of facilities, resources and services provided, and also in terms of improvements in performance of the students and staff who engage with them. Very recent developments have started to address the effects of learning support on the development of deeper approaches to learning, the affective domain and the development of communities of practice of both learners and teachers. This article intends to be a stimulus to those who work in mathematics and statistics support to gather even richer, more valuable, forms of data. It provides a 'toolkit' for those interested in evaluation of learning support and closes by referring to an on-line resource being developed to archive the growing body of evidence. © 2011 Taylor & Francis.

Paper, prices and politics : an evaluation of the Swedish support to the Bai Bang project in Vietnam : a Sida evaluation report

Sweden’s protest against the Vietnam War was given tangible form in 1969 through the decision to give economic aid to the Government of North Vietnam. The main outcome was an integrated pulp and paper mill in the Vinh Phu Province north-west of Hanoi. Known as Bai Bang after its location, the mill became the most costly, one of the longest lasting and the most controversial project in the history of Swedish development cooperation. In 1996 Bai Bang produced at its full capacity. Today the mill is exclusively managed and staffed by the Vietnamese and there are plans for future expansion. At the same time a substantial amount of money has been spent to reach these achievements. Looking back at the cumbersome history of the project the results are against many’s expectations. To learn more about the conditions for sustainable development Sida commissioned two studies of the Bai Bang project. Together they touch upon several important issues in development cooperation over a period of almost 30 years: the change of aid paradigms over time, the role of foreign policy in development cooperation, cultural obstacles, recipient responsibility versus donor led development etc. The two studies were commissioned by Sida’s Department for Evaluation and Internal Audit which is an independent department reporting directly to Sida’s Board of Directors. One study assesses the financial and economic viability of the pulp and paper mill and the broader development impact of the project in Vietnam. It has been carried out by the Centre for International Economics, an Australian private economic research agency. The other study analyses the decision-making processes that created and shaped the project over a period of two decades, and reflects on lessons from the project for development cooperation in general. This study has been carried out by the Chr. Michelsen Institute, a Norweigan independent research institution.

Get real : the development, implementation and evaluation of an intervention for primary aged children with High Functioning Autism

Children with Autism Spectrum Disorder experience difficulty in communication and in understanding the social world which can have negative consequences for their relationships, in managing emotions, and generally dealing with the challenges of everyday life. This thesis examines the effectiveness of the Active and Reflective components of the Get REAL program through the assessment of the detailed coding of video-recorded observations and longitudinal quantitative analysis. The aim of Get REAL is to increase the social, emotional, and cognitive learning of children with High Functioning Autism (HFA). Get REAL is a group program designed specifically for use in inclusive primary school settings. The Get REAL program was designed in response to the mixed success of generalisation of learning to new contexts of existing social skills programs. The theoretical foundation of Get REAL is based upon pedagogical theory and learning theory to facilitate transfer of learning, combined with experiential, individualised, evaluative and organisational approaches. This thesis is by publication and consists of four refereed journal papers; 1 accepted for publication and 3 that are under review. Paper 1 describes the development and theoretical basis of the Get REAL program and provides detail of the program structure and learning cycle. The focus of Paper 1 reflects the first question of interest in the thesis which is about the extent to which learning derived from participation in the program can be generalised to other contexts. Participants are 16 children with HFA ranging in age from 8-13 years. Results provided support for the generalisability of learning from Get REAL to home and school evidenced by parent and teacher data collected pre and post participation in Get REAL. Following establishment of the generalisation of learning from Get REAL, Papers 2 and 3 focus on the Active and Reflective components of the program in order to examine how individual and group learning takes place. Participants (N = 12) in the program are video-taped during the Active and Reflective Sessions. Using identical coding protocols of video data, improvements in prosocial behaviour and diminishing of inappropriate behaviours were apparent with the exception of perspective taking. Data also revealed that 2 of the participants had atypical trajectories. An in-depth case study analysis was then conducted with these 2 participants in Paper 4. Data included reports from health care and education professionals within the school and externally (e.g., paediatrician) and identified the multi-faceted nature of care needed for children with comorbid diagnoses and extremely challenging family circumstances as a complex task to effect change. Results of this research support the effectiveness of the Get REAL program in promoting pro social behaviours such as improvements in engaging with others and emotional regulation, and in diminishing unwanted behaviours such as conduct problems. Further, the gains made by the participating children were found to be generalisable beyond Get REAL to home and other school settings. The research contained in the thesis adds to current knowledge about how learning can take place for children with HFA. Results show that an experiential learning framework with a focus on social cognition, together with explicit teaching, scaffolded with video feedback, are key ingredients for the generalisation of social learning to broader contexts.

A pre-post evaluation of an ambulatory nutrition support service for malnourished patients post hospital discharge: a pilot study

Introduction Malnutrition is common among hospitalised patients, with poor follow-up of nutrition support post-discharge. Published studies on the efficacy of ambulatory nutrition support (ANS) for malnourished patients post-discharge are scarce. The aims of this study were to evaluate the rate of dietetics follow-up of malnourished patients post-discharge, before (2008) and after (2010) implementation of a new ANS service, and to evaluate nutritional outcomes post-implementation. Materials and Methods Consecutive samples of 261 (2008) and 163 (2010) adult inpatients referred to dietetics and assessed as malnourished using Subjective Global Assessment (SGA) were enrolled. All subjects received inpatient nutrition intervention and dietetic outpatient clinic follow-up appointments. For the 2010 cohort, ANS was initiated to provide telephone follow-up and home visits for patients who failed to attend the outpatient clinic. Subjective Global Assessment, body weight, quality of life (EQ-5D VAS) and handgrip strength were measured at baseline and five months post-discharge. Paired t-test was used to compare pre- and post-intervention results. Results In 2008, only 15% of patients returned for follow-up with a dietitian within four months post-discharge. After implementation of ANS in 2010, the follow-up rate was 100%. Mean weight improved from 44.0 ± 8.5kg to 46.3 ± 9.6kg, EQ-5D VAS from 61.2 ± 19.8 to 71.6 ± 17.4 and handgrip strength from 15.1 ± 7.1 kg force to 17.5 ± 8.5 kg force; p<0.001 for all. Seventy-four percent of patients improved in SGA score. Conclusion Ambulatory nutrition support resulted in significant improvements in follow-up rate, nutritional status and quality of life of malnourished patients post-discharge.

Development and formative evaluation of a motorcycle rider training intervention to address risk taking

The need to address on-road motorcycle safety in Australia is important due to the disproportionately high percentage of riders and pillions killed and injured each year. One approach to preventing motorcycle-related injury is through training and education. However, motorcycle rider training lacks empirical support as an effective road safety countermeasure to reduce crash involvement. Previous reviews have highlighted that risk-taking is a contributing factor in many motorcycle crashes, rather than merely a lack of vehicle-control skills (Haworth & Mulvihill, 2005; Jonah, Dawson & Bragg, 1982; Watson et al, 1996). Hence, though the basic vehicle-handling skills and knowledge of road rules that are taught in most traditional motorcycle licence training programs may be seen as an essential condition of safe riding, they do not appear to be sufficient in terms of crash reduction. With this in mind there is considerable scope for the improvement of program focus and content for rider training and education. This program of research examined an existing traditional pre-licence motorcycle rider training program and formatively evaluated the addition of a new classroom-based module to address risky riding; the Three Steps to Safer Riding program. The pilot program was delivered in the real world context of the Q-Ride motorcycle licensing system in the state of Queensland, Australia. Three studies were conducted as part of the program of research: Study 1, a qualitative investigation of delivery practices and student learning needs in an existing rider training course; Study 2, an investigation of the extent to which an existing motorcycle rider training course addressed risky riding attitudes and motives; and Study 3, a formative evaluation of the new program. A literature review as well as the investigation of learning needs for motorcyclists in Study 1 aimed to inform the initial planning and development of the Three Steps to Safer Riding program. Findings from Study 1 suggested that the training delivery protocols used by the industry partner training organisation were consistent with a learner-centred approach and largely met the learning needs of trainee riders. However, it also found that information from the course needs to be reinforced by on-road experiences for some riders once licensed and that personal meaning for training information was not fully gained until some riding experience had been obtained. While this research informed the planning and development of the new program, a project team of academics and industry experts were responsible for the formulation of the final program. Study 2 and Study 3 were conducted for the purpose of formative evaluation and program refinement. Study 2 served primarily as a trial to test research protocols and data collection methods with the industry partner organisation and, importantly, also served to gather comparison data for the pilot program which was implemented with the same rider training organisation. Findings from Study 2 suggested that the existing training program of the partner organisation generally had a positive (albeit small) effect on safety in terms of influencing attitudes to risk taking, the propensity for thrill seeking, and intentions to engage in future risky riding. However, maintenance of these effects over time and the effects on riding behaviour remain unclear due to a low response rate upon follow-up 24 months after licensing. Study 3 was a formative evaluation of the new pilot program to establish program effects and possible areas for improvement. Study 3a examined the short term effects of the intervention pilot on psychosocial factors underpinning risky riding compared to the effects of the standard traditional training program (examined in Study 2). It showed that the course which included the Three Steps to Safer Riding program elicited significantly greater positive attitude change towards road safety than the existing standard licensing course. This effect was found immediately following training, and mean scores for attitudes towards safety were also maintained at the 12 month follow-up. The pilot program also had an immediate effect on other key variables such as risky riding intentions and the propensity for thrill seeking, although not significantly greater than the traditional standard training. A low response rate at the 12 month follow-up unfortunately prevented any firm conclusions being drawn regarding the impact of the pilot program on self-reported risky riding once licensed. Study 3a further showed that the use of intermediate outcomes such as self-reported attitudes and intentions for evaluation purposes provides insights into the mechanisms underpinning risky riding that can be changed by education and training. A multifaceted process evaluation conducted in Study 3b confirmed that the intervention pilot was largely delivered as designed, with course participants also rating most aspects of training delivery highly. The complete program of research contributed to the overall body of knowledge relating to motorcycle rider training, with some potential implications for policy in the area of motorcycle rider licensing. A key finding of the research was that psychosocial influences on risky riding can be shaped by structured education that focuses on awareness raising at a personal level and provides strategies to manage future riding situations. However, the formative evaluation was mainly designed to identify areas of improvement for the Three Steps to Safer Riding program and found several areas of potential refinement to improve future efficacy of the program. This included aspects of program content, program delivery, resource development, and measurement tools. The planned future follow-up of program participants' official crash and traffic offence records over time may lend further support for the application of the program within licensing systems. The findings reported in this thesis offer an initial indication that the Three Steps to Safer Riding is a useful resource to accompany skills-based training programs.

Evaluation of an ambulatory nutrition support service for malnourished patients post hospital discharge

Introduction Malnutrition is common among hospitalised patients, with poor follow-up of nutrition support post-discharge. Published studies on the efficacy of ambulatory nutrition support (ANS) for malnourished patients post-discharge are scarce. The aims of this study were to evaluate the rate of dietetics follow-up of malnourished patients post-discharge, before (2008) and after (2010) implementation of a new ANS service, and to evaluate nutritional outcomes post-implementation. Materials and Methods Consecutive samples of 261 (2008) and 163 (2010) adult inpatients referred to dietetics and assessed as malnourished using Subjective Global Assessment (SGA) were enrolled. All subjects received inpatient nutrition intervention and dietetic outpatient clinic follow-up appointments. For the 2010 cohort, ANS was initiated to provide telephone follow-up and home visits for patients who failed to attend the outpatient clinic. Subjective Global Assessment, body weight, quality of life (EQ-5D VAS) and handgrip strength were measured at baseline and five months post-discharge. Paired t-test was used to compare pre- and post-intervention results. Results In 2008, only 15% of patients returned for follow-up with a dietitian within four months post-discharge. After implementation of ANS in 2010, the follow-up rate was 100%. Mean weight improved from 44.0 ± 8.5kg to 46.3 ± 9.6kg, EQ-5D VAS from 61.2 ± 19.8 to 71.6 ± 17.4 and handgrip strength from 15.1 ± 7.1 kg force to 17.5 ± 8.5 kg force; p<0.001 for all. Seventy-four percent of patients improved in SGA score. Conclusion Ambulatory nutrition support resulted in significant improvements in follow-up rate, nutritional status and quality of life of malnourished patients post-discharge.

Evaluation of Indigenous justice programs.: final report / Project D, Safe Aboriginal Youth Patrol Programs in New South Wales and Northbridge Policy and Juvenile Aid Group in Western Australia

The Australian Government has provided funding to evaluate the effectiveness of Indigenous law and justice programs across five subject areas to identify the best approaches to tackling crime and justice issues and better inform government funding decisions in the future. This report presents the findings of subject area "D", which examined two different approaches to delivering community and night patrol services for young people: the Safe Aboriginal Youth Patrol programs in New South Wales, and the Northbridge Policy project (the Young People in Northbridge project), in Western Australia. Night patrols can address crime either directly or indirectly, by prevention work or by addressing the social causes of crime through community development.

Evaluation of ANPR trials for traffic policing in Queensland. Report to the Queensland Police Service, State Traffic Support Branch

In response to the Travelsafe Committee Report No. 51 – report on the inquiry into Automatic Plate Recognition Technology – it was recommended that the Queensland Police Service continue to trial the deployment of ANPR technology for traffic enforcement work and to evaluate the road safety impacts and operational effectiveness of the technology. As such, the purpose of this report is to provide an independent evaluation of a trial of ANPR that was conducted by a project team within the State Traffic Support Branch of the Queensland Police Service (QPS) and provide recommendations as to the applicability and usability of the technology for use throughout Queensland...

Incentivizing innovation: A review of the Brazilian federal innovation support programs

Innovation is the transformation of knowledge of any kind into new products or services in the market. Its importance as a production factor is widely acknowledged. In the age of the knowledge-based economy innovation became critical for any company or even country to compete globally. Many countries are encouraging innovation through various mechanisms, and one of the most widely used is the provision of special incentives for innovation. This paper investigates incentive systems for the growth of technology companies as a strategy to promote knowledge-based economic development. As for the case investigations the study focuses on an emerging economy, Brazil. The research is based upon the available literature, best practices, government policy and review of incentive systems. The findings provide insights from the case study in a country context and some lessons learned for other countries using incentive systems to boost the innovation capabilities of their technology companies.

Evaluation of a school-based creative arts therapy programme for adolescents from refugee backgrounds

Creative arts therapy programs have been identified as effective interventions with adolescents affected by adversity. The current study provided a controlled trial of creative arts therapy to address the psychosocial needs of students from refugee backgrounds. Forty-two students participated in a therapy trial, comprising an intervention and control group. Mental health and behavioural difficulties were assessed pre and post intervention. Hopkins Symptoms Checklist-25 (HSCL-25) and the Strengths and Difficulties Questionnaire (SDQ) were used to assess wellbeing. Findings suggested an effect for a reduction in behavioural difficulties for the treatment group. A significant reduction in emotional symptoms was found for the treatment group. Findings provide empirical support for school-based creative arts therapy interventions specific to refugee young people.

Evaluation of weed eradication programs: containment and extirpation

Weed eradication programs often require 10 years or more to achieve their objective. It is important that progress is evaluated on a regular basis so that programs that are 'on track' can be distinguished from those that are unlikely to succeed. Earlier research has addressed conformity of eradication programs to the delimitation criterion. In this paper evaluation in relation to the containment and extirpation criteria is considered. Because strong evidence of containment failure (i.e. spread from infestations targeted for eradication) is difficult to obtain, it generally will not be practicable to evaluate how effective eradication programs are at containing the target species. However, chronic failure of containment will be reflected in sustained increases in cumulative infested area and thus a failure to delimit a weed invasion. Evaluating the degree of conformity to the delimitation and extirpation criteria is therefore sufficient to give an appraisal of progress towards the eradication objective. A significant step towards eradication occurs when a weed is no longer readily detectable at an infested site, signalling entry to the monitoring phase. This transition will occur more quickly if reproduction is prevented consistently. Where an invasion consists of multiple infestations, the monitoring profile (frequency distribution of time since detection) provides a summary of the overall effectiveness of the eradication program in meeting the extirpation criterion. Eradication is generally claimed when the target species has not been detected for a period equal to or greater than its seed longevity, although there is often considerable uncertainty in estimates of the latter. Recently developed methods, which take into consideration the cost of continued monitoring vs. the potential cost of damage should a weed escape owing to premature cessation of an eradication program, can assist managers to decide when to terminate weed eradication programs.

Development and evaluation of a spatial decision support system for malaria elimination in Bhutan

Background: Bhutan has reduced its malaria incidence significantly in the last 5 years, and is aiming for malaria elimination by 2016. To assist with the management of the Bhutanese malaria elimination programme a spatial decision support system (SDSS) was developed. The current study aims to describe SDSS development and evaluate SDSS utility and acceptability through informant interviews. Methods: The SDSS was developed based on the open-source Quantum geographical information system (QGIS) and piloted to support the distribution of long-lasting insecticidal nets (LLINs) and indoor residual spraying (IRS) in the two sub-districts of Samdrup Jongkhar District. It was subsequently used to support reactive case detection (RACD) in the two sub-districts of Samdrup Jongkhar and two additional sub-districts in Sarpang District. Interviews were conducted to ascertain perceptions on utility and acceptability of 11 informants using the SDSS, including programme and district managers, and field workers. Results: A total of 1502 households with a population of 7165 were enumerated in the four sub-districts, and a total of 3491 LLINs were distributed with one LLIN per 1.7 persons. A total of 279 households representing 728 residents were involved with RACD. Informants considered that the SDSS was an improvement on previous methods for organizing LLIN distribution, IRS and RACD, and could be easily integrated into routine malaria and other vector-borne disease surveillance systems. Informants identified some challenges at the programme and field level, including the need for more skilled personnel to manage the SDSS, and more training to improve the effectiveness of SDSS implementation and use of hardware. Conclusions: The SDSS was well accepted and informants expected its use to be extended to other malaria reporting districts and other vector-borne diseases. Challenges associated with efficient SDSS use included adequate skills and knowledge, access to training and support, and availability of hardware including computers and global positioning system receivers.