A combined impact-process evaluation of a program promoting active transport to school: understanding the factors that shaped program effectiveness

This mixed methods study was a comprehensive impact-process evaluation of the Ride2School program in metropolitan and regional areas in Victoria, Australia. The program aimed to promote transport to school for primary school children. Qualitative and quantitative data were collected at baseline and followup from two primary schools involved in the pilot phase of the program and two matched comparison schools, and a further 13 primary schools that participated in the implementation phase of the program. Classroom surveys, structured and unstructured observations, and interviews with Ride2School program staff were used to evaluate the pilot program. For the 13 schools in the second phase of the program, parents and students completed questionnaires at baseline (N = 889) and followup (N = 761). Based on the quantitative data, there was little evidence of an overall increase in active transport to school across participating schools, although impacts varied among individual schools. Qualitative data in the form of observations, interviews, and focus group discussions with students, school staff, and program staff provided insight into the reasons for variable program impacts. This paper highlights the benefits of undertaking a mixed methods approach to evaluating active transport to school programs that enables both measurement and understanding of program impacts.

Knowledge brokering between researchers and policymakers in Fiji to develop policies to reduce obesity: a process evaluation

The importance of using research evidence in decision making at the policy level has been increasingly recognized. However, knowledge brokering to engage researchers and policymakers in government and non-government organizations is challenging. This paper describes and evaluates the knowledge exchange processes employed by the Translational Research on Obesity Prevention in Communities (TROPIC) project that was conducted from July 2009 to April 2012 in Fiji. TROPIC aimed to enhance: the evidence-informed decision making skills of policy developers; and awareness and utilization of local and other obesity-related evidence to develop policies that could potentially improve the nation’s food and physical activity environments. The specific research question was: Can a knowledge brokering approach advance evidence-informed policy development to improve eating and physical activity environments in Fiji.

Process evaluation of a community-based intervention program: Healthy Youth Healthy Communities, an adolescent obesity prevention project in Fiji

Nearly one-half of the adult population in Fiji between the ages of 15–64 years is either overweight or obese; and rates amongst school children have, on average, doubled during the last decade. There is an urgent need to scale up the promotion of healthy behaviors and environments using a multi-sectoral approach. The Healthy Youth Healthy Community (HYHC) project in Fiji used a settings approach in secondary schools and faith-based organizations to increase the capacity of the whole community, including churches, mosques and temples, to promote healthy eating and regular physical activity, and to prevent unhealthy weight gain in adolescents aged 13–18 years. The team consisted of a study manager, project coordinator and four research assistants (RAs) committed to planning, designing and facilitating the implementation of intervention programs in collaboration with other stakeholders, such as the wider school communities, government and non-governmental organizations and business partners. Process data were collected on all intervention activities and analyzed by dose, frequency and reach for each specific strategy. The Fiji Action Plan included nine objectives for the school settings; four were based on nutrition and two on physical activity in schools, plus three general objectives, namely capacity building, social marketing and evaluation. Long-term change in nutritional behavior was difficult to achieve; a key contributor to this was the unhealthy food served in the school canteens. Whilst capacity-building proved to be one of the best mechanisms for intervening, it is important to consider the cultural and social factors influencing health behaviors and affecting specific groups.

Process evaluation of an integrated health promotion/occupational health model in WellWorks-2

Disparities in chronic disease risk by occupation call for newapproaches to health promotion. Well Works-2 was a randomized, controlled study comparing the effectiveness of a health promotion/occupational health program (HP/OHS) with a standard intervention (HP). Interventions in both studies were based on the same theoretical foundations. Results from process evaluation revealed that a similar number of activities were offered in both conditions and that in the HP/OHS condition there were higher levels of worker participation using three measures: mean participation per activity (HP: 14.2% vs. HP/OHS: 21.2%), mean minutes of worker exposure to the intervention/site (HP: 14.9 vs. HP/OHS: 33.3), and overall mean participation per site (HP: 34.4% vs. HP/ OHS: 45.8%). There were a greater number of contacts with management (HP: 8.8 vs. HP/OHS: 24.9) in the HP/ OHS condition. Addressing occupational health may have contributed to higher levels of worker and management participation and smoking cessation among blue-collar workers.

Linking secondary school physical education with community sport and recreation for girls: a process evaluation

The purpose of this study was to undertake a process evaluation to examine the reach, adoption and implementation of a school-community linked physical activity (PA) program for girls aged 12 - 15 years (School Years 7 - 9) using the RE-AIM framework.

Understanding feedback report uptake: process evaluation findings from a 13-month feedback intervention in long-term care settings

BACKGROUND: Long-term care settings provide care to a large proportion of predominantly older, highly disabled adults across the United States and Canada. Managing and improving quality of care is challenging, in part because staffing is highly dependent on relatively non-professional health care aides and resources are limited. Feedback interventions in these settings are relatively rare, and there has been little published information about the process of feedback intervention. Our objectives were to describe the key components of uptake of the feedback reports, as well as other indicators of participant response to the intervention. METHODS: We conducted this project in nine long-term care units in four facilities in Edmonton, Canada. We used mixed methods, including observations during a 13-month feedback report intervention with nine post-feedback survey cycles, to conduct a process evaluation of a feedback report intervention in these units. We included all facility-based direct care providers (staff) in the feedback report distribution and survey administration. We conducted descriptive analyses of the data from observations and surveys, presenting this in tabular and graphic form. We constructed a short scale to measure uptake of the feedback reports. Our analysis evaluated feedback report uptake by provider type over the 13 months of the intervention. RESULTS: We received a total of 1,080 survey responses over the period of the intervention, which varied by type of provider, facility, and survey month. Total number of reports distributed ranged from 103 in cycle 12 to 229 in cycle 3, although the method of delivery varied widely across the period, from 12% to 65% delivered directly to individuals and 15% to 84% left for later distribution. The key elements of feedback uptake, including receiving, reading, understanding, discussing, and reporting a perception that the reports were useful, varied by survey cycle and provider type, as well as by facility. Uptake, as we measured it, was consistently high overall, but varied widely by provider type and time period. CONCLUSIONS: We report detailed process data describing the aspects of uptake of a feedback report during an intensive, longitudinal feedback intervention in long-term care facilities. Uptake is a complex process for which we used multiple measures. We demonstrate the feasibility of conducting a complex longitudinal feedback intervention in relatively resource-poor long-term care facilities to a wider range of provider types than have been included in prior feedback interventions.

A process evaluation of the Supermarket Healthy Eating for Life (SHELf) randomized controlled trial

BACKGROUND: Supermarket Healthy Eating for Life (SHELf) was a randomized controlled trial that operationalized a socioecological approach to population-level dietary behaviour change in a real-world supermarket setting. SHELf tested the impact of individual (skill-building), environmental (20 % price reductions), and combined (skill-building + 20 % price reductions) interventions on women's purchasing and consumption of fruits, vegetables, low-calorie carbonated beverages and water. This process evaluation investigated the reach, effectiveness, implementation, and maintenance of the SHELf interventions.

METHODS: RE-AIM provided a conceptual framework to examine the processes underlying the impact of the interventions using data from participant surveys and objective sales data collected at baseline, post-intervention (3 months) and 6-months post-intervention. Fisher's exact, χ (2) and t-tests assessed differences in quantitative survey responses among groups. Adjusted linear regression examined the impact of self-reported intervention dose on food purchasing and consumption outcomes. Thematic analysis identified key themes within qualitative survey responses.

RESULTS: Reach of the SHELf interventions to disadvantaged groups, and beyond study participants themselves, was moderate. Just over one-third of intervention participants indicated that the interventions were effective in changing the way they bought, cooked or consumed food (p < 0.001 compared to control), with no differences among intervention groups. Improvements in purchasing and consumption outcomes were greatest among those who received a higher intervention dose. Most notably, participants who said they accessed price reductions on fruits and vegetables purchased (519 g/week) and consumed (0.5 servings/day) more vegetables. The majority of participants said they accessed (82 %) and appreciated discounts on fruits and vegetables, while there was limited use (40 %) and appreciation of discounts on low-calorie carbonated beverages and water. Overall reported satisfaction with, use, and impact of the skill-building resources was moderate. Maintenance of newly acquired behaviours was limited, with less than half of participants making changes or using study-provided resources during the 6-month post-intervention period.

CONCLUSIONS: SHELf's reach and perceived effectiveness were moderate. The interventions were more effective among those reporting greater engagement with them (an implementation-related construct). Maintenance of newly acquired behaviours proved challenging.

Screen Time Weight-loss Intervention Targeting Children at Home (SWITCH): process evaluation of a randomised controlled trial intervention.

BACKGROUND: The Screen Time Weight-loss Intervention Targeting Children at Home (SWITCH) trial tested a family intervention to reduce screen-based sedentary behaviour in overweight children. The trial found no significant effect of the intervention on children's screen-based sedentary behaviour. To explore these null findings, we conducted a pre-planned process evaluation, focussing on intervention delivery and uptake. METHODS: SWITCH was a randomised controlled trial of a 6-month family intervention to reduce screen time in overweight children aged 9-12 years (n = 251). Community workers met with each child's primary caregiver to deliver the intervention content. Community workers underwent standard training and were monitored once by a member of the research team to assess intervention delivery. The primary caregiver implemented the intervention with their child, and self-reported intervention use at 3 and 6 months. An exploratory analysis determined whether child outcomes at 6 months varied by primary caregiver use of the intervention. RESULTS: Monitoring indicated that community workers delivered all core intervention components to primary caregivers. However, two thirds of primary caregivers reported using any intervention component "sometimes" or less frequently at both time points, suggesting that intervention uptake was poor. Additionally, analyses indicated no effect of primary caregiver intervention use on child outcomes at 6 months, suggesting the intervention itself lacked efficacy. CONCLUSIONS: Poor uptake, and the efficacy of the intervention itself, may have played a role in the null findings of the SWITCH trial on health behaviour and body composition. TRIAL REGISTRATION: The trial was registered in the Australian and New Zealand Clinical Trials Registry (no. ACTRN12611000164998 ); registration date: 10/02/2011.

Evaluation of process capability indices of linear profiles

Purpose: In profile monitoring, which is a growing research area in the field of statistical process control, the relationship between response and explanatory variables is monitored over time. The purpose of this paper is to focus on the process capability analysis of linear profiles. Process capability indices give a quick indication of the capability of a manufacturing process. Design/methodology/approach: In this paper, the proportion of the non-conformance criteria is employed to estimate process capability index. The paper has considered the cases where specification limits is constant or is a function of explanatory variable X. Moreover, cases where both equal and random design schemes in profile data acquisition is required (as the explanatory variable) is considered. Profiles with the assumption of deterministic design points are usually used in the calibration applications. However, there are other applications where design points within a profile would be i.i.d. random variables from a given distribution. Findings: Simulation studies using simple linear profile processes for both fixed and random explanatory variable with constant and functional specification limits are considered to assess the efficacy of the proposed method. Originality/value: There are many cases in industries such as semiconductor industries where quality characteristics are in form of profiles. There is no method in the literature to analyze process capability for theses processes, however recently quite a few methods have been presented in monitoring profiles. Proposed methods provide a framework for quality engineers and production engineers to evaluate and analyze capability of the profile processes. © Emerald Group Publishing Limited.

Evaluation of deterministic and complex analytical hierarchy process methods for agricultural land suitability analysis in a changing climate

Land suitability analysis is employed to evaluate the appropriateness of land for a particular purpose whilst integrating both qualitative and quantitative inputs, which can be continuous in nature. However, in agricultural modelling there is often a disregard of this contiguous aspect. Therefore, some parametric procedures for suitability analysis compartmentalise units into defined membership classes. This imposition of crisp boundaries neglects the continuous formations found throughout nature and overlooks differences and inherent uncertainties found in the modelling. This research will compare two approaches to suitability analysis over three differing methods. The primary approach will use an Analytical Hierarchy Process (AHP), while the other approach will use a Fuzzy AHP over two methods; Fitted Fuzzy AHP and Nested Fuzzy AHP. Secondary to this, each method will be assessed into how it behaves in a climate change scenario to understand and highlight the role of uncertainties in model conceptualisation and structure. Outputs and comparisons between each method, in relation to area, proportion of membership classes and spatial representation, showed that fuzzy modelling techniques detailed a more robust and continuous output. In particular the Nested Fuzzy AHP was concluded to be more pertinent, as it incorporated complex modelling techniques, as well as the initial AHP framework. Through this comparison and assessment of model behaviour, an evaluation of each methods predictive capacity and relevance for decision-making purposes in agricultural applications is gained.

Evaluation of the usefulness of 'Marvin'; a computerized assessment tool for investigative interviewers of children

This study examined the usefulness of a computerized developmental assessment tool, designed to provide investigative interviewers with knowledge about children's understanding of spatial, temporal, numerical, and color terms that may be relevant to the forensic setting. Specifically, the effect of the computer program (compared to a standard verbal and no assessment) was examined on 4- to 5-year-old children's recall of an independent event and their enjoyment of the interview process. In addition, children's performance on the computerized and verbal developmental assessment was compared with their performance when recalling the independent event. Overall, the children rated the assessment conducted on computer more favorably than the verbal assessment. Further, the verbal assessment elicited responses that were more consistent with their responses about the event than the computerized assessment. However, there was no difference in the accuracy and detail of children's responses about the independent event, irrespective of whether the children received the computerized or verbal assessment, or no assessment. Directions for future research are discussed.

Proactive evaluation : new perspectives for ensuring quality in online learning applications

A major issue confronting educators is the extent to which they wish to conform to so-called paradigm shifts in teaching and learning. In the contemporary world of tertiary education these shifts embrace both pedagogy (from instructivist to constructivist) and technology (classroom to online). As teachers and learners are faced with the potential of these new learning environments, the extent to which the learning outcomes are achieved remains a high priority and subject to a wide range of evaluation strategies. Conventionally, evaluation is often conceptualised as occurring at the end of the development process, to assess first (formatively) whether or not the creative effort has achieved the original product goals and second (summatively) whether or not the desired learning outcomes were achieved. However, in the context of imperatives to implement online learning paradigms, the level of understanding teachers and developers have of the medium can impact the effectiveness of the product. This paper presents an additional perspective to the post-development, reactive evaluation processes in proposing the concept of proactive evaluation, a framework that identifies critical online learning factors and influences to better inform the development of learning resources. In essence, the proposal advocates an approach where development is undertaken within an environment where all activities are assessed using the evaluation criteria that would be applied when the product is assessed reactively. By performing these checks proactively, online learning resources will, in principle, work first time as all relevant factors and issues will have been considered and resolved.

Redesign of a multimedia course to reflect process not product

Universities are moving towards an environment of online teaching and learning. This is impacting the revision of degree programmes. A multimedia design course was revised to reflect and improve the design process inherent within multimedia production. The course was analyzed and revised using a critical reflection process. The outcome of the evaluation indicated that students were more satisfied but they had not grasped some of the fundamental concepts of design, and further revision of the course was needed for delivering materials and teaching in an online environment. The recommendation from the teaching team was to explore the range of research methods to formally revise the course again for future improvements to teaching practice and effective learning.