The inclusion of ‘then-test’ questions in post-test questionnaires alters post-test responses: a randomized study of bias in health program evaluation

Objectives Program evaluations are frequently based on ‘then-test’ data, i.e., pre-test collected in retrospect. While the application of the then-test has practical advantages, little is known about the validity of then-test data. Because of the collection of then-test in close proximity to post-test questions, this study was aimed at exploring whether the presence of then-test questions in post-test questionnaires influenced subjects’ responses to post-test.
Patients and methods To test the influence of then-test questions, we designed a randomized three-group study in the context of chronic disease self-management programs. Interventions had comparable goals and philosophies, and all 949 study participants filled out identical Health Education Impact Questionnaires (heiQ) at pre-test. At post-test, participants were then randomized to one of the following three groups: Group A responded to post-test questions only (n = 331); Group B filled out transition questions in addition to post-test (n = 304); and Group C filled out then-test questions in addition to post-test (n = 314).
Results Significant post-test differences were found in six of eight heiQ scales, with respondents who filled out then test questions reporting significantly higher post-test scores than respondents of the other groups.
Conclusions This study provides evidence that the inclusion of then-test questions alters post-test responses,
suggesting that change scores based on then-test data be interpreted with care.

Leading indigenous cultural inclusion 2014 program evaluation report

The Leading Indigenous Cultural Inclusion aims to support improvements in the learning outcomes and wellbeing of Aboriginal students by providing principals and school leaders with the understanding, skills and strategies to create and sustain school environments which support inclusion, engagement and achievement of Indigenous students.

A preliminary evaluation of the world's largest skills enhancement program for entrepreneurs

Start and Improve Your Business (SIYB) is an entrepreneurial training program created, sponsored and disseminated by the International Labour Organization. SIYB is, arguably, the world’s largest skills enhancement program for entrepreneurs. SIYB has been in operation since the early 1970s and currently operates in almost 80 countries. During the last decade, more than 100,000 entrepreneurs, thousands of trainers and hundreds of small enterprise development organizations around the world have participated in it (Samuelsen 2003). Surprisingly for an education program of this magnitude and potential global importance, there has been relatively little program evaluation performed. What evaluation has been conducted has been unsystematic, qualitative and unsatisfactory (Harper 1985).

Meanwhile, in the nation of Botswana, since the mid seventies, there has been a huge growth in programs devoted to the training of small and medium entrepreneurs. The SIYB program has formed a significant component of this entrepreneurial training effort in Botswana (ILO 1996). SIYB training was initially supported by the United Nations and later by the Botswana government. Since its introduction into Botswana in1983, the SIYB program has been evaluated only once, in 1993 (Samuelsen 2003). The methods used (subjective questionnaire administered to an unsystematic sample of past participants) fails to provide a satisfactory evaluation of the program. So, in Botswana and worldwide, the world’s largest entrepreneurial training program continues to run in the absence of any knowledge concerning its efficacy. This paper describes the various elements of the SIYB program and places it in the context of the data-rich Botswanan environment. The descriptive work is a necessary predicate to creation of a research design capable of providing the detailed, critical program evaluation that has been so conspicuously lacking throughout the history of the SIYB initiative.

Use of the program explication method to explore the benefits of a service for homeless and marginalized young people

Maintaining the alignment between the dynamic development of health and social services and the rapidly advancing scientific evaluation literature is a central challenge facing service administrators. We describe “program explication,” a consulting method designed to assist services to identify and review implicit program logic assumptions against the evaluation literature. Program explication initially facilitates agency staff to identify and document service components and activities considered critical for improving client outcomes. Program assumptions regarding the relationship between service activities and client outcomes are then examined against available scientific evidence. We demonstrate the application of this method using an example of its use in reviewing a service for homeless young people operating in Melbourne, Australia, known as the Young People's Health Service (YPHS). The YPHS involved 21 activities organized within 4 components. The intended benefits of each of the activities were coherently articulated and logically consistent. Our literature search revealed moderate to strong evidence for around 1 quarter of the activities. The program explication method proved feasible for describing and appraising the YPHS service assumptions, thereby enhancing service evaluability.

Adaptation and validation of the Japanese version of the health education impact questionnaire (heiQ-J) for the evaluation of self-management education interventions

Aim:  

Unpacking black boxes: Mechanisms and theory building in evaluation

There is growing interest in the concept of ‘‘mechanism’’ across many areas of the social sciences. In the field of program and policy evaluation, a number of scholars have also emphasized the importance of causal mechanisms for explaining how and why programs work. However, there appears to be some ambiguity about the meaning and uses of mechanism-based thinking in both the social science and evaluation literature. In this article we attempt to clarify what is meant by mechanisms in the context of program evaluation by identifying three main characteristics of mechanisms and outlining a possible typology of mechanisms. A number of theoretical and practical implications for evaluators are also discussed, along with some precautions to consider when investigating mechanisms that might plausibly account for program outcomes.

Mixed method evaluation of a community-based physical activity program using the RE-AIM framework: practical application in a real-world setting

BACKGROUND: Communities are a pivotal setting in which to promote increases in child and adolescent physical activity behaviours. Interventions implemented in these settings require effective evaluation to facilitate translation of findings to wider settings. The aims of this paper are to i) present findings from a RE-AIM evaluation of a community-based physical activity program, and ii) review the methodological challenges faced when applying RE-AIM in practice.

METHODS: A single mixed-methods case study was conducted based on a concurrent triangulation design. Five sources of data were collected via interviews, questionnaires, archival records, documentation and field notes. Evidence was triangulated within RE-AIM to assess individual and organisational-level program outcomes.

RESULTS: Inconsistent availability of data and a lack of robust reporting challenged assessment of all five dimensions. Reach, Implementation and setting-level Adoption were less successful, Effectiveness and Maintenance at an individual and organisational level were moderately successful. Only community-level Adoption was highly successful, reflecting the key program goal to provide community-wide participation in sport and physical activity.

CONCLUSIONS: This research highlighted important methodological constraints associated with the use of RE-AIM in practice settings. Future evaluators wishing to use RE-AIM may benefit from a mixed-method triangulation approach to offset challenges with data availability and reliability.

Assessing the sustained impact of a school-based obesity prevention program for adolescent boys: the ATLAS cluster randomized controlled trial

BACKGROUND: Obesity prevention interventions targeting 'at-risk' adolescents are urgently needed. The aim of this study is to evaluate the sustained impact of the 'Active Teen Leaders Avoiding Screen-time' (ATLAS) obesity prevention program.

METHODS: Cluster RCT in 14 secondary schools in low-income communities of New South Wales, Australia. Participants were 361 adolescent boys (aged 12-14 years) 'at risk' of obesity. The intervention was based on Self-Determination Theory and Social Cognitive Theory and involved: professional development, fitness equipment for schools, teacher-delivered physical activity sessions, lunch-time activity sessions, researcher-led seminars, a smartphone application, and parental strategies. Assessments for the primary (body mass index [BMI], waist circumference) and secondary outcomes were conducted at baseline, 8- (post-intervention) and 18-months (follow-up). Analyses followed the intention-to-treat principle using linear mixed models.

RESULTS: After 18-months, there were no intervention effects for BMI or waist circumference. Sustained effects were found for screen-time, resistance training skill competency, and motivational regulations for school sport.

CONCLUSIONS: There were no clinically meaningful intervention effects for the adiposity outcomes. However, the intervention resulted in sustained effects for secondary outcomes. Interventions that more intensively target the home environment, as well as other socio-ecological determinants of obesity may be needed to prevent unhealthy weight gain in adolescents from low-income communities.