The inclusion of ‘then-test’ questions in post-test questionnaires alters post-test responses: a randomized study of bias in health program evaluation

Objectives Program evaluations are frequently based on ‘then-test’ data, i.e., pre-test collected in retrospect. While the application of the then-test has practical advantages, little is known about the validity of then-test data. Because of the collection of then-test in close proximity to post-test questions, this study was aimed at exploring whether the presence of then-test questions in post-test questionnaires influenced subjects’ responses to post-test.
Patients and methods To test the influence of then-test questions, we designed a randomized three-group study in the context of chronic disease self-management programs. Interventions had comparable goals and philosophies, and all 949 study participants filled out identical Health Education Impact Questionnaires (heiQ) at pre-test. At post-test, participants were then randomized to one of the following three groups: Group A responded to post-test questions only (n = 331); Group B filled out transition questions in addition to post-test (n = 304); and Group C filled out then-test questions in addition to post-test (n = 314).
Results Significant post-test differences were found in six of eight heiQ scales, with respondents who filled out then test questions reporting significantly higher post-test scores than respondents of the other groups.
Conclusions This study provides evidence that the inclusion of then-test questions alters post-test responses,
suggesting that change scores based on then-test data be interpreted with care.

Leading indigenous cultural inclusion 2014 program evaluation report

The Leading Indigenous Cultural Inclusion aims to support improvements in the learning outcomes and wellbeing of Aboriginal students by providing principals and school leaders with the understanding, skills and strategies to create and sustain school environments which support inclusion, engagement and achievement of Indigenous students.

A preliminary evaluation of the world's largest skills enhancement program for entrepreneurs

Start and Improve Your Business (SIYB) is an entrepreneurial training program created, sponsored and disseminated by the International Labour Organization. SIYB is, arguably, the world’s largest skills enhancement program for entrepreneurs. SIYB has been in operation since the early 1970s and currently operates in almost 80 countries. During the last decade, more than 100,000 entrepreneurs, thousands of trainers and hundreds of small enterprise development organizations around the world have participated in it (Samuelsen 2003). Surprisingly for an education program of this magnitude and potential global importance, there has been relatively little program evaluation performed. What evaluation has been conducted has been unsystematic, qualitative and unsatisfactory (Harper 1985).

Meanwhile, in the nation of Botswana, since the mid seventies, there has been a huge growth in programs devoted to the training of small and medium entrepreneurs. The SIYB program has formed a significant component of this entrepreneurial training effort in Botswana (ILO 1996). SIYB training was initially supported by the United Nations and later by the Botswana government. Since its introduction into Botswana in1983, the SIYB program has been evaluated only once, in 1993 (Samuelsen 2003). The methods used (subjective questionnaire administered to an unsystematic sample of past participants) fails to provide a satisfactory evaluation of the program. So, in Botswana and worldwide, the world’s largest entrepreneurial training program continues to run in the absence of any knowledge concerning its efficacy. This paper describes the various elements of the SIYB program and places it in the context of the data-rich Botswanan environment. The descriptive work is a necessary predicate to creation of a research design capable of providing the detailed, critical program evaluation that has been so conspicuously lacking throughout the history of the SIYB initiative.

Mixed method evaluation of a community-based physical activity program using the RE-AIM framework: practical application in a real-world setting

BACKGROUND: Communities are a pivotal setting in which to promote increases in child and adolescent physical activity behaviours. Interventions implemented in these settings require effective evaluation to facilitate translation of findings to wider settings. The aims of this paper are to i) present findings from a RE-AIM evaluation of a community-based physical activity program, and ii) review the methodological challenges faced when applying RE-AIM in practice.

METHODS: A single mixed-methods case study was conducted based on a concurrent triangulation design. Five sources of data were collected via interviews, questionnaires, archival records, documentation and field notes. Evidence was triangulated within RE-AIM to assess individual and organisational-level program outcomes.

RESULTS: Inconsistent availability of data and a lack of robust reporting challenged assessment of all five dimensions. Reach, Implementation and setting-level Adoption were less successful, Effectiveness and Maintenance at an individual and organisational level were moderately successful. Only community-level Adoption was highly successful, reflecting the key program goal to provide community-wide participation in sport and physical activity.

CONCLUSIONS: This research highlighted important methodological constraints associated with the use of RE-AIM in practice settings. Future evaluators wishing to use RE-AIM may benefit from a mixed-method triangulation approach to offset challenges with data availability and reliability.

Quantitative evaluation in health promotion : evaluation with confidence

Over the past 10 years or so, confidence intervals have become increasingly recognised in program evaluation and quantitative health measurement generally as the preferred way of reporting the accuracy of statistical estimates. Statisticians have found that the more traditional ways of reporting results - using P-values and hypothesis tests - are often very difficult to interpret and can be misleading. This is particularly the case when sample sizes are small and results are 'negative' (ie P>0.05); in these cases, a confidence interval can communicate much more information about the sample and, by inference, about the population. Despite this trend among statisticians and health promotion evaluators towards the use of confidence intervals, it is surprisingly difficult to find succinct and reasonably simple methods to actually compute a confidence interval. This is particularly the case for proportions or percentages. Much of the data which are analysed in health promotion are binary or categorical, rather than the quantities and continuous variables often found in laboratories or other branches of science, so there is a need for health promotion evaluators to be able to present confidence intervals for percentages or proportions. However, the most popular statistical analysis computer package among health promotion professionals, SPSS does not have a routine to compute a simple confidence interval for a proportion! To address this shortcoming, I present in this paper some fairly simple strategies for computing confidence intervals for population percentages, both manually and using the right computer software.

Use of the program explication method to explore the benefits of a service for homeless and marginalized young people

Maintaining the alignment between the dynamic development of health and social services and the rapidly advancing scientific evaluation literature is a central challenge facing service administrators. We describe “program explication,” a consulting method designed to assist services to identify and review implicit program logic assumptions against the evaluation literature. Program explication initially facilitates agency staff to identify and document service components and activities considered critical for improving client outcomes. Program assumptions regarding the relationship between service activities and client outcomes are then examined against available scientific evidence. We demonstrate the application of this method using an example of its use in reviewing a service for homeless young people operating in Melbourne, Australia, known as the Young People's Health Service (YPHS). The YPHS involved 21 activities organized within 4 components. The intended benefits of each of the activities were coherently articulated and logically consistent. Our literature search revealed moderate to strong evidence for around 1 quarter of the activities. The program explication method proved feasible for describing and appraising the YPHS service assumptions, thereby enhancing service evaluability.